deepnoc bot

network optimized crawling

Follow us on linkedin or twitter to get the latest news!

Goals

- reduce traffic load for webservers
- collect up to 100% of all web pages worldwide
- replace up to 250 bots worldwide
- new search engines (thematic, regional) can start very easily without crawling

Currently we are testing our bot worldwide in the revised version 3.0

General notes

Our bot does NOT search for email addresses or personal data. It just parses and saves the website content for research purposes.

Our goal is to save lots of Internet traffic and cpu power for web servers in the future.

Follow us on linkedin or twitter to get the latest news!

In case you want to stop the deepnoc bot crawling your website: Just enter the following lines into your robots.txt:

User-agent: deepnoc
Disallow: /

Last update: June 2021