Development

Various topics related to Web Scraper, Web Crawler and Data Processing development

Content Grabber with free proxy account integration for business directories scrape

content grabber free nohodoProfessional data extraction requires adequate proxying to keep anonymity of scraping robots. When attempting to extract large data sets (over 1M records, ex. business directories) reliable and fast proxy service is needed.

Sequentum has released the Nohodo proxy service integration for Content Grabber. Nohodo provides a free account for Content Grabber users (up to 5000 requests monthly for free). The feature is available for both trial users and regular customers. Here’s how it works… more…

No CAPTCHA reCaptcha challenge

No CAPTCHA reCAPTCHA solvingSooner or later a new generation of spam protection methods will emerge to block all unwanted site visitors. The recently launched Google No CAPTHCA reCaptcha could just be such a method. This new “behaviour analysis” tool is getting more and more attention both from the site owners and from scraping engines who are trying to break it. Since Google does not reveal any secrets of its operation, we want to share with you the techniques used in this new smart analysis CAPTCHA that determines between bot and human. Let’s look inside. more…

How to detect your site is being scraped?

detect scrapeIn the age of the modern web there are a lot of data hunters – people who want to take the data that is on your website and re-use it. The reasons someone might want to scrape your site are incredibly varied, but regardless it is important for website owners to know if it is happening. You need to be able to identify any illegal bots and take necessary action to make sure they aren’t bringing down your site.  Not all bots are malicious (think search engine bots) so I’ve outlined some criteria for site owners and developers to use to identify if and how their site is being scraped.
more…

Back to top