webRobots_logoMost scraping solutions fall into two categories: ­Visual scraping platforms targeted at non-programmers (Import.io, Content Grabber, Dexi.io, etc.), and scraping code libraries like Scrapy or PhantomJS which require at least some knowledge of how to code.

Web Robots builds scraping IDE that fills the gap in between. Code is not hidden but instead made simple to create, run and debug.

Javascript scraping

Advance of HTML5 is changing web scraping. Websites are no longer just text with links but are instead advanced applications rivaling desktop applications.

Traditionally scraping by downloading HTML no longer works. Running JavaScript is necessary to get the content. Web Robots specialize in difficult to scrape websites and have created scraping tool chain that drives the latest version of Chrome browser. Therefore they can handle anything a modern browser can handle.

Here is the first look review of their tools.

Scraping IDE


webRobots scraping IDE

In a nutshell Web Robots scraping tool is a Chrome extension which drives Chrome browser.
Instead of building fully visual scraper builder Web Robots do not hide code from users. Visual scrapers are appealing in theory but often code writing is necessary for complex web scrapes.

Web Robots embraces that, and their tool is designed to make coding quick, easy and familiar. Code is written directly in the extension. It does syntax highlighting, code validation and linting. Users can write, run, view results and debug in one place.

Scraper robots are written inside the extension using familiar JavaScript and jQuery.

Compared to other tools there are many advantages:
● Easy development and debugging because robot’s code snippets can be tested in Chrome Developer Tools built into Chrome
● Doing test runs gives immediate visual feedback in browser
● Scraped data is visible in real time
● Chrome handles cookies and session management
● Browser fixes messy HTML and present valid DOM for scraping. So no need to worry about mismatched tags
● Scraping APIs is easy with jQuery AJAX calls
● No need to install anything on client
● No need to have a server

Feature comparison with full scraping frameworks:

Feature
Selenium
Scrapy
PhantomJS
Web Robots
Need to run server
Yes
No
No
No
Install client software
Yes
Yes
Yes
No
IDE
Yes
No
No
Yes
Robots in JavaScript
Yes
No
Yes
Yes
jQuery
No
No
No
Yes
Latest JavaScript runtime
No
No
No
Yes
Run robots on Cloud
No
Yes
No
Yes

Feature comparison with Visual Web scrapers

Feature 
import.io 
Mozenda
 
Visual Web Ripper
Web Robots 
Cloud portal 
Yes 
Yes 
No 
Yes 
Install client software 
Optional 
Yes 
Yes 
No 
Price 
Free 
Data amount based
$299 
Free 
Robots in JavaScript 
Yes 
No 
No 
Yes 
Code IDE 
No 
No 
No 
Yes 
Visual mode 
Yes 
Yes 
Yes 
No 

Data portal

webRobots data portal

Scraping code and extracted data are saved to Web Robots portal. All data is archived in portal and it can be downloaded in CSV or JSON.

Once robots are tested and reliable users can run them from portal and cloud based machines will do the scraping.

Data validation

Data is only valuable if it can be trusted. Web Robots has a lot of checks to make sure scraped data is correct. Some checks are mandatory like:

● Price field must be present, more than zero, less than 1000

Some are warnings:
● Price is significantly different from historic average.

Extra services

Web Robots provides extra services to users for whom self-service is not enough.

● Large scale crawling and fetching millions of pages using many parallel scrapers
● Data export directly to customer databases
● Data checking and validation
● Full­ service script writing and testing
● Proxies for high throughput parallel scraping

Conclusion

Web Robots offers a scraping tool that is a more advanced option than most visual web scrapers but simpler and easier to use than PhantomJS or Selenium. It can access dynamic websites that cannot be scraped using old school scraping libraries like Scrapy or Mechanize.

It requires some coding in JavaScript so it’s not for every user. But the reward is virtually unlimited scraping possibilities.
Tutorial: http://webrobots.io/support/