- Apply a webhook service to request your target data and store them to DB. more…
Recently I received a question on insurance leads:
Is this a legal method of acquiring insurance leads [from the web]? Are there any agent testimonials on the efficiency of this type of service?
Legality issue in web scraping
- Public info (prices, inventory info, public offers), i.e. everything that is not protected by copyright and available for scraping.
So far I have no insurance agent testimonies on the efficiency of any insurance lead scrape service. The web sites I searched [on the insurance leads] have given me the impression that the customer info they gather is highly secured (not viewable). I doubt that any sites are going to expose insurance leads. In most of them the leads are available by paid subscription plans.
If there are any such websites like insurance leads directories (public insurance quotes), we might develop a scraper that consistently grabs fresh or new info for further analysis. It does save the agent’s time for re-searching, re-visiting and so on. One scraper might work with multiple directory pages for scrape.
You might find it interesting to read about web page change tracking if you only need to see updates (no data storing applied).
Death By Captcha is a reputable CAPTCHA solving service with more than 7 years in the Captcha Solving business. They have recently updated all their API clients, so users can experience maximum efficiency and faster solving times.
They enthusiastically recommend that users and software developers visit the API page and update their DBC API implementation in order to get the most out of it (the API and docs are available for registered users only). The free credits are provided for users to test or implement the new client API!
Today I needed to enable a Charles proxy on my Windows PC. Later I have managed the Genymotion virtual device to be monitored by the Charles proxy. more…
Octoparse is a new, modern, visual web data extraction software. It has always committed itself to providing users with a more professional data scraping service and to becoming one of the most popular web scraper tools.
It has released a new version of the tool, 6.4.1, in March 2017 with some new features and a much faster and better user experience. more…
UiPath, one of the big providers of robotic process automation software, has some very interesting positioning. Unlike the other players on the market, they provide a free and fully featured community edition of their product for anybody to test and develop. The tool automates any application and is packed with all the web scraping and screen scraping capabilities for both desktop and web. The platform also has a lively community forum featuring jobs, automation contests and knowledge-sharing between UiPath users: www.forum.uipath.com. more…
- host: lx567.certain.com (SFTP)
- user: igor_user
- password: testPass
For SSH access in a terminal type:
$ ssh email@example.com
then enter the password (testPass) at a password prompt.
Recently I decided to work with pythonanywhere.com for running python scripts on JS stuffed websites.
Originally I tried to leverage the dryscrape library, but I failed to do it, and a nice support explained to me: “…unfortunately dryscrape depends on WebKit, and WebKit doesn’t work with our virtualisation system.”