Web Scraper
To extract data a scrape has to be created, this can be done using GrabzIt's point and click scraper wizard to select the parts of a web page or PDF document you are interested in. This wizard then automatically produces scrape instructions, in the form of JavaScript code with custom extensions specifically designed for scraping. These instructions tell the scraper how to extract the data, which most of the time is usually in the form of a dataset. To keep creating a scrape simple, every scrape instruction is represented graphically, however the JavaScript version can still be viewed if required.
The wizard even enables the web page to be manipulated by performing actions such as typing text in text boxes and clicking on links to reveal the required data to the web scraper. It is even possible to click on a series of HTML elements in a webpage to expose content hidden in inline pop-ups or to click multiple non-standard links.
As GrabzIt's web scraper is also integrated with all of GrabzIt's other web capture products the wizard also allows web pages to be captured as PDF, images or DOCX.
As you would expect the web scraper wizard enables structured data to be extracted from page metadata, HTML elements, attributes, text contained within images and can automatically download whole files. Such as images or PDF documents. However it also allows unstructured data that is contained within text to be extracted, such as automatically identifying organisations or people's names, by using natural language processing techniques.
While all these features makes the wizard very powerful if the automatically created scrape instructions do not produce the desired results they can be perfected manually, and because they are JavaScript based, normal JavaScript can be used to specify how the web scraper should behave. However if you are having any issues writing scrape instructions or are having problems with difficult to extract data you can always commission GrabzIt to write a scrape for you.
Once the scrape instructions are written the scrape is performed on one of GrabzIt’s servers using their custom scraper software, which uses a built-in web browser to scrape web pages, ensuring that dynamic content such as AJAX powered pages is captured properly. However you don't have to wait for a scrape to complete and can download the latest snapshot of the results from GrabzIt’s web scraper tool, this is especially useful to ensure long running scrapes are working correctly.
Once the scrape is complete the data is converted into the format or formats you requested, examples range from CSV and Excel to SQL scripts. The data is then sent using one of the following technologies: Amazon S3, API callback, email notification, FTP, Dropbox and WebDav.
If you are using information scraped from the web in your app you can integrate it more closely by using the scraper API to parse the scrape results and use the data directly in your application. Real-time results can also be accessed using the API giving your app up-to-the-minute data. So you get the exact data you want when you need it. Additionally the API also provides your app with the ability to alter, stop and start scrapes.
All of these features are available for free, although limited by a monthly web scraper page limit. However this limit can be lifted by signing up to GrabzIt’s free trial.
Finally as the web is an ever changing environment GrabzIt’s web scraper is constantly being enhanced with new features and more intelligent scraping techniques.
Share This Article
