Find New Market Insights Through Web Scraping

What is web scraping? It’s the act of extracting information from public-facing web sites. The process is necessary for obtaining alternative data — or unstructured information emanating from, but typically unannounced by, businesses and government agencies.  

At its most basic, web scraping consists of copying information from a web site and pasting it into a format you can use.  Sounds easy enough. But web scraping grows infinitely harder if, for example, you’re the chief development officer of a national big-box chain, seeking alternative data to help plan a significant Midwest expansion. To determine optimal store placement and inventory, you need information on housing trends at the ZIP code level, neighborhood demographics, foot traffic projections, and insight into what’s selling (and what’s not) at the competition.  

You can’t copy and paste all that. Capturing alternative data at scale takes proven technologies, methodologies, and know-how. The right web scraping tools can help organizations quickly collect the massive amounts of data needed to obtain market intelligence and other types of insight, empowering them to uncover opportunities, mitigate risk, and act with speed.  

How it works

How does web scraping work? Software engineers with specialized expertise program bots (software applications engineered to mimic human activity at a large scale) to quickly collect vast amounts of data. These bots vary in complexity from simple technologies that extract and index unstructured web page content to highly sophisticated parsing algorithms that extract fielded data in alignment with a company’s specific requirements.

Bots consist of web crawlers and web scrapers. Like humans visiting their favorite web sites, web crawlers send requests to host servers, fetching the web pages to be examined. Web scrapers extract the desired information by parsing content from the web page’s underlying code. After being extracted, pertinent content — which on any single web page can range from blobs of unstructured text to highly structured data fields — can be cleansed, structured, and stored in user-friendly file formats including CSV and JSON.

To provide optimal insight, the right web scraping solutions gather data from disparate web sites and coalesce it into a single file. A real estate company considering the construction of new shopping mall in the suburbs of Cleveland can scrape pertinent information from a variety of sources. These range from city planning and zoning web pages to online census data, along with additional information covering traffic patterns, foot traffic, and proximity to competing shopping centers. Storing this information in a single file, the builder can more easily determine the region’s buying patterns and rent to the types of stores at which area residents are most likely to shop.

Choosing the right web-scraping technology

Several web-scraping technologies are now on the market. How can organizations choose the best option for their needs?

For light use, some businesses may use the add-ons offered by popular web browsers. These web scrapers are typically either inexpensive or free, and they’re easy to use. But, limited by the capabilities of internet search engines, they typically cannot provide the breadth or depth of insight that larger organizations seek. If your search engine cannot return results for the question, “How many people between the ages of 40 and 60 live in Sullivan County, NY?” then that information can’t be scraped. This doesn’t mean that that information doesn’t exist. It just means that the search engine hasn’t indexed the anonymized demographic information available from the United States Census Bureau and other organizations.

Commercially available web scraping software is another choice. But these programs are also often sub-optimal for enterprise use. They may not scale appropriately. They may limit user customization capabilities. They may be unable to keep pace with frequent web site changes. Output may require significant cleansing and normalization.

Understanding these limitations, some companies try to engineer their own web scrapers. But this is a challenging process, requiring:

• Significant technical expertise to design web crawlers that can scrape accurate data without disrupting a web site’s operation
• Large financial outlays, including costs for engineering talent and infrastructure  
• Storage capacity for retaining all that scraped data

Even if an organization can obtain alternative data on its own, that data is typically unstructured, rendering it difficult to process and analyze. In addition, datasets may contain errors, along with incomplete, conflicting, or biased information. Analytical and technical expertise is needed to obtain legitimate insight from this imperfect data.

Exacerbating these challenges is the fact that most companies don’t want their data scraped. They implement anti-scraping measures that identify bot-like activity. They ban requests from flagged IP addresses. They implement CAPTCHA tests. Companies seeking insight can find it very difficult to work around these roadblocks.

For these reasons, many organizations find it easier and more cost-effective to work with third-party data providers. Established providers offer high-quality data, typically at a lower cost than building and maintaining an alternative data infrastructure in-house.

How Babel Street can help

Babel Street Data specializes in providing organizations with the data products and intelligence they need to uncover opportunities, mitigate risk, and act with speed. We offer hundreds of ready-to-use datasets covering virtually every industry. These are collected via our proprietary collection framework. This framework is built atop a mature connection infrastructure with IP addresses in 95 countries.

Our offerings include both raw data collections and refined information products. Both rely on our expertise in data discovery. Babel Street identifies sources of public data that can help customers answer their toughest data-driven questions. Knowing where to find data is a crucial first step to data collection — and one at which Babel Street excels.

Raw collections consist of data collected at scale, covering a vast array of industries. Datasets in our robust library often span more than five years, making them suitable for back testing and time-series analyses.

Based on our raw collections, refined information products are aggregated datasets combining multiple data sources in a single output. This output showcases important data elements, trends, or calculated fields. After collecting data, we cleanse it, normalize it, and enrich it for easier, more accurate interpretation.

These products are governed by a proven development process including:

• Collection and structuring: Leveraging our seasoned infrastructure, highly skilled Babel Street teams use their tradecraft to collect data from global web sites. They then extract key fields for analysis.
• Enrichment and auditing: Teams enhance data to ensure accuracy and fill in any missing information. Our auditing capabilities track data provenance. We also structure data to make it research ready — easily usable by both analysts and analytical tools.
• Storage: Data collections are securely stored on Babel Street infrastructure in alignment with compliance and security requirements for financial and governmental entities.
• Aggregation: Babel Street systematically cleanses, processes, and aggregates data across hundreds of collections to provide standardized, integrated, output.
• Delivery: Organizations receive data files as CSVs, JSONs, or other commonly used formats. Our research-ready data can be integrated directly into the customer's existing workflows and tools. For even sharper insight, data can be enhanced with Babel Street intelligence products, providing in-depth analysis and intelligence reporting conducted by our own subject matter experts.

Alternative data can provide organizations with deeper insight, helping them more quickly seize opportunity and mitigate risk. But getting that data is challenging. Working with Babel Street can ease the process.

Disclaimer:

All names, companies, and incidents portrayed in this document are fictitious. No identification with actual persons (living or deceased), places, companies, and products are intended or should be inferred.