The market for threat intelligence solutions totaled just $4 billion in 2018. It is expected to more than triple, to $13 billion, by 2025.[1] Increasingly, governments, businesses, and other organizations understand the dangers posed by threats — both internal and external — to their personnel, populations, infrastructure, operations, data, and IT systems. Hence the tremendous increase in spending on technology designed to manage threats.
The threat intelligence gleaned from publicly available information (PAI) and commercially available information (CAI) is often called open source intelligence — OSINT for short.
A closer look at OSINT
Open source intelligence is gleaned from publicly available information (readily accessible for free) and commercially available information (available for a price).
Some sources of PAI are websites (including those hosted on the deep- and dark web), social media platforms, message board interactions, online comments, certain news media articles and videos, government data, and legal data. In a world where people create more than 2.5 quintillion bytes of data daily,[2] there is a massive amount of PAI available for search.
Commercially available information includes market research, financial and investment analyses, consumer data, academic journals, geospatial information, intellectual property databases, social media analytics, industry newsletters, data available via subscription platforms, and more.
The threat landscape
What do governments and private enterprises have to fear?
External threats to a country’s security include threats of terrorism, cyberattacks, attacks against infrastructure, military incursions, foreign interference with elections, drug trafficking, human trafficking, trafficking in counterfeit goods, and disinformation campaigns. Many of these —terrorism, disinformation campaigns, cyberattacks, infrastructure attacks — double as internal threats. Additional internal threats include leaks of classified information, corruption, general crime and violence, and natural disasters.
The private sector is not immune to these and similar dangers. External threats can include cyberattacks, economic uncertainty, failure to comply with regulatory mandates, physical threats to employees or the workplace, frivolous lawsuits, supply chain disruptions, and event venue attacks. Internal business threats are most often posed by employees and former employees, partners, and contractors. Network vulnerabilities and the loss of corporate data — through theft, accident, or inadvertent disclosure — are significant concerns. Additional internal threats include sabotage of IT systems, operations failures, employee violence, and theft of devices.
Developing threat management programs
There’s no silver bullet for spotting, mitigating, and preventing danger. Rather, threat is best managed as part of an organization-wide program.
To protect against internal threats, organizations should develop holistic insider risk management programs. These programs identify insider threats across the enterprise; assess the impact of these threats on operations and missions; mitigate those threats; and consistently scan the digital landscape for emerging dangers. Holistic risk management typically consists of:
- Policy: Developing cohesive policies for identifying, assessing, and mitigating threats
- Leadership: Empowering strong leadership to engender employee buy-in, and to secure the financing needed for program implementation
- Robust security protocols and access control: Preventing unauthorized access to critical systems and data
- Employee education: Educating employees on the ways they and their colleagues may — intentionally or unintentionally — put their organizations and themselves at risk
- Adoption of OSINT technologies: Deploying technology to scan PAI and CAI for examination of personnel’s online behavior
To manage external threats, organizations should develop a program of threat assessment, prioritization, and prevention/mitigation, considering the likely business impact of each type of threat. These programs should include:
- Geopolitical risk assessment: Identifying the risks associated with wars, terrorism, and tensions among nations
- Cybersecurity measures: Protecting systems, networks, data, and applications from attack
- Supplier diversity: Minimizing the risk of supply chain disruptions
- Emergency response and business continuity planning: Ensuring the organization can continue operating in times of disaster or unrest
- Regulatory compliance: Avoiding the fines, reputational damage, and other repercussions of failing to comply with regulatory mandates
- Regular security audits and updates: Continually reassessing the organization’s security posture
- Adoption of OSINT technologies: Deploying technology to continuously scan PAI and CAI to spot situations that may threaten the organization
As you can see, open source intelligence tools are critical to both internal and external threat-management programs.
Using OSINT for threat detection
OSINT can help private- and public sector entities obtain operational threat intelligence, security threat intelligence, and strategic threat intelligence. Organizations compile a list of key words associated with known or suspected threats. Good OSINT platforms then rapidly and persistently scan PAI and CAI sources across the internet — including hard-to access sites on the deep and dark web, where information may be offered for sale. (Because the nature of the tools used to access the dark web ensure anonymity, it is a hotbed of illegal activity.) Searches are conducted in real time, so mitigation efforts can begin more quickly.
How do OSINT technologies differ from user activity monitoring?
To improve security, many organizations have deployed user activity monitoring software (UAM). This software tracks user behavior on employee-owned devices and networks. It can spot instances of unusual network access, which may indicate cyber hacking and other illegal behavior. It can also spot if Hal in R&D is using the email system on his office computer to sell the company’s biotech breakthroughs to a competitor.
You know what UAM can’t do? Detect if Hal is using his home computer to access a dark web marketplace and selling information there.
The tracking capabilities of UAM and related technologies are insufficient for the digital age; they provide information only on the use of enterprise-issued or authorized devices. They cannot track user activity on employee-owned devices. Supplementing the information obtained from UAM systems with AI-powered OSINT technology that actively scans PAI and CAI can close this security gap. OSINT technology enables organizations to examine personnel’s online behavior regardless of the device employed.
How can OSINT capabilities help protect against threats?
Here are just a few threat intelligence use cases from a handful of sectors and geographies.
- Immigration officials can use OSINT platforms to pre-screen travelers for visas. They can examine social media posts and other content to determine whether an applicant is in any way related to a criminal appearing on a watch list.
- Border security officers can use OSINT systems to detect and track illegal cross-border activity; monitor the movements of individuals and groups of interest; and begin response planning.
- National security agencies can use OSINT to monitor the social media activity of suspected terrorist organizations.
- Since mass shooters tend to announce their plans online[3], law enforcement can use OSINT platforms to monitor social media for potential mass shooters in their area.
- Airport security can deploy OSINT to learn more about the security of their facilities. If a PAI system detects someone tweeting, “Just saw a woman abandon a bag @Liverpool John Lennon Airport, Gate 8,” it can trigger an alert to airport authorities.
- The United States Department of Defense can search PAI and CAI worldwide to detect words and phrases associated with leaks of classified or sensitive information.
- OSINT platforms enable law enforcement to scour the deep and dark web to identify potential drug traffickers, potential human traffickers, and human trafficking victims. They can also analyze PAI and CAI for insights into trafficking patterns, criminal recruitment methods, and recruitment advertising.
- Business executives can use social media monitoring and other OSINT capabilities to monitor the online behavior of employees, ex-employees, contractors, and others suspected of malicious behavior.
- Public health and safety officials can deploy OSINT platforms to determine the scope of natural disasters, and coordinate responses appropriately. People often post about these events, even before calling emergency services. Information gleaned from monitoring these social media posts can inform governments and emergency services about what is happening where. They can then deploy resources accordingly.
Using OSINT at different stages of the threat intelligence lifecycle
Finding, understanding, and acting upon threat intelligence is a multi-stage process, ranging from collection to action and review. OSINT plays a significant role in many of these steps.
Collection
The first step in obtaining threat intelligence is collection of data. Analysts must gather relevant threat data — both structured and unstructured — from a huge array of disparate OSINT sources. The best OSINT platforms collect this PAI and CAI, translate it into the user’s language, then transform the information into enriched, relevant insights.
Processing
OSINT processing entails organizing and structuring collected data for improved threat intelligence management. OSINT systems now on the market categorize information, remove irrelevant data, and otherwise prepare it for analysis.
Analysis
Cutting-edge OSINT platforms help in analysis. They identify information themes and sentiment. They detect relationships, notably relationships that aren’t obvious to the human eye. They empower analysts to explore data through a wide range of analytical lenses. These include geospatial, temporal, and social relationships, along with topics of interest. In best-case scenarios, insights are presented via a single interface, enabling cross-team analysis and collaboration.
Visualization
The best OSINT platforms help analysts visualize data to better understand key connections between search terms and topics of interest. As part of this relationship mapping, OSINT platforms study social, business, and political networks to identify those influencers with the greatest potential to impact organizations or events.
Integration
Technologically advanced platforms can combine OSINT with internal data (including information from databases, network logs, and incident reports) to provide a more complete view of the potential threats facing an organization.
Dissemination
Sharing threat intelligence with relevant stakeholders and partners
Action
Implementing measures to combat threats
Iteration
Continuously evaluating the effectiveness of threat intelligence processes to improve future intelligence gathering
What to look for in an OSINT platform
Organizations can find a number of OSINT-centered threat intelligence platforms on the market. What should you look for in a solution?
To meet the challenge of detecting today’s threats, your threat intelligence platform must be capable of finding, analyzing, and coalescing vast amounts of data. Look for an automated solution that can access all layers of the internet, including the deep and dark web. Choose a platform that includes a large and diverse library of enriched data, originating from a broad array of free and commercially available sources. And don’t forget internal data. Threat intelligence solutions should be able to find data wherever it lives in your organization. To accomplish this in a cost-effective manner, you should consider an API-based solution, one that works on top of legacy systems to facilitate sharing from one application or data silo to another — avoiding the need to replace or re-tool older systems.
You should also look for:
Translation capabilities
All the OSINT in the world does organizations no good if it’s presented in a language they can’t understand. The best OSINT solutions automatically translate content from an array of different languages, helping organizations monitor content from across the globe.
Entity resolution
Entity resolution is the process of examining names appearing in unstructured text, then matching those names to entities appearing in a public knowledge base or the knowledge bases maintained by your organization. This capability helps you distinguish among multiple entities with the same or similar names.
Why is this capability important? There are 8 billion people on the planet. A lot of us are going to have the same name. Without entity resolution capabilities, State Department officials issuing B-1 business visas will struggle to distinguish between Wei Zhāng, the founder of a tech startup visiting Los Angeles to meet with potential investors, and Wei Zhāng, the owner of a factory that illicitly manufactures counterfeit designer handbags.
The entity resolution capabilities of the best OSINT platforms also coalesce various online identities to the real people behind the handles. For example, if law enforcement is worried about the possibility of a mass shooting, the right platforms can help them learn that Shoot2Kill587 and AimForTheHead91 coalesce to the same person — John Smith of Enid, Oklahoma.
Availability
Threats don’t occur only between 09:00 and 05:00 on weekdays. Because members of your organization may need immediate access to OSINT, your solution should work across dashboards, and across desktops and mobile devices.
Persistent search
Persistent search is a technology that keeps search operations running regardless of whether someone is actively using them, recording updates and changes, then automatically appending this information to search terms. This capability keeps your searches up to date without extra effort on the organization’s part.
Why Babel Street?
Babel Street Insights is an AI-powered OSINT platform that offers all the threat intelligence tools and capabilities discussed in this article. It rapidly and persistently searches PAI and CAI published in more than 200 languages. This data originates from more than a billion top-level domains; the deep-and dark web; and other commercially and publicly available sources. Among these sources are dozens of social media platforms; real time interactions generated on millions of message boards; and online comments. It searches in real time, and alerts according to user-determined thresholds.
Babel Street Insights’ AI-powered analytics capabilities can also help enrich data already appearing in governmental or enterprise databases. Geolocation and telemetry data is available to qualifying government organizations.
In providing these capabilities, Babel Street helps close the Risk-Confidence Gap, or the widening chasm between the escalating volume and variety of data that must be examined for improved threat intelligence, and the resources organizations have available to monitor that data. That’s why 84 percent of United States national security agencies, along with similar agencies worldwide, have partnered with us.
Endnotes
1. Global Market Insights, “Threat Intelligence Market Size,” accessed January 2024, https://www.gminsights.com/industry-analysis/threat-intelligence-market
2. Skelly, William, “Turning Quintillion Bytes of Data Into Opportunities,” Datanami, February 2023, https://www.datanami.com/2023/02/16/turning-quintillion-bytes-of-data-into-opportunities/#:~:text=Approximately%202.5%20quintillion%20bytes%20of,and%20opportunity%20of%20organized%20data.
3. Peterson, J., Densley, J., Spaulding, J., & Higgins, S., “How Mass Public Shooters Use Social Media: Exploring Themes and Future Direction,” Social Media + Society, accessed October 2023, https://doi.org/10.1177/20563051231155101
Disclaimer
All names, companies, and incidents portrayed in this document are fictitious. No identification with actual persons (living or deceased), places, companies, and products are intended or should be inferred.
