Open Source Intelligence: Public Data and the Rise of the New Sherlocks

“It is so long a chain, and yet every link rings true.”  
Arthur Conan Doyle, The Red-Headed League [1]

In “The Red-Headed League,” Sherlock Holmes meets a new client and, with a glance, concludes that the man had performed manual labor, inhaled snuff, spent many hours writing, and traveled to China.

How did the famous detective know all that about a stranger he’d just met?

Sherlock Holmes observed details that were openly available to anyone’s eyes. Using that information, the detective drew an intelligent portrait of the red-headed man standing before him. A portrait that answered the question: Who is this man?

Today we call that portrait “open source intelligence.”

What is open source intelligence?

Open source intelligence (OSINT for short) is digital data that has been collected, evaluated, and analyzed to answer a specific question. [2]

The process is important. Data does not equal intelligence. When Sherlock Holmes observed a fish tattoo on his client’s right wrist (i.e., the data), he processed that piece of information using his analytical mind and formed a valid conclusion about the man’s travels to China.

That analytical process turned the image of the tattoo into open source intelligence. Without analytical processing, the data remains an interesting piece of information that may or may not lead to a valid conclusion. Dr. Watson, for example, may have looked at the tattoo and thought the new client admired fish.

A modern equivalent of the tattoo is a satellite image that shows the insignia on a military uniform. [3] From that image, a seasoned analyst could determine which military unit was at the location in the image. By combining that insight with the knowledge gleaned from a succession of images showing, over time, the arrival and deployment of military equipment at the same location, the analyst could formulate valuable open source intelligence about troop movements.

Where does open source data come from?

OSINT is a product, not original source data. Given this distinction, where does the source data come from?

The data that feeds open source intelligence is publicly available. That data can come from websites, social media, satellite images, videos and from data vendors who collect, process, and analyze data. The distinguishing characteristic of open source data is that it is available to the public. Contrast that with data maintained by enterprises and governments such as PII (personally identifiable information), credit card data, and PHI (protected health information) and you’ll understand the difference between public and private data.

Some examples of open source data are:

• News websites
• Social media websites
• Satellite images
• Court records
• Internet domain data
• Airline flights
• Highway maps
• Census data
• Financial data

Open source data can be collected passively or actively. [4] Scraping a public website for airline flight data is an example of passive collection. Active data collection targets specific website and servers to retrieve data such as open ports, accessible endpoints, and security vulnerabilities. An example of active data collection is penetration testing.

How is open source intelligence used?

Open source intelligence, like all intellectual capital, can be used for good or evil. Just as Sherlock Holmes used his analytical powers to solve crime and bring justice to his clients, so too his nemesis, Dr. Moriarty, masterminded crime and brought grief to his victims.

Law enforcement, for example, can employ open source intelligence such as social media and satellite images to help foil an assassination plot that was planned with the help of OSINT such as online news articles, maps, and geolocation data.

Let’s review some examples of open source intelligence being used to both harm and help people. We’ll begin with the present-day Moriarty’s of the world: cyber criminals.

Open source intelligence and the rise of cybercrime

In the last decade we have, unfortunately, seen open source data increasingly used for two nefarious purposes: cybercrime and disinformation.

Cybercrime

In 2021, Marjorie Bloom was contacted by a man claiming to be a fraud investigator for PNC Bank, where Bloom kept her entire life’s savings. [5]

Telling Bloom that criminals using stolen person data (PII) had begun to withdraw funds from her accounts, this man convinced Bloom that to protect her assets she needed to transfer the $661,00 she held in savings, stocks, and an annuity to a cryptocurrency account.

After emptying her accounts, Marjorie Bloom never heard from this “investigator” again.

She had been swindled by a cybercriminal masquerading as a PNC Bank employee.

Cyber thieves target the elderly because these criminals know that older people have built wealth over decades. Often living alone and unfamiliar with technology, the elderly are also easy targets.

In 2022 cyber fraud scammed $3.1 billion from Americans over 60, according to FBI statistics. By 2025, the cost of cybercrime in the U.S. will exceed $10 trillion.

Disinformation

Merriam-Webster defines information as “knowledge obtained from investigation, study, or instruction” and the prefix dis- as “opposite or absence of.”

Spreading disinformation, therefore, is the attempt to spread the opposite of knowledge. In other words, disinformation equals deception.

More formally, disinformation has been defined as false information deliberately created and propagated to harm an individual, a company, or a country. [6] Disinformation is prevalent in open source data because it is camoflouged within “news” articles, blog posts, YouTube videos, and social media posts.

In short, disinformation deliberately encourages people to adopt a false belief.

When Russia reported, for example, that five Ukrainian soldiers in two armored personnel carriers (APCs) had crossed into Russian territory in February of 2022, it was pushing disinformation. [7]

How do we know?

Using video evidence, internet sleuths proved that the incident was staged by Russia and filmed in Ukrainian territory. And the APCs were Russian equipment that had been painted to look Ukrainian.

So, you may ask yourself, what are the good guys doing about all this cyber disinformation and cybercrime?

Open source intelligence and the rise of the new Sherlocks

“We imagine what might have happened, acted upon the supposition, and find ourselves justified. Let us proceed.”  
Arthur Conan Doyle, The Adventure of Silver Blaze [8]

Sherlock Holmes knew how to catch a criminal, even a criminal who’s well camouflaged, as we see in “The Adventure of Silver Blaze.” [9] The great detective was also adept at exposing disinformation, as he demonstrated in “The Red-Headed League.” [10]

In today’s online world, a new group of digital detectives is following the path created by Holmes — with a modern twist.

They gather, observe, and analyze not physical details but virtual data. These internet detectives analyze public data — data open to the world.

These present-day Sherlocks examine open source data, analyze that data, and transform it into open source intelligence. The new sleuths then use OSINT to reveal truths which help solve crimes and disarm disinformation.

Sometimes they do both at the same time.

Using OSINT to solve a crime and disarm disinformation

The Heinous Crime

On July 14, 2017, Malaysian Airlines Flight 17 (MH17) ascended from Amsterdam and headed east toward Kuala Lumpur. Aboard were 298 souls — 283 passengers and 15 crew. [11]

At 33,000 feet above eastern Ukraine, air traffic controllers lost contact with MH17. That’s because the plane was disintegrating in midair as it plummeted toward the ground, killing everyone aboard.

That same day, a Russian separatist leader in Ukraine claimed in a social media post that his forces had shot down a Ukrainian military aircraft:

"We did warn you — do not fly in our sky."

The Disinformation Campaign

After discovering it was a civilian airliner that had been struck, the separatists deleted the post and denied involvement. [12] And the leader of the Donetsk People’s Republic claimed his forces didn’t have the weapons to shoot down a plane flying at 33,000 feet. To confuse people even more, Russia claimed the missile had been fired from a Ukrainian military jet. [13]

The disinformation campaign was on.

OSINT Reveals the Truth  

Because 193 of the 298 aboard MH17 were Dutch nationals, Ukraine delegated the formal investigation into the downing of MH17 to the Netherlands. In a joint effort, the Dutch Safety Board coordinated the investigation with representatives from Australia, Malaysia, Belgium, and Ukraine. The Dutch called this group the Joint Investigation Team (JIT).

Meanwhile, a group of internet sleuths decided to take up the case. In less than two years, the work they did advanced OSINT by decades.

Bellingcat is an open source investigative team. Staffed mostly by volunteers, Bellingcat uses its OSINT skills to solve crimes, debunk disinformation, and bring justice to victims.

Knowing that debris from a Russian Buk-M1 missile launcher was found in the wreckage of MH17, Bellingcat focused on the missile launcher. [14]

Bellingcat used the following open source data to find and track the movement of the Buk-M1 missile launcher in the area of Donetsk, Ukraine on July 17, 2014 [15]:

• Satellite imagery
• Google Earth
• Social media posts with embedded video showing the Buk-M1 on the road  
• YouTube travel videos that identified landmarks such as storefronts  
• Court records in Russian that identified the addresses of buildings
• Eyewitness accounts from locals near Donetsk who described seeing the Buk-M1  
• A photograph of the Buk-M1 passing through the town of Torez
• A U.S. satellite image that depicted the path of the missile after it was launched  
• A photograph on Twitter that showed the smoke from the launcher
• Photographs showing an unmistakable numerical marking of 3x2 on the side of the Buk-M1
• Social medial profiles to identify the Russian soldiers in the missile brigade working the Buk-M1  

Through processing and analyzing all this public data, Bellingcat was able to document the journey the Buk-M1 took from Russia to a cleared farm field near Snizhne, Ukraine. In that field on the afternoon of July 17, 2014, the crew of the Buk-M1 fired one of the launcher’s four missiles.

Four days later, on July 21, the Russian Ministry of Defense attempted to push even more disinformation. [16] Russia claimed that video of the Buk-M1 being transported back to Russia was filmed in territory held by Ukrainian forces The Russians pointed to a billboard seen in the video as proof that the Buk-M1 was in Krasnoarmiisk (now Pokrovsk) and not in Luhansk, as Ukraine claimed. [17]

The two cities are 132 miles apart. Luhansk is 15 miles from the Russian border and was held by separatists loyal to Russia. Pokrovsk lies west of Luhansk and was controlled by Ukraine.

Using open source photographs provided by a Luhansk resident, the internet sleuths proved that the billboard seen in the video was indeed in Luhansk.

The Buk-M1 was on its way back to Russia.

Not only did open source intelligence prove where and when and how the crime was committed, it debunked the disinformation that tried to cover up that crime.

OSINT Tools Every Sleuth Needs

You may not possess Sherlock Holmes’ native ability to observe data at a glance and instantly formulate critical open source intelligence, but you have an advantage Holmes did not.

By using a powerful AI-driven OSINT tool, you can rapidly process and analyze large quantities of open source data to quickly form insights that can solve a crime or even prevent one. Equally important, you’ll gain the capability to debunk pernicious disinformation.

These new tools are what an internet sleuth needs to become the next great detective.

End notes

1. Arthur Conan Doyle, “The Red-Headed League”, 1891, https://sherlock-holm.es/stories/pdf/a4/1-sided/redh.pdf

2. Ritu Gill, “What Is Open-Source Intelligence?”, February 23, 2023, https://www.sans.org/blog/what-is-open-source-intelligence

3. Janes, “Russia/Ukraine – Coming of Age for OSINT?”, https://podcast.janes.com/public/68/The-World-of-Intelligence-50487d09/0fd51eaa

4. “Open-Source Intelligence (OSINT)”, https://www.imperva.com/learn/application-security/open-source-intelligence-osint/

5. Greg Iacurci, “How This 77-year-old Widow Lost $661,000 In a Common Tech Scam”, October 8, 2023, https://www.cnbc.com/2023/10/08/how-one-retired-woman-lost-her-life-savings-in-a-common-elder-fraud-scheme.html

6. Berlin Risk, “The Rise of Disinformation in OSINT”, https://berlinrisk.com/the-rise-of-disinformation-in-osint/

7. Matthew Gault, “The Internet Is Debunking Russian War Propaganda In Real Time”, February 22, 2022, https://www.vice.com/en/article/7kb75e/the-internet-is-debunking-russian-war-propaganda-in-real-time

8. Arthur Conan Doyle, “The Adventure of Silver Blaze”, 1892, https://sherlock-holm.es/stories/pdf/a4/1-sided/silv.pdf

9. Ibid

10. Doyle, “The Red-Headed League”

11. Wikipedia, “Malaysia Airlines Flight 17”, https://en.wikipedia.org/wiki/Malaysia_Airlines_Flight_17

12. Lizzie Dearden, “MH17 Crash: Fragments of Russian missile BUK missile launcher found at crash site”, August 11, 2015, https://www.independent.co.uk/news/world/europe/mh17-crash-investigators-find-parts-of-buk-missile-possibly-used-to-shoot-plane-down-10450053.html

13. Anna Holligan and Kate Vandy, “MH17: Three guilty as court finds Russia-controlled group downed airliner”, November 17, 2022, https://www.bbc.com/news/world-europe-63637625

14. Wikipedia, “Malaysia Airlines Flight 17”

15. Bellingcat Investigation Team, “MH17 The Open Source Evidence”, October 2015, https://www.bellingcat.com/app/uploads/2015/10/MH17-The-Open-Source-Evidence-EN.pdf

16. Ibid

17. Ibid