Collecting Open-Source Intelligence
When to Use
Use this skill when:
- Investigating external infrastructure associated with a phishing campaign targeting your organization
- Enriching threat actor profiles with publicly observable indicators (WHOIS, ASN data, SSL certificates)
- Conducting authorized attack surface discovery to understand your organization's external exposure
Do not use this skill for active scanning against targets without explicit written authorization — OSINT collection must remain passive (no packets sent to target systems) unless scope permits active recon.
Prerequisites
- Maltego CE or commercial license for graph-based link analysis
- Shodan API key (https://shodan.io) for internet-wide device/service discovery
- OSINT Framework familiarity (https://osintframework.com) for tool selection
- SpiderFoot HX or open-source SpiderFoot for automated OSINT correlation
Workflow
Step 1: Define Collection Requirements
Establish the intelligence requirement (IR) before collecting. Document:
- Target: threat actor group, malicious domain, IP range, or organization
- Priority Intelligence Requirements (PIRs): What specific questions need answering?
- Legal authority: Passive OSINT is legal; active probing requires authorization
- Data handling: TLP classification for collected intelligence
Step 2: Passive DNS and WHOIS Investigation
# Passive DNS via SecurityTrails API
curl "https://api.securitytrails.com/v1/domain/evil-domain.com/dns/a" \
-H "apikey: YOUR_KEY"
# WHOIS history via ARIN / RIPE
whois -h whois.arin.net evil-domain.com
# Certificate transparency logs (no API key required)
curl "https://crt.sh/?q=%.evil-domain.com&output=json" | jq '.[].name_value'Certificate transparency logs reveal all subdomains for a target domain, often exposing staging, VPN, or internal infrastructure inadvertently made public.
Step 3: Shodan Infrastructure Mapping
import shodan
api = shodan.Shodan("YOUR_SHODAN_API_KEY")
# Search for specific C2 framework signatures (Cobalt Strike beacon)
results = api.search('product:"Cobalt Strike" port:443')
for r in results['matches']:
print(r['ip_str'], r['port'], r['org'], r.get('ssl', {}).get('cert', {}).get('subject', ''))
# Find infrastructure associated with a known threat actor's ASN
results = api.search('asn:AS12345 http.title:"Redirector"')Correlate Shodan results with passive DNS to build infrastructure clusters.
Step 4: Maltego Graph Analysis
In Maltego, use these built-in transforms for threat actor infrastructure mapping:
- Start with a known malicious domain (Entity: Domain)
- Run "To IP Address [DNS]" → identifies hosting IPs
- Run "To Shared Hosting" → identifies co-hosted domains (potentially same threat actor)
- Run "To DNS Name [Reverse DNS]" → identifies PTR records
- Run "To Whois" → identifies registrant email/organization
- Pivot on registrant email → "To Domains [Registrant Email]" → expands to all domains registered with same email
Maltego Maltego Cyber threat intelligence transforms (VirusTotal, Shodan, PassiveTotal, URLScan) extend graph coverage.
Step 5: Dark Web and Paste Site Monitoring
Use SpiderFoot HX or manual searches for:
- Paste sites (Pastebin, Ghostbin): search for leaked credentials, IOCs, malware configs
- Dark web forums: via Tor browser with appropriate operational security
- GitHub/GitLab: search for exposed credentials or organization-specific strings
# SpiderFoot CLI for automated OSINT
python sf.py -s evil-domain.com -m sfp_shodan,sfp_virustotal,sfp_passivetotal \
-o TF -R result.jsonKey Concepts
| Term | Definition |
|---|---|
| Passive OSINT | Intelligence collection that does not send any packets to target systems — uses public databases, search engines, cached data |
| PIR | Priority Intelligence Requirement — specific question the intelligence collection must answer, preventing unfocused data gathering |
| Certificate Transparency | Public log of all SSL/TLS certificates issued by CAs, enabling discovery of subdomains via crt.sh |
| Pivoting | Using one data point (IP, email, registrant name) to discover related infrastructure or accounts |
| ASN | Autonomous System Number — block of IP addresses under a single routing policy; useful for clustering threat actor infrastructure |
| Co-hosted Domains | Multiple domains resolving to the same IP, potentially indicating shared attacker infrastructure |
Tools & Systems
- Maltego: Graph-based link analysis platform with 50+ data source transforms for IP, domain, email, and social media analysis
- Shodan: Internet-wide scanner database with 1B+ indexed devices; supports banner, port, SSL certificate, and vulnerability searches
- SpiderFoot: Automated OSINT tool with 200+ modules covering DNS, WHOIS, dark web, breach data, and social media
- Recon-ng: Python-based OSINT framework with modular design for domain, email, and social media reconnaissance
- crt.sh: Free certificate transparency search engine for subdomain and certificate discovery
- OSINT Framework (osintframework.com): Curated directory of OSINT tools organized by intelligence category
Common Pitfalls
- Leaving digital footprints: Visiting a threat actor's website or Shodan-queried IP can alert the adversary. Use Tor or VPN with a dedicated OSINT VM.
- Confirmation bias in graph analysis: Maltego graphs can create false connections. Verify each pivot independently before treating as confirmed.
- Outdated data: WHOIS privacy services and bulletproof hosting rotate frequently. Always check data timestamps — 6-month-old passive DNS may no longer be valid.
- Attribution overconfidence: Infrastructure overlap does not guarantee same threat actor. False flag operations deliberately share indicators across groups.
- Legal boundaries: Some OSINT tools perform active scans (port scanning, banner grabbing). Confirm tool behavior before use against external targets without authorization.