performing-open-source-intelligence-gathering

Performing Open Source Intelligence Gathering

Legal Notice: This skill is for authorized security testing and educational purposes only. Unauthorized use against systems you do not own or have written permission to test is illegal and may violate computer fraud laws.

Overview

Open Source Intelligence (OSINT) gathering is the first active phase of a red team engagement, where operators collect publicly available information about the target organization to identify attack surfaces, potential targets for social engineering, technology stacks, and credential exposures. Effective OSINT directly shapes initial access strategies and reduces operational risk.

When to Use

• When conducting security assessments that involve performing open source intelligence gathering
• When following incident response procedures for related security events
• When performing scheduled security testing or auditing activities
• When validating security controls through hands-on testing

Prerequisites

• Familiarity with red teaming concepts and tools
• Access to a test or lab environment for safe execution
• Python 3.8+ with required dependencies installed
• Appropriate authorization for any testing activities

Objectives

• Enumerate the target organization's external attack surface (domains, IPs, cloud assets)
• Identify employees and their roles for social engineering targeting
• Discover leaked credentials, API keys, and sensitive documents
• Map the organization's technology stack and vendors
• Identify physical locations, office layouts, and access control details
• Build target profiles for spearphishing campaign development

Core Concepts

OSINT Categories

Category	Sources	Value
Domain Intelligence	DNS records, WHOIS, CT logs, subdomain enumeration	Network attack surface
Personnel Intelligence	LinkedIn, social media, conference talks, publications	Social engineering targets
Credential Intelligence	Breach databases, paste sites, GitHub leaks	Valid credential discovery
Technology Intelligence	Job postings, Wappalyzer, Shodan, Censys	Vulnerability identification
Physical Intelligence	Google Maps, social media photos, Glassdoor	Physical access planning
Document Intelligence	SEC filings, public documents, metadata extraction	Organizational structure

MITRE ATT&CK Mapping

• T1595.001 - Active Scanning: Scanning IP Blocks
• T1595.002 - Active Scanning: Vulnerability Scanning
• T1592 - Gather Victim Host Information
• T1589 - Gather Victim Identity Information
• T1590 - Gather Victim Network Information
• T1591 - Gather Victim Org Information
• T1593 - Search Open Websites/Domains
• T1594 - Search Victim-Owned Websites
• T1596 - Search Open Technical Databases

Workflow

Phase 1: Domain and Network Reconnaissance

1. Perform WHOIS lookups for target domains
2. Enumerate subdomains using Certificate Transparency logs, DNS brute-force, and web scraping
3. Identify IP ranges and ASN ownership
4. Scan for exposed services using Shodan/Censys
5. Check for cloud storage buckets (S3, Azure Blob, GCS)
6. Map CDN and hosting providers

Phase 2: Personnel and Social Intelligence

1. Enumerate employees via LinkedIn, company website, and conference speaker lists
2. Identify email naming conventions
3. Discover personal social media accounts of key targets
4. Map organizational hierarchy and reporting structure
5. Identify recently hired IT/security personnel
6. Check for conference presentations and technical publications

Phase 3: Credential and Data Leak Discovery

1. Search breach databases (Have I Been Pwned, DeHashed)
2. Check paste sites (Pastebin, GitHub Gists)
3. Search GitHub/GitLab for leaked secrets and API keys
4. Look for exposed configuration files and backups
5. Check for leaked internal documents via Google dorking

Phase 4: Technology Stack Identification

1. Analyze job postings for technology mentions
2. Use Wappalyzer/BuiltWith for web technology fingerprinting
3. Check for exposed admin panels and development environments
4. Identify VPN and remote access technologies
5. Map cloud services and SaaS applications

Tools and Resources

Tool	Purpose	Type
Amass	Subdomain enumeration and network mapping	Open Source
Subfinder	Passive subdomain discovery	Open Source
theHarvester	Email, subdomain, and name harvesting	Open Source
Maltego	Visual link analysis and data correlation	Commercial
SpiderFoot	Automated OSINT collection	Open Source
Shodan	Internet-connected device search	Commercial
Censys	Internet asset discovery	Commercial
Recon-ng	Web reconnaissance framework	Open Source
GitDorker	GitHub secret scanning	Open Source
Photon	Web crawler for OSINT	Open Source

Validation Criteria

• [ ] Complete list of target domains and subdomains
• [ ] Employee list with roles and email addresses
• [ ] Technology stack identified
• [ ] Credential leak assessment completed
• [ ] Attack surface map documented
• [ ] OSINT report compiled for engagement team