All Projects → eschultze → Urlextractor

eschultze / Urlextractor

Licence: mit
Information gathering & website reconnaissance | https://phishstats.info/

Programming Languages

shell
77523 projects

Projects that are alternatives of or similar to Urlextractor

Threatpinchlookup
Documentation and Sharing Repository for ThreatPinch Lookup Chrome & Firefox Extension
Stars: ✭ 257 (-24.63%)
Mutual labels:  osint, whois, incident-response, shodan, virustotal
Infoga
Infoga - Email OSINT
Stars: ✭ 947 (+177.71%)
Mutual labels:  osint, information-extraction, information-gathering, shodan
Intelmq
IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.
Stars: ✭ 611 (+79.18%)
Mutual labels:  incident-response, malware, phishing
Whour
Tool for information gathering, IPReverse, AdminFInder, DNS, WHOIS, SQLi Scanner with google.
Stars: ✭ 18 (-94.72%)
Mutual labels:  information-extraction, whois, information-gathering
Reconspider
🔎 Most Advanced Open Source Intelligence (OSINT) Framework for scanning IP Address, Emails, Websites, Organizations.
Stars: ✭ 621 (+82.11%)
Mutual labels:  osint, information-gathering, shodan
E4GL30S1NT
E4GL30S1NT - Simple Information Gathering Tool
Stars: ✭ 139 (-59.24%)
Mutual labels:  osint, whois, information-gathering
Stargather
A fast GitHub stargazers information gathering tool
Stars: ✭ 30 (-91.2%)
Mutual labels:  osint, information-extraction, information-gathering
Deadtrap
An OSINT tool to gather information about the real owner of a phone number
Stars: ✭ 73 (-78.59%)
Mutual labels:  osint, information-extraction, information-gathering
Pentesting Bible
Learn ethical hacking.Learn about reconnaissance,windows/linux hacking,attacking web technologies,and pen testing wireless networks.Resources for learning malware analysis and reverse engineering.
Stars: ✭ 8,981 (+2533.72%)
Mutual labels:  osint, malware, information-gathering
ThePhish
ThePhish: an automated phishing email analysis tool
Stars: ✭ 676 (+98.24%)
Mutual labels:  incident-response, malware, phishing
TweetFeed
Collecting IOCs posted on Twitter
Stars: ✭ 181 (-46.92%)
Mutual labels:  osint, malware, phishing
freki
🐺 Malware analysis platform
Stars: ✭ 327 (-4.11%)
Mutual labels:  malware, virustotal
ShonyDanza
A customizable, easy-to-navigate tool for researching, pen testing, and defending with the power of Shodan.
Stars: ✭ 86 (-74.78%)
Mutual labels:  shodan, information-gathering
Vajra
Vajra is a highly customizable target and scope based automated web hacking framework to automate boring recon tasks and same scans for multiple target during web applications penetration testing.
Stars: ✭ 269 (-21.11%)
Mutual labels:  osint, information-gathering
Osintgram
Osintgram is a OSINT tool on Instagram. It offers an interactive shell to perform analysis on Instagram account of any users by its nickname
Stars: ✭ 312 (-8.5%)
Mutual labels:  osint, information-gathering
MalwareHashDB
Malware hashes for open source projects.
Stars: ✭ 31 (-90.91%)
Mutual labels:  malware, virustotal
Open-source-tools-for-CTI
Public Repository of Open Source Tools for Cyber Threat Intelligence Analysts and Researchers
Stars: ✭ 91 (-73.31%)
Mutual labels:  osint, malware
Stalkphish
StalkPhish - The Phishing kits stalker, harvesting phishing kits for investigations.
Stars: ✭ 256 (-24.93%)
Mutual labels:  osint, phishing
tenssens
Tenssens framework focused on gathering information from free tools or resources. The intention is to help people find free OSINT resources.
Stars: ✭ 19 (-94.43%)
Mutual labels:  osint, information-gathering
Dnstwist
Domain name permutation engine for detecting homograph phishing attacks, typo squatting, and brand impersonation
Stars: ✭ 3,124 (+816.13%)
Mutual labels:  osint, phishing

URLextractor

Information gathering & website reconnaissance

Usage: ./extractor http://www.hackthissite.org/

Tips:

  • Colorex: put colors to the ouput pip install colorex and use it like ./extractor http://www.hackthissite.org/ | colorex -g "INFO" -r "ALERT"
  • Tldextract: is used by dnsenumeration function pip install tldextract

Features:

  • IP and hosting info like city and country (using FreegeoIP)
  • DNS servers (using dig)
  • ASN, Network range, ISP name (using RISwhois)
  • Load balancer test
  • Whois for abuse mail (using Spamcop)
  • PAC (Proxy Auto Configuration) file
  • Compares hashes to diff code
  • robots.txt (recursively looking for hidden stuff)
  • Source code (looking for passwords and users)
  • External links (frames from other websites)
  • Directory FUZZ (like Dirbuster and Wfuzz - using Dirbuster) directory list)
  • URLvoid API - checks Google page rank, Alexa rank and possible blacklists
  • Provides useful links at other websites to correlate with IP/ASN
  • Option to open ALL results in browser at the end

Changelog to version 0.2.0:

  • [Fix] Changed GeoIP from freegeoip to ip-api
  • [Fix/Improvement] Remove duplicates from robots.txt
  • [Improvement] Better whois abuse contacts (abuse.net)
  • [Improvement] Top passwords collection added to sourcecode checking
  • [New feature] Firt run verification to install dependencies if need
  • [New feature] Log file
  • [New feature] Check for hostname on log file
  • [New feature] Check if hostname is listed on Spamaus Domain Blacklist
  • [New feature] Run a quick dnsenumeration with common server names

Changelog to version 0.1.9:

  • Abuse mail using lynx istead of curl
  • Target server name parsing fixed
  • More verbose about HTTP codes and directory discovery
  • MD5 collection for IP fixed
  • Links found now show unique URLs from array
  • [New feature] Google results
  • [New feature] Bing IP check for other hosts/vhosts
  • [New feature] Opened ports from Shodan
  • [New feature] VirusTotal information about IP
  • [New feature] Alexa Rank information about $TARGET_HOST

Requirements:

Tested on Kali light mini AND OSX 10.11.3 with brew

sudo apt-get install bc curl dnsutils libxml2-utils whois md5sha1sum lynx openssl -y

Configuration file:

CURL_TIMEOUT=15 #timeout in --connect-timeout
CURL_UA=Mozilla #user-agent (keep it simple)
INTERNAL=NO #YES OR NO (show internal network info)
URLVOID_KEY=your_API_key #using API from http://www.urlvoid.com/
FUZZ_LIMIT=10 #how many lines it will read from fuzz file
OPEN_TARGET_URLS=NO #open found URLs at the end of script
OPEN_EXTERNAL_LINKS=NO #open external links (frames) at the end of script
FIRST_TIME=YES #if first time check for dependecies

Todo list:

  • [x] Upload to github :)
  • [x] Check for installed packages
  • [ ] Integration with other APIs
  • [ ] Export to CSV
  • [ ] Integration with CipherScan

Stargazers over time

Stargazers over time

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].