All Projects → urbanadventurer → bing-ip2hosts

urbanadventurer / bing-ip2hosts

Licence: GPL-3.0 License
bingip2hosts is a Bing.com web scraper that discovers websites by IP address

Programming Languages

shell
77523 projects

Projects that are alternatives of or similar to bing-ip2hosts

Spiderfoot
SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
Stars: ✭ 6,882 (+6851.52%)
Mutual labels:  osint, reconnaissance, osint-reconnaissance, osint-tool
mailcat
Find existing email addresses by nickname using API/SMTP checking methods without user notification. Please, don't hesitate to improve cat's job! 🐱🔎 📬
Stars: ✭ 219 (+121.21%)
Mutual labels:  osint, reconnaissance, osint-reconnaissance, osint-tool
Mr.Holmes
🔍 A Complete Osint Tool
Stars: ✭ 307 (+210.1%)
Mutual labels:  osint, kali-linux, osint-tool
Sudomy
Sudomy is a subdomain enumeration tool to collect subdomains and analyzing domains performing automated reconnaissance (recon) for bug hunting / pentesting
Stars: ✭ 859 (+767.68%)
Mutual labels:  kali-linux, kali, reconnaissance
Theharvester
E-mails, subdomains and names Harvester - OSINT
Stars: ✭ 6,175 (+6137.37%)
Mutual labels:  osint, discovery, reconnaissance
Maryam
Maryam: Open-source Intelligence(OSINT) Framework
Stars: ✭ 371 (+274.75%)
Mutual labels:  search-engine, osint, reconnaissance
Intrigue Core
Discover Your Attack Surface!
Stars: ✭ 1,013 (+923.23%)
Mutual labels:  osint, discovery, reconnaissance
querytool
Querytool is an OSINT framework based on Google Spreadsheets. With this tool you can perform complex search of terms, people, email addresses, files and many more.
Stars: ✭ 104 (+5.05%)
Mutual labels:  osint, reconnaissance, osint-reconnaissance
Asnip
ASN target organization IP range attack surface mapping for reconnaissance, fast and lightweight
Stars: ✭ 126 (+27.27%)
Mutual labels:  osint, discovery, reconnaissance
Discover
Custom bash scripts used to automate various penetration testing tasks including recon, scanning, parsing, and creating malicious payloads and listeners with Metasploit.
Stars: ✭ 2,548 (+2473.74%)
Mutual labels:  osint, kali-linux, reconnaissance
Sudomy
Sudomy is a subdomain enumeration tool to collect subdomains and analyzing domains performing automated reconnaissance (recon) for bug hunting / pentesting
Stars: ✭ 1,572 (+1487.88%)
Mutual labels:  kali-linux, kali, reconnaissance
google-this
🔎 A simple yet powerful module to retrieve organic search results and much more from Google.
Stars: ✭ 88 (-11.11%)
Mutual labels:  search-engine, scraper
JAR
Just Another Repo
Stars: ✭ 53 (-46.46%)
Mutual labels:  kali-linux, kali
Pentest-Bookmarkz
A collection of useful links for Pentesters
Stars: ✭ 118 (+19.19%)
Mutual labels:  osint, kali-linux
openwrt-useful-tools
A repo containing different tools compiled specifically for the Hak5 WiFi Pineapple MK6 and MK7.
Stars: ✭ 155 (+56.57%)
Mutual labels:  kali-linux, kali
Reconky-Automated Bash Script
Reconky is an great Content Discovery bash script for bug bounty hunters which automate lot of task and organized in the well mannered form which help them to look forward.
Stars: ✭ 167 (+68.69%)
Mutual labels:  osint, reconnaissance
Converto
Installing Kali linux on Vps Server
Stars: ✭ 100 (+1.01%)
Mutual labels:  kali-linux, kali
Hacker-Playbook-Utility
A Python3 powered bash script written to slightly automate the tool installation process of the Hacker Playbook Book (http://thehackerplaybook.com).
Stars: ✭ 27 (-72.73%)
Mutual labels:  kali-linux, kali
AttackSurfaceManagement
Discover the attack surface and prioritize risks with our continuous Attack Surface Management (ASM) platform - Sn1per Professional #pentest #redteam #bugbounty
Stars: ✭ 45 (-54.55%)
Mutual labels:  osint, reconnaissance
Mimo-Crawler
A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.
Stars: ✭ 22 (-77.78%)
Mutual labels:  scraper, webscraping

License made-with-bash

Bing-ip2hosts

Bing-ip2hosts is a Bing.com web scraper to discover hostnames by IP address.

bing-ip2hosts-github-io

Description

Bing-ip2hosts is a Bing.com web scraper that discovers hostnames by IP address. Bing is the flagship Microsoft search engine formerly known as MSN Search and Live Search.

It provides a feature unique to search engines - it allows searching by IP address. Bing-ip2hosts uses this feature.

It can be used to discover subdomains and other related domains. It also helps to identify websites hosted in a shared hosting environment. This technique follows best practices during the reconnaissance phase of a penetration test or bug bounty, to expand the target's attack surface.

Unlike other many other recon tools that web scrape Bing, this tool has smart scraping behaviour to maximize the discovery of hostnames.

Features

  • Smart scraping behaviour to maximize hostname discovery.
  • Console user interface showing scraping progress.
  • Discovers subdomains and hostnames by IP address.
  • Can search by hostname or IP address.
  • Output with or without URL prefix.
  • Output to file, in list or CSV format.
  • Bing API key not required.
  • Select the search language and market.
  • Specify targets from the command line or from a file.
  • Lightweight Bash shell script without heavy dependencies.

Bing Web Scraping

Search by IP address with Bing

Bing provides a feature unique to search engines - it allows searching by IP address. To try this, go to Bing.com and search for IP:40.113.200.201. It should show you results from microsoft.com. If it shows empty results, then add a single dot.

Smart Scraping Behaviour

Unlike other Bing web scrapers that stop after scraping 10 result pages, bing-ip2hosts can scrape thousands of results. It continues scraping search result pages until it no longer finds new results.

Scraping completes when any of the following conditions are met:

  • After a configurable threshold of pages fail to return new results (default: 5).
  • A single page of search results, e.g. 10 or less results.
  • The last page of search results.
  • Empty results.

It also alerts the user when Bing reports that some results have been removed.

Avoid Empty Search Results

If searching by an IP address returns empty search results, add a single dot. Bing-ip2hosts always appends a single dot (%2e) to the query to avoid this issue.

Search Language and Market

By default this tool specifies the search language as "en-us". The market is left as unset, as this seems to maximize results.

The following URL parameters can be configured:

  • setlang (Language)
  • setmkt (Market code)

Both these parameters can affect how many results are returned.

A full list of market codes can be found at docs.microsoft.com/en-us/azure/cognitive-services/bing-web-search/language-support.

Repeating Search Result Pages

Sometimes Bing does not permit the user to reach the end of search result pages.

For example, in a search that shows 3 pages of results, it will not always allow the user to reach the 3rd page. Instead it will return the first page of results. This can be demonstrated by searching for ip:8.8.8.8 .. Note that it is not always the first page that it returned to.

Help

usage

Use the following command for usage information.

bing-ip2hosts is a Bing.com web scraper that discovers websites by IP address.
Use for OSINT and discovering attack-surface of penetration test targets.

Usage: ./bing-ip2hosts [OPTIONS] IP|hostname

OPTIONS are:
-o FILE	Output hostnames to FILE.
-i FILE	Input list of IP addresses or hostnames from FILE.
-n NUM	Stop after NUM scraped pages return no new results (Default: 5).
-l	Select the language for use in the setlang parameter (Default: en-us).
-m	Select the market for use in the setmkt parameter (Default is unset).
-u	Only display hostnames. Default is to include URL prefixes.
-c	CSV output. Outputs the IP and hostname on each line, separated by a comma.
-q	Quiet. Disable output except for final results.
-t DIR	Use this directory instead of /tmp.
-V	Display the version number of bing-ip2hosts and exit.

Installation

Dependencies

bing-ip2hosts requires wget. This is installed by default in Ubuntu Linux and Kali Linux.

It can be installed in macOS with homebrew.

homebrew install wget

It can be installed in Debian and Ubuntu Linux with apt.

sudo apt install wget

Install

Copy bing-ip2hosts into a folder in your $PATH.

sudo cp ./bing-ip2hosts /usr/local/bin/

Compatibility

Bing-ip2hosts uses the Bash scripting language.

It is known to work with the following systems.

  • Ubuntu Linux
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
  • macOS Catalina

The version of Bash bundled with macOS was published in 2007 and is the most recent version licensed with GPLv2. More recent versions are licensed with GPLv3, with licence terms that preclude Apple from bundling it in macOS.

GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19)
Copyright (C) 2007 Free Software Foundation, Inc.

Links

Related Projects

Here's a list of projects that also search Bing by IP address.

Here's a list of other related projects for recon using Bing. Note that these do not search Bing by IP address.

Author

Copyright Andrew Horton, aka urbanadventurer.

Licensing

This project is licensed under GPL version 3. See the attached LICENSE.txt.

Contributing

⚒️ If you have any ideas, just open an issue and tell me what you think.

If you'd like to contribute, please fork the repository and make changes as you'd like. Pull requests are warmly welcome.

Community contributors

  • @paralax
  • @mathieu-aubin (bug fix)

Acknowledgments

This project uses the following opensource packages.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].