roccomuso / Is Google
Licence: mit
Verify that a request is from Google crawlers using Google's DNS verification steps
Stars: ✭ 82
Programming Languages
javascript
184084 projects - #8 most used programming language
js
455 projects
Projects that are alternatives of or similar to Is Google
Scavenger
Crawler (Bot) searching for credential leaks on different paste sites.
Stars: ✭ 347 (+323.17%)
Mutual labels: bot, crawler
Autocrawler
Google, Naver multiprocess image web crawler (Selenium)
Stars: ✭ 957 (+1067.07%)
Mutual labels: google, crawler
Information collection handbook
Handbook of information collection for penetration testing and src
Stars: ✭ 447 (+445.12%)
Mutual labels: dns, ip
Line Bot Tutorial
line-bot-tutorial use python flask
Stars: ✭ 267 (+225.61%)
Mutual labels: bot, crawler
Simplenikeaccountcreator
A simple nike account creator I made using puppeteer and request in node js
Stars: ✭ 47 (-42.68%)
Mutual labels: bot, verify
Sitemap Generator
Easily create XML sitemaps for your website.
Stars: ✭ 273 (+232.93%)
Mutual labels: google, crawler
ebook
Third edition of the Computer Networking: Principles, Protocols and Practice ebook
Stars: ✭ 64 (-21.95%)
Mutual labels: dns, ip
Vulnx
vulnx 🕷️ is an intelligent bot auto shell injector that detect vulnerabilities in multiple types of cms { `wordpress , joomla , drupal , prestashop .. `}
Stars: ✭ 1,009 (+1130.49%)
Mutual labels: bot, crawler
Arachnid
Powerful web scraping framework for Crystal
Stars: ✭ 68 (-17.07%)
Mutual labels: bot, crawler
MicroDNSSrv
A micro DNS server for MicroPython to simply respond to A queries on multi-domains with or without wildcards (used on Pycom modules & ESP32)
Stars: ✭ 43 (-47.56%)
Mutual labels: dns, ip
domfind
A Python DNS crawler to find identical domain names under different TLDs.
Stars: ✭ 22 (-73.17%)
Mutual labels: dns, crawler
Encrypted Dns
Configuration profiles for DNS HTTPS and DNS over TLS for iOS 14 and MacOS Big Sur
Stars: ✭ 455 (+454.88%)
Mutual labels: google, dns
email-checker
Provides email verification on the go.
Stars: ✭ 116 (+41.46%)
Mutual labels: dns, verify
is-google
Verify that a request is from Google crawlers using Google's DNS verification steps
You may wish to verify that a web crawler accessing your server is Googlebot (or another Google user-agent) and not spammers or other bots scraping your site while claiming to be Googlebot. Since you cannot rely on the User-Agent
header which is easily spoofed, you need to use DNS look up to verify that the IP address belongs to Google.
This library implements Google's own verification steps outlined here.
Install
npm install --save is-google
Example
const isGoogle = require('is-google')
let ip = '66.249.66.1'
isGoogle(ip).then((outcome) => {
if (outcome) {
// it's google.
}
}).catch(console.error)
Example with express
app.enable('trust proxy')
app.use((req, res, next) => {
let ip = req.ip || req.connection.remoteAddress
isGoogle(ip).then(outcome => {
if (outcome) {
res.status(404).text('Nothing to scan') // block google crawler
} else {
next() // it's a user
}
})
})
Tests
npm test
License
MIT
Author
Rocco Musolino @roccomuso
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].