All Projects → A3h1nt → Grawler

A3h1nt / Grawler

Licence: mit
Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.

Projects that are alternatives of or similar to Grawler

Apify Js
Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
Stars: ✭ 3,154 (+3118.37%)
Mutual labels:  automation, scraping, crawling
Awesome Puppeteer
A curated list of awesome puppeteer resources.
Stars: ✭ 1,728 (+1663.27%)
Mutual labels:  automation, scraping, crawling
Crawly
Crawly, a high-level web crawling & scraping framework for Elixir.
Stars: ✭ 440 (+348.98%)
Mutual labels:  scraping, crawling
Dataflowkit
Extract structured data from web sites. Web sites scraping.
Stars: ✭ 456 (+365.31%)
Mutual labels:  scraping, crawling
Nickjs
Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)
Stars: ✭ 494 (+404.08%)
Mutual labels:  automation, scraping
Katana
A Python Tool For google Hacking
Stars: ✭ 355 (+262.24%)
Mutual labels:  scraping, proxy
Comic Dl
Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.
Stars: ✭ 365 (+272.45%)
Mutual labels:  automation, scraping
Ferret
Declarative web scraping
Stars: ✭ 4,837 (+4835.71%)
Mutual labels:  scraping, crawling
Scrapy Crawlera
Crawlera middleware for Scrapy
Stars: ✭ 281 (+186.73%)
Mutual labels:  scraping, proxy
Easy Scraping Tutorial
Simple but useful Python web scraping tutorial code.
Stars: ✭ 583 (+494.9%)
Mutual labels:  scraping, crawling
Headless Chrome Crawler
Distributed crawler powered by Headless Chrome
Stars: ✭ 5,129 (+5133.67%)
Mutual labels:  scraping, crawling
Lulu
[Unmaintained] A simple and clean video/music/image downloader 👾
Stars: ✭ 789 (+705.1%)
Mutual labels:  scraping, crawling
Autoscraper
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Stars: ✭ 4,077 (+4060.2%)
Mutual labels:  automation, scraping
Spidermon
Scrapy Extension for monitoring spiders execution.
Stars: ✭ 309 (+215.31%)
Mutual labels:  scraping, crawling
Undetected Chromedriver
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
Stars: ✭ 365 (+272.45%)
Mutual labels:  automation, scraping
Sasila
一个灵活、友好的爬虫框架
Stars: ✭ 286 (+191.84%)
Mutual labels:  scraping, crawling
Nginx Le
Nginx with automatic let's encrypt (docker image)
Stars: ✭ 475 (+384.69%)
Mutual labels:  automation, proxy
Api Store
Contains all the public APIs listed in Phantombuster's API store. Pull requests welcome!
Stars: ✭ 69 (-29.59%)
Mutual labels:  automation, scraping
Gopa
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Stars: ✭ 277 (+182.65%)
Mutual labels:  scraping, crawling
Oj
Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.
Stars: ✭ 517 (+427.55%)
Mutual labels:  automation, scraping

Grawler - v1.0

Grawler is the best tool ever, made for automating google dorks it's a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file, version 1.0 is the more powerful than ever supporting multiple proxies. ( Read in features )

Table of contents

General info

Grawler aims to automate the task of using google dorks with a web interface, the main idea is to provide a simple yet powerful tool that can be used by anyone, the thing that makes Grawler different in its category is its features.

Features

  • The biggest issue faced by tools that automate google dorks is CAPTCHA, but with Grawler, CAPTCHA is not an issue anymore, Grawler comes with a proxy feature which supports three different proxies.
    • Supported Proxies ( The mentioned proxies need you to signup and get the API key, without any credit card information and give you around one thousand free API calls each )
  • Grawler now supports two different modes.
    • Automatic Mode : Automatic mode now comes with many different dork files and supports multiple proxies to deliver a smooth experience.
    • Manual Mode : The manual mode has become more powerful with the Depth feature, now you can select the number of pages you want to scrape results from, proxy feature is also supported by manual mode.
  • Dorks are now categorized in the following categories:
    • Error Messages
    • Extension
    • Java
    • JavaScript
    • Login Panels
    • .Net
    • PHP
    • SQL Injection (7 different files with different dorks)
    • My_dorks file for users to add their own dorks.
  • API keys for proxies are first validated and added to the file.
  • Manual mode allows users to go up to depth 4, but I'd recommend using depth 2 or 3 because the best results are usually on the initial pages.
  • Grawler comes with it's own guide to learn google dorks.
  • The results are stored in a file ( filename needs to be specified with txt extension ).
  • URL scraping is better than ever with no garbage URL's at all.
  • Grawler supports three different search engines (Bing, Google, Yahoo), so if one blocks you another one is available.
  • Multiple proxy with multiple search engines delivers the best experience ever.

Setup

  • Download the ZIP file
  • Download XAMPP server
  • Move the folder to htdocs folder in XAMPP
  • Navigate to http://localhost/grawler
  • Results will be stored in same directory

Docker

  • docker pull jas9reet/grawler
  • docker run -d -p 8080:80 jas9reet/grawler

Application Access

  • YOUR-IP-ADDRESS:8080

Demo

Contribute

  • Report Bugs
  • Add more effective google dorks (which actually works)
  • Work on portability
  • Suggestions

Contact Me

You can contact me here A3h1nt regarding anything.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].