1969 Open source projects that are alternatives of or similar to Grawler

A curated list of awesome puppeteer resources.

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Intelligence and Reconnaissance Package/Bundle installer.

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

Elegant Scraper and Crawler Framework for Golang

6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

A Scrapy extension to log items coverage when the spider shuts down

Crawlera middleware for Scrapy

Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.

Crawly, a high-level web crawling & scraping framework for Elixir.

The web browser that's built for scraping.

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

internet monitoring osint telegram bot for windows

Distributed crawling framework for documents and structured data.

Tutorial for web scraping / crawling with Node.js.

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

ProxyCrawl Python library for scraping and crawling

💻 make your ssh client smarter

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

A Python Tool For google Hacking

一个灵活、友好的爬虫框架

Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.

Distributed crawler powered by Headless Chrome

[Unmaintained] A simple and clean video/music/image downloader 👾

Scrapy, a fast high-level web crawling & scraping framework for Python.

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Command-line client for WebSockets, like netcat (or curl) for ws:// with advanced socat-like functions

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

An Instagram bot developed using the Selenium Framework

PHP WHOIS provides parsed and raw whois lookup of domains and ASN routes. PHP 5.4+ and 7+ compatible

cdp4j - Chrome DevTools Protocol for Java

An automated e-mail OSINT tool

👨‍👩‍👦 Social account detection and extraction in Python, e.g. for crawling/scraping.

Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.

An open source web crawling platform

Easily crawl news portals or blog sites using Storm Crawler.

Web crawling and scraping framework for Golang

Nginx with automatic let's encrypt (docker image)

Selenium powered Python script to automate searching for vulnerable web apps.

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

No description or website provided.

Contains all the public APIs listed in Phantombuster's API store. Pull requests welcome!

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Scrapy Extension for monitoring spiders execution.

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

Screen scraping and web crawling framework

Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)

Declarative web scraping

Simple but useful Python web scraping tutorial code.

Crawl sites for RSS, Atom, and JSON feeds.

Extract structured data from web sites. Web sites scraping.

Web scraping and automation using python

Python framework to scrape Pastebin pastes and analyze them

An opinionated React Framework. [Rename to pea.js]

Clash of Clans library, proxy and server written in .NET [Unmaintained]

Tiny script to automate everyday task