528 Open source projects that are alternatives of or similar to crawlkit

PhantomJS/Node.js web scraper for AWS Lambda

Crawly, a high-level web crawling & scraping framework for Elixir.

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

[Unmaintained] A simple and clean video/music/image downloader 👾

Extract structured data from web sites. Web sites scraping.

A tutorial-based introduction to web scraping with Python.

ProxyCrawl Python library for scraping and crawling

Python Web Crawler with Selenium and PhantomJS

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

No description or website provided.

Distributed crawler powered by Headless Chrome

Elegant Scraper and Crawler Framework for Golang

News, full-text, and article metadata extraction in Python 3. Advanced docs:

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL

HTTP API for Scrapy spiders

Declarative web scraping

Universal scrapping tool, which allows you to extract data using multiple environments

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

A PHP Scraping Class

Web service and CLI tool for SEO site audit: crawl site, lighthouse all pages, view public reports in browser. Also output to console, json, csv, xlsx, Google Drive.

京东App自动登录，在线商品自动下单

No description or website provided.

🌌 High productivity semi-automatic crawler generator 🛠️🧰

Code relating to scraping public police data.

There was no free Dictionary API on the web when I wanted one for my friend, so I created one.

Page metadata as a service

No description or website provided.

Install Selenium-compatible Chrome/Firefox/Opera/PhantomJS/Edge webdrivers automatically.

A web scraper starter project

Create a ZIM file from a Youtube channel/username/playlist

🔎 A simple yet powerful module to retrieve organic search results and much more from Google.

Monitors for restocks

Get Aliexpress product details in JSON

A simple web scraper built around the JavaFX WebEngine

Open bank data for Leumi bank and Leumi card credit card

Web scraper for grabing data from Linkedin profiles or company pages (personal project)

Access parts of your account unavailable through normal YouTube API access.

Search and download mangas from the command line

Scraper for downloading the entire ebooks repository of project Gutenberg

Node.js based scraper using headless chrome

Monitor instagram user account and automatically post new images to discord channel via a webhook. Working 2022!

Using this tool you can easily add so many members from any group to your group. Less than 2 minutes. Super easy. Time saver. But this tool is only for educational purpose. You could be banned from Telegram. So be careful. Recommanded to use this tool only on Termux.

Twitter Troll & Fake News Hunter - Crawls news websites and twitter to identify fake news

An open source webapp for scraping: towards a public service for webscraping

Web crawling and scraping framework for Golang

A monadic web scraping library

Automate Javascript/Ajax websites with Perl and PhantomJS

At present contains scraped data from around 1500 problems present on the site. More to follow....

Scrapes VK user's photos

❄️ Extract links, ids, and names from a youtube playlist

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

📷 Automate full website screenshots and PDF generation with multiple viewport support.

A free TVTime API based on scraping TVTime website. No API key required

Tools for downloading agendas, minutes and other documents produced by local government

By serving course 📚 data that is more "edible" 🍰 for developers, we hope CourseCake offers a smooth approach to build useful tools for students.

Handy API for Headless Chromium

Extract rich metadata from URLs

Download public videos on TikTok using Python with Selenium

Command line program to download documents from web portals.