1043 Open source projects that are alternatives of or similar to Mimo-Crawler

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

A crawler based on Phantom. Allows discovery of dynamic content and supports custom scrapers.

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Be nice on the web

The simple, easy to use command line web crawler.

bingip2hosts is a Bing.com web scraper that discovers websites by IP address

No description or website provided.

Create agents that monitor and act on your behalf. Your agents are standing by!

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

Web Extension for Firefox/Chrome/MS Edge and CLI tool to save a faithful copy of an entire web page in a self-extracting HTML/ZIP polyglot file

Continuous scalable web crawler built on top of Flink and crawler-commons

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

API for fetching data from news websites.

PHP Metacritic API - Mirrored by my GitLab

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Apache Nutch is an extensible and scalable web crawler

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

[Unmaintained] A simple and clean video/music/image downloader 👾

This repository contains all the code I use in my YouTube tutorials.

Creating Scrapy scrapers via the Django admin interface

🌎 🏨 Scrape Booking.com 🏨 🌎

A web crawler for Go

Social media research and promotion, semi-autonomous CLI bot

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

Python Web Crawler with Selenium and PhantomJS

단 몇줄의 코드로 다양한 쇼핑 플랫폼에서 결제 내역을 긁어오자!

Declarative web scraping

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Elegant Scraper and Crawler Framework for Golang

News extraction and scraping. Article Parsing

An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL

Interactive CLI Web Crawler

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

An R web crawler and scraper

Scrape Instagram content and stories anonymously, using a new technique based on the har file (No Token + No public API).

Crawly, a high-level web crawling & scraping framework for Elixir.

Golang pkg to quickly return a preview of a webpage (title/description/images)

HTTP API for Scrapy spiders

Distributed crawler powered by Headless Chrome

A python web scraper for public email lists.

A collection of awesome web crawler,spider in different languages

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Extract structured data from web sites. Web sites scraping.

A library that scrapes Linkedin for user data

robots.txt file parsing and checking for R

ProxyCrawl Python library for scraping and crawling

This repo contains regression and classification projects. Examples: development of predictive models for comments on social media websites; building classifiers to predict outcomes in sports competitions; churn analysis; prediction of clicks on online ads; analysis of the opioids crisis and an analysis of retail store expansion strategies using…

Bypass repubblica.it and lastampa.it paywall

Download Impartus lectures, convert to mkv for offline viewing.

A simple web scraping engine supporting concurrent and anonymous scraping

Projeto de calculo de Imposto de Renda em operacoes na bovespa automaticamente. Tags:canal eletronico do investidor, CEI, selenium, bovespa, IRPF, IR, imposto de renda, finance, yahoo finance, acao, fii, etf, python, crawler, webscraping, calculadora ir

Containerise compatible domain lists

Scraps historical stock market data from Yahoo Finance (https://finance.yahoo.com/)

Get Aliexpress product details in JSON

State of the art object detection in real-time using YOLOV3 algorithm. Augmented with a process that allows easy training of the classifier as a plug & play solution . Provides alert if an item in an alert list is detected.