288 Open source projects that are alternatives of or similar to Raspagem-de-dados-para-iniciantes

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Parser and database to index the terpene profile of different strains of Cannabis from online databases

A collection of awesome web scaper, crawler.

Building a model to recognize incentives for landscape restoration in environmental policies from Latin America, the US and India. Bringing NLP to the world of policy analysis through an extensible framework that includes scraping, preprocessing, active learning and text analysis pipelines.

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

A set of reusable Java components that implement functionality common to any web crawler

使用asyncio和aiohttp开发的轻量级异步协程web爬虫框架

A simple tool for fetching usable proxies from several websites.

Apache Nutch is an extensible and scalable web crawler

Crawling the data from lazada, websosanh, compare.vn, cdiscount and cungmua with flexible configs

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Explore a website recursively and download all the wanted documents (PDF, ODT…)

Dynamic configurable crawl (动态可配置化爬虫)

简单易用的Python爬虫框架，QQ交流群：597510560

A simple but powerful web crawler library for .NET

开源矢量地理数据获取与预处理工具(POI/AOI/行政区/路网/土地利用)

Inventus is a spider designed to find subdomains of a specific domain by crawling it and any subdomains it discovers.

A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Using scrapy to scrape tripadvisor in order to get users' reviews.

Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.

Analyze scraped data

This is a project of build knowledge graph course. The project leverages historical stock price, and integrates social media listening from customers to predict market Trend On Dow Jones Industrial Average (DJIA).

快速登录大连理工大学统一身份认证系统（SSO）的Python模块，可轻松实现成绩提醒、抢课、玉兰卡信息、个人信息查询等功能。

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Araignée LeBonCoin avec Scrapy et ElasticSearch

A simple distributed crawler for zhihu && data analysis

Classificador de poemas do Fernando Pessoa de acordo com os seus heterônimos

python爬虫项目集合

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Norconex HTTP Collector is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.

A web crawler for Go

Simply scrape / download all the media from an fansly account

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Iota is a web scraper which can find all of the images and links/suburls on a webpage

Turn large Web sites into tables and charts using simple SQLs.

Common interface for data container classes

Web Crawler

A scrapy middleware to use rotated proxy ip list.

Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.

Distributed crawling/scraping, Kafka And Redis based components for Scrapy

Rust Web Crawler saving pages on Redis

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Django to build a Search Engine website --- README_zh.md (including: implementation roadmap, distributed-crawler and coping with anti-crawling strategies).

A scalable, mature and versatile web crawler based on Apache Storm

python+scrapy+splash 爬取拉勾全站职位信息

A collection of awesome web crawler,spider in different languages

scrapy mysql pipeline

World’s single largest Internet domains dataset

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

ACHE is a web crawler for domain-specific search.

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Scraping the real estate agencies for up-to-date house listings as soon as they arrive!

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

The simple, easy to use command line web crawler.

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

虎扑步行街爬虫