1958 Open source projects that are alternatives of or similar to Scrapple

lots of spider (很多爬虫）

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Netflix like full-stack application with SPA client and backend implemented in service oriented architecture

Scrapy crawler to collect data on the back catalog of songs listed for sale.

Declarative web scraping

Distributed crawler powered by Headless Chrome

A Facebook crawler

Redis-based components for Scrapy.

🕸 List of mini projects that involve web scraping 🕸

[Unmaintained] A simple and clean video/music/image downloader 👾

Web scraper for grabing data from Linkedin profiles or company pages (personal project)

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

用严肃的数据来回答“什么样的企业会到什么样的大学招聘”？

Random User-Agent middleware based on fake-useragent

Scrapy-based Crawlers for news of Taiwan

Some scrapy and web.py exmaples

Scrapy, a fast high-level web crawling & scraping framework for Python.

A scalable web crawler framework for Java.

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

Parser and database to index the terpene profile of different strains of Cannabis from online databases

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

Universal scrapping tool, which allows you to extract data using multiple environments

Elegant Scraper and Crawler Framework for Golang

Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

Tool to scrape linkedin

🌎 🏨 Scrape Booking.com 🏨 🌎

Chopper is a tool to extract elements from HTML by preserving ancestors and CSS rules

Web Scraping Framework

A Scrapy extension to log items coverage when the spider shuts down

Simplify your ETL processes with these hands-on data sanitation tips, tricks, and best practices

⚽ Instantly find 🏆EURO 2016 live-streams & highlights, now a Web App!

A Node.js library to easily manage and rotate a pool of web browsers, using any of the popular browser automation libraries like Puppeteer, Playwright, or SecretAgent.

🐤️ Lost Ark wait notifier

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

Simple and friendly Bot for Instagram, using Selenium and Scrapy with Python.

Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).

API for scrapping common meme sites

Building a model to recognize incentives for landscape restoration in environmental policies from Latin America, the US and India. Bringing NLP to the world of policy analysis through an extensible framework that includes scraping, preprocessing, active learning and text analysis pipelines.

Build and deploy a machine learning app from scratch 🚀

Basic setup with random user agents and IP addresses for Python Scrapy Framework.

PTT 網路版爬蟲

A Scraper made 100% in Python using BeautifulSoup and Tor. It can be used to scrape both normal and onion links. Happy Scraping :)

📓 Minicurso de raspagem de dados web com Python ministrado na Semana de Tecnologia da FATEC Jundiaí

Scraping articles of medium and providing audio versions 📑 to 🔊 using django

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

No description or website provided.

line-bot-tutorial use python flask

NBA Stats API via Basketball Reference

一个灵活、友好的爬虫框架

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

Simple yet powerful automation stuffs.

Pythonic HTML Parsing for Humans™