496 Open source projects that are alternatives of or similar to Spider_python

camaro is an utility to transform XML to JSON, using Node.js binding to native XML parser pugixml, one of the fastest XML parser around.

Convenient way to request HTTP APIs

🎭 Twisted Deferred Thread backend for Requests.

This package is meant to make http request of an easy way inspiren in the architecture of Moya package. This package is 100% free of dependencies and works with Combine api + Codable

BaseX Main Repository.

Generating spiders dynamically to crawl and check those free proxy ip on the internet with scrapy.

a class that uses scraped proxies to make http GET/POST requests (Python requests)

python-Reptile

A mocking library for requests

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

swiss army knife for hackers

a simple & tiny scrapy clustering solution, considered a drop-in replacement for scrapyd

Redis-based components for Scrapy.

Building a model to recognize incentives for landscape restoration in environmental policies from Latin America, the US and India. Bringing NLP to the world of policy analysis through an extensible framework that includes scraping, preprocessing, active learning and text analysis pipelines.

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

基于Scrapy框架的豆瓣电影爬虫

Minimalist Requests wrapper to work within rate limits of any amount of services simultaneously. Parallel processing friendly.

PTT 網路版爬蟲

htmlquery is golang XPath package for HTML query.

知名網站avgle下載器

Borrow cookies from your browser's authenticated session for use in Python scripts.

HTTP-based DSL for for validating RESTful APIs

💂 Wrap NodeJS request module to retry http requests in case of errors

爬虫管理平台

Kotlin HTTP requests library. Similar to Python requests.

A Visual Studio Extension which can run any XPath and XPath function; navigates through results at the click of a button. Can show and copy any XPath incl. XML namespaces, avoiding XML namespace induced headaches. Keeps track of the current XPath via the statusbar.

python powered Intelligent System

📺 Youtube字幕下载脚本

Custom PHP curl library for the Laravel 5 framework - developed by Ixudra

Python related technologies used in work: crawler, data analysis, timing tasks, RPC, page parsing, decorator, built-in functions, Python objects, multi-threading, multi-process, asynchronous, redis, mongodb, mysql, openstack, etc.

BeGoneAds is a script that puts some popular hosts file lists into the systems hosts file as a adblocker measure.

🕷️some python spiders with BeautifulSoup or scarpy

基于 scrapy-redis 的通用分布式爬虫框架

Pycon 2021 Tutorial to help Spreadsheet (Excel) Users learn Python

Android 本地网络小说爬虫，基于jsoup及xpath

HTTP requests library for WASM Apps

A framework for creating semi-automatic web content extractors

秀人网爬虫 55156爬虫

Türkçe olarak hazırlanmış Python programlama dili ile ilgili içeriklerin derlendiği sayfa.

Lightweight XPath2 for .NET

The Zipru scraper developed in the Advanced Web Scraping Tutorial.

No description or website provided.

大众点评评论文本挖掘，包括点评数据爬取、数据清洗入库、数据分析、评论情感分析等的完整挖掘项目

downloads photos and videos from twitter

Web Content Retrieval for Humans™

Crawlera middleware for Scrapy

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

📷 web scrapping in python: multiple libraries -requests, beautifulsoup, mechanize, selenium

Web application hosted on Heroku cloud platform based on web scraping in python using lxml library (XML Path Language).

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

A set of spiders and scrapers to extract location information from places that post their location on the internet.

Web scraper project to collect the useful Hong Kong weather data from HKO website

V2EX爬虫

爬取 www.mzitu.com 全站图片，截至目前共5162个图集，16.5万多张美女图片，使用 asyncio 和 aiohttp 实现的异步版本只需要不到2小时就能爬取完成。按日期创建图集目录，保存更合理。控制台只显示下载的进度条，详细信息保存在日志文件中。支持异常处理，不会终止爬虫程序。失败的请求，下次再执行爬虫程序时会自动下载

A Scrapy demo : Download all images from a site

REST API calls made easier

京东定时自动预约,抢购.(只针对性支持要先预约,到点抢购.然后直接下单的商品!)

A Python library for automating interaction with websites.

Powerful, modern HTTP/REST client built on top of the Requests library

surfstudio.github.io/nodekit