Querylist🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Stars: ✭ 2,392 (+12489.47%)
Freshonions TorscraperFresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
Stars: ✭ 348 (+1731.58%)
unfurlExtract rich metadata from URLs
Stars: ✭ 41 (+115.79%)
XidelCommand line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Stars: ✭ 335 (+1663.16%)
JvppeteerHeadless Chrome For Java (Java 爬虫)
Stars: ✭ 193 (+915.79%)
AutoscraperA Smart, Automatic, Fast and Lightweight Web Scraper for Python
Stars: ✭ 4,077 (+21357.89%)
CryptocmdCryptocurrency historical price data library in Python. Data from https://coinmarketcap.com.
Stars: ✭ 299 (+1473.68%)
UnfurlScraper for oEmbed, Twitter Cards and Open Graph metadata - fast and Promise-based ⚡️
Stars: ✭ 193 (+915.79%)
Socialmanagertools Gui🤖 👻 Desktop application for Instagram Bot, Twitter Bot and Facebook Bot
Stars: ✭ 293 (+1442.11%)
Java Spider一个基于webmagic框架二次开发的java爬虫框架实战,已实现能爬取腾讯,搜狐,今日头条(单独集成功能)等资讯内容,配合elasticsearch框架用法,实现了自动爬虫,已投入线上生产使用。
Stars: ✭ 276 (+1352.63%)
Anime DlAnime-dl is a command-line program to download anime from CrunchyRoll and Funimation.
Stars: ✭ 190 (+900%)
Weibo terminator workflowUpdate Version of weibo_terminator, This is Workflow Version aim at Get Job Done!
Stars: ✭ 259 (+1263.16%)
fb-scraperScrape a Facebook profile and turn it into a JSON file
Stars: ✭ 18 (-5.26%)
GmdbGMDB is the ultra-simple, cross-platform Movie Library with Features (Search, Take Note, Watch Later, Like, Import, Learn, Instantly Torrent Magnet Watch)
Stars: ✭ 189 (+894.74%)
kaa.si-cliStream anime from kaa.si and sync with anilist
Stars: ✭ 12 (-36.84%)
ts-ebmlEBML encoder and decoder
Stars: ✭ 130 (+584.21%)
Unhtml.rsA magic html parser
Stars: ✭ 180 (+847.37%)
Instagram-Scraper-2021Scrape Instagram content and stories anonymously, using a new technique based on the har file (No Token + No public API).
Stars: ✭ 57 (+200%)
diffbot-php-client[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library
Stars: ✭ 53 (+178.95%)
SearchScraperAPIAiohttp web server API, which scrapes Google and returns scrape results as response. Supports proxies, multiple geos and number of results.
Stars: ✭ 31 (+63.16%)
Linkedin Profile Scraper🕵️♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
Stars: ✭ 171 (+800%)
facebook-discussion-tkA collection of tools to (semi-)automatically collect and analyze data from online discussions on Facebook groups and pages.
Stars: ✭ 33 (+73.68%)
go-jd京东App自动登录,在线商品自动下单
Stars: ✭ 158 (+731.58%)
trainline-pythonNon-official Python wrapper and CLI tool for Trainline
Stars: ✭ 41 (+115.79%)
Novel基于 Laravel 5.2 的小说网站
Stars: ✭ 172 (+805.26%)
dijnet-botAz összes számlád még egy helyen :)
Stars: ✭ 17 (-10.53%)
scrapeerEssential PHP library that scrapes HTTP(S) and UDP trackers for torrent information.
Stars: ✭ 81 (+326.32%)
scraperNodejs web scraper. Contains a command line, docker container, terraform module and ansible roles for distributed cloud scraping. Supported databases: SQLite, MySQL, PostgreSQL. Supported headless clients: Puppeteer, Playwright, Cheerio, JSdom.
Stars: ✭ 37 (+94.74%)
Scrapelib⛏ a library for scraping things
Stars: ✭ 164 (+763.16%)
SocialInfo4Jfetch data from Facebook, Instagram and LinkedIn
Stars: ✭ 44 (+131.58%)
sotokiStackExchange websites to ZIM scraper
Stars: ✭ 64 (+236.84%)
ttc subway timesA scraper to grab and publish TTC subway arrival times.
Stars: ✭ 40 (+110.53%)
OpensanctionsAn open database of international sanctions data, persons of interest and politically exposed persons
Stars: ✭ 157 (+726.32%)
RPICovidScraperscraper for Rensselaer Polytechnic Institute (RPI)'s Covid Dashboard
Stars: ✭ 12 (-36.84%)
roseAnalyse all kinds of data for a TV series
Stars: ✭ 34 (+78.95%)
Instagram Scraperscrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot
Stars: ✭ 2,209 (+11526.32%)
quoters📝 Random quotes generator package. Available on npm and PyPi
Stars: ✭ 17 (-10.53%)
wget-luaWget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
Stars: ✭ 52 (+173.68%)
SerpscrapSEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.
Stars: ✭ 153 (+705.26%)
arachnodHigh performance crawler for Nodejs
Stars: ✭ 17 (-10.53%)
gHarvesterProof of concept for a security issue (in my opinion) that I found in accounts.google.com
Stars: ✭ 20 (+5.26%)
extract-cssExtract all CSS from a webpage, packaged as a Now V2 Lambda
Stars: ✭ 23 (+21.05%)
PhpscraperPHP Scraper - an highly opinionated web-interface for PHP
Stars: ✭ 148 (+678.95%)
papercutPapercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.
Stars: ✭ 15 (-21.05%)
Scanlessonline port scan scraper
Stars: ✭ 875 (+4505.26%)
Google2csvGoogle2Csv a simple google scraper that saves the results on a csv/xlsx/jsonl file
Stars: ✭ 145 (+663.16%)
OLX Scraper📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (-21.05%)
PDAP-ScrapersCode relating to scraping public police data.
Stars: ✭ 72 (+278.95%)
esajScrapers for many e-SAJ systems
Stars: ✭ 35 (+84.21%)
Instagram-to-discordMonitor instagram user account and automatically post new images to discord channel via a webhook. Working 2022!
Stars: ✭ 113 (+494.74%)
reinforcement learning course materialsLecture notes, tutorial tasks including solutions as well as online videos for the reinforcement learning course hosted by Paderborn University
Stars: ✭ 765 (+3926.32%)
jd-autobuyPython爬虫,京东自动登录,在线抢购商品
Stars: ✭ 1,262 (+6542.11%)