AutoscraperA Smart, Automatic, Fast and Lightweight Web Scraper for Python
Stars: ✭ 4,077 (+1832.23%)
GeziyorGeziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Stars: ✭ 1,246 (+490.52%)
FerretDeclarative web scraping
Stars: ✭ 4,837 (+2192.42%)
NickjsWeb scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)
Stars: ✭ 494 (+134.12%)
Hquery.phpAn extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
Stars: ✭ 295 (+39.81%)
CrawlyCrawly, a high-level web crawling & scraping framework for Elixir.
Stars: ✭ 440 (+108.53%)
Parser JavascriptBrowser sniffing gone too far — A useragent parser library for JavaScript
Stars: ✭ 66 (-68.72%)
bots-zooNo description or website provided.
Stars: ✭ 59 (-72.04%)
Scrapysharpreborn of https://bitbucket.org/rflechner/scrapysharp
Stars: ✭ 226 (+7.11%)
CollyElegant Scraper and Crawler Framework for Golang
Stars: ✭ 15,535 (+7262.56%)
papercutPapercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.
Stars: ✭ 15 (-92.89%)
Lulu[Unmaintained] A simple and clean video/music/image downloader 👾
Stars: ✭ 789 (+273.93%)
Linkedin Profile Scraper🕵️♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
Stars: ✭ 171 (-18.96%)
Lodestone NodejsCharacter tracking and parser library for nodejs
Stars: ✭ 81 (-61.61%)
Blockchain ParserThe simpliest script for parsing Bitcoin blockchain. It made convertion of blk*****.dat files to the simple text.
Stars: ✭ 84 (-60.19%)
GooglescraperA Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
Stars: ✭ 2,363 (+1019.91%)
ScrapoxyScrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!
Stars: ✭ 1,322 (+526.54%)
Graphql Go ToolsTools to write high performance GraphQL applications using Go/Golang.
Stars: ✭ 96 (-54.5%)
Media ScraperScrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok
Stars: ✭ 206 (-2.37%)
WebmagicA scalable web crawler framework for Java.
Stars: ✭ 10,186 (+4727.49%)
ScrapyScrapy, a fast high-level web crawling & scraping framework for Python.
Stars: ✭ 42,343 (+19967.77%)
Instagram Profilecrawl💻 Quickly crawl the information (e.g. followers, tags, etc...) of an instagram profile. No login required!
Stars: ✭ 110 (-47.87%)
Black WidowGUI based offensive penetration testing tool (Open Source)
Stars: ✭ 124 (-41.23%)
Mini YamlSingle header YAML 1.0 C++11 serializer/deserializer.
Stars: ✭ 79 (-62.56%)
DotnetcrawlerDotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c
Stars: ✭ 100 (-52.61%)
Dan Jurafsky Chris Manning NlpMy solution to the Natural Language Processing course made by Dan Jurafsky, Chris Manning in Winter 2012.
Stars: ✭ 124 (-41.23%)
Tianyanchapip安装的天眼查爬虫API,指定的单个/多个企业工商信息一键保存为Excel/JSON格式。A Battery-included Scraper API of Tianyancha, the best Chinese business data and investigation platform.
Stars: ✭ 206 (-2.37%)
LibdparseLibrary for lexing and parsing D source code
Stars: ✭ 91 (-56.87%)
Email ExtractorThe main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url
Stars: ✭ 81 (-61.61%)
LiefAuthors
Stars: ✭ 2,730 (+1193.84%)
Querylist🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Stars: ✭ 2,392 (+1033.65%)
WombatLightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Stars: ✭ 1,220 (+478.2%)
Jikan RestThe REST API for Jikan
Stars: ✭ 200 (-5.21%)
Sywac🚫 🐭 Asynchronous, single package CLI framework for Node
Stars: ✭ 109 (-48.34%)
D4n155OWASP D4N155 - Intelligent and dynamic wordlist using OSINT
Stars: ✭ 105 (-50.24%)
Whois ParserGo(Golang) module for domain whois information parsing.
Stars: ✭ 123 (-41.71%)
SeleniumcrawlerAn example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
Stars: ✭ 117 (-44.55%)
NewspaperNews, full-text, and article metadata extraction in Python 3. Advanced docs:
Stars: ✭ 11,545 (+5371.56%)
GoscraperGolang pkg to quickly return a preview of a webpage (title/description/images)
Stars: ✭ 72 (-65.88%)
ParjsJavaScript parser-combinator library
Stars: ✭ 145 (-31.28%)
Youtube ProjectsThis repository contains all the code I use in my YouTube tutorials.
Stars: ✭ 144 (-31.75%)
PhpscraperPHP Scraper - an highly opinionated web-interface for PHP
Stars: ✭ 148 (-29.86%)
AntchAntch, a fast, powerful and extensible web crawling & scraping framework for Go
Stars: ✭ 198 (-6.16%)
Google Play ScraperGoogle play scraper for Python inspired by <facundoolano/google-play-scraper>
Stars: ✭ 143 (-32.23%)
SerpscrapSEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.
Stars: ✭ 153 (-27.49%)
Secret AgentThe web browser that's built for scraping.
Stars: ✭ 151 (-28.44%)
RatsMovie Ratings Synchronization with Python
Stars: ✭ 156 (-26.07%)
AbotCross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Stars: ✭ 1,961 (+829.38%)
Datmusic ApiAlternative for VK Audio API
Stars: ✭ 160 (-24.17%)
Jsonframe Cheeriosimple multi-level scraper json input/output for Cheerio
Stars: ✭ 196 (-7.11%)
OnegramThis repository is no longer maintained.
Stars: ✭ 137 (-35.07%)
Instagram Scraperscrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot
Stars: ✭ 2,209 (+946.92%)
Command Line ApiCommand line parsing, invocation, and rendering of terminal output.
Stars: ✭ 2,418 (+1045.97%)
WhispersIdentify hardcoded secrets and dangerous behaviours
Stars: ✭ 66 (-68.72%)
Parse XmlA fast, safe, compliant XML parser for Node.js and browsers.
Stars: ✭ 184 (-12.8%)