scraperNodejs web scraper. Contains a command line, docker container, terraform module and ansible roles for distributed cloud scraping. Supported databases: SQLite, MySQL, PostgreSQL. Supported headless clients: Puppeteer, Playwright, Cheerio, JSdom.
Stars: ✭ 37 (+48%)
OnegramThis repository is no longer maintained.
Stars: ✭ 137 (+448%)
SocialInfo4Jfetch data from Facebook, Instagram and LinkedIn
Stars: ✭ 44 (+76%)
immo-feedA extensible app for scraping property listings
Stars: ✭ 35 (+40%)
ttc subway timesA scraper to grab and publish TTC subway arrival times.
Stars: ✭ 40 (+60%)
NewspaperNews, full-text, and article metadata extraction in Python 3. Advanced docs:
Stars: ✭ 11,545 (+46080%)
RPICovidScraperscraper for Rensselaer Polytechnic Institute (RPI)'s Covid Dashboard
Stars: ✭ 12 (-52%)
cat-messageFinds cat images/videos/gifs on reddit, sends them to my mom via applescript
Stars: ✭ 35 (+40%)
ScraperA scraper that switches between normal mode and gentleman mode, built on Eletron, React
Stars: ✭ 127 (+408%)
ArxivscraperA python module to scrape arxiv.org for specific date range and categories
Stars: ✭ 121 (+384%)
arachnodHigh performance crawler for Nodejs
Stars: ✭ 17 (-32%)
stweetAdvanced python library to scrap Twitter (tweets, users) from unofficial API
Stars: ✭ 287 (+1048%)
extract-cssExtract all CSS from a webpage, packaged as a Now V2 Lambda
Stars: ✭ 23 (-8%)
SeleniumcrawlerAn example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
Stars: ✭ 117 (+368%)
papercutPapercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.
Stars: ✭ 15 (-40%)
EsriRESTScraperA Python class that scrapes ESRI Rest Endpoints and exports data to a geodatabase
Stars: ✭ 43 (+72%)
Android-Apps-Downloader📱 A tool to download android apps from Google Play Store and Xiaomi App Store (the famous Chinese Store).
Stars: ✭ 16 (-36%)
Ridereceipts🚕 Simple automation desktop app to download and organize your receipts from Uber/Lyft. Try out our new Ride Receipts PRO !
Stars: ✭ 117 (+368%)
scrapy-LBCAraignée LeBonCoin avec Scrapy et ElasticSearch
Stars: ✭ 14 (-44%)
spiderMultithreaded Web spider crawler written in Rust.
Stars: ✭ 81 (+224%)
HeadlesschromeA Go package for working with headless Chrome. Run interactive JavaScript commands on web pages with Go and Chrome.
Stars: ✭ 112 (+348%)
ogcheckr-apiAn api to check social media username availability on a variety of services
Stars: ✭ 18 (-28%)
OnlyFansScrape all the media from an OnlyFans account - Updated regularly
Stars: ✭ 573 (+2192%)
scoopi-scraperScoopi Web Scraper is a heavy duty tool to extract data from HTML pages.
Stars: ✭ 18 (-28%)
fb-page-chat-downloadPython script to download messages from a Facebook page to a CSV file
Stars: ✭ 51 (+104%)
scrapy facebookerCollection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.
Stars: ✭ 22 (-12%)
TikTokDownloader PyWebIO🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok数据爬取工具,支持API调用,在线批量解析及下载。
Stars: ✭ 919 (+3576%)
evineInteractive CLI Web Crawler
Stars: ✭ 140 (+460%)
SillyniumAutomate the creation of Python Selenium Scripts by drawing coloured boxes on webpage elements
Stars: ✭ 100 (+300%)
patreon-scraperWIP Patreon attachment download written in TypeScript
Stars: ✭ 25 (+0%)
web-crawlerPython Web Crawler with Selenium and PhantomJS
Stars: ✭ 19 (-24%)
Scraper-Projects🕸 List of mini projects that involve web scraping 🕸
Stars: ✭ 25 (+0%)
ScrapoxyScrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!
Stars: ✭ 1,322 (+5188%)
metacritic apiPHP Metacritic API - Mirrored by my GitLab
Stars: ✭ 31 (+24%)
TelegramScraperUsing this tool you can easily add so many members from any group to your group. Less than 2 minutes. Super easy. Time saver. But this tool is only for educational purpose. You could be banned from Telegram. So be careful. Recommanded to use this tool only on Termux.
Stars: ✭ 234 (+836%)
wishlistRead an Amazon wishlist programmatically with Python
Stars: ✭ 44 (+76%)
AzurLaneWikiScrapersA console application that can scrape the Azur Lane wiki and export the data to Json files
Stars: ✭ 12 (-52%)
awesome-interfaceAngularJS SPA interface for awesome lists. Awesome lists parsed using python.
Stars: ✭ 25 (+0%)
ceiba-dlNTU CEIBA 資料下載工具
Stars: ✭ 80 (+220%)
Instagram ScraperScrapes an instagram user's photos and videos
Stars: ✭ 5,664 (+22556%)
InstaloctrackAn Instagram OSINT tool to collect all the geotagged locations available on an Instagram profile in order to plot them on a map, and dump them in a JSON.
Stars: ✭ 85 (+240%)
freeDictionaryAPIThere was no free Dictionary API on the web when I wanted one for my friend, so I created one.
Stars: ✭ 1,352 (+5308%)
proxycrawl-pythonProxyCrawl Python library for scraping and crawling
Stars: ✭ 51 (+104%)
CourseCakeBy serving course 📚 data that is more "edible" 🍰 for developers, we hope CourseCake offers a smooth approach to build useful tools for students.
Stars: ✭ 21 (-16%)
YT-DLP-SCRIPTS...Just a place for me to share my various YT-DLP & related bash scripts.
Stars: ✭ 70 (+180%)
twpyTwitter High level scraper for humans.
Stars: ✭ 58 (+132%)
Thepiratebay💀 The Pirate Bay node.js client
Stars: ✭ 191 (+664%)