newspaperjsNews extraction and scraping. Article Parsing
Stars: ✭ 59 (+90.32%)
HuginnCreate agents that monitor and act on your behalf. Your agents are standing by!
Stars: ✭ 33,694 (+108590.32%)
AutoscraperA Smart, Automatic, Fast and Lightweight Web Scraper for Python
Stars: ✭ 4,077 (+13051.61%)
bing-ip2hostsbingip2hosts is a Bing.com web scraper that discovers websites by IP address
Stars: ✭ 99 (+219.35%)
newsembleAPI for fetching data from news websites.
Stars: ✭ 42 (+35.48%)
XidelCommand line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Stars: ✭ 335 (+980.65%)
RcrawlerAn R web crawler and scraper
Stars: ✭ 274 (+783.87%)
Mimo-CrawlerA web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.
Stars: ✭ 22 (-29.03%)
Youtube ProjectsThis repository contains all the code I use in my YouTube tutorials.
Stars: ✭ 144 (+364.52%)
Instagram-Scraper-2021Scrape Instagram content and stories anonymously, using a new technique based on the har file (No Token + No public API).
Stars: ✭ 57 (+83.87%)
PoliteBe nice on the web
Stars: ✭ 253 (+716.13%)
robotstxtrobots.txt file parsing and checking for R
Stars: ✭ 65 (+109.68%)
scraperA web scraper starter project
Stars: ✭ 18 (-41.94%)
youtube-unofficialAccess parts of your account unavailable through normal YouTube API access.
Stars: ✭ 33 (+6.45%)
supervised-machine-learningThis repo contains regression and classification projects. Examples: development of predictive models for comments on social media websites; building classifiers to predict outcomes in sports competitions; churn analysis; prediction of clicks on online ads; analysis of the opioids crisis and an analysis of retail store expansion strategies using…
Stars: ✭ 34 (+9.68%)
impartus-downloaderDownload Impartus lectures, convert to mkv for offline viewing.
Stars: ✭ 19 (-38.71%)
Instagram-Comments-ScraperInstagram comment scraper using python and selenium. Save the comments into excel.
Stars: ✭ 73 (+135.48%)
Trakt-UserscriptsUserscripts to improve and add features to Trakt.tv
Stars: ✭ 39 (+25.81%)
stock-market-scraperScraps historical stock market data from Yahoo Finance (https://finance.yahoo.com/)
Stars: ✭ 110 (+254.84%)
web-scraping-engineA simple web scraping engine supporting concurrent and anonymous scraping
Stars: ✭ 27 (-12.9%)
google-this🔎 A simple yet powerful module to retrieve organic search results and much more from Google.
Stars: ✭ 88 (+183.87%)
scraperA simple web scraper built around the JavaFX WebEngine
Stars: ✭ 13 (-58.06%)
yt-videos-listCreate and **automatically** update a list of all videos on a YouTube channel (in txt/csv/md form) via YouTube bot with end-to-end web scraping - no API tokens required. Multi-threaded support for YouTube videos list updates.
Stars: ✭ 64 (+106.45%)
WaGpScraperA Python Oriented tool to Scrap WhatsApp Group Link using Google Dork it Scraps Whatsapp Group Links From Google Results And Gives Working Links.
Stars: ✭ 18 (-41.94%)
AzurLaneWikiScrapersA console application that can scrape the Azur Lane wiki and export the data to Json files
Stars: ✭ 12 (-61.29%)
imdb-scraper🎬 An attempt at the most complete IMDb API
Stars: ✭ 24 (-22.58%)
crawlkitA crawler based on Phantom. Allows discovery of dynamic content and supports custom scrapers.
Stars: ✭ 23 (-25.81%)
wget-luaWget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
Stars: ✭ 52 (+67.74%)
ps store helperchrome extension for injecting metacritic score to Playstation store page
Stars: ✭ 27 (-12.9%)
OLX Scraper📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (-51.61%)
VideoRecognition-realtime-autotrainer-alertsState of the art object detection in real-time using YOLOV3 algorithm. Augmented with a process that allows easy training of the classifier as a plug & play solution . Provides alert if an item in an alert list is detected.
Stars: ✭ 36 (+16.13%)
SpydanA web spider for shodan.io without using the Developer API.
Stars: ✭ 30 (-3.23%)
site-audit-seoWeb service and CLI tool for SEO site audit: crawl site, lighthouse all pages, view public reports in browser. Also output to console, json, csv, xlsx, Google Drive.
Stars: ✭ 91 (+193.55%)
awesome-interfaceAngularJS SPA interface for awesome lists. Awesome lists parsed using python.
Stars: ✭ 25 (-19.35%)
PDAP-ScrapersCode relating to scraping public police data.
Stars: ✭ 72 (+132.26%)
Android-Web-ScraperAndroid Web Scraper is a simple library for android web automation. You can perform web task in background to fetch website data programmatically.
Stars: ✭ 38 (+22.58%)
wishlistRead an Amazon wishlist programmatically with Python
Stars: ✭ 44 (+41.94%)
ScrapeMA monadic web scraping library
Stars: ✭ 17 (-45.16%)
TrollHunterTwitter Troll & Fake News Hunter - Crawls news websites and twitter to identify fake news
Stars: ✭ 38 (+22.58%)
VK-ScraperScrapes VK user's photos
Stars: ✭ 42 (+35.48%)
scraped-tvtime-apiA free TVTime API based on scraping TVTime website. No API key required
Stars: ✭ 23 (-25.81%)
opensea-scraperScrapes nft floor prices and additional information from opensea. Used for https://nftfloorprice.info
Stars: ✭ 129 (+316.13%)
tieba-zhuaqu百度贴吧分布式爬虫,用于贴吧数据挖掘。从贴吧维度和用户维度进行数据分析
Stars: ✭ 56 (+80.65%)
diostsA Go scraper that validates security.txt files and outputs them in the disclose.io JSON format.
Stars: ✭ 18 (-41.94%)
esajScrapers for many e-SAJ systems
Stars: ✭ 35 (+12.9%)
scrapisma work-in-progress guide to web scraping as an artistic and critical practice
Stars: ✭ 43 (+38.71%)
Captcha-ToolsAll-in-one Python (And now Go!) module to help solve captchas with Capmonster, 2captcha and Anticaptcha API's!
Stars: ✭ 23 (-25.81%)
irProjeto de calculo de Imposto de Renda em operacoes na bovespa automaticamente. Tags:canal eletronico do investidor, CEI, selenium, bovespa, IRPF, IR, imposto de renda, finance, yahoo finance, acao, fii, etf, python, crawler, webscraping, calculadora ir
Stars: ✭ 120 (+287.1%)
ogePage metadata as a service
Stars: ✭ 22 (-29.03%)
anime-scraper[partially working] Scrape and add anime episode stream URLs to uGet (Linux) or IDM (Windows) ~ Python3
Stars: ✭ 21 (-32.26%)
aliexscrapeGet Aliexpress product details in JSON
Stars: ✭ 80 (+158.06%)