YoongiKim / Autocrawler
Licence: apache-2.0
Google, Naver multiprocess image web crawler (Selenium)
Stars: ✭ 957
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Autocrawler
Instagram Profilecrawl
💻 Quickly crawl the information (e.g. followers, tags, etc...) of an instagram profile. No login required!
Stars: ✭ 110 (-88.51%)
Mutual labels: crawler, selenium, chromedriver
headless-chrome
Implementation of the new headless chrome with chromedriver and selenium.
Stars: ✭ 34 (-96.45%)
Mutual labels: selenium, chromedriver
Sitemap Generator
Easily create XML sitemaps for your website.
Stars: ✭ 273 (-71.47%)
Mutual labels: google, crawler
scribd-dl
Command-line program to download Scribd documents in pdf format
Stars: ✭ 23 (-97.6%)
Mutual labels: selenium, chromedriver
dusker
Stand alone Laravel Dusk test suit, which do not require Laravel framework itself
Stars: ✭ 28 (-97.07%)
Mutual labels: selenium, chromedriver
TikTok
Download public videos on TikTok using Python with Selenium
Stars: ✭ 37 (-96.13%)
Mutual labels: selenium, chromedriver
Undetected Chromedriver
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
Stars: ✭ 365 (-61.86%)
Mutual labels: selenium, chromedriver
Node Chromedriver
An installer and wrapper for Chromedriver.
Stars: ✭ 378 (-60.5%)
Mutual labels: selenium, chromedriver
Docker Python Chromedriver
Dockerfile for running Python Selenium in headless Chrome (Python 2.7 / 3.6 / 3.7 / 3.8 / Alpine based Python / Chromedriver / Selenium / Xvfb included in different versions)
Stars: ✭ 385 (-59.77%)
Mutual labels: selenium, chromedriver
Insta-Bot
Python bot using Selenium increasing Instagram Followers.
Stars: ✭ 62 (-93.52%)
Mutual labels: selenium, chromedriver
pyderman
Install Selenium-compatible Chrome/Firefox/Opera/PhantomJS/Edge webdrivers automatically.
Stars: ✭ 24 (-97.49%)
Mutual labels: selenium, chromedriver
yt-videos-list
Create and **automatically** update a list of all videos on a YouTube channel (in txt/csv/md form) via YouTube bot with end-to-end web scraping - no API tokens required. Multi-threaded support for YouTube videos list updates.
Stars: ✭ 64 (-93.31%)
Mutual labels: selenium, chromedriver
devtools-proxy
Multiplexing proxy for Chrome DevTools. Fully compatible with Selenium and ChromeDriver
Stars: ✭ 64 (-93.31%)
Mutual labels: selenium, chromedriver
lostark-wait-notifier
🐤️ Lost Ark wait notifier
Stars: ✭ 38 (-96.03%)
Mutual labels: crawler, selenium
Price Monitor
京东商品价格监控:监控用户设定商品价格,降价邮件/微信提醒。技术:Python爬虫/IP代理池/JS接口爬取/Selenium页面爬取
Stars: ✭ 634 (-33.75%)
Mutual labels: crawler, selenium
chameleon-crawler
Browser automation for Chameleon.
Stars: ✭ 17 (-98.22%)
Mutual labels: selenium, chromedriver
docker-selenium-lambda
The simplest demo of chrome automation by python and selenium in AWS Lambda
Stars: ✭ 172 (-82.03%)
Mutual labels: selenium, chromedriver
Instagramcrawler
A non API python program to crawl public photos, posts or followers
Stars: ✭ 349 (-63.53%)
Mutual labels: crawler, selenium
AutoCrawler
Google, Naver multiprocess image crawler (High Quality & Speed & Customizable)
How to use
-
Install Chrome
-
pip install -r requirements.txt
-
Write search keywords in keywords.txt
-
Run "main.py"
-
Files will be downloaded to 'download' directory.
Arguments
usage:
python3 main.py [--skip true] [--threads 4] [--google true] [--naver true] [--full false] [--face false] [--no_gui auto] [--limit 0]
--skip true Skips keyword if downloaded directory already exists. This is needed when re-downloading.
--threads 4 Number of threads to download.
--google true Download from google.com (boolean)
--naver true Download from naver.com (boolean)
--full false Download full resolution image instead of thumbnails (slow)
--face false Face search mode
--no_gui auto No GUI mode. (headless mode) Acceleration for full_resolution mode, but unstable on thumbnail mode.
Default: "auto" - false if full=false, true if full=true
(can be used for docker linux system)
--limit 0 Maximum count of images to download per site. (0: infinite)
Full Resolution Mode
You can download full resolution image of JPG, GIF, PNG files by specifying --full true
Data Imbalance Detection
Detects data imbalance based on number of files.
When crawling ends, the message show you what directory has under 50% of average files.
I recommend you to remove those directories and re-download.
Remote crawling through SSH on your server
sudo apt-get install xvfb <- This is virtual display
sudo apt-get install screen <- This will allow you to close SSH terminal while running.
screen -S s1
Xvfb :99 -ac & DISPLAY=:99 python3 main.py
Customize
You can make your own crawler by changing collect_links.py
Issues
As google site consistently changes, please make issues if it doesn't work.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].