Lodour / XMQ-BackUp

Licence: MIT License

小密圈备份，圈子/话题/图片/文件。

Programming Languages

python

139335 projects - #7 most used programming language

Projects that are alternatives of or similar to XMQ-BackUp

Scrapy Selenium

Scrapy middleware to handle javascript pages using selenium

Stars: ✭ 550 (+2400%)

Mutual labels: selenium, scrapy

Python Spider

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

Stars: ✭ 615 (+2695.45%)

Mutual labels: selenium, scrapy

E Commerce Crawlers

🚀电商网站爬虫合集，淘宝京东亚马逊等

Stars: ✭ 377 (+1613.64%)

Mutual labels: selenium, scrapy

Post Tuto Deployment

Build and deploy a machine learning app from scratch 🚀

Stars: ✭ 368 (+1572.73%)

Mutual labels: selenium, scrapy

Python3 Spider

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Stars: ✭ 2,129 (+9577.27%)

Mutual labels: selenium, scrapy

Alipayspider Scrapy

AlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)

Stars: ✭ 70 (+218.18%)

Mutual labels: selenium, scrapy

Pythonspidernotes

Python入门网络爬虫之精华版

Stars: ✭ 5,634 (+25509.09%)

Mutual labels: selenium, scrapy

python-crawler

爬虫学习仓库，适合零基础的人学习，对新手比较友好

Stars: ✭ 37 (+68.18%)

Mutual labels: selenium, scrapy

Seleniumcrawler

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

Stars: ✭ 117 (+431.82%)

Mutual labels: selenium, scrapy

Wswp

Code for the second edition Web Scraping with Python book by Packt Publications

Stars: ✭ 112 (+409.09%)

Mutual labels: selenium, scrapy

RARBG-scraper

With Selenium headless browsing and CAPTCHA solving

Stars: ✭ 38 (+72.73%)

Mutual labels: selenium, scrapy

InstaBot

Simple and friendly Bot for Instagram, using Selenium and Scrapy with Python.

Stars: ✭ 32 (+45.45%)

Mutual labels: selenium, scrapy

Ucampus

解放双手，u校园的题再也不用写啦(暂停维护

Stars: ✭ 28 (+27.27%)

Mutual labels: selenium

scrapy facebooker

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

Stars: ✭ 22 (+0%)

Mutual labels: scrapy

V2EX Spider

V2EX爬虫

Stars: ✭ 21 (-4.55%)

Mutual labels: scrapy

ImageGrabber

A Scrapy demo : Download all images from a site

Stars: ✭ 33 (+50%)

Mutual labels: scrapy

geetest test

极验滑动验证码研究报告

Stars: ✭ 66 (+200%)

Mutual labels: selenium

pyscrapper

📷 web scrapping in python: multiple libraries -requests, beautifulsoup, mechanize, selenium

Stars: ✭ 50 (+127.27%)

Mutual labels: selenium

EHX

Realtime Browser Element Verification Tool [Stable]

Stars: ✭ 29 (+31.82%)

Mutual labels: selenium

rec-a-sketch

content discovery... IN 3D

Stars: ✭ 45 (+104.55%)

Mutual labels: selenium

View All Similar Projects ➔

XMQ-BackUp

小密圈备份，圈子/话题/图片/文件。

Usage

安装 chromedriver

仅用于自动登录，如果你愿意自己抓包，则不需要安装

brew install chromedriver
或前往官网/镜像下载
- 将包含可执行文件的目录添加至环境变量
- 或设置settings.py/CHROME_DRIVER_PATH为完整执行路径

安装 XMQ-BackUp

git clone [email protected]:Lodour/XMQ-BackUp.git
cd XMQ-BackUp
mv xmq/settings.exammple.py xmq/settings.py
virtualenv env -p python3.5
source ./env/bin/activate
pip install -r requirements.txt

运行

scrapy crawl backup
手动指定token及User-Agent
- 浏览器端登录后抓包获取request headers中的authorization和User-Agent字段
- 在xmq/settings.py末尾将其设置为XMQ_ACCESS_TOKEN和XMQ_USER_AGENT

Note

phantomjs渲染所得到的access_token不合法，所以换成了chromedriver
virtualenv下使用scrapy有问题的请参照这里
如果你的浏览器版本有更新，是需要重新设置UA的。
欢迎交流

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Lodour / XMQ-BackUp

Programming Languages

Labels

Projects that are alternatives of or similar to XMQ-BackUp

XMQ-BackUp

Usage

Note