GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → bouxinLou → Company Crawler

bouxinLou / Company Crawler

Licence: mit
天眼查爬虫&企查查爬虫，指定关键字爬取公司信息

Programming Languages

python
139335 projects - #7 most used programming language
python3
1442 projects

Labels

proxy

Projects that are alternatives of or similar to Company Crawler

Mtail
extract internal monitoring data from application logs for collection in a timeseries database
Stars: ✭ 3,028 (+962.46%)
Mutual labels:  proxy
Pushpin
Proxy server for adding push to your API
Stars: ✭ 3,050 (+970.18%)
Mutual labels:  proxy
Proxy Manager Bridge
Provides integration for ProxyManager with various Symfony components.
Stars: ✭ 274 (-3.86%)
Mutual labels:  proxy
Shadowsocks Windows
A C# port of shadowsocks
Stars: ✭ 53,572 (+18697.19%)
Mutual labels:  proxy
Manba
HTTP API Gateway
Stars: ✭ 3,000 (+952.63%)
Mutual labels:  proxy
Radioactive State
☢ Make Your React App Truly Reactive!
Stars: ✭ 273 (-4.21%)
Mutual labels:  proxy
Bus
Bus 是一个基础框架、服务套件，它基于Java8编写，参考、借鉴了大量已有框架、组件的设计，可以作为后端服务的开发基础中间件。代码简洁，架构清晰，非常适合学习使用。
Stars: ✭ 253 (-11.23%)
Mutual labels:  proxy
Mosn
The Cloud-Native Network Proxy Platform.
Stars: ✭ 3,451 (+1110.88%)
Mutual labels:  proxy
Proxyspider
代理IP 采集程序
Stars: ✭ 264 (-7.37%)
Mutual labels:  proxy
Spring Cloud Gateway
A Gateway built on Spring Framework 5.x and Spring Boot 2.x providing routing and more.
Stars: ✭ 3,305 (+1059.65%)
Mutual labels:  proxy
Dorknet
Selenium powered Python script to automate searching for vulnerable web apps.
Stars: ✭ 256 (-10.18%)
Mutual labels:  proxy
Multiple Host
虚拟host解决方案，轻松实现两套host环境
Stars: ✭ 258 (-9.47%)
Mutual labels:  proxy
Open Proxy
一键部署被墙网站反向代理; 免翻墙访问被禁网站
Stars: ✭ 274 (-3.86%)
Mutual labels:  proxy
Websocketd
Turn any program that uses STDIN/STDOUT into a WebSocket server. Like inetd, but for WebSockets.
Stars: ✭ 15,828 (+5453.68%)
Mutual labels:  proxy
Httptunnel
Bidirectional data stream tunnelled in HTTP requests.
Stars: ✭ 279 (-2.11%)
Mutual labels:  proxy
Websockify
Websockify is a WebSocket to TCP proxy/bridge. This allows a browser to connect to any application/server/service.
Stars: ✭ 2,942 (+932.28%)
Mutual labels:  proxy
Infini Gateway
INFINI-GATEWAY(极限网关), a high performance and lightweight gateway written in golang, for elasticsearch and his friends.
Stars: ✭ 272 (-4.56%)
Mutual labels:  proxy
Socks5
A full-fledged high-performance socks5 proxy server written in C#. Plugin support included.
Stars: ✭ 286 (+0.35%)
Mutual labels:  proxy
Scrapy Crawlera
Crawlera middleware for Scrapy
Stars: ✭ 281 (-1.4%)
Mutual labels:  proxy
Cloudbunny
CloudBunny is a tool to capture the real IP of the server that uses a WAF as a proxy or protection. In this tool we used three search engines to search domain information: Shodan, Censys and Zoomeye.
Stars: ✭ 273 (-4.21%)
Mutual labels:  proxy
View All Similar Projects ➔

天眼查、企查查

公司信息爬虫

使用说明

  1. 设置数据源 
    MysqlConfig = {
    'develop': {
        'host': '192.168.1.103',
        'port': 3306,
        'db': 'enterprise',
        'username': 'root',
        'password': '[email protected]'
    }
}
  2. 执行db/data.sql生成数据结构
  3. 配置IP代理config/settings 
    # 全局代理控制
GLOBAL_PROXY = True
PROXY_POOL_URL = "http://localhost:5010"
  4. 设置爬取关键字qichacha&tianyancha 
    keys = ['Google'] # 设置爬取列表
crawler.load_keys(keys)
crawler.start()

PS：建议使用IP代理 + 随机UA，否者一定会被ban

  1. 随机UA推荐fake_useragent
  2. 代理池推荐proxy_pool
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].