All Projects → chairco → 2017_PyConTW_Talk

chairco / 2017_PyConTW_Talk

Licence: other
tw.pycon.org/2017/events/talk/314386410792550475/

Programming Languages

javascript
184084 projects - #8 most used programming language
python
139335 projects - #7 most used programming language
HTML
75241 projects
CSS
56736 projects

Projects that are alternatives of or similar to 2017 PyConTW Talk

sse-option-crawler
SSE 50 index options crawler 上证50期权数据爬虫
Stars: ✭ 17 (-5.56%)
Mutual labels:  crawler
ptt-web-crawler
PTT 網路版爬蟲
Stars: ✭ 20 (+11.11%)
Mutual labels:  crawler
tasuku
✅ タスク — The minimal task runner for Node.js
Stars: ✭ 1,488 (+8166.67%)
Mutual labels:  task
Sharingan
We will try to find your visible basic footprint from social media as much as possible - 😤 more sites is comming soon
Stars: ✭ 13 (-27.78%)
Mutual labels:  crawler
domfind
A Python DNS crawler to find identical domain names under different TLDs.
Stars: ✭ 22 (+22.22%)
Mutual labels:  crawler
timer.cljs
Scheduling async operations in Clojurescript
Stars: ✭ 22 (+22.22%)
Mutual labels:  task
auto crawler ptt beauty image
Auto Crawler Ptt Beauty Image Use Python Schedule
Stars: ✭ 35 (+94.44%)
Mutual labels:  crawler
WeiboCrawler
无cookie版微博爬虫,可以连续爬取一个或多个新浪微博用户信息、用户微博及其微博评论转发。
Stars: ✭ 45 (+150%)
Mutual labels:  crawler
vania
A module which fairly distributes a list of arbitrary objects among a set of targets, considering weights.
Stars: ✭ 75 (+316.67%)
Mutual labels:  task
lostark-wait-notifier
🐤️ Lost Ark wait notifier
Stars: ✭ 38 (+111.11%)
Mutual labels:  crawler
lets
CLI task runner for developers - a better alternative to make
Stars: ✭ 50 (+177.78%)
Mutual labels:  task
medium-stat-box
Practical pinned gist which show your latest medium status 📌
Stars: ✭ 29 (+61.11%)
Mutual labels:  crawler
TripAdvisor-Crawling-Suite
Fetching hotel data from TripAdvisor.
Stars: ✭ 17 (-5.56%)
Mutual labels:  crawler
arachnod
High performance crawler for Nodejs
Stars: ✭ 17 (-5.56%)
Mutual labels:  crawler
BilibiliCrawler
🌀 crawl bilibili user info and video info for data analysis | BiliBili爬虫
Stars: ✭ 25 (+38.89%)
Mutual labels:  crawler
flink-crawler
Continuous scalable web crawler built on top of Flink and crawler-commons
Stars: ✭ 48 (+166.67%)
Mutual labels:  crawler
spiderable-middleware
🤖 Prerendering for JavaScript powered websites. Great solution for PWAs (Progressive Web Apps), SPAs (Single Page Applications), and other websites based on top of front-end JavaScript frameworks
Stars: ✭ 29 (+61.11%)
Mutual labels:  crawler
Crawling-CV-Conference-Papers
Crawling CV conference papers with Python.
Stars: ✭ 32 (+77.78%)
Mutual labels:  crawler
resc
A task orchestrator using redis, written in rust
Stars: ✭ 27 (+50%)
Mutual labels:  task
concurrent-tasks
A simple task runner which will run all tasks till completion, while maintaining concurrency limits.
Stars: ✭ 27 (+50%)
Mutual labels:  task

用 Django 建立一個可以設定爬蟲排程任務與監控的網站

範例環境設定

  • 建立 log 資料夾和 .log 檔案

    cd code/pycontw/
    mkdir -p logs
    cd logs
    touch django.log ghostdriver.log project.log
  • 建立環境變數 local.env, local.py

    cp pycontw/settings/local_sample.env pycontw/settings/local.env
    cp pycontw/settings/local_sample.py pycontw/settings/local.py
  • 設定環境變數 在執行終端機模式執行下面指令取得 Secret_key:

    $ python -c 'import random; import string; print("".join([random.SystemRandom().choice(string.digits + string.ascii_letters + string.punctuation) for i in range(100)]))'

    vim local.env 將剛剛產生的 key 填入 SECRET_KEY= 不用加引號 ''

  • 確認 requirments.txt 內套件都安裝,執行 migrate 就完成了。 因為 lazy_logger 最新版本 0.1.2 尚未上傳到 PyPi 所以請先到 lazy-logger 資料夾內 python setup.py install 安裝最新版本 lazy_logger 確認 lazy_logger 的版本是 0.1.2 萬一低於 0.1.2 版本會造成使用 @logger.patch 的 hook 回傳值是 None。

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].