lucasxlu / Lagoujob
Licence: apache-2.0
Job data mining repo for lagou.com
Stars: ✭ 256
Programming Languages
Projects that are alternatives of or similar to Lagoujob
Datascience
Curated list of Python resources for data science.
Stars: ✭ 3,051 (+1091.8%)
Mutual labels: data-analysis, data-mining
Loan-Approval-Prediction
Loan Application Data Analysis
Stars: ✭ 61 (-76.17%)
Mutual labels: data-mining, data-analysis
PracticalMachineLearning
A collection of ML related stuff including notebooks, codes and a curated list of various useful resources such as books and softwares. Almost everything mentioned here is free (as speech not free food) or open-source.
Stars: ✭ 60 (-76.56%)
Mutual labels: data-mining, data-analysis
Python practice of data analysis and mining
《Python数据分析与挖掘实战》随书源码与数据
Stars: ✭ 172 (-32.81%)
Mutual labels: data-analysis, data-mining
Amazing Feature Engineering
Feature engineering is the process of using domain knowledge to extract features from raw data via data mining techniques. These features can be used to improve the performance of machine learning algorithms. Feature engineering can be considered as applied machine learning itself.
Stars: ✭ 218 (-14.84%)
Mutual labels: data-analysis, data-mining
python-notebooks
A collection of Jupyter Notebooks used in conferences or just to have some snippets.
Stars: ✭ 14 (-94.53%)
Mutual labels: data-mining, data-analysis
Sourced Ce
source{d} Community Edition (CE)
Stars: ✭ 153 (-40.23%)
Mutual labels: data-analysis, data-mining
Spotify-Song-Recommendation-ML
UC Berkeley team's submission for RecSys Challenge 2018
Stars: ✭ 70 (-72.66%)
Mutual labels: data-mining, data-analysis
genie
Genie: A Fast and Robust Hierarchical Clustering Algorithm (this R package has now been superseded by genieclust)
Stars: ✭ 21 (-91.8%)
Mutual labels: data-mining, data-analysis
Data Science Resources
👨🏽🏫You can learn about what data science is and why it's important in today's modern world. Are you interested in data science?🔋
Stars: ✭ 171 (-33.2%)
Mutual labels: data-analysis, data-mining
genieclust
Genie++ Fast and Robust Hierarchical Clustering with Noise Point Detection - for Python and R
Stars: ✭ 34 (-86.72%)
Mutual labels: data-mining, data-analysis
Pipeline
the `pipeline` shell command
Stars: ✭ 168 (-34.37%)
Mutual labels: data-analysis, data-mining
Deepgraph
Analyze Data with Pandas-based Networks. Documentation:
Stars: ✭ 232 (-9.37%)
Mutual labels: data-analysis, data-mining
Awesome Ts Anomaly Detection
List of tools & datasets for anomaly detection on time-series data.
Stars: ✭ 2,027 (+691.8%)
Mutual labels: data-analysis, data-mining
Machine learning for good
Machine learning fundamentals lesson in interactive notebooks
Stars: ✭ 142 (-44.53%)
Mutual labels: data-analysis, data-mining
taller SparkR
Taller SparkR para las Jornadas de Usuarios de R
Stars: ✭ 12 (-95.31%)
Mutual labels: data-mining, data-analysis
popular restaurants from officials
서울시 공무원의 업무추진비를 분석하여 진짜 맛집 찾기 프로젝트
Stars: ✭ 22 (-91.41%)
Mutual labels: data-mining, data-analysis
Data Analysis of Lagou Job
Introduction
This repository holds the code for job data analysis of Lagou. The main functions included are listed as follows:
- Crawling job data from Lagou, and get the latest information of jobs about Internet.
- Proxies are collected from XiCiDaiLi.
- Data analysis and visualization.
- Crawling job details info and generate word cloud as Job Impression.
- In order to train a NLP task with machine learning, the data of interviewee's comments will be stored in mongodb
Prerequisites
-
Install 3rd party libraries
sudo pip3 install -r requirements.txt -
Install mongodb and start mongodb service [optional]
sudo service mongod start
How to Use
- clone this project from github.
- Lagou's anti-spider strategy has been upgrade frequently recently. I suggest you run proxy_crawler.py to get IP proxies and execute the code with PhantomJS.
- run m_lagou_spider.py to crawl job data, it will generate a collection of Excel files in
./datadirectory. - run hot_words_generator.py to cut sentences, it will return TOP-30 hot words and wordcloud figure.
Analysis Results
Report
- For technical details, please refer to my answer at Zhihu.
- The PDF report can be downloaded from here.
Change Log
- [V2.0] - 2019.04. Upgraded to PhantomJS and IP proxies.
- [V1.2] - 2017.05. Rewrite WordCloud visualization module.
- [V1.0] - 2017.04. Upgraded to mobile Lagou.
- [V0.8] - 2016.05. Finish Lagou PC web spider.
LICENSE
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].