hardikp / Scrapy Finance
Licence: mit
[OUTDATED] scrapy spiders to crawl the financial text data 📚 📜 pertinent to train word vectors 🚀
Stars: ✭ 17
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Scrapy Finance
Scrape-Finance-Data
My code for scraping financial data in Vietnam
Stars: ✭ 13 (-23.53%)
Mutual labels: finance, scrapy
Twstock
台灣股市股票價格擷取 (含即時股票資訊) - Taiwan Stock Opendata with realtime
Stars: ✭ 763 (+4388.24%)
Mutual labels: finance
Icrawler
A multi-thread crawler framework with many builtin image crawlers provided.
Stars: ✭ 629 (+3600%)
Mutual labels: scrapy
Tweetscraper
TweetScraper is a simple crawler/spider for Twitter Search without using API
Stars: ✭ 694 (+3982.35%)
Mutual labels: scrapy
House Renting
Possibly the best practice of Scrapy 🕷 and renting a house 🏡
Stars: ✭ 741 (+4258.82%)
Mutual labels: scrapy
Iex Api
The IEX API provides any individual or academic, public or private institution looking to develop applications that require stock market data to access near real-time quote and trade data for all stocks trading on IEX.
Stars: ✭ 683 (+3917.65%)
Mutual labels: finance
Webhubbot
Python + Scrapy + MongoDB . 5 million data per day !!!💥 The world's largest website.
Stars: ✭ 5,427 (+31823.53%)
Mutual labels: scrapy
Awesome Quant
A curated list of insanely awesome libraries, packages and resources for Quants (Quantitative Finance)
Stars: ✭ 8,205 (+48164.71%)
Mutual labels: finance
Awesome Streamlit
The purpose of this project is to share knowledge on how awesome Streamlit is and can be
Stars: ✭ 769 (+4423.53%)
Mutual labels: finance
Python Spider
豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
Stars: ✭ 615 (+3517.65%)
Mutual labels: scrapy
Fecon235
Notebooks for financial economics. Keywords: Jupyter notebook pandas Federal Reserve FRED Ferbus GDP CPI PCE inflation unemployment wage income debt Case-Shiller housing asset portfolio equities SPX bonds TIPS rates currency FX euro EUR USD JPY yen XAU gold Brent WTI oil Holt-Winters time-series forecasting statistics econometrics
Stars: ✭ 708 (+4064.71%)
Mutual labels: finance
Funpyspidersearchengine
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
Stars: ✭ 782 (+4500%)
Mutual labels: scrapy
scrapy-finance
scrapy spiders to crawl the financial data pertinent to train word vectors.
List of sources
-
bloomberg- Bloomberg news articles -
investopedia- Definitions of finance terms from investopedia.com -
wikipedia- Finance pages from wikipedia - all wiki pages reachable from https://en.wikipedia.org/wiki/Outline_of_finance with at most 2 hops. -
qplum- Investment articles from https://www.qplum.co/investing-library
How to use this
- Install scrapy.
pip3 install scrapy
- Run the
scrapy crawlcommand.
(py3) [email protected]:~/scrapy-finance$ scrapy crawl bloomberg
How to modify spiders for your use
Please look at the specific spider files like wikipedia.py. They are relatively easy to follow and modify.
.
├── LICENSE
├── README.md
├── scrapy.cfg
└── text
├── __init__.py
├── items.py
├── middlewares.py
├── pipelines.py
├── settings.py
└── spiders
├── bloomberg.py
├── __init__.py
├── investopedia.py
├── qplum.py
└── wikipedia.py
Notes
- The text data is written in the lower case at the moment in all spiders.
- This is not checked with python2.
Contributing
Please feel free to submit a pull request to add relevant spiders.
LICENSE
MIT
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].