Financial Center: open financial data framework

• Financial Center: open financial data framework
• Overview
  • Running Shot
  • Logging Shot
  • Database Shot
  • Flask Demo Shot
• Description
• Installation guide
  • 1. Packages / Dependencies
  • 2. Database
  • 3. Redis
• Configure FinDy Settings
• Updates:
  • (25 Dec 2021)
  • (01 Mar 2021)
  • (16 Sep 2020):
• How to use
  • Fetch data
  • Read data
• Docker Usage
  • Docker Execute
  • View Log
• To Do

Overview

Running Shot

Logging Shot

Database Shot

Flask Demo Shot

Description

Based on registry framework, project can easily compat various of thrid-party data sources.
Below shows all data sources that project already supported.

To add more third party data sources, please reference (findy/database/plugins/***) modules for modification.

Installation guide

The FinDy installation consists of setting up the following components:

1. Packages / Dependencies

Command line tools

xcode-select --install #xcode command line tools

Homebrew

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install git cmake pkg-config openssl
brew link openssl --force

2. Database

FinDy recommends using a PostgreSQL database. But you can use MySQL too, see MySQL setup guide.

brew install postgresql
ln -sfv /usr/local/opt/postgresql/*.plist ~/Library/LaunchAgents
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist

Login to PostgreSQL

psql -d postgres

Create a user for FinDy.

CREATE USER xxx;

Create the FinDy production database & grant all privileges on database

CREATE DATABASE findy OWNER xxx;

Quit the database session

\q

Try connecting to the new database with the new user

sudo -u git -H psql -d findy

3. Redis

brew install redis
ln -sfv /usr/local/opt/redis/*.plist ~/Library/LaunchAgents

Redis config is located in /usr/local/etc/redis.conf. Make a copy:

cp /usr/local/etc/redis.conf /usr/local/etc/redis.conf.orig

Disable Redis listening on TCP by setting 'port' to 0

sed 's/^port .*/port 0/' /usr/local/etc/redis.conf.orig | sudo tee /usr/local/etc/redis.conf

Edit file (nano /usr/local/etc/redis.conf) and uncomment:

unixsocket /tmp/redis.sock
unixsocketperm 777

Start Redis

launchctl load ~/Library/LaunchAgents/homebrew.mxcl.redis.plist

Configure FinDy Settings

Default (config.json) setting

{
  "version": "0.0.2",

  "debug": 0,
  "processes": 1,
  "batch_size": 10000,
  
  "db_name": "findy",
  "db_host": "192.168.1.133",
  "db_port": "15432",
  "db_user": "postgres",
  "db_pass": "123",

  "redis_pass": "",
  "kafka": "localhost:9092",

  "jq_username": "",
  "jq_password": "",

  "tushare_token": "",

  "trade_host": "127.0.0.1",
  "trade_port": "11111",
  "trade_socket_port": "33333",
  "trade_socket_key": "",

  "http_proxy": "127.0.0.1:1087",
  "https_proxy": "127.0.0.1:1087",
  "smtp_host": "smtpdm.aliyun.com",
  "smtp_port": "80",
  "email_username": "",
  "email_password": "",
  "wechat_app_id": "",
  "wechat_app_secrect": "",
  "wechat_agent_id": ""
}

change database host address to following setting if you install your database locally:

  "db_host": "127.0.0.1",
  "db_port": "5432",

set database user and password to your custom settings:

  "db_user": "xxx",
  "db_pass": "xxx",

chinese stock market user are required to obtain joinquant and tushare authentication.

JoinQuant
TuShare

Updates:

(25 Dec 2021)

• rewrite yahoo finance lib (yfinance) to support aiohttp requests
• remove proxy pool
• add wechat notify (see utils/notify.py for more details)
• dev/async_db branch: add async sqlalchemy ORM
• many bugfix

(01 Mar 2021)

• database session decoupling
• entity provider removed
• database querying speed boost up
• asyncio logic framework (implementations requires rewrite request logic)
• multiprocessing logic rewrite, use processes to replace pool
• add proxy pool (see proxy_pool)
• add baostock thrid party data source

(16 Sep 2020):

• Switch Sql to Postgresql, which allow remote database access, separate fetching and analysis models to two individal project(see FinanceAnalysis project for more detail).
• Centralized http request interface，add Session and retry logic (Rxpy may be involved to rewrite request model pretty soon).
• multiprocessing is involved to boost up request speed.
• request progress display.
• auto joinquant account switching (banned)
• log to console -> log to file.
• US stock market data.
• global timezone support.
• tiny bug fix.

How to use

Fetch data

from findy.interface import Region
from findy.interface.fetch import fetching

fetching(Region.CHN)  # Region.US

Read data

from datetime import datetime, timedelta

from findy.interface import Region, Provider, EntityType, SP_500_TICKER
from findy.interface.reader import DataReader
from findy.database.schema.meta.stock_meta import Stock, StockDetail
from findy.database.schema.quotes.stock.stock_1d_kdata import Stock1dKdata
from findy.database.schema.register import get_entity_schema_by_type
from findy.database.context import get_db_session
from findy.database.quote import get_entities


end = datetime.now()
start = end - timedelta(days=365)

entity_schema = get_entity_schema_by_type(EntityType.StockDetail)
db_session = get_db_session(Region.US, Provider.Yahoo, entity_schema)
entities, column_names = get_entities(
        region=Region.US,
        provider=Provider.Yahoo,
        entity_schema=entity_schema,
        db_session=db_session,
        codes=SP_500_TICKER,
        columns=[StockDetail.code],
        filters=[StockDetail.market_cap > 10_000_000_000])

codes_cap = [entity.code for entity in entities]

stock_reader = DataReader(region=Region.US,
                          provider=Provider.Yahoo,
                          data_schema=Stock1dKdata,
                          entity_schema=Stock,
                          codes=codes_cap,
                          start_timestamp=start.strftime('%Y-%m-%d'),
                          end_timestamp=end.strftime('%Y-%m-%d'))

stock_reader.load_data()
df = stock_reader.data_df.reset_index(drop=True)
gb = df.groupby('code', sort=False)
stocks = [(code, gb.get_group(code)) for code in gb.groups]

Docker Usage

Docker Execute

docker-compose stop; docker-compose rm -f ; docker-compose build --no-cache
docker-compose up -d

View Log

docker-compose logs -f 

To Do

1. rewrite vendor/baostock package to support async logic
2. add more 3th-party data source packages
3. dev/async_db branch optimization