GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → yandexmobile → appmetrica-logsapi-loader

yandexmobile / appmetrica-logsapi-loader

Licence: other
A tool for automatic data loading from AppMetrica LogsAPI into (local) ClickHouse

Programming Languages

python
139335 projects - #7 most used programming language
Dockerfile
14818 projects

Labels

automationyandexdocker-composeclickhousedocker-imageyandex-appmetricaappmetricalogs-api

Projects that are alternatives of or similar to appmetrica-logsapi-loader

Clickhouse Sqlalchemy
ClickHouse dialect for SQLAlchemy
Stars: ✭ 166 (+822.22%)
Mutual labels:  yandex, clickhouse
Clickhouse Net
Yandex ClickHouse fully managed .NET client
Stars: ✭ 142 (+688.89%)
Mutual labels:  yandex, clickhouse
Clickhouse Driver
ClickHouse Python Driver with native interface support
Stars: ✭ 562 (+3022.22%)
Mutual labels:  yandex, clickhouse
aioch
aioch - is a library for accessing a ClickHouse database over native interface from the asyncio
Stars: ✭ 145 (+705.56%)
Mutual labels:  yandex, clickhouse
ClickHouseTools
Инструменты обслуживания и разработки для Yandex ClickHouse, а также другие интересности
Stars: ✭ 16 (-11.11%)
Mutual labels:  clickhouse
db-migrator.go
DB migrations. CLI and Golang
Stars: ✭ 13 (-27.78%)
Mutual labels:  clickhouse
yabr.os
Чтение скобочного формата файлов 1С (oscript)
Stars: ✭ 33 (+83.33%)
Mutual labels:  clickhouse
sql exporter
Database agnostic SQL exporter for Prometheus
Stars: ✭ 72 (+300%)
Mutual labels:  clickhouse
Yandex.Music.Api
Client Yandex.Music.Api for Yandex.Music
Stars: ✭ 53 (+194.44%)
Mutual labels:  yandex
trickster
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator
Stars: ✭ 1,753 (+9638.89%)
Mutual labels:  clickhouse
xy2xy
A list of technologies similar to inner Yandex technologies
Stars: ✭ 112 (+522.22%)
Mutual labels:  yandex
robots-txt-parser
PHP class for parse all directives from robots.txt files according to specifications
Stars: ✭ 38 (+111.11%)
Mutual labels:  yandex
yandex-checkout-node
Node.js SDK for Yandex.Checkout (unofficial)
Stars: ✭ 64 (+255.56%)
Mutual labels:  yandex
metrica-plugin-unity
Unity plugin for Yandex AppMetrica SDK
Stars: ✭ 24 (+33.33%)
Mutual labels:  yandex-appmetrica
clickhouse-ast-parser
AST parser and visitor for ClickHouse SQL
Stars: ✭ 60 (+233.33%)
Mutual labels:  clickhouse
fense
Fense is a database proxy written in Java, which can connect DB of different engines at the same time. The key features are: authority management, query cache, audit security, current limiting fuse, onesql and so on
Stars: ✭ 22 (+22.22%)
Mutual labels:  clickhouse
YaSeeker
Yandex OSINT tool
Stars: ✭ 104 (+477.78%)
Mutual labels:  yandex
alice-renderer
Node.js библиотека для формирования ответов в навыках Яндекс Алисы.
Stars: ✭ 27 (+50%)
Mutual labels:  yandex
uptrace
Open source APM: OpenTelemetry traces, metrics, and logs
Stars: ✭ 1,187 (+6494.44%)
Mutual labels:  clickhouse
openreplay
📺 OpenReplay is developer-friendly, open-source session replay.
Stars: ✭ 6,131 (+33961.11%)
Mutual labels:  clickhouse
View All Similar Projects ➔

AppMetrica LogsAPI Loader

Python-script for automatic loading from AppMetrica LogsAPI into local ClickHouse DB. And yes, there is a Docker container too.

How to use this image

To use this image you should generate OAuth token for AppMetrica Logs API. This token should be provided via environment variable TOKEN.

Also you should copy app's numeric IDs. You could find them in General Settings of your app ("Application ID"). All IDs should be provided as JSON-array via environment variable APP_IDS.

Start manualy

docker run -d \
    --name clickhouse \
    yandex/clickhouse-server
docker run -d \
    --name appmetrica-logsapi-loader \
    --link clickhouse \
    --env 'CH_HOST=http://clickhouse:8123' \
    --env 'TOKEN=YOUR_OAUTH_TOKEN' \
    --env 'APP_IDS=["YOUR_APP_ID"]' \
    yandex/appmetrica-logsapi-loader

More information about ClickHouse server image.

Start with Docker Compose

Download this repository (or just docker-compose.yml file) and run:

TOKEN=YOUR_OAUTH_TOKEN \
APP_IDS='["YOUR_APP_ID"]' \
docker-compose up -d

Configuration

All configuration properties can be passed through environment variables.

Main variables

  • TOKEN - (required) Logs API OAuth token.
  • APP_IDS - (required) JSON-array of numeric AppMetrica app identifiers.
  • SOURCES - Logs API endpoints to download from. See available endpoints.

ClickHouse related

  • CH_HOST - Host of ClickHouse DB to store events. (default: http://localhost:8123)
  • CH_USER - Login of ClickHouse DB. (default: empty)
  • CH_PASSWORD - Password of ClickHouse DB. (default: empty)
  • CH_DATABASE - Database in ClickHouse to create tables in. (default: mobile)

LogsAPI related

  • LOGS_API_HOST - Base host of LogsAPI endpoints. (default: https://api.appmetrica.yandex.ru)
  • REQUEST_CHUNK_ROWS - Size of chunks to process at once. (default: 25000)
  • ALLOW_CACHED - Flag that allows cached LogsAPI data. Possible values: 0, 1. (default: 0)

Scheduling configuration

  • UPDATE_LIMIT - Count of days for the first events fetch. (default: 30)
  • FRESH_LIMIT - Count of days which still can have new events. (default: 7)
  • UPDATE_INTERVAL - Interval of time in hours between events fetches from Logs API. (default: 12)

Other variables

  • DEBUG - Enables extended logging. Possible values: 0, 1. (default: 0)
  • STATE_FILE_PATH - Path to file with script state. (default: data/state.json)

License

License agreement on use of Yandex AppMetrica is available at EULA site

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].