Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details

All Projects → IlyaGusev → Tgcontest

IlyaGusev / Tgcontest

Licence: apache-2.0

Telegram Data Clustering contest solution by Mindful Squirrel

Programming Languages

1120 projects

Labels

html machine-learning nlp data-science classification clustering fasttext

Projects that are alternatives of or similar to Tgcontest

🍊 📊 💡 Orange: Interactive data analysis

Stars: ✭ 3,152 (+4159.46%)

Mutual labels: data-science, classification, clustering

A high-level machine learning and deep learning library for the PHP language.

Stars: ✭ 1,270 (+1616.22%)

Mutual labels: data-science, classification, clustering

Statistical Machine Intelligence & Learning Engine

Stars: ✭ 5,412 (+7213.51%)

Mutual labels: data-science, classification, clustering

Machine Learning in R

Stars: ✭ 1,542 (+1983.78%)

Mutual labels: data-science, classification, clustering

Machine Learning With Python

Practice and tutorial-style notebooks covering wide variety of machine learning techniques

Stars: ✭ 2,197 (+2868.92%)

Mutual labels: data-science, classification, clustering

A Julia machine learning framework

Stars: ✭ 982 (+1227.03%)

Mutual labels: data-science, classification, clustering

Artificial Neural Networks for Scala

Stars: ✭ 105 (+41.89%)

Mutual labels: data-science, classification, clustering

Simple API for UCI Machine Learning Dataset Repository (search, download, analyze)

Stars: ✭ 190 (+156.76%)

Mutual labels: data-science, classification, clustering

An open-source, low-code machine learning library in Python

Stars: ✭ 4,594 (+6108.11%)

Mutual labels: data-science, clustering, classification

Tensorflow Book

Accompanying source code for Machine Learning with TensorFlow. Refer to the book for step-by-step explanations.

Stars: ✭ 4,448 (+5910.81%)

Mutual labels: classification, clustering

Automated Machine Learning [AutoML] with Python, scikit-learn, Keras, XGBoost, LightGBM, and CatBoost

Stars: ✭ 564 (+662.16%)

Mutual labels: data-science, classification

ELKI Data Mining Toolkit

Stars: ✭ 613 (+728.38%)

Mutual labels: data-science, clustering

Measure and visualize machine learning model performance without the usual boilerplate.

Stars: ✭ 71 (-4.05%)

Mutual labels: data-science, classification

mlr3: Machine Learning in R - next generation

Stars: ✭ 463 (+525.68%)

Mutual labels: data-science, classification

Food Recipe Cnn

food image to recipe with deep convolutional neural networks.

Stars: ✭ 448 (+505.41%)

Mutual labels: data-science, classification

pyclustring is a Python, C++ data mining library.

Stars: ✭ 806 (+989.19%)

Mutual labels: data-science, clustering

Awesome Fraud Detection Papers

A curated list of data mining papers about fraud detection.

Stars: ✭ 843 (+1039.19%)

Mutual labels: data-science, classification

Stats Maths With Python

General statistics, mathematical programming, and numerical/scientific computing scripts and notebooks in Python

Stars: ✭ 381 (+414.86%)

Mutual labels: data-science, clustering

Scikit Multilearn

A scikit-learn based module for multi-label et. al. classification

Stars: ✭ 638 (+762.16%)

Mutual labels: classification, clustering

Text classification

all kinds of text classification models and more with deep learning

Stars: ✭ 7,179 (+9601.35%)

Mutual labels: classification, fasttext

View All Similar Projects ➔

TGNews

Links

Description in English: https://medium.com/@phoenixilya/news-aggregator-in-2-weeks-5b38783b95e3
Description in Russian: https://habr.com/ru/post/487324/

Demo

Russian: https://ilyagusev.github.io/tgcontest/ru/main.html
English: https://ilyagusev.github.io/tgcontest/en/main.html

Install

Prerequisites: CMake, Boost

$ sudo apt-get install cmake libboost-all-dev build-essential libjsoncpp-dev uuid-dev protobuf-compiler libprotobuf-dev

For MacOS

$ brew install boost jsoncpp ossp-uuid protobuf

If you got zip archive, just go to building binary

To download code and models:

$ git clone https://github.com/IlyaGusev/tgcontest
$ cd tgcontest
$ git submodule update --init --recursive
$ bash download_models.sh
$ wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.5.0%2Bcpu.zip
$ unzip libtorch-cxx11-abi-shared-with-deps-1.5.0+cpu.zip

For MacOS use https://download.pytorch.org/libtorch/cpu/libtorch-macos-1.5.0.zip

To build binary (in "tgcontest" dir):

$ mkdir build && cd build && Torch_DIR="../libtorch" cmake -DCMAKE_BUILD_TYPE=Release .. && make -j4

To download datasets:

$ bash download_data.sh

Run on sample:

./build/tgnews top data --ndocs 10000

Training

Russian FastText vectors training: VectorsRu.ipynb
Russian fasttext category classifier training: CatTrainRu.ipynb
Russian text embedder with triplet loss training (v3):
English FastText vectors training: VectorsEn.ipynb
English fasttext category classifier training: CatTrainEn.ipynb
English text embedder with triplet loss training (v3):
PageRank rating calculation: PageRankRating.ipynb
Russian ELMo-based sentence embedder training (not used):
XLM-RoBERTa pseudo-labeling for categorization:

Models

Language detection model (2 round): lang_detect_v10.ftz
Russian FastText vectors (2 round): ru_vectors_v3.bin
Russian categories detection model (2 round): ru_cat_v5.ftz
English FastText vectors (2 round): en_vectors_v3.bin
English categories detection model (2 round): en_cat_v5.ftz
PageRank-based agency rating: pagerank_rating.txt
Alexa agency rating: alexa_rating_4_fixed.txt
XLM-RoBERTa for categorization (pytorch-lightning checkpoint): xlmr_en_ru_cat_v1.tar.gz

Data

Russian news from 11.01.2019 to 10.05.2020 with gaps: ru_tg_1101_0510.jsonl.tar.gz
Russian news from 11.05.2020 to 17.05.2020: ru_tg_0511_0517.jsonl.tar.gz
English news from 11.01.2019 to 10.05.2020 with gaps: en_tg_1101_0510.jsonl.tar.gz
English news from 11.05.2020 to 17.05.2020: en_tg_0511_0517.jsonl.tar.gz

Markup

Russian categories raw train markup: ru_cat_v4_train_raw_markup.tsv
Russian categories aggregated train markup: ru_cat_v4_train_annot.json
Russian categories aggregated train markup in fastText format: ft_ru_cat_v4_train.txt
Russian categories manual train markup: ru_cat_v4_train_manual_annot.json
Russian categoreis manual train markup in fastText format: ft_ru_cat_v4_train_manual.txt
Russian categoreis raw test markup: ru_cat_v4_test_raw_markup.tsv
Russian categories aggregated test markup: ru_cat_v4_test_annot.json
Russian categories aggregated test markup in fastText format: ft_ru_cat_v4_test.txt
English categories aggregated train markup: en_cat_v4_train_annot.json
English categories aggregated train markup in fastText format: ft_en_cat_v4_train.txt
English categories aggregated test markup: en_cat_v4_test_annot.json
English categories aggregated test markup in fastText format: ft_en_cat_v4_test.txt
Russian clustering pairs: ru_pairs_raw_markup.tsv
English clustering pairs: en_pairs_raw_markup.tsv
Russian clustering pairs for one day (0517): ru_clustering_0517.tsv

Misc

Flamegraph: https://ilyagusev.github.io/tgcontest/flamegraph.svg

Other contestants

Round 2
- II place
  - Daring Frog: https://github.com/a-l-e-x-k/data_clustering_contest, article: https://medium.com/@alexkuznetsov/2nd-place-solution-for-telegram-data-clustering-contest-f28d55b98d30
  - Swift Skunk: https://github.com/sorrge/tg_news_cluster
- III place
  - Mindful Kitten: https://danlark.org/2020/07/31/news-aggregator-from-scratch-in-2-weeks/
- IV place
  - Bossy Gnu: https://github.com/maxoodf/tgnews
- Other:
  - Large Crab: https://github.com/ilya-ustinov/tgcontest
Round 1
- III place
  - Kooky Dragon: https://github.com/nick-baliesnyi/tgnews
- IV place
  - Sharp Sloth: https://github.com/thehemen/telegram-data-clustering
- Other
  - Desert Python: https://github.com/crazyleg/telegram_data_clustering_2019
  - Funky Peacock: https://github.com/Stepka/telegram_clustering_contest
  - Unknown animal: https://github.com/roman-rybalko/telegram-data-clustering-contest
  - Unknown animal: https://github.com/MarcoBuster/data-clustering-contest
  - Unknown animal: https://github.com/sudevschiz/tgnews
  - Unknown animal: https://github.com/crazyleg/telegram_data_clustering_2019
  - Unknown animal: https://github.com/77ph/tgnews
  - Unknown animal: https://github.com/akash-joshi/telegram-cluster
  - Unknown animal: https://github.com/dremovd/telegram-clustering

Contacts

Telegram: @YallenGusev

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].

Stars: ✭ 74

Visit Git Page 🔗Visit User Page 🔗Visit Issues Page (0) 🔗