GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → Stream-AD → MemStream

Stream-AD / MemStream

Licence: Apache-2.0 license
MemStream: Memory-Based Streaming Anomaly Detection

Programming Languages

python
139335 projects - #7 most used programming language

Labels

streamingreal-timedenial-of-serviceconcept-driftfraud-detectionanomaly-detectionmemory-basedmulti-aspect-recordintrusion-dectection

Projects that are alternatives of or similar to MemStream

MStream
Anomaly Detection on Time-Evolving Streams in Real-time. Detecting intrusions (DoS and DDoS attacks), frauds, fake rating anomalies.
Stars: ✭ 68 (+17.24%)
Mutual labels:  real-time, anomaly-detection, multi-aspect-record
benfordslaw
benfordslaw is about the frequency distribution of leading digits.
Stars: ✭ 29 (-50%)
Mutual labels:  fraud-detection, anomaly-detection
Pyod
A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)
Stars: ✭ 5,083 (+8663.79%)
Mutual labels:  fraud-detection, anomaly-detection
Feature-Engineering-for-Fraud-Detection
Implementation of feature engineering from Feature engineering strategies for credit card fraud
Stars: ✭ 31 (-46.55%)
Mutual labels:  fraud-detection, anomaly-detection
Faster-Grad-CAM
Faster and more precisely than Grad-CAM
Stars: ✭ 33 (-43.1%)
Mutual labels:  real-time, anomaly-detection
DGFraud-TF2
A Deep Graph-based Toolbox for Fraud Detection in TensorFlow 2.X
Stars: ✭ 84 (+44.83%)
Mutual labels:  fraud-detection, anomaly-detection
Pysad
Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)
Stars: ✭ 87 (+50%)
Mutual labels:  real-time, anomaly-detection
deepAD
Detection of Accounting Anomalies in the Latent Space using Adversarial Autoencoder Neural Networks - A lab we prepared for the KDD'19 Workshop on Anomaly Detection in Finance that will walk you through the detection of interpretable accounting anomalies using adversarial autoencoder neural networks. The majority of the lab content is based on J…
Stars: ✭ 65 (+12.07%)
Mutual labels:  fraud-detection, anomaly-detection
ECBSR
Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices, ACM Multimedia 2021
Stars: ✭ 216 (+272.41%)
Mutual labels:  real-time
efficient-descriptors
🚀🚀 Revisiting Binary Local Image Description for Resource Limited Devices
Stars: ✭ 76 (+31.03%)
Mutual labels:  real-time
jeelizGlanceTracker
JavaScript/WebGL lib: detect if the user is looking at the screen or not from the webcam video feed. Lightweight and robust to all lighting conditions. Great for play/pause videos if the user is looking or not, or for person detection. Link to live demo.
Stars: ✭ 68 (+17.24%)
Mutual labels:  real-time
RTFM
Official code for 'Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning' [ICCV 2021]
Stars: ✭ 216 (+272.41%)
Mutual labels:  anomaly-detection
anomaly-seg
The Combined Anomalous Object Segmentation (CAOS) Benchmark
Stars: ✭ 115 (+98.28%)
Mutual labels:  anomaly-detection
Real-time-Image-Smoothing-via-Iterative-Least-Squares
This is the released code for the paper Real-time Image Smoothing via Iterative Least Squares accepted to ACM Transactions on Graphics
Stars: ✭ 60 (+3.45%)
Mutual labels:  real-time
Tensorflow-YOLACT
Implementation of the paper "YOLACT Real-time Instance Segmentation" in Tensorflow 2
Stars: ✭ 97 (+67.24%)
Mutual labels:  real-time
Samsung-TV-Denial-of-Service-DoS-Attack
Samsung TV Denial of Service (DoS) Attack
Stars: ✭ 28 (-51.72%)
Mutual labels:  denial-of-service
loggregator
Archived: Now bundled in https://github.com/cloudfoundry/loggregator-release
Stars: ✭ 61 (+5.17%)
Mutual labels:  real-time
kidnapped-vehicle
Implemented a C++ particle filter for real-time vehicle localization with only current visual observations and a map.
Stars: ✭ 18 (-68.97%)
Mutual labels:  real-time
soketi
Just another simple, fast, and resilient open-source WebSockets server. 📣
Stars: ✭ 2,202 (+3696.55%)
Mutual labels:  real-time
Embedded UKF Library
A compact Unscented Kalman Filter (UKF) library for Teensy4/Arduino system (or any real time embedded system in general)
Stars: ✭ 31 (-46.55%)
Mutual labels:  real-time
View All Similar Projects ➔

MemStream

Implementation of

MemStream detects anomalies from a multi-aspect data stream. We output an anomaly score for each record. MemStream is a memory augmented feature extractor, allows for quick retraining, gives a theoretical bound on the memory size for effective drift handling, is robust to memory poisoning, and outperforms 11 state-of-the-art streaming anomaly detection baselines.

After an initial training of the feature extractor on a small subset of normal data, MemStream processes records in two steps: (i) It outputs anomaly scores for each record by querying the memory for K-nearest neighbours to the record encoding and calculating a discounted distance and (ii) It updates the memory, in a FIFO manner, if the anomaly score is within an update threshold β.

Demo

  1. KDDCUP99: Run python3 memstream.py --dataset KDD --beta 1 --memlen 256
  2. NSL-KDD: Run python3 memstream.py --dataset NSL --beta 0.1 --memlen 2048
  3. UNSW-NB 15: Run python3 memstream.py --dataset UNSW --beta 0.1 --memlen 2048
  4. CICIDS-DoS: Run python3 memstream.py --dataset DOS --beta 0.1 --memlen 2048
  5. SYN: Run python3 memstream-syn.py --dataset SYN --beta 1 --memlen 16
  6. Ionosphere: Run python3 memstream.py --dataset ionosphere --beta 0.001 --memlen 4
  7. Cardiotocography: Run python3 memstream.py --dataset cardio --beta 1 --memlen 64
  8. Statlog Landsat Satellite: Run python3 memstream.py --dataset statlog --beta 0.01 --memlen 32
  9. Satimage-2: Run python3 memstream.py --dataset satimage-2 --beta 10 --memlen 256
  10. Mammography: Run python3 memstream.py --dataset mammography --beta 0.1 --memlen 128
  11. Pima Indians Diabetes: Run python3 memstream.py --dataset pima --beta 0.001 --memlen 64
  12. Covertype: Run python3 memstream.py --dataset cover --beta 0.0001 --memlen 2048

Command line options

  • --dataset: The dataset to be used for training. Choices 'NSL', 'KDD', 'UNSW', 'DOS'. (default 'NSL')
  • --beta: The threshold beta to be used. (default: 0.1)
  • --memlen: The size of the Memory Module (default: 2048)
  • --dev: Pytorch device to be used for training like "cpu", "cuda:0" etc. (default: 'cuda:0')
  • --lr: Learning rate (default: 0.01)
  • --epochs: Number of epochs (default: 5000)

Input file format

MemStream expects the input multi-aspect record stream to be stored in a contains , separated file.

Datasets

Processed Datasets can be downloaded from here. Please unzip and place the files in the data folder of the repository.

  1. KDDCUP99
  2. NSL-KDD
  3. UNSW-NB 15
  4. CICIDS-DoS
  5. Synthetic Dataset (Introduced in paper)
  6. Ionosphere
  7. Cardiotocography
  8. Statlog Landsat Satellite
  9. Satimage-2
  10. Mammography
  11. Pima Indians Diabetes
  12. Covertype

Environment

This code has been tested on Debian GNU/Linux 9 with a 12GB Nvidia GeForce RTX 2080 Ti GPU, CUDA Version 10.2 and PyTorch 1.5.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].