All Projects → fjxmlzn → Doppelganger

fjxmlzn / Doppelganger

Licence: bsd-3-clause-clear
[IMC 2020 (Best Paper Finalist)] Using GANs for Sharing Networked Time Series Data: Challenges, Initial Promise, and Open Questions

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Doppelganger

Sdv
Synthetic Data Generation for tabular, relational and time series data.
Stars: ✭ 360 (+271.13%)
Mutual labels:  time-series, gan, generative-adversarial-network, gans
Ad examples
A collection of anomaly detection methods (iid/point-based, graph and time series) including active learning for anomaly detection/discovery, bayesian rule-mining, description for diversity/explanation/interpretability. Analysis of incorporating label feedback with ensemble and tree-based detectors. Includes adversarial attacks with Graph Convolutional Network.
Stars: ✭ 641 (+560.82%)
Mutual labels:  time-series, timeseries, gan, generative-adversarial-network
A Pytorch Tutorial To Super Resolution
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network | a PyTorch Tutorial to Super-Resolution
Stars: ✭ 157 (+61.86%)
Mutual labels:  gan, generative-adversarial-network, gans
Iseebetter
iSeeBetter: Spatio-Temporal Video Super Resolution using Recurrent-Generative Back-Projection Networks | Python3 | PyTorch | GANs | CNNs | ResNets | RNNs | Published in Springer Journal of Computational Visual Media, September 2020, Tsinghua University Press
Stars: ✭ 202 (+108.25%)
Mutual labels:  gan, generative-adversarial-network, gans
Gif
GIF is a photorealistic generative face model with explicit 3D geometric and photometric control.
Stars: ✭ 233 (+140.21%)
Mutual labels:  gan, generative-adversarial-network, gans
Tagan
An official PyTorch implementation of the paper "Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language", NeurIPS 2018
Stars: ✭ 97 (+0%)
Mutual labels:  gan, generative-adversarial-network, gans
Generative adversarial networks 101
Keras implementations of Generative Adversarial Networks. GANs, DCGAN, CGAN, CCGAN, WGAN and LSGAN models with MNIST and CIFAR-10 datasets.
Stars: ✭ 138 (+42.27%)
Mutual labels:  gan, generative-adversarial-network, gans
Pacgan
[NeurIPS 2018] [JSAIT] PacGAN: The power of two samples in generative adversarial networks
Stars: ✭ 67 (-30.93%)
Mutual labels:  gan, generative-adversarial-network, gans
Cyclegan
Software that can generate photos from paintings, turn horses into zebras, perform style transfer, and more.
Stars: ✭ 10,933 (+11171.13%)
Mutual labels:  gan, generative-adversarial-network, gans
DLSS
Deep Learning Super Sampling with Deep Convolutional Generative Adversarial Networks.
Stars: ✭ 88 (-9.28%)
Mutual labels:  generative-adversarial-network, gan, gans
Faceswap Gan
A denoising autoencoder + adversarial losses and attention mechanisms for face swapping.
Stars: ✭ 3,099 (+3094.85%)
Mutual labels:  gan, generative-adversarial-network, gans
Pytorch Cyclegan And Pix2pix
Image-to-Image Translation in PyTorch
Stars: ✭ 16,477 (+16886.6%)
Mutual labels:  gan, generative-adversarial-network, gans
AvatarGAN
Generate Cartoon Images using Generative Adversarial Network
Stars: ✭ 24 (-75.26%)
Mutual labels:  generative-adversarial-network, gan, gans
Anycost Gan
[CVPR 2021] Anycost GANs for Interactive Image Synthesis and Editing
Stars: ✭ 367 (+278.35%)
Mutual labels:  gan, generative-adversarial-network, gans
Gans In Action
Companion repository to GANs in Action: Deep learning with Generative Adversarial Networks
Stars: ✭ 748 (+671.13%)
Mutual labels:  gan, generative-adversarial-network, gans
Hypergan
Composable GAN framework with api and user interface
Stars: ✭ 1,104 (+1038.14%)
Mutual labels:  gan, generative-adversarial-network
Cyclegan Tensorflow
An implementation of CycleGan using TensorFlow
Stars: ✭ 1,096 (+1029.9%)
Mutual labels:  gan, generative-adversarial-network
Cyclegan Qp
Official PyTorch implementation of "Artist Style Transfer Via Quadratic Potential"
Stars: ✭ 59 (-39.18%)
Mutual labels:  gan, generative-adversarial-network
Stingray
Anything can happen in the next half hour (including spectral timing made easy)!
Stars: ✭ 94 (-3.09%)
Mutual labels:  time-series, timeseries
Acgan Pytorch
Pytorch implementation of Conditional Image Synthesis with Auxiliary Classifier GANs
Stars: ✭ 57 (-41.24%)
Mutual labels:  gan, generative-adversarial-network

Using GANs for Sharing Networked Time Series Data: Challenges, Initial Promise, and Open Questions

Previous title: Generating High-fidelity, Synthetic Time Series Datasets with DoppelGANger

[paper (arXiv)] [paper (IMC 2020, Best Paper Finalist)] [talk] [code]

Authors: Zinan Lin (CMU), Alankar Jain (CMU), Chen Wang (IBM), Giulia Fanti (CMU), Vyas Sekar (CMU)

Abstract: Limited data access is a longstanding barrier to data-driven research and development in the networked systems community. In this work, we explore if and how generative adversarial networks (GANs) can be used to incentivize data sharing by enabling a generic framework for sharing synthetic datasets with minimal expert knowledge. As a specific target, our focus in this paper is on time series datasets with metadata (e.g., packet loss rate measurements with corresponding ISPs). We identify key challenges of existing GAN approaches for such workloads with respect to fidelity (e.g., long-term dependencies, complex multidimensional relationships, mode collapse) and privacy (i.e., existing guarantees are poorly understood and can sacrifice fidelity). To improve fidelity, we design a custom workflow called DoppelGANger (DG) and demonstrate that across diverse real-world datasets (e.g., bandwidth measurements, cluster requests, web sessions) and use cases (e.g., structural characterization, predictive modeling, algorithm comparison), DG achieves up to 43% better fidelity than baseline models. Although we do not resolve the privacy problem in this work, we identify fundamental challenges with both classical notions of privacy and recent advances to improve the privacy properties of GANs, and suggest a potential roadmap for addressing these challenges. By shedding light on the promise and challenges, we hope our work can rekindle the conversation on workflows for data sharing.

Users

DoppelGANger has been used by several independent users/companies. Check the following links for more information:


This repo contains the codes of DoppelGANger. The codes were tested under Python 2.7.5 and Python 3.5.2, TensorFlow 1.4.0.

Dataset format

Note that metadata in the paper are denoted as attribute in the code; measurement in the paper are denoted as feature in the code. To train DoppelGANger for your data, you need to prepare your data according to the following format, which contains three files:

  • data_feature_output.pkl: A pickle dump of a list of gan.output.Output objects, indicating the dimension, type, normalization of each feature.
  • data_attribute_output.pkl: A pickle dump of a list of gan.output.Output objects, indicating the dimension, type, normalization of each attribute.
  • data_train.npz: A numpy .npz archive of the following three arrays:
    • data_feature: Training features, in numpy float32 array format. The size is [(number of training samples) x (maximum length) x (total dimension of features)]. Categorical features are stored by one-hot encoding; for example, if a categorical feature has 3 possibilities, then it can take values between [1., 0., 0.], [0., 1., 0.], and [0., 0., 1.]. Each continuous feature should be normalized to [0, 1] or [-1, 1]. The array is padded by zeros after the time series ends.
    • data_attribute: Training attributes, in numpy float32 array format. The size is [(number of training samples) x (total dimension of attributes)]. Categorical attributes are stored by one-hot encoding; for example, if a categorical attribute has 3 possibilities, then it can take values between [1., 0., 0.], [0., 1., 0.], and [0., 0., 1.]. Each continuous attribute should be normalized to [0, 1] or [-1, 1].
    • data_gen_flag: Flags indicating the activation of features, in numpy float32 array format. The size is [(number of training samples) x (maximum length)]. 1 means the time series is activated at this time step, 0 means the time series is inactivated at this timestep.

Let's look at a concrete example. Assume that there are two features (a 1-dimension continuous feature normalized to [0,1] and a 2-dimension categorical feature) and two attributes (a 2-dimension continuous attribute normalized to [-1, 1] and a 3-dimension categorical attributes). Then data_feature_output and data_attribute_output should be:

data_feature_output = [
	Output(type_=CONTINUOUS, dim=1, normalization=ZERO_ONE, is_gen_flag=False),
	Output(type_=DISCRETE, dim=2, normalization=None, is_gen_flag=False)]
	
data_attribute_output = [
	Output(type_=CONTINUOUS, dim=2, normalization=MINUSONE_ONE, is_gen_flag=False),
	Output(type_=DISCRETE, dim=3, normalization=None, is_gen_flag=False)]

Note that is_gen_flag should always set to False (default). is_gen_flag=True is for internal use only (see comments in doppelganger.py for details).

Assume that there are two samples, whose lengths are 2 and 4, and assume that the maximum length is set to 4. Then data_feature, data_attribute, and data_gen_flag could be:

data_feature = [
	[[0.2, 1.0, 0.0], [0.4, 0.0, 1.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
	[[0.9, 0.0, 1.0], [0.3, 0.0, 1.0], [0.2, 0.0, 1.0], [0.8, 1.0, 0.0]]]
	
data_attribute = [
	[-0.2, 0.3, 1.0, 0.0, 0.0],
	[0.2, 0.3, 0.0, 1.0, 0.0]]
	
data_gen_flag = [
	[1.0, 1.0, 0.0, 0.0],
	[1.0, 1.0, 1.0, 1.0]]

The datasets we used in the paper (Wikipedia Web Traffic, Google Cluster Usage Traces, Measuring Broadband America) can be found here.

Run DoppelGANger

The codes are based on GPUTaskScheduler library, which helps you automatically schedule jobs among GPU nodes. Please install it first. You may need to change GPU configurations according to the devices you have. The configurations are set in config*.py in each directory. Please refer to GPUTaskScheduler's GitHub page for details of how to make proper configurations.

You may also run these codes without GPUTaskScheduler. See the main.py in example_training(without_GPUTaskScheduler) for an example.

The implementation of DoppelGANger is at gan/doppelganger.py. You may refer to the comments in it for details. Here we provide our code for training DoppelGANger on the three datasets (Wikipedia Web Traffic, Google Cluster Usage Traces, Measuring Broadband America) in the paper, and give examples on using DoppelGANger to generate data and retraining the attribute generation network.

Download dataset

Before running the code, please download the three datasets here and put it under data folder.

Train DoppelGANger

cd example_training
python main.py

Generate data by DoppelGANger

cd example_generating_data
python main_generate_data.py

Retrain attribute generation network of DoppelGANger

Put your data with the desired attribute distribution in data/web_retraining, and then

cd example_retraining_attribute
python main.py

Differentially private (DP) version

To run the differentially private version of DoppelGANger (Section 6.2 in the paper), please first install TensorFlow Privacy library.

  • Train DP DoppelGANger
cd example_dp_training
python main.py
  • Generate data by DP DoppelGANger
cd example_dp_generating_data
python main_generate_data.py

Customize DoppelGANger

You can play with the configurations (e.g., whether to have the auxiliary discriminator) in config*.py.

The meaning of the key parameters are:

  • epoch: Number of training epochs.
  • batch_size: Training batch size.
  • sample_len: The time series batch size, i.e. the number of time steps generated at each RNN rollout (parameter S in Section 4.1).
  • aux_disc: Whether to have auxiliary discriminator (Section 4.3).
  • self_norm: Whether to do auto-normalization for each time series (Section 4.2).
  • num_packing: Packing degree in PacGAN (a method for solving mode collapse in NeurIPS 2018, see the paper and code). Setting it to 1 is equivalent to the vanilla GAN without packing.
  • noise: Whether to input noise at each RNN rollout.
  • feed_back: Whether to input the generated values from the previous rollout at each RNN rollout.
  • g_lr: The learning rate in Adam for training the generator
  • d_lr: The learning rate in Adam for training the discriminator.
  • d_gp_coe: Weight of gradient penalty loss in Wasserstein GAN for the discriminator.
  • attr_d_lr: The learning rate in Adam for training the auxiliary discriminator.
  • attr_d_gp_coe: Weight of gradient penalty loss in Wasserstein GAN for the auxiliary discriminator.
  • d_rounds: Number of discriminator steps per batch.
  • g_rounds: Number of generator steps per batch.
  • gen_feature_num_layers: Number of layers in the time series generator (RNN).
  • gen_feature_num_units: Number of units in each layer of the time series generator (RNN).
  • gen_attribute_num_layers: Number of layers in the attribute (metadata) generator.
  • gen_attribute_num_units: Number of units in each layer of the attribute (metadata) generator.
  • attr_disc_num_layers: Number of layers in the auxiliary discriminator.
  • attr_disc_num_units: Number of units in each layer of the auxiliary discriminator.
  • disc_num_layers: Number of layers in the discriminator.
  • disc_num_units: Number of units in each layer of the auxiliary discriminator.
  • initial_state: The initial state for the RNN: "random" means setting the initial state to random numbers; "zero" means setting the initial state to zeros; "variable" means setting the inital state to be learnable parameters.
  • extra_checkpoint_freq: The frequency of saving the trained model in a separated folder (unit: epoch).
  • epoch_checkpoint_freq: The frequency of saving the trained model (unit: epoch).
  • vis_freq: The frequency of visualizing generated samples during training (unit: training batch).
  • vis_num_sample: The number of samples to visualize each time during training.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].