Variational Inference with Tail-adaptive f-Divergence
This repository contains a tensorflow implementation for tail-adaptive f-divergence, as presented in the followng paper:
Dilin Wang, Hao Liu, Qiang Liu, "Variational Inference with Tail-adaptive f-Divergence", NIPS 2018.
Our Algorithm
Our tail-adaptive weights ($\gamma_f$ in Eqn 8) could be easily calculated with the following function,
def get_tail_adaptive_weights(self, l_p, l_q, beta=-1.):
"""returns the tail-adaptive weights
Args:
l_p: log p(x), 1-d tensor, log probability of p
l_q: log q(x), 1-d tensor, log probability of q
beta: magnitude, default -1
Returns:
Tail-adaptive weights
"""
diff = l_p - l_q
diff -= tf.reduce_max(diff)
dx = tf.exp(diff)
prob = tf.sign(tf.expand_dims(dx, 1) - tf.expand_dims(dx, 0))
prob = tf.cast(tf.greater(prob, 0.5), tf.float32)
wx = tf.reduce_sum(prob, axis=1) / tf.cast(tf.size(l_p), tf.float32)
wx = (1. - wx) ** beta # beta = -1; or beta = -0.5
wx /= tf.reduce_sum(wx) # self-normalization
return tf.stop_gradient(wx)Citation
If you find tail-adaptive f-divergence useful in your research, please consider citing:
@article{wang2018variational,
title={Variational Inference with Tail-adaptive f-Divergence},
author={Wang, Dilin and Liu, Hao and Liu, Qiang},
journal={arXiv preprint arXiv:1810.11943},
year={2018}
}