OxCSML : Projects : BigBayes

Publications

2019

J. Lee , Y. Lee , J. Kim , A. Kosiorek , S. Choi , Y. W. Teh , Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks, in International Conference on Machine Learning (ICML), 2019.
Project: bigbayes

Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant. We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set. The model consists of an encoder and a decoder, both of which rely on attention mechanisms. In an effort to reduce computational complexity, we introduce an attention scheme inspired by inducing point methods from sparse Gaussian process literature. It reduces the computation time of self-attention from quadratic to linear in the number of elements in the set. We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating the state-of-the-art performance compared to recent methods for set-structured data.
```
@inproceedings{pmlr-v97-lee19d,
  title = {Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks},
  author = {Lee, Juho and Lee, Yoonho and Kim, Jungtaek and Kosiorek, Adam and Choi, Seungjin and Teh, Yee Whye},
  booktitle = {International Conference on Machine Learning (ICML)},
  year = {2019},
  series = {Proceedings of Machine Learning Research},
  month = jun,
  publisher = {PMLR}
}
```
L. T. Elliott , M. De Iorio , S. Favaro , K. Adhikari , Y. W. Teh , Modeling Population Structure Under Hierarchical Dirichlet Processes, Bayesian Analysis, Jun. 2019.
Project: bigbayes

We propose a Bayesian nonparametric model to infer population admixture, extending the hierarchical Dirichlet process to allow for correlation between loci due to linkage disequilibrium. Given multilocus genotype data from a sample of individuals, the proposed model allows inferring and classifying individuals as unadmixed or admixed, inferring the number of subpopulations ancestral to an admixed population and the population of origin of chromosomal regions. Our model does not assume any specific mutation process, and can be applied to most of the commonly used genetic markers. We present a Markov chain Monte Carlo (MCMC) algorithm to perform posterior inference from the model and we discuss some methods to summarize the MCMC output for the analysis of population admixture. Finally, we demonstrate the performance of the proposed model in a real application, using genetic data from the ectodysplasin-A receptor (EDAR) gene, which is considered to be ancestry-informative due to well-known variations in allele frequency as well as phenotypic effects across ancestry. The structure analysis of this dataset leads to the identification of a rare haplotype in Europeans. We also conduct a simulated experiment and show that our algorithm outperforms parametric methods.
```
@article{elliott2019,
  author = {Elliott, Lloyd T. and De Iorio, Maria and Favaro, Stefano and Adhikari, Kaustubh and Teh, Yee Whye},
  doi = {10.1214/17-BA1093},
  firstavailable = {2018-05-19T02:03:09Z},
  fjournal = {Bayesian Analysis},
  journal = {Bayesian Analysis},
  publisher = {International Society for Bayesian Analysis},
  title = {Modeling Population Structure Under Hierarchical Dirichlet Processes},
  month = jun,
  year = {2019}
}
```
S. Webb , T. Rainforth , Y. W. Teh , M. P. Kumar , A Statistical Approach to Assessing Neural Network Robustness, in International Conference on Learning Representations (ICLR), 2019.
Project: bigbayes

We present a new approach to assessing the robustness of neural networks based on estimating the proportion of inputs for which a property is violated. Specifically, we estimate the probability of the event that the property is violated under an input model. Our approach critically varies from the formal verification framework in that when the property can be violated, it provides an informative notion of how robust the network is, rather than just the conventional assertion that the network is not verifiable. Furthermore, it provides an ability to scale to larger networks than formal verification approaches. Though the framework still provides a formal guarantee of satisfiability whenever it successfully finds one or more violations, these advantages do come at the cost of only providing a statistical estimate of unsatisfiability whenever no violation is found. Key to the practical success of our approach is an adaptation of multi-level splitting, a Monte Carlo approach for estimating the probability of rare events, to our statistical robustness framework. We demonstrate that our approach is able to emulate formal verification procedures on benchmark problems, while scaling to larger networks and providing reliable additional information in the form of accurate estimates of the violation probability.
```
@inproceedings{webb2018statistical,
  title = {{A Statistical Approach to Assessing Neural Network Robustness}},
  author = {Webb, Stefan and Rainforth, Tom and Teh, Yee Whye and Kumar, M Pawan},
  booktitle = {International Conference on Learning Representations (ICLR)},
  month = may,
  year = {2019}
}
```

J. Lee , L. James , S. Choi , F. Caron , A Bayesian model for sparse graphs with flexible degree distributionand overlapping community structure, in Artificial Intelligence and Statistics (AISTATS), 2019.

Project: bigbayes

@inproceedings{Lee2019,
  author = {Lee, Juho and James, Lancelot and Choi, Seungjin and Caron, Fran\c cois},
  title = {A Bayesian model for sparse graphs with flexible degree distributionand overlapping community structure},
  booktitle = {Artificial Intelligence and Statistics (AISTATS)},
  note = {ArXiv e-prints: 1810.01778},
  year = {2019},
  month = apr
}

B. Bloem-Reddy , Y. W. Teh , Probabilistic symmetry and invariant neural networks, Jan. 2019.

Project: bigbayes

@article{Bloem-Reddy:Teh:2019,
  archiveprefix = {arXiv},
  author = {Bloem-Reddy, Benjamin and Teh, Yee Whye},
  eprint = {1901.06082},
  month = jan,
  primaryclass = {stat.ML},
  title = {Probabilistic symmetry and invariant neural networks},
  year = {2019}
}

S. Flaxman , M. Chirico , P. Pereira , C. Loeffler , Scalable high-resolution forecasting of sparse spatiotemporal events with kernel methods: a winning solution to the NIJ “Real-Time Crime Forecasting Challenge,” Revised and resubmit at Annals of Applied Statistics, 2019.
Project: bigbayes

We propose a generic spatiotemporal event forecasting method, which we developed for the National Institute of Justice’s (NIJ) Real-Time Crime Forecasting Challenge. Our solution to the challenge is a spatiotemporal forecasting model combining scalable randomized Reproducing Kernel Hilbert Space (RKHS) methods for approximating Gaussian processes with autoregressive smoothing kernels in a regularized supervised learning framework. While the smoothing kernels capture the two main approaches in current use in the field of crime forecasting, kernel density estimation (KDE) and self-exciting point process (SEPP) models, the RKHS component of the model can be understood as an approximation to the popular log-Gaussian Cox Process model. For inference, we discretize the spatiotemporal point pattern and learn a log intensity function using the Poisson likelihood and highly efficient gradient-based optimization methods. Model hyperparameters including quality of RKHS approximation, spatial and temporal kernel lengthscales, number of autoregressive lags, bandwidths for smoothing kernels, as well as cell shape, size, and rotation, were learned using crossvalidation. Resulting predictions significantly exceeded baseline KDE estimates and SEPP models for sparse events.
```
@article{flaxman2019scalable,
  title = {Scalable high-resolution forecasting of sparse spatiotemporal events with kernel methods: a winning solution to the NIJ ``Real-Time Crime Forecasting Challenge''},
  author = {Flaxman, Seth and Chirico, Michael and Pereira, Pau and Loeffler, Charles},
  journal = {Revised and resubmit at Annals of Applied Statistics},
  year = {2019}
}
```
A. Raj , H. Law , D. Sejdinovic , M. Park , A Differentially Private Kernel Two-Sample Test, in European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), 2019, to appear.
Project: bigbayes

Kernel two-sample testing is a useful statistical tool in determining whether data samples arise from different distributions without imposing any parametric assumptions on those distributions. However, raw data samples can expose sensitive information about individuals who participate in scientific studies, which makes the current tests vulnerable to privacy breaches. Hence, we design a new framework for kernel two-sample testing conforming to differential privacy constraints, in order to guarantee the privacy of subjects in the data. Unlike existing differentially private parametric tests that simply add noise to data, kernel-based testing imposes a challenge due to a complex dependence of test statistics on the raw data, as these statistics correspond to estimators of distances between representations of probability measures in Hilbert spaces. Our approach considers finite dimensional approximations to those representations. As a result, a simple chi-squared test is obtained, where a test statistic depends on a mean and covariance of empirical differences between the samples, which we perturb for a privacy guarantee. We investigate the utility of our framework in two realistic settings and conclude that our method requires only a relatively modest increase in sample size to achieve a similar level of power to the non-private tests in both settings.
```
@inproceedings{RajLawSejPar2019,
  author = {Raj, A. and Law, H.C.L. and Sejdinovic, D. and Park, M.},
  title = {{{A Differentially Private Kernel Two-Sample Test}}},
  booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD)},
  pages = {to appear},
  year = {2019}
}
```
E. Mathieu , T. Rainforth , N. Siddharth , Y. W. Teh , Disentangling Disentanglement in Variational Autoencoders, in Proceedings of the 36th International Conference on Machine Learning, Long Beach, California, USA, 2019, vol. 97, 4402–4412.
Project: bigbayes

We develop a generalisation of disentanglement in VAEs—decomposition of the latent representation—characterising it as the fulfilment of two factors: a) the latent encodings of the data having an appropriate level of overlap, and b) the aggregate encoding of the data conforming to a desired structure, represented through the prior. Decomposition permits disentanglement, i.e. explicit independence between latents, as a special case, but also allows for a much richer class of properties to be imposed on the learnt representation, such as sparsity, clustering, independent subspaces, or even intricate hierarchical dependency relationships. We show that the β-VAE varies from the standard VAE predominantly in its control of latent overlap and that for the standard choice of an isotropic Gaussian prior, its objective is invariant to rotations of the latent representation. Viewed from the decomposition perspective, breaking this invariance with simple manipulations of the prior can yield better disentanglement with little or no detriment to reconstructions. We further demonstrate how other choices of prior can assist in producing different decompositions and introduce an alternative training objective that allows the control of both decomposition factors in a principled manner.
```
@inproceedings{pmlr-v97-mathieu19a,
  title = {Disentangling Disentanglement in Variational Autoencoders},
  author = {Mathieu, Emile and Rainforth, Tom and Siddharth, N and Teh, Yee Whye},
  booktitle = {Proceedings of the 36th International Conference on Machine Learning},
  pages = {4402--4412},
  year = {2019},
  volume = {97},
  series = {Proceedings of Machine Learning Research},
  address = {Long Beach, California, USA},
  month = {09--15 Jun},
  publisher = {PMLR}
}
```
A. Foster , M. Jankowiak , E. Bingham , P. Horsfall , Y. W. Teh , T. Rainforth , N. Goodman , Variational Bayesian Optimal Experimental Design, Advances in Neural Information Processing Systems (NeurIPS, spotlight), 2019.
Project: bigbayes

Bayesian optimal experimental design (BOED) is a principled framework for making efficient use of limited experimental resources. Unfortunately, its applicability is hampered by the difficulty of obtaining accurate estimates of the expected information gain (EIG) of an experiment. To address this, we introduce several classes of fast EIG estimators by building on ideas from amortized variational inference. We show theoretically and empirically that these estimators can provide significant gains in speed and accuracy over previous approaches. We further demonstrate the practicality of our approach on a number of end-to-end experiments.
```
@article{foster2019variational,
  title = {{Variational Bayesian Optimal Experimental Design}},
  author = {Foster, Adam and Jankowiak, Martin and Bingham, Eli and Horsfall, Paul and Teh, Yee Whye and Rainforth, Tom and Goodman, Noah},
  journal = {Advances in Neural Information Processing Systems (NeurIPS, spotlight)},
  year = {2019}
}
```
F. Locatello , G. Abbati , T. Rainforth , S. Bauer , B. Schölkopf , O. Bachem , On the Fairness of Disentangled Representations, Advances in Neural Information Processing Systems (NeurIPS, to appear), 2019.
Project: bigbayes

Recently there has been a significant interest in learning disentangled representations, as they promise increased interpretability, generalization to unseen scenarios and faster learning on downstream tasks. In this paper, we investigate the usefulness of different notions of disentanglement for improving the fairness of downstream prediction tasks based on representations. We consider the setting where the goal is to predict a target variable based on the learned representation of high-dimensional observations (such as images) that depend on both the target variable and an unobserved sensitive variable. We show that in this setting both the optimal and empirical predictions can be unfair, even if the target variable and the sensitive variable are independent. Analyzing more than 12600 trained representations of state-of-the-art disentangled models, we observe that various disentanglement scores are consistently correlated with increased fairness, suggesting that disentanglement may be a useful property to encourage fairness when sensitive variables are not observed.
```
@article{locatello2019fairness,
  title = {{On the Fairness of Disentangled Representations}},
  author = {Locatello, Francesco and Abbati, Gabriele and Rainforth, Tom and Bauer, Stefan and Sch{\"o}lkopf, Bernhard and Bachem, Olivier},
  journal = {Advances in Neural Information Processing Systems (NeurIPS, to appear)},
  year = {2019}
}
```
A. Golinski , F. Wood , T. Rainforth , Amortized Monte Carlo Integration, International Conference on Machine Learning (ICML, Best Paper honorable mention), 2019.
Project: bigbayes

Current approaches to amortizing Bayesian inference focus solely on approximating the posterior distribution. Typically, this approximation is, in turn, used to calculate expectations for one or more target functions—a computational pipeline which is inefficient when the target function(s) are known upfront. In this paper, we address this inefficiency by introducing AMCI, a method for amortizing Monte Carlo integration directly. AMCI operates similarly to amortized inference but produces three distinct amortized proposals, each tailored to a different component of the overall expectation calculation. At run-time, samples are produced separately from each amortized proposal, before being combined to an overall estimate of the expectation. We show that while existing approaches are fundamentally limited in the level of accuracy they can achieve, AMCI can theoretically produce arbitrarily small errors for any integrable target function using only a single sample from each proposal at run-time. We further show that it is able to empirically outperform the theoretically optimal self-normalized importance sampler on a number of example problems. Furthermore, AMCI allows not only for amortizing over datasets but also amortizing over target functions.
```
@article{golinski2018amci,
  title = {{Amortized Monte Carlo Integration}},
  author = {Golinski, Adam and Wood, Frank and Rainforth, Tom},
  journal = {International Conference on Machine Learning (ICML, Best Paper honorable mention)},
  year = {2019}
}
```
Y. Zhou , B. Gram-Hansen , T. Kohn , T. Rainforth , H. Yang , F. Wood , A Low-Level Probabilistic Programming Language for Non-Differentiable Models, International Conference on Artificial Intelligence and Statistics (AISTATS), 2019.
Project: bigbayes

We develop a new Low-level, First-order Prob- abilistic Programming Language (LF-PPL) suited for models containing a mix of contin- uous, discrete, and/or piecewise-continuous variables. The key success of this language and its compilation scheme, is in its ability to automatically distinguish the discontinuous and continuous parameters in the density func- tion, while further providing runtime checks of when discontinuity boundaries have been crossed. This enables the introduction of new inference engines that are able to exploit gra- dient information, while remaining efficient for models which are not everywhere differen- tiable. We demonstrate this ability by intro- ducing a discontinuous Hamiltonian Monte Carlo (DHMC) inference engine that is able to deliver automated and efficient inference for non-differentiable models. Our system is backed up by a mathematical formalism that ensures that any model expressed in this lan- guage has a density with a sufficiently low measure of discontinuities to maintain the validity of the inference engine.
```
@article{zhou2018lfppla,
  title = {{A Low-Level Probabilistic Programming Language
    for Non-Differentiable Models}},
  author = {Zhou, Yuan and Gram-Hansen, Bradley and Kohn, Tobias and Rainforth, Tom and Yang, Hongseok and Wood, Frank},
  year = {2019},
  journal = {International Conference on Artificial Intelligence and Statistics (AISTATS)}
}
```
A. Golinski* , M. Lezcano-Casado* , T. Rainforth , Improving Normalizing Flows via Better Orthogonal Parameterizations, ICML Workshop on Invertible Neural Nets and Normalizing Flows, 2019.
Project: bigbayes

Normalizing flows have to be designed in a manner that permits efficient computation of the determinant of the transformation Jacobian, while ensuring that the transformation remains invertible. Many methods achieve this by using transformations based on matrix decompositions, with decompositions that produce an orthogonal matrix being a popular such approach. However, this introduces the additional difficulty of performing a constrained optimization over the space of orthogonal matrices. Current approaches for achieving this use methods which are prone to numerical instability and which may introduce additional local minima into the optimization landscape. To address this, we introduce two orthogonal matrix parameterizations stemming from Lie group theory—the exponential map and the Cayley map—which alleviate these issues. In particular, they are proven to not introduce new local minima. Focusing on the example architecture of Sylvester normalizing flows (van den Berg et al. (2018)), we show empirically that using our suggested parameterizations lead to significantly improved optimization and, in turn, more effective normalizing flows.
```
@article{golinski2019snf,
  title = {{Improving Normalizing Flows via Better Orthogonal Parameterizations}},
  author = {Golinski*, Adam and Lezcano-Casado*, Mario and Rainforth, Tom},
  journal = {ICML Workshop on Invertible Neural Nets and Normalizing Flows},
  year = {2019}
}
```
J. Ton , L. Chan , Y. W. Teh , D. Sejdinovic , Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings, ArXiv e-prints:1906.02236, 2019.
Project: bigbayes tencent-lsml

Current meta-learning approaches focus on learning functional representations of relationships between variables, i.e. on estimating conditional expectations in regression. In many applications, however, we are faced with conditional distributions which cannot be meaningfully summarized using expectation only (due to e.g. multimodality). Hence, we consider the problem of conditional density estimation in the meta-learning setting. We introduce a novel technique for meta-learning which combines neural representation and noise-contrastive estimation with the established literature of conditional mean embeddings into reproducing kernel Hilbert spaces. The method is validated on synthetic and real-world problems, demonstrating the utility of sharing learned representations across multiple conditional density estimation tasks.
```
@unpublished{TonChaTehSej2019,
  author = {Ton, Jean-Francois and Chan, Lucian and Teh, Yee Whye and Sejdinovic, Dino},
  title = {{{Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings}}},
  journal = {ArXiv e-prints:1906.02236},
  year = {2019}
}
```

Y. Zhou , B. Gram-Hansen , T. Kohn , T. Rainforth , H. Yang , F. Wood , LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models, in The 22nd International Conference on Artificial Intelligence and Statistics, 2019, 148–157.

Project: bigbayes

@inproceedings{zhou2019lf,
  title = {LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models},
  author = {Zhou, Yuan and Gram-Hansen, Bradley and Kohn, Tobias and Rainforth, Tom and Yang, Hongseok and Wood, Frank},
  booktitle = {The 22nd International Conference on Artificial Intelligence and Statistics},
  pages = {148--157},
  year = {2019}
}

Y. Zhou , H. Yang , Y. W. Teh , T. Rainforth , Divide, Conquer, and Combine: a New Inference Strategy for Probabilistic Programs with Stochastic Support, International Conference on Machine Learning (ICML, to appear), 2019.

Project: bigbayes

@article{zhou2019divide,
  title = {Divide, Conquer, and Combine: a New Inference Strategy for Probabilistic Programs with Stochastic Support},
  author = {Zhou, Yuan and Yang, Hongseok and Teh, Yee Whye and Rainforth, Tom},
  journal = {International Conference on Machine Learning (ICML, to appear)},
  year = {2019}
}

2018

X. Miscouridou , F. Caron , Y. W. Teh , Modelling sparsity, heterogeneity, reciprocity and community structure in temporal interaction data, in Advances in Neural Information Processing Systems (NeurIPS), 2018.

Project: bigbayes

@inproceedings{HawkesInteractions,
  author = {Miscouridou, Xenia and Caron, Fran\c ois and Teh, Yee Whye},
  title = {Modelling sparsity, heterogeneity, reciprocity and community structure in temporal interaction data},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  note = {ArXiv e-prints: 1803.06070},
  year = {2018},
  month = dec
}

A. Golinski , Y. W. Teh , F. Wood , T. Rainforth , Amortized Monte Carlo Integration, in Symposium on Advances in Approximate Bayesian Inference, 2018.
Project: bigbayes

Current approaches to amortizing Bayesian inference focus solely on approximating the posterior distribution. Typically, this approximation is in turn used to calculate expec- tations for one or more target functions. In this paper, we address the inefficiency of this computational pipeline when the target function(s) are known upfront. To this end, we introduce a method for amortizing Monte Carlo integration. Our approach operates in a similar manner to amortized inference, but tailors the produced amortization arti- facts to maximize the accuracy of the resulting expectation calculation(s). We show that while existing approaches have fundamental limitations in the level of accuracy that can be achieved for a given run time computational budget, our framework can produce arbitrary small errors for a wide range of target functions with O(1) computational cost at run time. Furthermore, our framework allows not only for amortizing over possible datasets, but also over possible target functions.
```
@inproceedings{golinski2018amcj,
  title = {{Amortized Monte Carlo Integration}},
  author = {Golinski, Adam and Teh, Yee Whye and Wood, Frank and Rainforth, Tom},
  booktitle = {Symposium on Advances in Approximate Bayesian Inference},
  year = {2018},
  month = dec
}
```
J. Mitrovic , D. Sejdinovic , Y. Teh , Causal Inference via Kernel Deviance Measures, in Advances in Neural Information Processing Systems (NeurIPS), 2018.
Project: bigbayes

Discovering the causal structure among a set of variables is a fundamental problem in many areas of science. In this paper, we propose Kernel Conditional Deviance for Causal Inference (KCDC) a fully nonparametric causal discovery method based on purely observational data. From a novel interpretation of the notion of asymmetry between cause and effect, we derive a corresponding asymmetry measure using the framework of reproducing kernel Hilbert spaces. Based on this, we propose three decision rules for causal discovery. We demonstrate the wide applicability of our method across a range of diverse synthetic datasets. Furthermore, we test our method on real-world time series data and the real-world benchmark dataset Tubingen Cause-Effect Pairs where we outperform existing state-of-the-art methods.
```
@inproceedings{MitSejTeh2018,
  author = {Mitrovic, J. and Sejdinovic, D. and Teh, Y.W.},
  title = {{{Causal Inference via Kernel Deviance Measures}}},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2018},
  month = dec
}
```

J. Chen , J. Zhu , Y. W. Teh , T. Zhang , Stochastic Expectation Maximization with Variance Reduction, in Advances in Neural Information Processing Systems (NeurIPS), 2018, 7978–7988.

Project: bigbayes tencent-lsml

@inproceedings{NIPS2018_8021,
  author = {Chen, Jianfei and Zhu, Jun and Teh, Yee Whye and Zhang, Tong},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  editor = {Bengio, S. and Wallach, H. and Larochelle, H. and Grauman, K. and Cesa-Bianchi, N. and Garnett, R.},
  pages = {7978--7988},
  publisher = {Curran Associates, Inc.},
  title = {Stochastic Expectation Maximization with Variance Reduction},
  year = {2018},
  month = dec,
  bdsk-url-1 = {http://papers.nips.cc/paper/8021-stochastic-expectation-maximization-with-variance-reduction.pdf}
}

B. Bloem-Reddy , A. Foster , E. Mathieu , Y. W. Teh , Sampling and Inference for Beta Neutral-to-the-Left Models of Sparse Networks, in Conference on Uncertainty in Artificial Intelligence, 2018.

Project: bigbayes

@inproceedings{BloemReddy:etal:2018,
  author = {Bloem-Reddy, Benjamin and Foster, Adam and Mathieu, Emile and Teh, Yee Whye},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  title = {Sampling and Inference for Beta Neutral-to-the-Left Models of Sparse Networks},
  month = aug,
  year = {2018}
}

B. Bloem-Reddy , P. Orbanz , Random-Walk Models of Network Formation and Sequential Monte Carlo Methods for Graphs, Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 80, no. 5, 871–898, Aug. 2018.

Project: bigbayes

@article{Bloem-Reddy:Orbanz:2018,
  archiveprefix = {arXiv},
  author = {Bloem-Reddy, Benjamin and Orbanz, Peter},
  eprint = {1612.06404},
  month = aug,
  primaryclass = {stat.ME},
  journal = {Journal of the Royal Statistical Society: Series B (Statistical Methodology)},
  title = {Random-Walk Models of Network Formation and Sequential Monte Carlo Methods for Graphs},
  volume = {80},
  number = {5},
  pages = {871--898},
  year = {2018}
}

T. Rainforth , A. R. Kosiorek , T. A. Le , C. J. Maddison , M. Igl , F. Wood , Y. W. Teh , Tighter Variational Bounds are Not Necessarily Better, in International Conference on Machine Learning (ICML), 2018.
Project: bigbayes

We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted auto-encoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Moreover, PIWAE can simultaneously deliver improvements in both the quality of the inference network and generative network, relative to IWAE.
```
@inproceedings{rainforth2018tighter,
  title = {Tighter Variational Bounds are Not Necessarily Better},
  author = {Rainforth, Tom and Kosiorek, Adam R. and Le, Tuan Anh and Maddison, Chris J. and Igl, Maximilian and Wood, Frank and Teh, Yee Whye},
  booktitle = {International Conference on Machine Learning (ICML)},
  year = {2018},
  month = jul
}
```

X. Miscouridou , A. Perotte , N. Elhadad , R. Ranganath , Deep Survival Analysis: Nonparametrics and Missingness, in pmlr, 2018.

Project: bigbayes

@inproceedings{DGMforSurvival,
  author = {Miscouridou, Xenia and Perotte, Adler and Elhadad, Noemie and Ranganath, Rajesh},
  title = {Deep Survival Analysis: Nonparametrics and Missingness},
  booktitle = pmlr,
  note = {Proceedings of Machine Learning Research, 85:1-12},
  year = {2018},
  month = jun
}

M. Battiston , S. Favaro , D. M. Roy , Y. W. Teh , A Characterization of Product-Form Exchangeable Feature Probability Functions, Annals of Applied Probability, vol. 28, Jun. 2018.
Project: bigbayes

We characterize the class of exchangeable feature allocations assigning probability V_n,k ∏^k_l=1 W_m_lU_n−m_l to a feature allocation of n individuals, displaying k features with counts (m_1,\ldots,m_k) for these features. Each element of this class is parametrized by a countable matrix V and two sequences U and W of non-negative weights. Moreover, a consistency condition is imposed to guarantee that the distribution for feature allocations of n−1 individuals is recovered from that of n individuals, when the last individual is integrated out. In Theorem 1.1, we prove that the only members of this class satisfying the consistency condition are mixtures of the Indian Buffet Process over its mass parameter γ and mixtures of the Beta–Bernoulli model over its dimensionality parameter N. Hence, we provide a characterization of these two models as the only, up to randomization of the parameters, consistent exchangeable feature allocations having the required product form.
```
@article{BatFavRoy2016a,
  author = {Battiston, M. and Favaro, S. and Roy, D. M. and Teh, Y. W.},
  journal = {Annals of Applied Probability},
  title = {A Characterization of Product-Form Exchangeable Feature Probability Functions},
  volume = {28},
  year = {2018},
  month = jun,
  bdsk-url-1 = {https://arxiv.org/pdf/1607.02066.pdf}
}
```
H. Kim , Y. W. Teh , Scaling up the Automatic Statistician: Scalable Structure Discovery using Gaussian Processes, in Artificial Intelligence and Statistics (AISTATS), 2018.
Project: bigbayes

Automating statistical modelling is a challenging problem in artificial intelligence. The Automatic Statistician takes a first step in this direction, by employing a kernel search algorithm with Gaussian Processes (GP) to provide interpretable statistical models for regression problems. However this does not scale due to its O(N^3) running time for the model selection. We propose Scalable Kernel Composition (SKC), a scalable kernel search algorithm that extends the Automatic Statistician to bigger data sets. In doing so, we derive a cheap upper bound on the GP marginal likelihood that sandwiches the marginal likelihood with the variational lower bound . We show that the upper bound is significantly tighter than the lower bound and thus useful for model selection.
```
@inproceedings{KimTeh18,
  author = {Kim, H. and Teh, Y. W.},
  booktitle = {Artificial Intelligence and Statistics (AISTATS)},
  title = {Scaling up the Automatic Statistician: Scalable Structure Discovery using Gaussian Processes},
  month = apr,
  year = {2018}
}
```
Q. Zhang , S. Filippi , A. Gretton , D. Sejdinovic , Large-Scale Kernel Methods for Independence Testing, Statistics and Computing, vol. 28, no. 1, 113–130, Jan. 2018.
Project: bigbayes

Representations of probability measures in reproducing kernel Hilbert spaces provide a flexible framework for fully nonparametric hypothesis tests of independence, which can capture any type of departure from independence, including nonlinear associations and multivariate interactions. However, these approaches come with an at least quadratic computational cost in the number of observations, which can be prohibitive in many applications. Arguably, it is exactly in such large-scale datasets that capturing any type of dependence is of interest, so striking a favourable tradeoff between computational efficiency and test performance for kernel independence tests would have a direct impact on their applicability in practice. In this contribution, we provide an extensive study of the use of large-scale kernel approximations in the context of independence testing, contrasting block-based, Nystrom and random Fourier feature approaches. Through a variety of synthetic data experiments, it is demonstrated that our novel large scale methods give comparable performance with existing methods whilst using significantly less computation time and memory.
```
@article{ZhaFilGreSej2018,
  author = {Zhang, Q. and Filippi, S. and Gretton, A. and Sejdinovic, D.},
  journal = {Statistics and Computing},
  year = {2018},
  month = jan,
  volume = {28},
  number = {1},
  pages = {113--130},
  title = {{Large-Scale Kernel Methods for Independence Testing}}
}
```

F. Ayed , M. Battiston , F. Camerlenghi , S. Favaro , Consistent estimation of the missing mass for feature models, 2018.

Project: bigbayes

@unpublished{Ayed:Battiston:Camerlenghi:Favaro2018b,
  author = {Ayed, F. and Battiston, M. and Camerlenghi, F. and Favaro, S.},
  title = {Consistent estimation of the missing mass for feature models},
  year = {2018}
}

M. Battiston , S. Favaro , Y. W. Teh , Bayesian nonparametric approaches to sample-size estimation for finding unseen species, 2018.

Project: bigbayes

@unpublished{Battiston:Favaro:Teh:2018b,
  author = {Battiston, M. and Favaro, S. and Teh, Y. W.},
  title = {Bayesian nonparametric approaches to sample-size estimation for finding unseen species},
  year = {2018}
}

F. Ayed , M. Battiston , F. Camerlenghi , S. Favaro , On the consistent estimation of the missing mass, 2018.

Project: bigbayes

@unpublished{Ayed:Battiston:Camerlenghi:Favaro2018a,
  author = {Ayed, F. and Battiston, M. and Camerlenghi, F. and Favaro, S.},
  title = {On the consistent estimation of the missing mass},
  year = {2018}
}

F. Ayed , M. Battiston , F. Camerlenghi , S. Favaro , On the Good-Turing estimator for feature allocation models, 2018.

Project: bigbayes

@unpublished{Ayed:Battiston:Camerlenghi:Favaro2018c,
  author = {Ayed, F. and Battiston, M. and Camerlenghi, F. and Favaro, S.},
  title = {On the {Good-Turing} estimator for feature allocation models},
  year = {2018}
}

B. Bloem-Reddy , Y. W. Teh , Neural network models of exchangeable sequences, NeurIPS Workshop on Bayesian Deep Learning, 2018.

Project: bigbayes

@article{BloemReddy:Teh:2018aa,
  author = {Bloem-Reddy, Benjamin and Teh, Yee Whye},
  journal = {NeurIPS Workshop on Bayesian Deep Learning},
  title = {Neural network models of exchangeable sequences},
  year = {2018}
}

C. Loeffler , S. Flaxman , Is gun violence contagious? A spatiotemporal test, Journal of Quantitative Criminology, vol. 34, no. 4, 999–1017, 2018.
Project: bigbayes

Existing theories of gun violence predict stable spatial concentrations and contagious diffusion of gun violence into surrounding areas. Recent empirical studies have reported confirmatory evidence of such spatiotemporal diffusion of gun violence. However, existing tests cannot readily distinguish spatiotemporal clustering from spatiotemporal diffusion. This leaves as an open question whether gun violence actually is contagious or merely clusters in space and time. Compounding this problem, gun violence is subject to considerable measurement error with many nonfatal shootings going unreported to police. Using point process data from an acoustical gunshot locator system and a combination of Bayesian spatiotemporal point process modeling and space/time interaction tests, this paper demonstrates that contemporary urban gun violence does diffuse, but only slightly, suggesting that a disease model for infectious spread of gun violence is a poor fit for the geographically stable and temporally stochastic process observed.
```
@article{loeffler2018gun,
  title = {Is gun violence contagious? A spatiotemporal test},
  author = {Loeffler, Charles and Flaxman, Seth},
  journal = {Journal of Quantitative Criminology},
  volume = {34},
  number = {4},
  pages = {999--1017},
  year = {2018},
  publisher = {Springer}
}
```
A. Foster , M. Jankowiak , E. Bingham , Y. W. Teh , T. Rainforth , N. Goodman , Variational Optimal Experiment Design: Efficient Automation of Adaptive Experiments, NeurIPS Workshop on Bayesian Deep Learning, 2018.
Project: bigbayes

Bayesian optimal experimental design (OED) is a principled framework for making efficient use of limited experimental resources. Unfortunately, the applicability of OED is hampered by the difficulty of obtaining accurate estimates of the expected information gain (EIG) for different experimental designs. We introduce a class of fast EIG estimators that leverage amortised variational inference and show that they provide substantial empirical gains over previous approaches. We integrate our approach into a deep probabilistic programming framework, thus making OED accessible to practitioners at large.
```
@article{foster2018voed,
  title = {{Variational Optimal Experiment Design: Efficient Automation of Adaptive Experiments}},
  author = {Foster, Adam and Jankowiak, Martin and Bingham, Eli and Teh, Yee Whye and Rainforth, Tom and Goodman, Noah},
  journal = {NeurIPS Workshop on Bayesian Deep Learning},
  year = {2018}
}
```
S. Webb , A. Golinski , R. Zinkov , N. Siddharth , T. Rainforth , Y. W. Teh , F. Wood , Faithful Inversion of Generative Models for Effective Amortized Inference, in Advances in Neural Information Processing Systems (NeurIPS), 2018.
Project: bigbayes

Inference amortization methods share information across multiple posterior-inference problems, allowing each to be carried out more efficiently. Generally, they require the inversion of the dependency structure in the generative model, as the modeller must learn a mapping from observations to distributions approximating the posterior. Previous approaches have involved inverting the dependency structure in a heuristic way that fails to capture these dependencies correctly, thereby limiting the achievable accuracy of the resulting approximations. We introduce an algorithm for faithfully, and minimally, inverting the graphical model structure of any generative model. Such inverses have two crucial properties: (a) they do not encode any independence assertions that are absent from the model and; (b) they are local maxima for the number of true independencies encoded. We prove the correctness of our approach and empirically show that the resulting minimally faithful inverses lead to better inference amortization than existing heuristic approaches.
```
@inproceedings{webb2018minimal,
  title = {Faithful Inversion of Generative Models for Effective Amortized Inference},
  author = {Webb, Stefan and Golinski, Adam and Zinkov, Robert and Siddharth, N. and Rainforth, Tom and Teh, Yee Whye and Wood, Frank},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2018}
}
```
A. R. Kosiorek , H. Kim , Y. W. Teh , I. Posner , Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects, in Advances in Neural Information Processing Systems (NeurIPS), 2018.
Project: bigbayes

We present Sequential Attend, Infer, Repeat (SQAIR), an interpretable deep generative model for videos of moving objects. It can reliably discover and track objects throughout the sequence of frames, and can also generate future frames conditioning on the current frame, thereby simulating expected motion of objects. This is achieved by explicitly encoding object presence, locations and appearances in the latent variables of the model. SQAIR retains all strengths of its predecessor, Attend, Infer, Repeat (AIR, Eslami et. al., 2016), including learning in an unsupervised manner, and addresses its shortcomings. We use a moving multi-MNIST dataset to show limitations of AIR in detecting overlapping or partially occluded objects, and show how SQAIR overcomes them by leveraging temporal consistency of objects. Finally, we also apply SQAIR to real-world pedestrian CCTV data, where it learns to reliably detect, track and generate walking pedestrians with no supervision.
```
@inproceedings{koskimposteh18,
  title = {Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects},
  author = {Kosiorek, Adam R. and Kim, Hyunjik and Teh, Yee Whye and Posner, Ingmar},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2018}
}
```
H. Kim , A. Mnih , Disentangling by Factorising, in International Conference on Machine Learning (ICML), 2018.
Project: bigbayes

We define and address the problem of unsupervised learning of disentangled representations on data generated from independent factors of variation. We propose FactorVAE, a method that disentangles by encouraging the distribution of representations to be factorial and hence independent across the dimensions. We show that it improves upon beta-VAE by providing a better trade-off between disentanglement and reconstruction quality. Moreover, we highlight the problems of a commonly used disentanglement metric and introduce a new metric that does not suffer from them.
```
@inproceedings{KimMnih18,
  author = {Kim, H. and Mnih, A.},
  booktitle = {International Conference on Machine Learning (ICML)},
  title = {Disentangling by Factorising},
  year = {2018}
}
```
H. Law , D. Sejdinovic , E. Cameron , T. Lucas , S. Flaxman , K. Battle , K. Fukumizu , Variational Learning on Aggregate Outputs with Gaussian Processes, in Advances in Neural Information Processing Systems (NeurIPS), 2018, to appear.
Project: bigbayes

While a typical supervised learning framework assumes that the inputs and the outputs are measured at the same levels of granularity, many applications, including global mapping of disease, only have access to outputs at a much coarser level than that of the inputs. Aggregation of outputs makes generalization to new inputs much more difficult. We consider an approach to this problem based on variational learning with a model of output aggregation and Gaussian processes, where aggregation leads to intractability of the standard evidence lower bounds. We propose new bounds and tractable approximations, leading to improved prediction accuracy and scalability to large datasets, while explicitly taking uncertainty into account. We develop a framework which extends to several types of likelihoods, including the Poisson model for aggregated count data. We apply our framework to a challenging and important problem, the fine-scale spatial modelling of malaria incidence, with over 1 million observations.
```
@inproceedings{LawSejCamLucFlaBatFuk2018,
  author = {Law, H.C.L. and Sejdinovic, D. and Cameron, E. and Lucas, T.C.D. and Flaxman, S. and Battle, K. and Fukumizu, K.},
  title = {{{Variational Learning on Aggregate Outputs with Gaussian Processes}}},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  pages = {to appear},
  year = {2018}
}
```

J. Heo , H. Lee , S. Kim , J. Lee , K. Kim , E. Yang , S. Hwang , Uncertainty-aware attention for reliable interpretation and prediction, in Advances in Neural Information Processing Systems (NeurIPS), 2018.

Project: bigbayes

@inproceedings{Heo2018,
  title = {Uncertainty-aware attention for reliable interpretation and prediction},
  author = {Heo, J. and Lee, H. and Kim, S. and Lee, J. and Kim, K. and Yang, E. and Hwang, S.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2018}
}

H. Lee , J. Lee , S. Kim , E. Yang , S. Hwang , DropMax: adaptive variational softmax, in Advances in Neural Information Processing Systems (NeurIPS), 2018.

Project: bigbayes

@inproceedings{Lee2018,
  title = {DropMax: adaptive variational softmax},
  author = {Lee, H. and Lee, J. and Kim, S. and Yang, E. and Hwang, S.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2018}
}

T. Rainforth , Y. Zhou , X. Lu , Y. W. Teh , F. Wood , H. Yang , J. Meent , Inference Trees: Adaptive Inference with Exploration, arXiv preprint arXiv:1806.09550, 2018.
Project: bigbayes

We introduce inference trees (ITs), a new class of inference methods that build on ideas from Monte Carlo tree search to perform adaptive sampling in a manner that balances exploration with exploitation, ensures consistency, and alleviates pathologies in existing adaptive methods. ITs adaptively sample from hierarchical partitions of the parameter space, while simultaneously learning these partitions in an online manner. This enables ITs to not only identify regions of high posterior mass, but also maintain uncertainty estimates to track regions where significant posterior mass may have been missed. ITs can be based on any inference method that provides a consistent estimate of the marginal likelihood. They are particularly effective when combined with sequential Monte Carlo, where they capture long-range dependencies and yield improvements beyond proposal adaptation alone.
```
@article{rainforth2018it,
  title = {Inference Trees: Adaptive Inference with Exploration},
  author = {Rainforth, Tom and Zhou, Yuan and Lu, Xiaoyu and Teh, Yee Whye and Wood, Frank and Yang, Hongseok and van de Meent, Jan-Willem},
  journal = {arXiv preprint arXiv:1806.09550},
  year = {2018}
}
```
X. Lu , T. Rainforth , Y. Zhou , J. Meent , Y. W. Teh , On Exploration, Exploitation and Learning in Adaptive Importance Sampling, arXiv preprint arXiv:1810.13296, 2018.
Project: bigbayes

We study adaptive importance sampling (AIS) as an online learning problem and argue for the importance of the trade-off between exploration and exploitation in this adaptation. Borrowing ideas from the bandits literature, we propose Daisee, a partition-based AIS algorithm. We further introduce a notion of regret for AIS and show that Daisee has O((log T)^(3/4) √T) cumulative pseudo-regret, where T is the number of iterations. We then extend Daisee to adaptively learn a hierarchical partitioning of the sample space for more efficient sampling and confirm the performance of both algorithms empirically.
```
@article{lu2018exploration,
  title = {{On Exploration, Exploitation and Learning in Adaptive Importance Sampling}},
  author = {Lu, Xiaoyu and Rainforth, Tom and Zhou, Yuan and van de Meent, Jan-Willem and Teh, Yee Whye},
  journal = {arXiv preprint arXiv:1810.13296},
  year = {2018}
}
```
T. Rainforth , Nesting Probabilistic Programs, Conference on Uncertainty in Artificial Intelligence (UAI), 2018.
Project: bigbayes

We formalize the notion of nesting probabilistic programming queries and investigate the resulting statistical implications. We demonstrate that query nesting allows the definition of models which could not otherwise be expressed, such as those involving agents reasoning about other agents, but that existing systems take approaches that lead to inconsistent estimates. We show how to correct this by delineating possible ways one might want to nest queries and asserting the respective conditions required for convergence. We further introduce, and prove the correctness of, a new online nested Monte Carlo estimation method that makes it substantially easier to ensure these conditions are met, thereby providing a simple framework for designing statistically correct inference engines.
```
@article{rainforth2018nesting,
  title = {Nesting Probabilistic Programs},
  author = {Rainforth, Tom},
  journal = {Conference on Uncertainty in Artificial Intelligence (UAI)},
  year = {2018}
}
```
T. Rainforth , R. Cornish , H. Yang , A. Warrington , F. Wood , On Nesting Monte Carlo Estimators, International Conference on Machine Learning (ICML), 2018.
Project: bigbayes

Many problems in machine learning and statistics involve nested expectations and thus do not permit conventional Monte Carlo (MC) estimation. For such problems, one must nest estimators, such that terms in an outer estimator themselves involve calculation of a separate, nested, estimation. We investigate the statistical implications of nesting MC estimators, including cases of multiple levels of nesting, and establish the conditions un- der which they converge. We derive corresponding rates of convergence and provide empirical evidence that these rates are observed in practice. We further establish a number of pitfalls that can arise from naive nesting of MC estimators, provide guidelines about how these can be avoided, and lay out novel methods for reformulating certain classes of nested expectation problems into single expectations, leading to improved convergence rates. We demonstrate the applicability of our work by using our results to develop a new estimator for discrete Bayesian experimental design problems and derive error bounds for a class of variational objectives.
```
@article{rainforth2017opportunities,
  title = {{On Nesting Monte Carlo Estimators}},
  author = {Rainforth, Tom and Cornish, Robert and Yang, Hongseok and Warrington, Andrew and Wood, Frank},
  journal = {International Conference on Machine Learning (ICML)},
  year = {2018}
}
```
J. Ton , S. Flaxman , D. Sejdinovic , S. Bhatt , Spatial Mapping with Gaussian Processes and Nonstationary Fourier Features, Spatial Statistics, vol. 28, 59–78, 2018.
Project: bigbayes

The use of covariance kernels is ubiquitous in the field of spatial statistics. Kernels allow data to be mapped into high-dimensional feature spaces and can thus extend simple linear additive methods to nonlinear methods with higher order interactions. However, until recently, there has been a strong reliance on a limited class of stationary kernels such as the Matern or squared exponential, limiting the expressiveness of these modelling approaches. Recent machine learning research has focused on spectral representations to model arbitrary stationary kernels and introduced more general representations that include classes of nonstationary kernels. In this paper, we exploit the connections between Fourier feature representations, Gaussian processes and neural networks to generalise previous approaches and develop a simple and efficient framework to learn arbitrarily complex nonstationary kernel functions directly from the data, while taking care to avoid overfitting using state-of-the-art methods from deep learning. We highlight the very broad array of kernel classes that could be created within this framework. We apply this to a time series dataset and a remote sensing problem involving land surface temperature in Eastern Africa. We show that without increasing the computational or storage complexity, nonstationary kernels can be used to improve generalisation performance and provide more interpretable results.
```
@article{TonFlaSejBha2018,
  author = {Ton, J.-F. and Flaxman, S. and Sejdinovic, D. and Bhatt, S.},
  title = {{Spatial Mapping with Gaussian Processes and Nonstationary Fourier Features}},
  journal = {Spatial Statistics},
  year = {2018},
  volume = {28},
  pages = {59--78}
}
```
H. Law , D. Sutherland , D. Sejdinovic , S. Flaxman , Bayesian Approaches to Distribution Regression, in Artificial Intelligence and Statistics (AISTATS), 2018.
Project: bigbayes

Distribution regression has recently attracted much interest as a generic solution to the problem of supervised learning where labels are available at the group level, rather than at the individual level. Current approaches, however, do not propagate the uncertainty in observations due to sampling variability in the groups. This effectively assumes that small and large groups are estimated equally well, and should have equal weight in the final regression. We account for this uncertainty with a Bayesian distribution regression formalism, improving the robustness and performance of the model when group sizes vary. We frame our models in a neural network style, allowing for simple MAP inference using backpropagation to learn the parameters, as well as MCMC-based inference which can fully propagate uncertainty. We demonstrate our approach on illustrative toy datasets, as well as on a challenging problem of predicting age from images.
```
@inproceedings{LawSutSejFla2018,
  author = {Law, H.C.L. and Sutherland, D.J. and Sejdinovic, D. and Flaxman, S.},
  title = {{Bayesian Approaches to Distribution Regression}},
  booktitle = {Artificial Intelligence and Statistics (AISTATS)},
  year = {2018}
}
```

2017

V. Perrone , P. A. Jenkins , D. Spano , Y. W. Teh , Poisson Random Fields for Dynamic Feature Models, Journal of Machine Learning Research (JMLR), Dec. 2017.

Project: bigbayes

@article{PerJenSpa2017a,
  author = {Perrone, V. and Jenkins, P. A. and Spano, D. and Teh, Y. W.},
  journal = {Journal of Machine Learning Research (JMLR)},
  month = dec,
  year = {2017},
  title = {{P}oisson Random Fields for Dynamic Feature Models},
  bdsk-url-1 = {https://arxiv.org/abs/1611.07460},
  bdsk-url-2 = {https://arxiv.org/pdf/1611.07460v1.pdf}
}

G. Di Benedetto , F. Caron , Y. W. Teh , Non-exchangeable random partition models for microclustering, Nov-2017.

Project: bigbayes

@unpublished{DiBenedetto2017,
  author = {Di Benedetto, Giuseppe and Caron, Fran{\c{c}}ois and Teh, Yee Whye},
  title = {Non-exchangeable random partition models for microclustering},
  note = {ArXiv e-prints:1711.07287},
  archiveprefix = {arXiv},
  year = {2017},
  month = nov
}

B. Bloem-Reddy , P. Orbanz , Preferential Attachment and Vertex Arrival Times, Oct. 2017.

Project: bigbayes

@article{Bloem-Reddy:Orbanz:2017,
  archiveprefix = {arXiv},
  author = {Bloem-Reddy, Benjamin and Orbanz, Peter},
  eprint = {1710.02159},
  month = oct,
  primaryclass = {math.PR},
  title = {Preferential Attachment and Vertex Arrival Times},
  year = {2017}
}

A. Todeschini , X. Miscouridou , F. Caron , Exchangeable Random Measures for Sparse and Modular Graphs with Overlapping Communities, Aug-2017.

Project: bigbayes

@unpublished{OverlappingCommunityGraph,
  author = {Todeschini, Adrien and Miscouridou, Xenia and Caron, Fran\c ois},
  title = {Exchangeable Random Measures for Sparse and Modular Graphs with Overlapping Communities},
  note = {ArXiv e-prints: 1602.02114},
  archiveprefix = {arXiv},
  year = {2017},
  month = aug
}

J. Arbel , S. Favaro , B. Nipoti , Y. W. Teh , Bayesian nonparametric inference for discovery probabilities: credible intervals and large sample asymptotics, Statistica Sinica, Apr. 2017.
Project: bigbayes

Given a sample of size n from a population of individuals belonging to different species with unknown proportions, a popular problem of practical interest consists in making inference on the probability Dn(l) that the (n+1)-th draw coincides with a species with frequency l in the sample, for any l = 0,1,...,n. This paper contributes to the methodology of Bayesian nonparametric inference for Dn(l). Specifically, under the general framework of Gibbs-type priors we show how to derive credible intervals for a Bayesian nonparametric estimation of Dn(l), and we investigate the large n asymptotic behaviour of such an estimator. Of particular interest are special cases of our results obtained under the specification of the two parameter Poisson–Dirichlet prior and the normalized generalized Gamma prior, which are two of the most commonly used Gibbs-type priors. With respect to these two prior specifications, the proposed results are illustrated through a simulation study and a benchmark Expressed Sequence Tags dataset. To the best our knowledge, this illustration provides the first comparative study between the two parameter Poisson–Dirichlet prior and the normalized generalized Gamma prior in the context of Bayesian nonparemetric inference for Dn(l).
```
@article{ArbFavNip2017a,
  author = {Arbel, J. and Favaro, S. and Nipoti, B. and Teh, Y. W.},
  journal = {Statistica Sinica},
  title = {{B}ayesian nonparametric inference for discovery probabilities: credible intervals and large sample asymptotics},
  year = {2017},
  month = apr
}
```
X. Lu , V. Perrone , L. Hasenclever , Y. W. Teh , S. J. Vollmer , Relativistic Monte Carlo, in Artificial Intelligence and Statistics (AISTATS), 2017.
Project: bigbayes

Hamiltonian Monte Carlo (HMC) is a popular Markov chain Monte Carlo (MCMC) algorithm that generates proposals for a Metropolis-Hastings algorithm by simulating the dynamics of a Hamiltonian system. However, HMC is sensitive to large time discretizations and performs poorly if there is a mismatch between the spatial geometry of the target distribution and the scales of the momentum distribution. In particular the mass matrix of HMC is hard to tune well. In order to alleviate these problems we propose relativistic Hamiltonian Monte Carlo, a version of HMC based on relativistic dynamics that introduce a maximum velocity on particles. We also derive stochastic gradient versions of the algorithm and show that the resulting algorithms bear interesting relationships to gradient clipping, RMSprop, Adagrad and Adam, popular optimisation methods in deep learning. Based on this, we develop relativistic stochastic gradient descent by taking the zero-temperature limit of relativistic stochastic gradient Hamiltonian Monte Carlo. In experiments we show that the relativistic algorithms perform better than classical Newtonian variants and Adam.
```
@inproceedings{LuPerHas2016a,
  author = {Lu, X. and Perrone, V. and Hasenclever, L. and Teh, Y. W. and Vollmer, S. J.},
  booktitle = {Artificial Intelligence and Statistics (AISTATS)},
  title = {Relativistic {M}onte {C}arlo},
  month = apr,
  year = {2017},
  bdsk-url-1 = {https://arxiv.org/pdf/1609.04388v1.pdf}
}
```

M. Battiston , S. Favaro , Discussion of F. Caron and E. B. Fox, "Sparse graphs using exchangeable random measures.", Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 79, no. 5, 2017.

Project: bigbayes

@article{Battiston:Favaro:2017a,
  author = {Battiston, M. and Favaro, S.},
  journal = {Journal of the Royal Statistical Society: Series B (Statistical Methodology)},
  number = {5},
  title = {Discussion of F. Caron and E. B. Fox, "Sparse graphs using exchangeable random measures."},
  volume = {79},
  year = {2017}
}

S. Bacallado , M. Battiston , S. Favaro , L. Trippa , Sufficientness postulates for Gibbs-type priors and hierarchial generalizations, Statistical Sciences, vol. 32, 487–500, 2017.

Project: bigbayes

@article{Battiston:Bacallado:Favaro:Trippa:2017,
  author = {Bacallado, S. and Battiston, M. and Favaro, S. and Trippa, L.},
  journal = {Statistical Sciences},
  pages = {487–-500},
  title = {Sufficientness postulates for Gibbs-type priors and hierarchial generalizations},
  volume = {32},
  year = {2017}
}

B. Goodman , S. Flaxman , European Union Regulations on Algorithmic Decision Making and a “Right to Explanation,” AI Magazine, vol. 38, no. 3, 50–58, 2017.

Project: bigbayes

@article{goodman2017european,
  author = {Goodman, Bryce and Flaxman, Seth},
  journal = {AI Magazine},
  number = {3},
  pages = {50--58},
  publisher = {American Association for Artificial Intelligence},
  title = {{European Union} Regulations on Algorithmic Decision Making and a ``Right to Explanation''},
  volume = {38},
  year = {2017}
}

L. Hasenclever , S. Webb , T. Lienart , S. Vollmer , B. Lakshminarayanan , C. Blundell , Y. W. Teh , Distributed Bayesian Learning with Stochastic Natural Gradient Expectation Propagation and the Posterior Server, Journal of Machine Learning Research, vol. 18, no. 106, 1–37, 2017.
Project: bigbayes sgmcmc

This paper makes two contributions to Bayesian machine learning algorithms. Firstly, we propose stochastic natural gradient expectation propagation (SNEP), a novel alternative to expectation propagation (EP), a popular variational inference algorithm. SNEP is a black box variational algorithm, in that it does not require any simplifying assumptions on the distribution of interest, beyond the existence of some Monte Carlo sampler for estimating the moments of the EP tilted distributions. Further, as opposed to EP which has no guarantee of convergence, SNEP can be shown to be convergent, even when using Monte Carlo moment estimates. Secondly, we propose a novel architecture for distributed Bayesian learning which we call the posterior server. The posterior server allows scalable and robust Bayesian learning in cases where a dataset is stored in a distributed manner across a cluster, with each compute node containing a disjoint subset of data. An independent Monte Carlo sampler is run on each compute node, with direct access only to the local data subset, but which targets an approximation to the global posterior distribution given all data across the whole cluster. This is achieved by using a distributed asynchronous implementation of SNEP to pass messages across the cluster. We demonstrate SNEP and the posterior server on distributed Bayesian learning of logistic regression and neural networks.
```
@article{HasWebLie2015a,
  author = {Hasenclever, Leonard and Webb, Stefan and Lienart, Thibaut and Vollmer, Sebastian and Lakshminarayanan, Balaji and Blundell, Charles and Teh, Yee Whye},
  title = {Distributed Bayesian Learning with Stochastic Natural Gradient Expectation Propagation and the Posterior Server},
  journal = {Journal of Machine Learning Research},
  year = {2017},
  volume = {18},
  number = {106},
  pages = {1-37}
}
```

K. Palla , D. Belgrave , A Birth-Death Modelling Framework for Inferring Disease Causality within the Context of Allergy Development., in 16th IEEE International Conference on Machine Learning and Applications (ICMLA), 2017.

Project: bigbayes

@inproceedings{PallaBel17,
  author = {Palla, Konstantina and Belgrave, Danielle},
  title = {A Birth-Death Modelling Framework for Inferring Disease Causality within the Context of Allergy Development.},
  year = {2017},
  booktitle = {16th IEEE International Conference on Machine Learning and Applications (ICMLA)}
}

K. Palla , D. A. Knowles , Z. Ghahramani , A birth-death process for feature allocation., in Proceedings of the 34th International Conference on Machine Learning, 2017.
Project: bigbayes

We propose a Bayesian nonparametric prior over feature allocations for sequential data, the birth-death feature allocation process (BDFP). The BDFP models the evolution of the feature allocation of a set of objects N across a covariate (e.g. time) by creating and deleting features. A BDFP is exchangeable, projective, stationary and reversible, and its equilibrium distribution is given by the Indian buffet process (IBP). We also present the Beta Event Process (BEP) and we show that it is the de Finetti mixing distribution underlying the BDFP. This results shows that the BEP plays the role for the BDFP that the Beta process plays for the Indian buffet process. Moreover, we show that the BEP permits simplified inference. The utility of this prior is demonstrated on synthetic and real world data.
```
@inproceedings{PalKnoGha2016,
  author = {Palla, Konstantina and Knowles, David A. and Ghahramani, Zoubin},
  title = {A birth-death process for feature allocation.},
  booktitle = {Proceedings of the 34th International Conference on Machine Learning},
  year = {2017}
}
```
B. Bloem-Reddy , E. Mathieu , A. Foster , T. Rainforth , H. Ge , M. Lomelí , Z. Ghahramani , Y. W. Teh , Sampling and inference for discrete random probability measures in probabilistic programs, NIPS Workshop on Advances in Approximate Bayesian Inference, 2017.
Project: bigbayes

We consider the problem of sampling a sequence from a discrete random probability measure (RPM) with countable support, under (probabilistic) constraints of finite memory and computation. A canonical example is sampling from the Dirichlet Process, which can be accomplished using its stick-breaking representation and lazy initialization of its atoms. We show that efficiently lazy initialization is possible if and only if a size-biased representation of the discrete RPM is used. For models constructed from such discrete RPMs, we consider the implications for generic particle-based inference methods in probabilistic programming systems. To demonstrate, we implement SMC for Normalized Inverse Gaussian Process mixture models in Turing.
```
@article{bloemreddy2017rpm,
  title = {Sampling and inference for discrete random probability measures in probabilistic programs},
  author = {Bloem-Reddy, Benjamin and Mathieu, Emile and Foster, Adam and Rainforth, Tom and Ge, Hong and Lomelí, María and Ghahramani, Zoubin and Teh, Yee Whye},
  journal = {NIPS Workshop on Advances in Approximate Bayesian Inference},
  year = {2017}
}
```
S. Flaxman , Y. Teh , D. Sejdinovic , Poisson Intensity Estimation with Reproducing Kernels, Electronic Journal of Statistics, vol. 11, no. 2, 5081–5104, 2017.
Project: bigbayes

Despite the fundamental nature of the inhomogeneous Pois- son process in the theory and application of stochastic processes, and its attractive generalizations (e.g. Cox process), few tractable nonparametric modeling approaches of intensity functions exist, especially when observed points lie in a high-dimensional space. In this paper we develop a new, computationally tractable Reproducing Kernel Hilbert Space (RKHS) for- mulation for the inhomogeneous Poisson process. We model the square root of the intensity as an RKHS function. Whereas RKHS models used in su- pervised learning rely on the so-called representer theorem, the form of the inhomogeneous Poisson process likelihood means that the representer theorem does not apply. However, we prove that the representer theorem does hold in an appropriately transformed RKHS, guaranteeing that the optimization of the penalized likelihood can be cast as a tractable finite- dimensional problem. The resulting approach is simple to implement, and readily scales to high dimensions and large-scale datasets.
```
@article{FlaTehSej2017ejs,
  author = {Flaxman, S. and Teh, Y.W. and Sejdinovic, D.},
  title = {{{Poisson Intensity Estimation with Reproducing Kernels}}},
  journal = {Electronic Journal of Statistics},
  year = {2017},
  volume = {11},
  number = {2},
  pages = {5081--5104}
}
```
Q. Zhang , S. Filippi , S. Flaxman , D. Sejdinovic , Feature-to-Feature Regression for a Two-Step Conditional Independence Test, in Uncertainty in Artificial Intelligence (UAI), 2017.
Project: bigbayes

The algorithms for causal discovery and more broadly for learning the structure of graphical models require well calibrated and consistent conditional independence (CI) tests. We revisit the CI tests which are based on two-step procedures and involve regression with subsequent (unconditional) independence test (RESIT) on regression residuals and investigate the assumptions under which these tests operate. In particular, we demonstrate that when going beyond simple functional relationships with additive noise, such tests can lead to an inflated number of false discoveries. We study the relationship of these tests with those based on dependence measures using reproducing kernel Hilbert spaces (RKHS) and propose an extension of RESIT which uses RKHS-valued regression. The resulting test inherits the simple two-step testing procedure of RESIT, while giving correct Type I control and competitive power. When used as a component of the PC algorithm, the proposed test is more robust to the case where hidden variables induce a switching behaviour in the associations present in the data.
```
@inproceedings{ZhaFilFlaSej2017,
  author = {Zhang, Q. and Filippi, S. and Flaxman, S. and Sejdinovic, D.},
  title = {{Feature-to-Feature Regression for a Two-Step Conditional Independence Test}},
  booktitle = {Uncertainty in Artificial Intelligence (UAI)},
  year = {2017}
}
```
J. Mitrovic , D. Sejdinovic , Y. W. Teh , Deep Kernel Machines via the Kernel Reparametrization Trick, in International Conference on Learning Representations (ICLR) Workshop Track, 2017.
Project: bigbayes

While deep neural networks have achieved state-of-the-art performance on many tasks across varied domains, they still remain black boxes whose inner workings are hard to interpret and understand. In this paper, we develop a novel method for efficiently capturing the behaviour of deep neural networks using kernels. In particular, we construct a hierarchy of increasingly complex kernels that encode individual hidden layers of the network. Furthermore, we discuss how our framework motivates a novel supervised weight initialization method that discovers highly discriminative features already at initialization.
```
@inproceedings{MitSejTeh2017,
  author = {Mitrovic, J. and Sejdinovic, D. and Teh, Y. W.},
  booktitle = {International Conference on Learning Representations (ICLR) Workshop Track},
  title = {{Deep Kernel Machines via the Kernel Reparametrization Trick}},
  year = {2017},
  bdsk-url-1 = {https://openreview.net/forum?id=Bkiqt3Ntg&noteId=Bkiqt3Ntg}
}
```
S. Flaxman , Y. W. Teh , D. Sejdinovic , Poisson Intensity Estimation with Reproducing Kernels, in Artificial Intelligence and Statistics (AISTATS), 2017.
Project: bigbayes

Despite the fundamental nature of the inhomogeneous Poisson process in the theory and application of stochastic processes, and its attractive generalizations (e.g. Cox process), few tractable nonparametric modeling approaches of intensity functions exist, especially in high dimensional settings. In this paper we develop a new, computationally tractable Reproducing Kernel Hilbert Space (RKHS) formulation for the inhomogeneous Poisson process. We model the square root of the intensity as an RKHS function. The modeling challenge is that the usual representer theorem arguments no longer apply due to the form of the inhomogeneous Poisson process likelihood. However, we prove that the representer theorem does hold in an appropriately transformed RKHS, guaranteeing that the optimization of the penalized likelihood can be cast as a tractable finite-dimensional problem. The resulting approach is simple to implement, and readily scales to high dimensions and large-scale datasets.
```
@inproceedings{FlaTehSej2017,
  author = {Flaxman, S. and Teh, Y. W. and Sejdinovic, D.},
  booktitle = {Artificial Intelligence and Statistics (AISTATS)},
  title = {{Poisson Intensity Estimation with Reproducing Kernels}},
  year = {2017}
}
```
M. Lomeli , S. Favaro , Y. W. Teh , A Marginal Sampler for σ-Stable Poisson-Kingman Mixture Models, Journal of Computational and Graphical Statistics, 2017.
Project: bigbayes

We investigate the class of σ-stable Poisson-Kingman random probability measures (RPMs) in the context of Bayesian nonparametric mixture modeling. This is a large class of discrete RPMs which encompasses most of the popular discrete RPMs used in Bayesian nonparametrics, such as the Dirichlet process, Pitman-Yor process, the normalized inverse Gaussian process and the normalized generalized Gamma process. We show how certain sampling properties and marginal characterisations of σ-stable Poisson-Kingman RPMs can be usefully exploited for devising a Markov chain Monte Carlo (MCMC) algorithm for performing posterior inference with a Bayesian nonparametric mixture model. Specifically, we introduce a novel and efficient MCMC sampling scheme in an augmented space that has a fixed number of auxiliary variables per iteration. We apply our sampling scheme to a density estimation and clustering tasks with unidimensional and multidimensional datasets, and compare it against competing MCMC sampling schemes.
```
@article{LomFavTeh2015b,
  author = {Lomeli, M. and Favaro, S. and Teh, Y. W.},
  doi = {10.1080/10618600.2015.1110526},
  journal = {Journal of Computational and Graphical Statistics},
  title = {A Marginal Sampler for {$\sigma$}-Stable {P}oisson-{K}ingman Mixture Models},
  year = {2017},
  bdsk-url-1 = {http://www.tandfonline.com/doi/abs/10.1080/10618600.2015.1110526},
  bdsk-url-2 = {http://dx.doi.org/10.1080/10618600.2015.1110526},
  bdsk-url-3 = {https://arxiv.org/pdf/1407.4211v3.pdf}
}
```

2016

S. Flaxman , D. Sutherland , Y. Wang , Y. W. Teh , Understanding the 2016 US Presidential Election using ecological inference and distribution regression with census microdata, Arxiv e-prints, Nov-2016.
Project: bigbayes

We combine fine-grained spatially referenced census data with the vote outcomes from the 2016 US presidential election. Using this dataset, we perform ecological inference using distribution regression (Flaxman et al, KDD 2015) with a multinomial-logit regression so as to model the vote outcome Trump, Clinton, Other / Didn’t vote as a function of demographic and socioeconomic features. Ecological inference allows us to estimate "exit poll" style results like what was Trump’s support among white women, but for entirely novel categories. We also perform exploratory data analysis to understand which census variables are predictive of voting for Trump, voting for Clinton, or not voting for either. All of our methods are implemented in python and R and are available online for replication.
```
@unpublished{flaxsuthetal2016,
  author = {Flaxman, Seth and Sutherland, Dougal and Wang, Yu-Xiang and Teh, Yee Whye},
  title = {Understanding the 2016 US Presidential Election using ecological inference and distribution regression with census microdata},
  journal = {Arxiv e-prints},
  note = {ArXiv e-prints: 1611.03787},
  year = {2016},
  month = nov
}
```
K. Palla , F. Caron , Y. W. Teh , A Bayesian nonparametric model for sparse dynamic networks, Jun-2016.
Project: bigbayes

We propose a Bayesian nonparametric prior for time-varying networks. To each node of the network is associated a positive parameter, modeling the sociability of that node. Sociabilities are assumed to evolve over time, and are modeled via a dynamic point process model. The model is able to (a) capture smooth evolution of the interaction between nodes, allowing edges to appear/disappear over time (b) capture long term evolution of the sociabilities of the nodes (c) and yield sparse graphs, where the number of edges grows subquadratically with the number of nodes. The evolution of the sociabilities is described by a tractable time-varying gamma process. We provide some theoretical insights into the model and apply it to three real world datasets.
```
@unpublished{PallaCaronTeh2016,
  author = {Palla, Konstantina and Caron, Francois and Teh, Yee Whye},
  title = {A Bayesian nonparametric model for sparse dynamic networks},
  note = {ArXiv e-prints: 1607.01624},
  archiveprefix = {arXiv},
  year = {2016},
  month = jun
}
```
N. Heard , K. Palla , M. Skoularidou , Topic modelling of authentication events in an enterprise computer network, 2016.
Project: bigbayes

The possibility for theft or misuse of legitimate user credentials is a potential cyber-security weakness in any enterprise computer network which is almost impossible to eradicate. However, by monitoring the network traffic patterns, it can be possible to detect misuse of credentials. This article presents an initial investigation into deconvolving the mixture behaviour of several individuals within a network, to see if individual users can be identified. Towards that, a technique used for document classification is deployed, the Latent Dirichlet allocation model. A pilot study is conducted on authentication events taken from real data from the enterprise network of Los Alamos National Laboratory.
```
@inproceedings{Heard:2016:10.1109/ISI.2016.7745466,
  author = {Heard, NA and Palla, K and Skoularidou, M},
  doi = {10.1109/ISI.2016.7745466},
  publisher = {IEEE},
  title = {Topic modelling of authentication events in an enterprise computer network},
  year = {2016}
}
```
J. Mitrovic , D. Sejdinovic , Y. W. Teh , DR-ABC: Approximate Bayesian Computation with Kernel-Based Distribution Regression, in International Conference on Machine Learning (ICML), 2016, 1482–1491.
Project: bigbayes

Performing exact posterior inference in complex generative models is often difficult or impossible due to an expensive to evaluate or intractable likelihood function. Approximate Bayesian computation (ABC) is an inference framework that constructs an approximation to the true likelihood based on the similarity between the observed and simulated data as measured by a predefined set of summary statistics. Although the choice of appropriate problem-specific summary statistics crucially influences the quality of the likelihood approximation and hence also the quality of the posterior sample in ABC, there are only few principled general-purpose approaches to the selection or construction of such summary statistics. In this paper, we develop a novel framework for this task using kernel-based distribution regression. We model the functional relationship between data distributions and the optimal choice (with respect to a loss function) of summary statistics using kernel-based distribution regression. We show that our approach can be implemented in a computationally and statistically efficient way using the random Fourier features framework for large-scale kernel learning. In addition to that, our framework shows superior performance when compared to related methods on toy and real-world problems.
```
@inproceedings{MitSejTeh2016,
  author = {Mitrovic, J. and Sejdinovic, D. and Teh, Y. W.},
  booktitle = {International Conference on Machine Learning (ICML)},
  pages = {1482--1491},
  title = {{DR-ABC: Approximate Bayesian Computation with Kernel-Based Distribution Regression}},
  year = {2016},
  bdsk-url-1 = {http://jmlr.org/proceedings/papers/v48/mitrovic16.html}
}
```
S. Flaxman , D. Sejdinovic , J. Cunningham , S. Filippi , Bayesian Learning of Kernel Embeddings, in Uncertainty in Artificial Intelligence (UAI), 2016, 182–191.
Project: bigbayes

Kernel methods are one of the mainstays of machine learning, but the problem of kernel learning remains challenging, with only a few heuristics and very little theory. This is of particular importance in methods based on estimation of kernel mean embeddings of probability measures. For characteristic kernels, which include most commonly used ones, the kernel mean embedding uniquely determines its probability measure, so it can be used to design a powerful statistical testing framework, which includes nonparametric two-sample and independence tests. In practice, however, the performance of these tests can be very sensitive to the choice of kernel and its lengthscale parameters. To address this central issue, we propose a new probabilistic model for kernel mean embeddings, the Bayesian Kernel Embedding model, combining a Gaussian process prior over the Reproducing Kernel Hilbert Space containing the mean embedding with a conjugate likelihood function, thus yielding a closed form posterior over the mean embedding. The posterior mean of our model is closely related to recently proposed shrinkage estimators for kernel mean embeddings, while the posterior uncertainty is a new, interesting feature with various possible applications. Critically for the purposes of kernel learning, our model gives a simple, closed form marginal pseudolikelihood of the observed data given the kernel hyperparameters. This marginal pseudolikelihood can either be optimized to inform the hyperparameter choice or fully Bayesian inference can be used.
```
@inproceedings{FlaSejCunFil2016,
  author = {Flaxman, S. and Sejdinovic, D. and Cunningham, J.P. and Filippi, S.},
  booktitle = {Uncertainty in Artificial Intelligence (UAI)},
  pages = {182--191},
  title = {{Bayesian Learning of Kernel Embeddings}},
  year = {2016},
  bdsk-url-1 = {http://www.auai.org/uai2016/proceedings/papers/145.pdf},
  bdsk-url-2 = {http://www.auai.org/uai2016/proceedings/supp/145_supp.pdf}
}
```

T. Fernandez , Y. W. Teh , Posterior Consistency for a Non-parametric Survival Model under a Gaussian Process Prior, 2016.

Project: bigbayes

@unpublished{FerTeh2016a,
  author = {Fernandez, T. and Teh, Y. W.},
  note = {ArXiv e-prints: 1611.02335},
  title = {Posterior Consistency for a Non-parametric Survival Model under a {G}aussian Process Prior},
  year = {2016},
  bdsk-url-1 = {https://arxiv.org/abs/1611.02335},
  bdsk-url-2 = {https://arxiv.org/pdf/1611.02335v1.pdf}
}

T. Fernandez , N. Rivera , Y. W. Teh , Gaussian Processes for Survival Analysis, in Advances in Neural Information Processing Systems (NeurIPS), 2016.
Project: bigbayes

We introduce a semi-parametric Bayesian model for survival analysis. The model is centred on a parametric baseline hazard, and uses a Gaussian process to model variations away from it nonparametrically, as well as dependence on covariates. As opposed to many other methods in survival analysis, our framework does not impose unnecessary constraints in the hazard rate or in the survival function. Furthermore, our model handles left, right and interval censoring mechanisms common in survival analysis. We propose a MCMC algorithm to perform inference and an approximation scheme based on random Fourier features to make computations faster. We report experimental results on synthetic and real data, showing that our model performs better than competing models such as Cox proportional hazards, ANOVA-DDP and random survival forests.
```
@inproceedings{FerRivTeh2016a,
  author = {Fernandez, T. and Rivera, N. and Teh, Y. W.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  title = {Gaussian Processes for Survival Analysis},
  year = {2016},
  bdsk-url-1 = {http://papers.nips.cc/paper/6443-gaussian-processes-for-survival-analysis},
  bdsk-url-2 = {http://papers.nips.cc/paper/6443-gaussian-processes-for-survival-analysis.pdf}
}
```
H. Kim , Y. W. Teh , Scalable Structure Discovery in Regression using Gaussian Processes, in Proceedings of the 2016 Workshop on Automatic Machine Learning, 2016.
Project: bigbayes

Automatic Bayesian Covariance Discovery(ABCD) in Lloyd et. al (2014) provides a framework for automating statistical modelling as well as exploratory data analysis for regression problems. However ABCD does not scale due to its O(N^3) running time. This is undesirable not only because the average size of data sets is growing fast, but also because there is potentially more information in bigger data, implying a greater need for more expressive models that can discover sophisticated structure. We propose a scalable version of ABCD, to encompass big data within the boundaries of automated statistical modelling.
```
@inproceedings{KimTeh2016a,
  author = {Kim, H. and Teh, Y. W.},
  booktitle = {Proceedings of the 2016 Workshop on Automatic Machine Learning},
  title = {Scalable Structure Discovery in Regression using Gaussian Processes},
  year = {2016},
  bdsk-url-1 = {http://www.jmlr.org/proceedings/papers/v64/kim_scalable_2016.html},
  bdsk-url-2 = {http://www.jmlr.org/proceedings/papers/v64/kim_scalable_2016.pdf}
}
```
L. T. Elliott , Y. W. Teh , A Nonparametric HMM for Genetic Imputation and Coalescent Inference, Electronic Journal of Statistics, 2016.
Project: bigbayes

Genetic sequence data are well described by hidden Markov models (HMMs) in which latent states correspond to clusters of similar mutation patterns. Theory from statistical genetics suggests that these HMMs are nonhomogeneous (their transition probabilities vary along the chromosome) and have large support for self transitions. We develop a new nonparametric model of genetic sequence data, based on the hierarchical Dirichlet process, which supports these self transitions and nonhomogeneity. Our model provides a parameterization of the genetic process that is more parsimonious than other more general nonparametric models which have previously been applied to population genetics. We provide truncation-free MCMC inference for our model using a new auxiliary sampling scheme for Bayesian nonparametric HMMs. In a series of experiments on male X chromosome data from the Thousand Genomes Project and also on data simulated from a population bottleneck we show the benefits of our model over the popular finite model fastPHASE, which can itself be seen as a parametric truncation of our model. We find that the number of HMM states found by our model is correlated with the time to the most recent common ancestor in population bottlenecks. This work demonstrates the flexibility of Bayesian nonparametrics applied to large and complex genetic data.
```
@article{EllTeh2016a,
  author = {Elliott, L. T. and Teh, Y. W.},
  journal = {Electronic Journal of Statistics},
  title = {A Nonparametric {HMM} for Genetic Imputation and Coalescent Inference},
  year = {2016}
}
```
S. Favaro , A. Lijoi , C. Nava , B. Nipoti , I. Prüenster , Y. W. Teh , On the Stick-Breaking Representation for Homogeneous NRMIs, Bayesian Analysis, vol. 11, 697–724, 2016.
Project: bigbayes

In this paper, we consider homogeneous normalized random measures with independent increments (hNRMI), a class of nonparametric priors recently introduced in the literature. Many of their distributional properties are known by now but their stick-breaking representation is missing. Here we display such a representation, which will feature dependent stick-breaking weights, and then derive explicit versions for noteworthy special cases of hNRMI. Posterior characterizations are also discussed. Finally, we devise an algorithm for slice sampling mixture models based on hNRMIs, which relies on the representation we have obtained, and implement it to analyze real data.
```
@article{FavLijNav2016a,
  author = {Favaro, S. and Lijoi, A. and Nava, C. and Nipoti, B. and Pr\"uenster, I. and Teh, Y. W.},
  journal = {Bayesian Analysis},
  pages = {697-724},
  title = {On the Stick-Breaking Representation for Homogeneous {NRMIs}},
  volume = {11},
  year = {2016},
  bdsk-url-1 = {http://projecteuclid.org/euclid.ba/1440594949},
  bdsk-url-2 = {http://projecteuclid.org/download/pdfview_1/euclid.ba/1440594949}
}
```

Y. W. Teh , Bayesian Nonparametric Modelling and the Ubiquitous Ewens Sampling Formula, Statistical Science, vol. 31, no. 1, 34–36, 2016.

Project: bigbayes

@article{Teh2016a,
  author = {Teh, Y. W.},
  journal = {Statistical Science},
  number = {1},
  pages = {34-36},
  title = {Bayesian Nonparametric Modelling and the Ubiquitous {Ewens} Sampling Formula},
  volume = {31},
  year = {2016},
  bdsk-url-1 = {http://projecteuclid.org/euclid.ss/1455115911},
  bdsk-url-2 = {http://projecteuclid.org/download/pdfview_1/euclid.ss/1455115911}
}

M. Balog , B. Lakshminarayanan , Z. Ghahramani , D. M. Roy , Y. W. Teh , The Mondrian Kernel, in Uncertainty in Artificial Intelligence (UAI), 2016.
Project: bigbayes

We introduce the Mondrian kernel, a fast random feature approximation to the Laplace kernel. It is suitable for both batch and online learning, and admits a fast kernel-width-selection procedure as the random features can be re-used efficiently for all kernel widths. The features are constructed by sampling trees via a Mondrian process [Roy and Teh, 2009], and we highlight the connection to Mondrian forests [Lakshminarayanan et al., 2014], where trees are also sampled via a Mondrian process, but fit independently. This link provides a new insight into the relationship between kernel methods and random forests.
```
@inproceedings{BalLakGha2016a,
  author = {Balog, M. and Lakshminarayanan, B. and Ghahramani, Z. and Roy, D. M. and Teh, Y. W.},
  booktitle = {Uncertainty in Artificial Intelligence (UAI)},
  title = {The {M}ondrian Kernel},
  year = {2016},
  bdsk-url-1 = {http://auai.org/uai2016/proceedings/papers/236.pdf},
  bdsk-url-2 = {http://auai.org/uai2016/proceedings/supp/236_supp.pdf}
}
```
B. Lakshminarayanan , D. M. Roy , Y. W. Teh , Mondrian Forests for Large-Scale Regression when Uncertainty Matters, in Artificial Intelligence and Statistics (AISTATS), 2016.
Project: bigbayes

Many real-world regression problems demand a measure of the uncertainty associated with each prediction. Standard decision forests deliver efficient state-of-the-art predictive performance, but high-quality uncertainty estimates are lacking. Gaussian processes (GPs) deliver uncertainty estimates, but scaling GPs to large-scale data sets comes at the cost of approximating the uncertainty estimates. We extend Mondrian forests, first proposed by Lakshminarayanan et al. (2014) for classification problems, to the large-scale nonparametric regression setting. Using a novel hierarchical Gaussian prior that dovetails with the Mondrian forest framework, we obtain principled uncertainty estimates, while still retaining the computational advantages of decision forests. Through a combination of illustrative examples, real-world large-scale datasets and Bayesian optimization benchmarks, we demonstrate that Mondrian forests outperform approximate GPs on large-scale regression tasks and deliver better-calibrated uncertainty assessments than decision-forest-based methods.
```
@inproceedings{LakRoyTeh2016a,
  author = {Lakshminarayanan, B. and Roy, D. M. and Teh, Y. W.},
  booktitle = {Artificial Intelligence and Statistics (AISTATS)},
  title = {{M}ondrian Forests for Large-Scale Regression when Uncertainty Matters},
  year = {2016},
  bdsk-url-1 = {http://jmlr.org/proceedings/papers/v51/lakshminarayanan16.html},
  bdsk-url-2 = {http://www.jmlr.org/proceedings/papers/v51/lakshminarayanan16.pdf},
  bdsk-url-3 = {http://jmlr.org/proceedings/papers/v51/lakshminarayanan16-supp.pdf}
}
```
K. Palla , F. Caron , Y. W. Teh , Bayesian Nonparametrics for Sparse Dynamic Networks, 2016.
Project: bigbayes

We propose a Bayesian nonparametric prior for time-varying networks. To each node of the network is associated a positive parameter, modeling the sociability of that node. Sociabilities are assumed to evolve over time, and are modeled via a dynamic point process model. The model is able to (a) capture smooth evolution of the interaction between nodes, allowing edges to appear/disappear over time (b) capture long term evolution of the sociabilities of the nodes (c) and yield sparse graphs, where the number of edges grows subquadratically with the number of nodes. The evolution of the sociabilities is described by a tractable time-varying gamma process. We provide some theoretical insights into the model and apply it to three real world datasets.
```
@unpublished{PalCarTeh2016a,
  author = {Palla, K. and Caron, F. and Teh, Y. W.},
  note = {ArXiv e-prints: 1607.01624},
  title = {Bayesian Nonparametrics for Sparse Dynamic Networks},
  year = {2016},
  bdsk-url-1 = {https://arxiv.org/pdf/1607.01624.pdf}
}
```
M. Battiston , S. Favaro , Y. W. Teh , Multi-armed bandit for species discovery: A Bayesian nonparametric approach, Journal of the American Statistical Association, 2016.
Project: bigbayes

Let (P1,...,PJ) denote J populations of animals from distinct regions. A priori, it is unknown which species are present in each region and what are their corresponding frequencies. Species are shared among populations and each species can be present in more than one region with its frequency varying across populations. In this paper we consider the problem of sequentially sampling these populations in order to observe the greatest number of different species. We adopt a Bayesian nonparametric approach and endow (P1,...,PJ) with a Hierarchical Pitman-Yor process prior. As a consequence of the hierarchical structure, the J unknown discrete probability measures share the same support, that of their common random base measure. Given this prior choice, we propose a sequential rule that, at every time step, given the information available up to that point, selects the population from which to collect the next observation. Rather than picking the population with the highest posterior estimate of producing a new value, the proposed rule includes a Thompson sampling step to better balance the exploration-exploitation trade-off. We also propose an extension of the algorithm to deal with incidence data, where multiple observations are collected in a time period. The performance of the proposed algorithms is assessed through a simulation study and compared to three other strategies. Finally, we compare these algorithms using a dataset of species of trees, collected from different plots in South America.
```
@article{BatFavTeh2016a,
  author = {Battiston, M. and Favaro, S. and Teh, Y. W.},
  doi = {http://dx.doi.org/10.1080/01621459.2016.1261711},
  journal = {Journal of the American Statistical Association},
  title = {Multi-armed bandit for species discovery: A Bayesian nonparametric approach},
  year = {2016},
  bdsk-url-1 = {http://www.tandfonline.com/doi/full/10.1080/01621459.2016.1261711},
  bdsk-url-2 = {http://www.tandfonline.com/doi/pdf/10.1080/01621459.2016.1261711?needAccess=true}
}
```

2015

A. G. Deshwar , L. Boyles , J. Wintersinger , P. C. Boutros , Y. W. Teh , Q. Morris , Abstract B2-59: PhyloSpan: using multimutation reads to resolve subclonal architectures from heterogeneous tumor samples, AACR Special Conference on Computational and Systems Biology of Cancer, vol. 75, 2015.

Project: bigbayes

@article{BouBoyDes2015a,
  author = {Deshwar, A. G. and Boyles, L. and Wintersinger, J. and Boutros, P. C. and Teh, Y. W. and Morris, Q.},
  booktitle = {AACR Special Conference on Computational and Systems Biology of Cancer},
  doi = {10.1158/1538-7445.AM2015-4865},
  journal = {Cancer Research},
  title = {Abstract {B2-59}: {PhyloSpan}: using multimutation reads to resolve subclonal architectures from heterogeneous tumor samples},
  volume = {75},
  year = {2015},
  bdsk-url-1 = {http://cancerres.aacrjournals.org/content/75/22_Supplement_2/B2-59.short},
  bdsk-url-2 = {http://dx.doi.org/10.1158/1538-7445.AM2015-4865}
}

S. Favaro , B. Nipoti , Y. W. Teh , Rediscovery of Good-Turing Estimators via Bayesian Nonparametrics, Biometrics, 2015.
Project: bigbayes

The problem of estimating discovery probabilities originated in the context of statistical ecology, and in recent years it has become popular due to its frequent appearance in challenging applications arising in genetics, bioinformatics, linguistics, designs of experiments, machine learning, etc. A full range of statistical approaches, parametric and nonparametric as well as frequentist and Bayesian, has been proposed for estimating discovery probabilities. In this article, we investigate the relationships between the celebrated Good–Turing approach, which is a frequentist nonparametric approach developed in the 1940s, and a Bayesian nonparametric approach recently introduced in the literature. Specifically, under the assumption of a two parameter Poisson-Dirichlet prior, we show that Bayesian nonparametric estimators of discovery probabilities are asymptotically equivalent, for a large sample size, to suitably smoothed Good–Turing estimators. As a by-product of this result, we introduce and investigate a methodology for deriving exact and asymptotic credible intervals to be associated with the Bayesian nonparametric estimators of discovery probabilities. The proposed methodology is illustrated through a comprehensive simulation study and the analysis of Expressed Sequence Tags data generated by sequencing a benchmark complementary DNA library.
```
@article{FavNipTeh2015b,
  author = {Favaro, S. and Nipoti, B. and Teh, Y. W.},
  doi = {10.1111/biom.12366},
  journal = {Biometrics},
  title = {Rediscovery of {Good-Turing} Estimators via {B}ayesian Nonparametrics},
  year = {2015},
  bdsk-url-1 = {http://onlinelibrary.wiley.com/doi/10.1111/biom.12366/abstract},
  bdsk-url-2 = {http://dx.doi.org/10.1111/biom.12366},
  bdsk-url-3 = {https://arxiv.org/pdf/1401.0303.pdf}
}
```
P. G. Moreno , A. Artés-Rodríguez , Y. W. Teh , F. Perez-Cruz , Bayesian Nonparametric Crowdsourcing, Journal of Machine Learning Research (JMLR), 2015.
Project: bigbayes

Crowdsourcing has been proven to be an effective and efficient tool to annotate large data-sets. User annotations are often noisy, so methods to combine the annotations to produce reliable estimates of the ground truth are necessary. We claim that considering the existence of clusters of users in this combination step can improve the performance. This is especially important in early stages of crowdsourcing implementations, where the number of annotations is low. At this stage there is not enough information to accurately estimate the bias introduced by each annotator separately, so we have to resort to models that consider the statistical links among them. In addition, finding these clusters is interesting in itself as knowing the behavior of the pool of annotators allows implementing efficient active learning strategies. Based on this, we propose in this paper two new fully unsupervised models based on a Chinese restaurant process (CRP) prior and a hierarchical structure that allows inferring these groups jointly with the ground truth and the properties of the users. Efficient inference algorithms based on Gibbs sampling with auxiliary variables are proposed. Finally, we perform experiments, both on synthetic and real databases, to show the advantages of our models over state-of-the-art algorithms.
```
@article{MorArtTeh2015a,
  author = {Moreno, P. G. and Art\'es-Rodr\'iguez, A. and Teh, Y. W. and Perez-Cruz, F.},
  journal = {Journal of Machine Learning Research (JMLR)},
  title = {{B}ayesian Nonparametric Crowdsourcing},
  year = {2015},
  bdsk-url-1 = {http://www.jmlr.org/papers/v16/moreno15a.html},
  bdsk-url-2 = {http://www.jmlr.org/papers/volume16/moreno15a/moreno15a.pdf}
}
```
M. Lomeli , S. Favaro , Y. W. Teh , A hybrid sampler for Poisson-Kingman mixture models, in Advances in Neural Information Processing Systems (NeurIPS), 2015.
Project: bigbayes

This paper concerns the introduction of a new Markov Chain Monte Carlo scheme for posterior sampling in Bayesian nonparametric mixture models with priors that belong to the general Poisson-Kingman class. We present a novel and compact way of representing the infinite dimensional component of the model such that while explicitly representing this infinite component it has less memory and storage requirements than previous MCMC schemes. We describe comparative simulation results demonstrating the efficacy of the proposed MCMC algorithm against existing marginal and conditional MCMC samplers.
```
@inproceedings{LomFavTeh2015a,
  author = {Lomeli, M. and Favaro, S. and Teh, Y. W.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  title = {A hybrid sampler for {Poisson-Kingman} mixture models},
  year = {2015},
  bdsk-url-1 = {http://papers.nips.cc/paper/5799-a-hybrid-sampler-for-poisson-kingman-mixture-models},
  bdsk-url-2 = {http://papers.nips.cc/paper/5799-a-hybrid-sampler-for-poisson-kingman-mixture-models.pdf},
  bdsk-url-3 = {http://papers.nips.cc/paper/5799-a-hybrid-sampler-for-poisson-kingman-mixture-models-supplemental.zip}
}
```
M. De Iorio , S. Favaro , Y. W. Teh , Bayesian Inference on Population Structure: From Parametric to Nonparametric Modeling, in Nonparametric Bayesian Inference in Biostatistics, Springer, 2015.
Project: bigbayes

Making inference on population structure from genotype data requires to identify the actual subpopulations and assign individuals to these populations. The source populations are assumed to be in Hardy-Weinberg equilibrium, but the allelic frequencies of these populations and even the number of populations present in a sample are unknown. In this chapter we present a review of some Bayesian parametric and nonparametric models for making inference on population structure, with emphasis on model-based clustering methods. Our aim is to show how recent developments in Bayesian nonparametrics have been usefully exploited in order to introduce natural nonparametric counterparts of some of the most celebrated parametric approaches for inferring population structure. We use data from the 1000 Genomes project (http://www.1000genomes.org/) to provide a brief illustration of some of these nonparametric approaches.
```
@incollection{De-FavTeh2015a,
  author = {{De Iorio}, M. and Favaro, S. and Teh, Y. W.},
  booktitle = {Nonparametric {B}ayesian Inference in Biostatistics},
  doi = {10.1007/978-3-319-19518-6_7},
  publisher = {Springer},
  title = {{B}ayesian Inference on Population Structure: From Parametric to Nonparametric Modeling},
  year = {2015},
  bdsk-url-1 = {http://link.springer.com/chapter/10.1007/978-3-319-19518-6_7},
  bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-319-19518-6_7}
}
```
S. Favaro , B. Nipoti , Y. W. Teh , Random variate generation for Laguerre-type exponentially tilted α-stable distributions, Electronic Journal of Statistics, vol. 9, 1230–1242, 2015.
Project: bigbayes

Exact sampling methods have been recently developed for generating random variates for exponentially tilted α-stable distributions. In this paper we show how to generate, exactly, random variates for a more general class of tilted α-stable distributions, which is referred to as the class of Laguerre-type exponentially tilted α-stable distributions. Beside the exponentially tilted α-stable distribution, such a class includes also the Erlang tilted α-stable distribution. This is a special case of the so-called gamma tilted α-stable distribution, for which an efficient exact random variate generator is currently not available in the literature. Our result fills this gap.
```
@article{FavNipTeh2015a,
  author = {Favaro, S. and Nipoti, B. and Teh, Y. W.},
  doi = {10.1214/15-EJS1033},
  journal = {Electronic Journal of Statistics},
  pages = {1230-1242},
  title = {Random variate generation for {L}aguerre-type exponentially tilted {$\alpha$}-stable distributions},
  volume = {9},
  year = {2015},
  bdsk-url-1 = {http://projecteuclid.org/euclid.ejs/1433982944},
  bdsk-url-2 = {http://dx.doi.org/10.1214/15-EJS1033},
  bdsk-url-3 = {http://projecteuclid.org/download/pdfview_1/euclid.ejs/1433982944}
}
```
M. Balog , Y. W. Teh , The Mondrian Process for Machine Learning, 2015.
Project: bigbayes

This report is concerned with the Mondrian process and its applications in machine learning. The Mondrian process is a guillotine-partition-valued stochastic process that possesses an elegant self-consistency property. The first part of the report uses simple concepts from applied probability to define the Mondrian process and explore its properties. The Mondrian process has been used as the main building block of a clever online random forest classification algorithm that turns out to be equivalent to its batch counterpart. We outline a slight adaptation of this algorithm to regression, as the remainder of the report uses regression as a case study of how Mondrian processes can be utilized in machine learning. In particular, the Mondrian process will be used to construct a fast approximation to the computationally expensive kernel ridge regression problem with a Laplace kernel. The complexity of random guillotine partitions generated by a Mondrian process and hence the complexity of the resulting regression models is controlled by a lifetime hyperparameter. It turns out that these models can be efficiently trained and evaluated for all lifetimes in a given range at once, without needing to retrain them from scratch for each lifetime value. This leads to an efficient procedure for determining the right model complexity for a dataset at hand. The limitation of having a single lifetime hyperparameter will motivate the final Mondrian grid model, in which each input dimension is endowed with its own lifetime parameter. In this model we preserve the property that its hyperparameters can be tweaked without needing to retrain the modified model from scratch.
```
@unpublished{BalTeh2015a,
  author = {Balog, M. and Teh, Y. W.},
  note = {ArXiv e-prints: 1507.05181},
  title = {The {M}ondrian Process for Machine Learning},
  year = {2015},
  bdsk-url-1 = {https://arxiv.org/pdf/1507.05181.pdf}
}
```
P. Orbanz , L. James , Y. W. Teh , Scaled subordinators and generalizations of the Indian buffet process, 2015.
Project: bigbayes

We study random families of subsets of ℕ that are similar to exchangeable random partitions, but do not require constituent sets to be disjoint: Each element of ℕ may be contained in multiple subsets. One class of such objects, known as Indian buffet processes, has become a popular tool in machine learning. Based on an equivalence between Indian buffet and scale-invariant Poisson processes, we identify a random scaling variable whose role is similar to that played in exchangeable partition models by the total mass of a random measure. Analogous to the construction of exchangeable partitions from normalized subordinators, random families of sets can be constructed from randomly scaled subordinators. Coupling to a heavy-tailed scaling variable induces a power law on the number of sets containing the first n elements. Several examples, with properties desirable in applications, are derived explicitly. A relationship to exchangeable partitions is made precise as a correspondence between scaled subordinators and Poisson-Kingman measures, generalizing a result of Arratia, Barbour and Tavare on scale-invariant processes.
```
@unpublished{OrbJamTeh2015a,
  author = {Orbanz, P. and James, L. and Teh, Y. W.},
  note = {ArXiv e-prints: 1510.07309},
  title = {Scaled subordinators and generalizations of the {I}ndian buffet process},
  year = {2015},
  bdsk-url-1 = {https://arxiv.org/pdf/1510.07309v1.pdf}
}
```
M. De Iorio , L. Elliott , S. Favaro , Y. W. Teh , Bayesian Nonparametric Inference of Population Admixtures, 2015.
Project: bigbayes

We propose a Bayesian nonparametric model to infer population admixture, extending the Hierarchical Dirichlet Process to allow for correlation between loci due to Linkage Disequilibrium. Given multilocus genotype data from a sample of individuals, the model allows inferring classifying individuals as unadmixed or admixed, inferring the number of subpopulations ancestral to an admixed population and the population of origin of chromosomal regions. Our model does not assume any specific mutation process and can be applied to most of the commonly used genetic markers. We present a MCMC algorithm to perform posterior inference from the model and discuss methods to summarise the MCMC output for the analysis of population admixture. We demonstrate the performance of the proposed model in simulations and in a real application, using genetic data from the EDAR gene, which is considered to be ancestry-informative due to well-known variations in allele frequency as well as phenotypic effects across ancestry. The structure analysis of this dataset leads to the identification of a rare haplotype in Europeans.
```
@unpublished{De-EllFav2015a,
  author = {{De Iorio}, M. and Elliott, L. and Favaro, S. and Teh, Y. W.},
  note = {ArXiv e-prints: 1503.08278},
  title = {{B}ayesian Nonparametric Inference of Population Admixtures},
  year = {2015},
  bdsk-url-1 = {https://arxiv.org/pdf/1503.08278v1.pdf}
}
```
B. Lakshminarayanan , D. M. Roy , Y. W. Teh , Particle Gibbs for Bayesian Additive Regression Trees, in Proceedings of the International Conference on Artificial Intelligence and Statistics, 2015.
Project: bigbayes sgmcmc

Additive regression trees are flexible non-parametric models and popular off-the-shelf tools for real-world non-linear regression. In application domains, such as bioinformatics, where there is also demand for probabilistic predictions with measures of uncertainty, the Bayesian additive regression trees (BART) model, introduced by Chipman et al. (2010), is increasingly popular. As data sets have grown in size, however, the standard Metropolis–Hastings algorithms used to per- form inference in BART are proving inadequate. In particular, these Markov chains make local changes to the trees and suffer from slow mixing when the data are high- dimensional or the best-fitting trees are more than a few layers deep. We present a novel sampler for BART based on the Particle Gibbs (PG) algorithm (Andrieu et al., 2010) and a top-down particle filtering algorithm for Bayesian decision trees (Lakshminarayanan et al., 2013). Rather than making local changes to individual trees, the PG sampler proposes a complete tree to fit the residual. Experiments show that the PG sampler outperforms existing samplers in many settings.
```
@inproceedings{LakRoyTeh2015b,
  author = {Lakshminarayanan, B. and Roy, D. M. and Teh, Y. W.},
  booktitle = {Proceedings of the International Conference on Artificial Intelligence and Statistics},
  title = {Particle {G}ibbs for {B}ayesian Additive Regression Trees},
  year = {2015},
  bdsk-url-1 = {http://jmlr.org/proceedings/papers/v38/lakshminarayanan15.html},
  bdsk-url-2 = {http://jmlr.org/proceedings/papers/v38/lakshminarayanan15.pdf},
  bdsk-url-3 = {http://jmlr.org/proceedings/papers/v38/lakshminarayanan15-supp.pdf}
}
```

2014

S. Favaro , M. Lomeli , Y. W. Teh , On a Class of σ-stable Poisson-Kingman Models and an Effective Marginalized Sampler, Statistics and Computing, 2014.
Project: bigbayes

We investigate the use of a large class of discrete random probability measures, which is referred to as the class Q, in the context of Bayesian nonparametric mixture modeling. The class Q encompasses both the the two-parameter Poisson–Dirichlet process and the normalized generalized Gamma process, thus allowing us to comparatively study the inferential advantages of these two well-known nonparametric priors. Apart from a highly flexible parameterization, the distinguishing feature of the class Q is the availability of a tractable posterior distribution. This feature, in turn, leads to derive an efficient marginal MCMC algorithm for posterior sampling within the framework of mixture models. We demonstrate the efficacy of our modeling framework on both one-dimensional and multi-dimensional datasets.
```
@article{FavLomTeh2014a,
  author = {Favaro, S. and Lomeli, M. and Teh, Y. W.},
  doi = {10.1007/s11222-014-9499-4},
  journal = {Statistics and Computing},
  title = {On a Class of {$\sigma$}-stable {P}oisson-{K}ingman Models and an Effective Marginalized Sampler},
  year = {2014},
  bdsk-url-1 = {http://projecteuclid.org/euclid.ejs/1433982944},
  bdsk-url-2 = {http://dx.doi.org/10.1007/s11222-014-9499-4}
}
```
S. Favaro , M. Lomeli , B. Nipoti , Y. W. Teh , On the Stick-Breaking Representation of σ-stable Poisson-Kingman Models, Electronic Journal of Statistics, vol. 8, 1063–1085, 2014.
Project: bigbayes

In this paper we investigate the stick-breaking representation for the class of σ-stable Poisson-Kingman models, also known as Gibbs-type random probability measures. This class includes as special cases most of the discrete priors commonly used in Bayesian nonparametrics, such as the two parameter Poisson-Dirichlet process and the normalized generalized Gamma process. Under the assumption σ=u/v, for any coprime integers 1≤u<v such that u/v≤1/2, we show that a σ-stable Poisson-Kingman model admits an explicit stick-breaking representation in terms of random variables which are obtained by suitably transforming Gamma random variables and products of independent Beta and Gamma random variables.
```
@article{FavLomNip2014a,
  author = {Favaro, S. and Lomeli, M. and Nipoti, B. and Teh, Y. W.},
  doi = {10.1214/14-EJS921},
  journal = {Electronic Journal of Statistics},
  pages = {1063-1085},
  title = {On the Stick-Breaking Representation of {$\sigma$}-stable {P}oisson-{K}ingman Models},
  volume = {8},
  year = {2014},
  bdsk-url-1 = {http://projecteuclid.org/euclid.ejs/1407243242},
  bdsk-url-2 = {http://dx.doi.org/10.1214/14-EJS921},
  bdsk-url-3 = {http://projecteuclid.org/download/pdfview_1/euclid.ejs/1407243242}
}
```

B. Lakshminarayanan , D. Roy , Y. W. Teh , Mondrian Forests: Efficient Online Random Forests, in Advances in Neural Information Processing Systems (NeurIPS), 2014.

Project: bigbayes

@inproceedings{LakRoyTeh2014a,
  author = {Lakshminarayanan, B. and Roy, D. and Teh, Y. W.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  title = {{M}ondrian Forests: Efficient Online Random Forests},
  year = {2014},
  bdsk-url-1 = {http://papers.nips.cc/paper/5234-mondrian-forests-efficient-online-random-forests},
  bdsk-url-2 = {http://papers.nips.cc/paper/5234-mondrian-forests-efficient-online-random-forests.pdf},
  bdsk-url-3 = {http://papers.nips.cc/paper/5234-mondrian-forests-efficient-online-random-forests-supplemental.zip},
  bdsk-url-4 = {https://github.com/balajiln/mondrianforest}
}

Software

2019

Y. Zhou , B. Gram-Hansen , T. Kohn , T. Rainforth , H. Yang , F. Wood , A Low-Level Probabilistic Programming Language for Non-Differentiable Models, International Conference on Artificial Intelligence and Statistics (AISTATS). 2019.
Project: bigbayes

We develop a new Low-level, First-order Prob- abilistic Programming Language (LF-PPL) suited for models containing a mix of contin- uous, discrete, and/or piecewise-continuous variables. The key success of this language and its compilation scheme, is in its ability to automatically distinguish the discontinuous and continuous parameters in the density func- tion, while further providing runtime checks of when discontinuity boundaries have been crossed. This enables the introduction of new inference engines that are able to exploit gra- dient information, while remaining efficient for models which are not everywhere differen- tiable. We demonstrate this ability by intro- ducing a discontinuous Hamiltonian Monte Carlo (DHMC) inference engine that is able to deliver automated and efficient inference for non-differentiable models. Our system is backed up by a mathematical formalism that ensures that any model expressed in this lan- guage has a density with a sufficiently low measure of discontinuities to maintain the validity of the inference engine.
```
@software{zhou2018lfpplb,
  title = {{A Low-Level Probabilistic Programming Language for Non-Differentiable Models}},
  author = {Zhou, Yuan and Gram-Hansen, Bradley and Kohn, Tobias and Rainforth, Tom and Yang, Hongseok and Wood, Frank},
  booktitle = {International Conference on Artificial Intelligence and Statistics (AISTATS)},
  year = {2019},
  bdsk-url-1 = {https://github.com/bradleygramhansen/PyLFPPL}
}
```

2017

S. Flaxman , M. Chirico , P. Pereira , C. Loeffler , Forecasting Crime in Portland. 2017.

Project: bigbayes

@software{FlaScal2017,
  author = {Flaxman, Seth and Chirico, Michael and Pereira, Pau and Loeffler, Charles},
  title = {Forecasting Crime in Portland},
  year = {2017}
}

S. Flaxman , Y. W. Teh , D. Sejdinovic , Kernel Poisson. 2017.

Project: bigbayes

@software{FlaTehSej2017b,
  author = {Flaxman, Seth and Teh, Yee Whye and Sejdinovic, Dino},
  title = {Kernel {P}oisson},
  year = {2017}
}

A. Todeschini , X. Miscouridou , F. Caron , SNetOC. 2017.

Project: bigbayes

@software{SNetOC2017,
  author = {Todeschini, Adrien and Miscouridou, Xenia and Caron, Fran\c ois},
  title = {SNetOC},
  year = {2017},
  bdsk-url-1 = { https://github.com/misxenia/SNetOC}
}

2016

V. Perrone , P. A. Jenkins , D. Spano , Y. W. Teh , NIPS 1987-2015 dataset. 2016.

Project: bigbayes

@software{PerJenSpa2016b,
  author = {Perrone, V. and Jenkins, P. A. and Spano, D. and Teh, Y. W.},
  title = {NIPS 1987-2015 dataset},
  year = {2016},
  bdsk-url-1 = {https://archive.ics.uci.edu/ml/datasets/NIPS+Conference+Papers+1987-2015}
}

B. Lakshminarayanan , D. M. Roy , Y. W. Teh , Mondrian Forest. 2016.

Project: bigbayes

@software{LakRoyTeh2016b,
  author = {Lakshminarayanan, B. and Roy, D. M. and Teh, Y. W.},
  title = {{M}ondrian Forest},
  year = {2016},
  bdsk-url-1 = {https://github.com/balajiln/mondrianforest}
}

L. Elliott , Y. W. Teh , BNPPhase. 2016.

Project: bigbayes

@software{EllTeh2016b,
  author = {Elliott, L. and Teh, Y. W.},
  title = {BNPPhase},
  year = {2016},
  bdsk-url-1 = {https://github.com/BigBayes/BNPPhase}
}

2015

B. Lakshminarayanan , D. M. Roy , Y. W. Teh , PGBart. 2015.

Project: bigbayes sgmcmc

@software{LakRoyTeh2015a,
  author = {Lakshminarayanan, B. and Roy, D. M. and Teh, Y. W.},
  title = {PGBart},
  year = {2015},
  bdsk-url-1 = {https://github.com/balajiln/pgbart}
}

M. De Iorio , L. T. Elliott , S. Favaro , Y. W. Teh , HDPStructure. 2015.

Project: bigbayes

@software{De-EllFav2015b,
  author = {{De Iorio}, M. and Elliott, L. T. and Favaro, S. and Teh, Y. W.},
  title = {HDPStructure},
  year = {2015},
  bdsk-url-1 = {https://github.com/BigBayes/HDPStructure}
}

L. Boyles , Y. W. Teh , CPABS: Cancer Phylogenetic Reconstruction with Aldous’ Beta Splitting. 2015.

Project: bigbayes

@software{BoyTeh2015a,
  author = {Boyles, L. and Teh, Y. W.},
  title = {{CPABS}: Cancer Phylogenetic Reconstruction with {A}ldous' Beta Splitting},
  year = {2015},
  bdsk-url-1 = {https://github.com/BigBayes/CPABS}
}