I was a DPhil student supervised by Dino Sejdinovic. My primary interest is in scalable kernel method with Random Fourier Features and Nystrom Method with their theoretical properties. Currently I am trying to understand the possible connections between RFF with deep neural network along with their generalization and optimization property. I also did some research in fair learning.
Publications
2021
Z. Li
,
J. Ton
,
D. Oglic
,
D. Sejdinovic
,
Towards A Unified Analysis of Random Fourier Features, Journal of Machine Learning Research (JMLR), vol. 22, no. 108, 1–51, 2021.
Random Fourier features is a widely used, simple, and effective technique for scaling up kernel methods. The existing theoretical analysis of the approach, however, remains focused on specific learning tasks and typically gives pessimistic bounds which are at odds with the empirical results. We tackle these problems and provide the first unified risk analysis of learning with random Fourier features using the squared error and Lipschitz continuous loss functions. In our bounds, the trade-off between the computational cost and the learning risk convergence rate is problem specific and expressed in terms of the regularization parameter and the number of effective degrees of freedom. We study both the standard random Fourier features method for which we improve the existing bounds on the number of features required to guarantee the corresponding minimax risk convergence rate of kernel ridge regression, as well as a data-dependent modification which samples features proportional to ridge leverage scores and further reduces the required number of features. As ridge leverage scores are expensive to compute, we devise a simple approximation scheme which provably reduces the computational cost without loss of statistical efficiency. Our empirical results illustrate the effectiveness of the proposed scheme relative to the standard random Fourier features method.
@article{LiTonOglSej2021,
author = {Li, Zhu and Ton, Jean-Francois and Oglic, Dino and Sejdinovic, Dino},
title = {{{Towards A Unified Analysis of Random Fourier Features}}},
journal = {Journal of Machine Learning Research (JMLR)},
volume = {22},
number = {108},
year = {2021},
pages = {1--51}
}
2019
Z. Li
,
A. Perez-Suay
,
G. Camps-Valls
,
D. Sejdinovic
,
Kernel Dependence Regularizers and Gaussian Processes with Applications to Algorithmic Fairness, ArXiv e-prints:1911.04322, 2019.
Current adoption of machine learning in industrial, societal and economical activities has raised concerns about the fairness, equity and ethics of automated decisions. Predictive models are often developed using biased datasets and thus retain or even exacerbate biases in their decisions and recommendations. Removing the sensitive covariates, such as gender or race, is insufficient to remedy this issue since the biases may be retained due to other related covariates. We present a regularization approach to this problem that trades off predictive accuracy of the learned models (with respect to biased labels) for the fairness in terms of statistical parity, i.e. independence of the decisions from the sensitive covariates. In particular, we consider a general framework of regularized empirical risk minimization over reproducing kernel Hilbert spaces and impose an additional regularizer of dependence between predictors and sensitive covariates using kernel-based measures of dependence, namely the Hilbert-Schmidt Independence Criterion (HSIC) and its normalized version. This approach leads to a closed-form solution in the case of squared loss, i.e. ridge regression. Moreover, we show that the dependence regularizer has an interpretation as modifying the corresponding Gaussian process (GP) prior. As a consequence, a GP model with a prior that encourages fairness to sensitive variables can be derived, allowing principled hyperparameter selection and studying of the relative relevance of covariates under fairness constraints. Experimental results in synthetic examples and in real problems of income and crime prediction illustrate the potential of the approach to improve fairness of automated decisions.
@unpublished{LiPerCamSej2019,
author = {Li, Zhu and Perez-Suay, Adrian and Camps-Valls, Gustau and Sejdinovic, Dino},
title = {{{Kernel Dependence Regularizers and Gaussian Processes with Applications to Algorithmic Fairness}}},
journal = {ArXiv e-prints:1911.04322},
year = {2019}
}
Z. Li
,
J. Ton
,
D. Oglic
,
D. Sejdinovic
,
Towards A Unified Analysis of Random Fourier Features, in International Conference on Machine Learning (ICML), 2019, PMLR 97:3905–3914.
Random Fourier features is a widely used, simple, and effective technique for scaling up kernel methods. The existing theoretical analysis of the approach, however, remains focused on specific learning tasks and typically gives pessimistic bounds which are at odds with the empirical results. We tackle these problems and provide the first unified risk analysis of learning with random Fourier features using the squared error and Lipschitz continuous loss functions. In our bounds, the trade-off between the computational cost and the expected risk convergence rate is problem specific and expressed in terms of the regularization parameter and the \emphnumber of effective degrees of freedom. We study both the standard random Fourier features method for which we improve the existing bounds on the number of features required to guarantee the corresponding minimax risk convergence rate of kernel ridge regression, as well as a data-dependent modification which samples features proportional to \emphridge leverage scores and further reduces the required number of features. As ridge leverage scores are expensive to compute, we devise a simple approximation scheme which provably reduces the computational cost without loss of statistical efficiency.
@inproceedings{LiTonOglSej2019,
author = {Li, Z. and Ton, J.-F. and Oglic, D. and Sejdinovic, D.},
title = {{{Towards A Unified Analysis of Random Fourier Features}}},
booktitle = {International Conference on Machine Learning (ICML)},
pages = {PMLR 97:3905-3914},
year = {2019}
}
