Quantile of a random variable

Let ${\displaystyle Y}$ be a real-valued random variable with cumulative distribution function ${\displaystyle F_{Y}(y)=P(Y\leq y)}$. The ${\displaystyle \tau }$th quantile of Y is given by

${\displaystyle q_{Y}(\tau )=F_{Y}^{-1}(\tau )=\inf \left\{y:F_{Y}(y)\geq \tau \right\}}$

where ${\displaystyle \tau \in (0,1).}$

Define the loss function as ${\displaystyle \rho _{\tau }(m)=m(\tau -\mathbb {I} _{(m<0)})}$, where ${\displaystyle \mathbb {I} }$ is an indicator function. A specific quantile can be found by minimizing the expected loss of ${\displaystyle Y-u}$ with respect to ${\displaystyle u}$:^[1](pp. 5–6):

${\displaystyle q_{Y}(\tau )={\underset {u}{\mbox{arg min}}}E(\rho _{\tau }(Y-u))={\underset {u}{\mbox{arg min}}}{\biggl \{}(\tau -1)\int _{-\infty }^{u}(y-u)dF_{Y}(y)+\tau \int _{u}^{\infty }(y-u)dF_{Y}(y){\biggr \}}.}$

This can be shown by computing the derivative of the expected loss with respect to ${\displaystyle u}$ via an application of the Leibniz integral rule, setting it to 0, and letting ${\displaystyle q_{\tau }}$ be the solution of

${\displaystyle 0=(1-\tau )\int _{-\infty }^{q_{\tau }}dF_{Y}(y)-\tau \int _{q_{\tau }}^{\infty }dF_{Y}(y).}$

This equation reduces to

${\displaystyle 0=F_{Y}(q_{\tau })-\tau ,}$

and then to

${\displaystyle F_{Y}(q_{\tau })=\tau .}$

If the solution ${\displaystyle q_{\tau }}$ is not unique, then we have to take the smallest such solution to obtain the ${\displaystyle \tau }$th quantile of the random variable Y.

Sample quantile

The ${\displaystyle \tau }$ sample quantile can be obtained by using an importance sampling estimate and solving the following minimization problem

${\displaystyle {\hat {q}}_{\tau }={\underset {q\in \mathbb {R} }{\mbox{arg min}}}\sum _{i=1}^{n}\rho _{\tau }(y_{i}-q),}$

${\displaystyle ={\underset {q\in \mathbb {R} }{\mbox{arg min}}}\left[(1-\tau )\sum _{y_{i}<q}(y_{i}-q)+\tau \sum _{y_{i}\geq q}(y_{i}-q)\right]}$,

where the function ${\displaystyle \rho _{\tau }}$ is the tilted absolute value function. The intuition is the same as for the population quantile.

Scale equivariance

For any ${\displaystyle a>0}$ and ${\displaystyle \tau \in [0,1]}$

${\displaystyle {\hat {\beta }}(\tau ;aY,X)=a{\hat {\beta }}(\tau ;Y,X),}$

${\displaystyle {\hat {\beta }}(\tau$ ;-aY,X)=-a{\hat {\beta }}(1-\tau ;Y,X).}

Shift equivariance

For any ${\displaystyle \gamma \in R^{k}}$ and ${\displaystyle \tau \in [0,1]}$

${\displaystyle {\hat {\beta }}(\tau ;Y+X\gamma ,X)={\hat {\beta }}(\tau ;Y,X)+\gamma .}$

Equivariance to reparameterization of design

Let ${\displaystyle A}$ be any ${\displaystyle p\times p}$ nonsingular matrix and ${\displaystyle \tau \in [0,1]}$

${\displaystyle {\hat {\beta }}(\tau ;Y,XA)=A^{-1}{\hat {\beta }}(\tau ;Y,X).}$

Invariance to monotone transformations

If ${\displaystyle h}$ is a nondecreasing function on ${\displaystyle \mathbb {R} }$, the following invariance property applies:

${\displaystyle h(Q_{Y|X}(\tau ))\equiv Q_{h(Y)|X}(\tau ).}$

Example (1):

If ${\displaystyle W=\exp(Y)}$ and ${\displaystyle Q_{Y|X}(\tau )=X\beta _{\tau }}$, then ${\displaystyle Q_{W|X}(\tau )=\exp(X\beta _{\tau })}$. The mean regression does not have the same property since ${\displaystyle \operatorname {E} (\ln(Y))\neq \ln(\operatorname {E} (Y)).}$

Interpretation of the slope parameters

The linear model ${\displaystyle Q_{Y|X}(\tau )=X\beta _{\tau }}$ mis-specifies the true systematic relation ${\displaystyle Q_{Y|X}(\tau )=f(X,\tau )}$ when ${\displaystyle f(\cdot ,\tau )}$ is nonlinear. However, ${\displaystyle Q_{Y|X}(\tau )=X\beta _{\tau }}$ minimizes a weighted distanced to ${\displaystyle f(X,\tau )}$ among linear models.^[10] Furthermore, the slope parameters ${\displaystyle \beta _{\tau }}$ of the linear model can be interpreted as weighted averages of the derivatives ${\displaystyle \nabla f(X,\tau )}$ so that ${\displaystyle \beta _{\tau }}$ can be used for causal inference.^[11] Specifically, the hypothesis ${\displaystyle H_{0}:\nabla f(x,\tau )=0}$ for all ${\displaystyle x}$ implies the hypothesis ${\displaystyle H_{0}:\beta _{\tau }=0}$, which can be tested using the estimator ${\displaystyle {\hat {\beta _{\tau }}}}$ and its limit distribution.

Goodness of fit

The goodness of fit for quantile regression for the ${\displaystyle \tau }$ quantile can be defined as:^[12] ${\displaystyle R^{2}(\tau )=1-{\frac {{\hat {V}}_{\tau }}{{\tilde {V}}_{\tau }}},}$ where ${\displaystyle {\hat {V}}_{\tau }}$ is the sum of squares of the conditional quantile, while ${\displaystyle {\tilde {V}}_{\tau }}$ is the sum of squares of the unconditional quantile.

Bayesian methods for quantile regression

Because quantile regression does not normally assume a parametric likelihood for the conditional distributions of Y|X, the Bayesian methods work with a working likelihood. A convenient choice is the asymmetric Laplacian likelihood,^[13] because the mode of the resulting posterior under a flat prior is the usual quantile regression estimates. The posterior inference, however, must be interpreted with care. Yang, Wang and He^[14] provided a posterior variance adjustment for valid inference. In addition, Yang and He^[15] showed that one can have asymptotically valid posterior inference if the working likelihood is chosen to be the empirical likelihood.

Machine learning methods for quantile regression

Beyond simple linear regression, there are several machine learning methods that can be extended to quantile regression. A switch from the squared error to the tilted absolute value loss function (a.k.a. the pinball loss^[16]) allows gradient descent-based learning algorithms to learn a specified quantile instead of the mean. It means that we can apply all neural network and deep learning algorithms to quantile regression,^[17][18] which is then referred to as nonparametric quantile regression.^[19] Tree-based learning algorithms are also available for quantile regression (see, e.g., Quantile Regression Forests,^[20] as a simple generalization of Random Forests).

Censored quantile regression

If the response variable is subject to censoring, the conditional mean is not identifiable without additional distributional assumptions, but the conditional quantile is often identifiable. For recent work on censored quantile regression, see: Portnoy^[21] and Wang and Wang^[22]

Example (2):

Let ${\displaystyle Y^{c}=\max(0,Y)}$ and ${\displaystyle Q_{Y|X}=X\beta _{\tau }}$. Then ${\displaystyle Q_{Y^{c}|X}(\tau )=\max(0,X\beta _{\tau })}$. This is the censored quantile regression model: estimated values can be obtained without making any distributional assumptions, but at the cost of computational difficulty,^[23] some of which can be avoided by using a simple three step censored quantile regression procedure as an approximation.^[24]

For random censoring on the response variables, the censored quantile regression of Portnoy (2003)^[21] provides consistent estimates of all identifiable quantile functions based on reweighting each censored point appropriately.

Censored quantile regression has close links to survival analysis.

Heteroscedastic errors

The quantile regression loss needs to be adapted in the presence of heteroscedastic errors in order to be efficient.^[25]