Skip to main content

Nonparametric Statistics

by Eduardo García-Portugués
Last built on 2026-02-23, v7.0.0

Welcome

Nonparametric Statistics Welcome to Nonparametric Statistics! This book provides an introduction to the mathematical theory and computational practice of several nonparametric methods, i.e., statistical techniques that make minimal assumptions about the underlying data-generating process.

Rather than committing to a fully parametric model that may be inadequate to describe reality, nonparametric methods aim to “let the data speak” and reveal structure with minimal assumptions, providing flexibility and robustness that parametric approaches cannot match. This makes them indispensable in modern statistical practice: from density and regression estimation, where they capture features that no single parametric family can describe, to hypothesis testing, where they yield conclusions free from potentially misspecified distributional assumptions and allow for the validation of parametric models through goodness-of-fit tests.

The book strikes a balance between mathematical rigor and practical implementation. Each chapter develops the theoretical foundations, including asymptotic properties and technical details, while providing extensive R code examples with simulated and real data. Interactive Shiny applications complement the exposition, allowing the reader to visualize key concepts and develop intuition for how methods behave in practice. The book is designed to be relatively self-contained, providing the necessary probabilistic, statistical, and analytical tools, particularly through the introduction and appendices.

A collection of more than 130 exercises is included to help the reader test their understanding and apply the methods. These exercises vary in difficulty and type, including theoretical problems, data analysis tasks, and programming exercises. The reader should close each chapter with a clear understanding of why and how the methods work, together with the practical ability to deploy them.

Scope of the book

The central focus of the book is kernel smoothing methods for density and regression estimation, as well as classical nonparametric hypothesis tests for one and two samples. Recurring themes are the asymptotic properties of estimators and tests, and the use of resampling methods. A distinctive feature of the book is its substantial coverage of multivariate kernel methods and their connections to classical multivariate analysis. Kernel density estimation leads to principled nonparametric alternatives to techniques traditionally reliant on normality: mean shift clustering as a density-mode-based approach with a clean population interpretation; kernel discriminant analysis as a generalization of LDA and QDA; density ridges as a nonparametric counterpart to principal components; and level set estimation for outlier detection without Mahalanobis-distance assumptions. The regression treatment extends to mixed-type predictors via Aitchison–Aitken and Li–Racine kernels, and to local likelihood methods for non-Gaussian responses. The hypothesis testing chapter develops distance correlation as a characterization of independence, bootstrap goodness-of-fit testing for general parametric families, and a nuanced re-examination of classical tests.

The book does not cover every topic in nonparametric statistics. Other important nonparametric regression techniques, such as spline smoothing, wavelets, and random forests, are beyond the scope of this text. Similarly, the treatment of hypothesis testing is heavily focused on tests based on the empirical distribution function and does not cover chi-squared-type tests. Nevertheless, the book provides a solid foundation for further exploration of the wide realm of nonparametric statistics.

The book begins with Chapter 1, Introduction, which reviews the essential probabilistic and analytical tools needed throughout. This includes probability fundamentals, some useful distributional facts, key stochastic convergence results, and analytical tools such as Taylor’s theorem. A less standard part of this chapter is the \(O_\mathbb{P}\) and \(o_\mathbb{P}\) notations, together with their intuition and uses. The chapter concludes by motivating why nonparametric statistics matter and when they offer advantages over parametric alternatives.

Chapter 2, Kernel density estimation I, introduces the fundamental problem of estimating a univariate probability density function \(f\) from data. This is a more challenging problem than estimating the cumulative distribution function, requiring some type of smoothing. Starting from the familiar histogram, we motivate and develop the kernel density estimator as a natural smoothed successor. The chapter examines the bias–variance tradeoff, derives the asymptotically optimal bandwidth, and presents practical bandwidth selection methods including rule-of-thumb, plug-in, and cross-validation approaches. Transformation and simulation of the kernel density estimator are discussed. The principles of smoothing learned in this chapter will be essential for the subsequent chapters.

Chapter 3, Kernel density estimation II, extends kernel density estimation to the multivariate case where \(f\) is supported on \(\mathbb{R}^p\). The chapter also covers estimation of density derivatives, with high-order derivatives tamed through Kronecker products \(\mathrm{D}^{\otimes r}\) of the differential operator \(\mathrm{D}\). This enables a compact version of the multivariate Taylor’s theorem of arbitrary order. A similar agenda to that of Chapter 2 is followed to analyse the asymptotic properties of the estimator and perform bandwidth selection. A neat insight into the difficulty of the estimation of \(\mathrm{D}^{\otimes r} f\) in terms of \((r,p)\) is offered by the rule-of-thumb selector. Density (derivative) estimation is applied to derive nonparametric alternatives to classical topics in multivariate analysis often approached through some form of Gaussianity: outlier detection, clustering, discriminant analysis, and dimension reduction.

Chapter 4, Kernel regression estimation I, shifts focus to estimating the regression function \(m(x) = \mathbb{E}[Y \mid X = x]\), for the case where both \(Y\) and \(X\) are univariate and absolutely continuous random variables. We introduce the classical Nadaraya–Watson estimator using density estimation ideas from Chapter 2 and then develop local polynomial regression as an extension. Asymptotic properties for these estimators are derived, and the advantage of odd polynomial orders over even is presented. Bandwidth selection strategies based on plug-in and cross-validation are introduced, with more emphasis on the latter due to its scalability to more challenging settings. The regressogram is also presented as the “histogram approach” to regression estimation still used in certain applied fields.

Chapter 5, Kernel regression estimation II, tackles more advanced regression scenarios where \(m(\mathbf{x}) = \mathbb{E}[Y \mid \mathbf{X} = \mathbf{x}]\) features a multivariate random vector \(\mathbf{X}\) and a more general response \(Y\). The first generalization handles the case when \(\mathbf{X}\) is a mixed-type random vector with continuous, discrete, and categorical variables by defining appropriate kernels. Bandwidth selection through cross-validation is studied in that situation. Marginal effects of each predictor are introduced to add interpretability to the nonparametric fit. Bootstrap confidence intervals for \(m(\mathbf{x})\) serve to quantify the uncertainty. The second generalization presents local likelihood methods for more general responses, specializing in local logistic regression for binary outcomes.

Finally, Chapter 6, Nonparametric tests, covers hypothesis testing without parametric assumptions, avoiding the need for smoothing. First, we study classical one-sample goodness-of-fit tests based on the empirical cumulative distribution function, both for simple and composite hypotheses. Special attention is given to normality testing and to bootstrap-based approaches that extend these methods to general parametric families. Then, we cover two-sample tests for equality of distributions, also based on the empirical cumulative distribution function, and the Wilcoxon–Mann–Whitney test for distribution shifts. The general approach to permutation-based testing of homogeneity is also presented. The chapter concludes with tests of independence between two random variables, including tests based on concordance measures, the modern distance-correlation test, and the general permutation-based approach.

Software & code

The book contains a substantial amount of code, with snippets that are intended to be self-contained within the chapter in which they appear. This helps illustrate how the methods and theory translate to practice. The software employed throughout this book is the statistical language R and the RStudio IDE (Integrated Development Environment). Basic knowledge of both is assumed.1

The Shiny interactive apps in the book can be downloaded and run locally, which also allows inspection of their source code. Check out this GitHub repository for the sources.

Several packages that are not included within R by default are used throughout the book. The main ones are ks and np. All of the packages can be installed with the following commands:

# Installation of required packages
packages <- c("boot", "circular", "dgof", "emstreeR", "energy",
              "geometry", "goftest", "ISLR", "KernSmooth", "ks",
              "latex2exp", "locfit", "manipulate", "maps", "MASS",
              "microbenchmark", "misc3d", "mvtnorm", "nor1mix", "nortest",
              "np", "numDeriv", "OceanView", "rgl", "viridis")
install.packages(packages)

The book makes explicit mention of the package to which a function belongs by using the operator ::, except when the use of the functions of a package is very repetitive and that package is loaded. You can load all the packages by running:

# Load packages
lapply(packages, library, character.only = TRUE)

The book’s R code snippets are collected in the following scripts:

To download them, use your browser’s Save link as… option.

Citation

You may use the following \(\text{B}{\scriptstyle\text{IB}}\mkern-2mu \text{T}\mkern-3mu\lower.5ex\hbox{E}\mkern-2mu\text{X}\) entry when citing this book:

@book{NonparametricStatistics2026,
    title        = {Nonparametric Statistics},
    author       = {Garc\'ia-Portugu\'es, E.},
    year         = {2026},
    note         = {Version 7.0.0. ISBN 978-84-09-29537-1},
    url          = {https://egarpor.github.io/NP-UC3M/}
}

You may also want to use the following template:

García-Portugués, E. (2026). Nonparametric Statistics. Version 7.0.0. ISBN 978-84-09-29537-1. Available at https://egarpor.github.io/NP-UC3M/.

A previous version of the book at the now-discontinued bookdown.org hosting service was known as Notes for Nonparametric Statistics.

License

All the material in this book is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License (CC BY-NC-ND 4.0). You may not use this material except in compliance with the aforementioned license. The human-readable summary of the license states that:

  • You are free to:
    • Share – Copy and redistribute the material in any medium or format.
  • Under the following terms:
    • Attribution – You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
    • NonCommercial – You may not use the material for commercial purposes.
    • NoDerivatives – If you remix, transform, or build upon the material, you may not distribute the modified material.

Contributions

Contributions, reporting of typos, and feedback on the book are very welcome. Send an email to and I will gladly add your name to the list of contributors.

Credits

Several excellent references were used in preparing this book. The following list presents the books that have been consulted:

This book was made possible thanks to the excellent open-source software Xie (2016), Xie (2020), Allaire et al. (2020), Xie and Allaire (2020), and R Core Team (2020). In addition, some layout improvements build on the outstanding work of Úcar (2018).

Last but not least, the book has benefited from contributions from the following people (in alphabetical order):

References

Allaire, J. J., Y. Xie, J. McPherson, J. Luraschi, K. Ushey, A. Atkins, H. Wickham, J. Cheng, W. Chang, and R. Iannone. 2020. rmarkdown: Dynamic Documents for R. https://github.com/rstudio/rmarkdown.
Chacón, J. E., and T. Duong. 2018. Multivariate Kernel Smoothing and Its Applications. Vol. 160. Monographs on Statistics and Applied Probability. Boca Raton: CRC Press. https://doi.org/10.1201/9780429485572.
D’Agostino, R. B., and M. A. Stephens, eds. 1986. Goodness-of-Fit Techniques. Vol. 68. Statistics: Textbooks and Monographs. New York: Marcel Dekker. https://www.routledge.com/Goodness-of-Fit-Techniques/DAgostino/p/book/9780367580346.
DasGupta, A. 2008. Asymptotic Theory of Statistics and Probability. Springer Texts in Statistics. New York: Springer. https://doi.org/10.1007/978-0-387-75971-5.
Fan, J., and I. Gijbels. 1996. Local Polynomial Modelling and Its Applications. Vol. 66. Monographs on Statistics and Applied Probability. London: Chapman & Hall. https://doi.org/10.1201/9780203748725.
Li, Q., and J. S. Racine. 2007. Nonparametric Econometrics. Princeton: Princeton University Press. https://press.princeton.edu/books/hardcover/9780691121611/nonparametric-econometrics.
Loader, C. 1999. Local Regression and Likelihood. Statistics and Computing. New York: Springer. https://doi.org/10.2307/1270956.
Nelsen, R. B. 2006. An Introduction to Copulas. Second. Springer Series in Statistics. New York: Springer-Verlag. https://doi.org/10.1007/0-387-28678-0.
R Core Team. 2020. R: A Language and Environment for Statistical Computing. Vienna. https://www.R-project.org/.
Scott, D. W. 2015. Multivariate Density Estimation. Second. Wiley Series in Probability and Statistics. Hoboken: John Wiley & Sons. https://doi.org/10.1002/9781118575574.
Sheskin, D. J. 2011. Handbook of Parametric and Nonparametric Statistical Procedures. Fifth. Boca Raton: CRC Press. https://doi.org/10.1201/9780429186196.
Silverman, B. W. 1986. Density Estimation for Statistics and Data Analysis. Monographs on Statistics and Applied Probability. London: Chapman & Hall. https://doi.org/10.1007/978-1-4899-3324-9.
Úcar, I. 2018. “Energy Efficiency in Wireless Communications for Mobile User Devices.” PhD thesis, Universidad Carlos III de Madrid. https://enchufa2.github.io/thesis/.
van der Vaart, A. W. 1998. Asymptotic Statistics. Vol. 3. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511802256.
Wand, M. P., and M. C. Jones. 1995. Kernel Smoothing. Vol. 60. Monographs on Statistics and Applied Probability. London: Chapman & Hall. https://doi.org/10.1201/b14876.
Wasserman, L. 2004. All of Statistics. Springer Texts in Statistics. New York: Springer. https://doi.org/10.1007/978-0-387-21736-9.
———. 2006. All of Nonparametric Statistics. Springer Texts in Statistics. New York: Springer. https://doi.org/10.1007/0-387-30623-4.
Xie, Y. 2016. Bookdown: Authoring Books and Technical Documents with R Markdown. The r Series. Boca Raton: CRC Press. https://bookdown.org/yihui/bookdown/.
———. 2020. knitr: A General-Purpose Package for Dynamic Report Generation in R. https://CRAN.R-project.org/package=knitr.
Xie, Y., and J. J. Allaire. 2020. tufte: Tufte’s Styles for R Markdown Documents. https://CRAN.R-project.org/package=tufte.

  1. Among others: basic programming in R, ability to work with objects and data structures, ability to produce graphics, knowledge of the main statistical functions, and ability to run scripts in RStudio.↩︎