1 Introduction, 1
1.1 Estimation Problem and Method, 2
1.1.1 Cubic Smoothing Spline, 2
1.1.2 Penalized Likelihood Method, 4
1.2 Notation, 5
1.3 Decomposition of Multivariate Functions, 6
1.3.1 ANOVA and Averaging Operator, 6
1.3.2 Multiway ANOVA Decomposition, 7
1.3.3 Multivariate Statistical Models, 10
1.4 Case Studies, 12
1.4.1 Water Acidity in Lakes, 12
1.4.2 AIDS Incubation, 14
1.4.3 Survival After Heart Transplant, 15
1.5 Scope, 17
1.6 Bibliographic Notes, 18
1.7 Problems, 19

2 Model Construction, 21
2.1 Reproducing Kernel Hilbert Spaces, 22
2.1.1 Hilbert Spaces and Linear Subspaces, 22
2.1.2 Riesz Representation Theorem, 27
2.1.3 Reproducing Kernel and Non-negative Definite Function, 27
2.2 Smoothing Splines on {1,...,K}, 30
2.3 Polynomial Smoothing Splines on [0,1], 32
2.3.1 A Reproducing Kernel in C^(m)[0,1], 32
2.3.2 Computation of Polynomial Smoothing Splines, 34
2.3.3 Another Reproducing Kernel in C^(m)[0,1], 35
2.4 Smoothing Splines on Product Domains, 38
2.4.1 Tensor Product Reproducing Kernel Hilbert Spaces, 38
2.4.2 Reproducing Kernel Hilbert Spaces on {1,...,K}², 39
2.4.3 Reproducing Kernel Hilbert Spaces on [0,1]², 40
2.4.4 Reproducing Kernel Hilbert Spaces on {1,...,K}x[0,1], 43
2.4.5 Multiple-Term Reproducing Kernel Hilbert Spaces, 43
2.5 Bayes Model, 46
2.5.1 Shrinkage Estimates as Bayes Estimates, 46
2.5.2 Polynomial Splines as Bayes Estimates, 47
2.5.3 Smoothing Splines as Bayes Estimates, 49
2.6 Minimization of Penalized Functional, 50
2.6.1 Existence of Minimizer, 50
2.6.2 Penalized and Constrained Optimization, 52
2.7 Bibliographic Notes, 53
2.8 Problems, 55

3 Regression with Gaussian-Type Responses, 59
3.1 Preliminaries, 60
3.2 Smoothing Parameter Selection, 62
3.2.1 Unbiased Estimate of Relative Loss, 63
3.2.2 Generalized Cross-Validation, 65
3.2.3 Restricted Maximum Likelihood, 68
3.2.4 Weighted and Replicated Data, 69
3.2.5 Empirical Performance, 70
3.3 Bayesian Confidence Intervals, 72
3.3.1 Posterior Distribution, 72
3.3.2 Confidence Intervals on Sampling Points, 74
3.3.3 Across-the-Function Coverage, 75
3.4 Computation: Generic Algorithms, 76
3.4.1 Algorithm for Fixed Smoothing Parameters, 76
3.4.2 Algorithm for Single Smoothing Parameter, 77
3.4.3 Algorithm for Multiple Smoothing Parameters, 79
3.4.4 Calculation of Posterior Variances, 80
3.5 Software, 81
3.5.1 RKPACK, 81
3.5.2 R Package gss, 82
3.6 Model Checking Tools, 86
3.6.1 Cosine Diagnostics, 87
3.6.2 Examples, 87
3.6.3 Concepts and Heuristics, 91
3.7 Case Studies, 93
3.7.1 Nitrogen Oxides in Engine Exhaust, 93
3.7.2 Ozone Concentration in Los Angeles Basin, 94
3.8 Computation: Special Algorithms, 99
3.8.1 Fast Algorithm for Polynomial Splines, 100
3.8.2 Monte Carlo Cross-Validation, 101
3.9 Bibliographic Notes, 103
3.10 Problems, 105

4 More Splines, 111
4.1 Partial Splines, 112
4.2 Splines on the Circle, 113
4.2.1 Periodic Reproducing Kernel Hilbert Spaces, 114
4.2.2 Splines as Low-Pass Filters, 114
4.2.3 More on Asymptotics of Sec.3.2, 116
4.3 L-Splines, 119
4.3.1 Trigonometric Splines, 120
4.3.2 Chebyshev Splines, 122
4.3.3 General Construction, 126
4.3.4 Case Study: Weight Loss of Obese Patient, 130
4.3.5 Fast Algorithm, 134
4.4 Thin-Plate Splines, 135
4.4.1 Semi-Kernels for Thin-Plate Splines, 136
4.4.2 Reproducing Kernels for Thin-Plate Splines, 138
4.4.3 Tensor Product Thin-Plate Splines, 140
4.4.4 Case Study: Water Acidity in Lakes, 141
4.5 Bibliographic Notes, 143
4.6 Problems, 144

5 Regression with Exponential Families, 149
5.1 Preliminaries, 150
5.2 Smoothing Parameter Selection, 151
5.2.1 Performance-Oriented Iteration, 152
5.2.2 Direct Cross-Validation, 155
5.2.3 Empirical Performance, 158
5.3 Approximate Bayesian Confidence Intervals, 159
5.4 Software: R Package gss, 161
5.4.1 Binomial Family, 162
5.4.2 Poisson Family, 163
5.4.3 Gamma Family, 164
5.4.4 Inverse Gaussian Family, 165
5.4.5 Negative Binomial Family, 166
5.5 Case Studies, 168
5.5.1 Eruption Time of Old Faithful, 168
5.5.2 Spectrum of Yearly Sunspots, 170
5.5.3 Progression of Diabetic Retinopathy, 171
5.6 Bibliographic Notes, 173
5.7 Problems, 175

6 Probability Density Estimation, 177
6.1 Preliminaries, 178
6.2 Poisson Intensity, 182
6.3 Smoothing Parameter Selection, 183
6.3.1 Kullback-Leibler Loss and Cross-Validation, 183
6.3.2 Modifications of Cross-Validation Score, 185
6.3.3 Empirical Performance, 187
6.4 Computation, 187
6.5 Case Studies, 190
6.5.1 Buffalo Snowfall, 190
6.5.2 Eruption Time of Old Faithful, 191
6.5.3 AIDS Incubation, 191
6.6 Biased Sampling and Random Truncation, 192
6.6.1 Biased and Truncated Samples, 193
6.6.2 Penalized Likelihood Estimation, 194
6.6.3 Empirical Performance, 196
6.6.4 Case Study: AIDS Incubation, 198
6.7 Conditional Densities, 198
6.7.1 Penalized Likelihood Estimation, 199
6.7.2 Case Study: Penny Thickness, 200
6.7.3 Logistic Regression, 202
6.8 Response-Based Sampling, 204
6.8.1 Response-Based Samples, 204
6.8.2 Penalized Likelihood Estimation, 206
6.9 Bibliographic Notes, 206
6.10 Problems, 209

7 Hazard Rate Estimation, 211
7.1 Preliminaries, 212
7.2 Smoothing Parameter Selection, 214
7.2.1 Kullback-Leibler Loss and Cross-Validation, 215
7.2.2 Empirical Performance, 217
7.3 Case Studies, 218
7.3.1 Treatments of Gastric Cancer, 218
7.3.2 Survival After Heart Transplant, 219
7.4 Penalized Partial Likelihood, 221
7.4.1 Partial Likelihood and Biased Sampling, 221
7.4.2 Case Study: Survival After Heart Transplant, 221
7.5 Models Parametric in Time, 222
7.5.1 Accelerated Life Models, 222
7.5.2 Weibull Family, 224
7.5.3 Log Normal Family, 225
7.5.4 Log Logistic Family, 226
7.5.5 Case Study: Survival After Heart Transplant, 227
7.6 Bibliographic Notes, 228
7.7 Problems, 230

8 Asymptotic Convergence, 231
8.1 Preliminaries, 231
8.2 Rates for Density Estimates, 234
8.2.1 Linear Approximation, 235
8.2.2 Approximation Error and Main Results, 237
8.2.3 Semiparametric Approximation, 239
8.2.4 Convergence Under Incorrect Model, 242
8.2.5 Estimation Under Biased Sampling, 243
8.2.6 Estimation of Conditional Density, 244
8.2.7 Estimation Under Response-Based Sampling, 245
8.3 Rates for Hazard Estimates, 245
8.3.1 Martingale Structure, 246
8.3.2 Linear Approximation, 247
8.3.3 Approximation Error and Main Result, 248
8.3.4 Semiparametric Approximation, 250
8.3.5 Convergence Under Incorrect Model, 253
8.4 Rates for Regression Estimates, 253
8.4.1 General Formulation, 254
8.4.2 Linear Approximation, 254
8.4.3 Approximation Error and Main Result, 255
8.4.4 Convergence Under Incorrect Model, 257
8.5 Bibliographic Notes, 258
8.6 Problems, 259