Singular Value Decomposition and Its Visualization

Lingsong Zhang¹, J. S. Marron, Haipeng Shen and Zhengyuan Zhu

Abstract

Singular Value Decomposition (SVD) is a useful tool in Functional Data Analysis (FDA). Comparing to Principal Component Analysis (PCA), SVD is more fundamental, because SVD simultaneously provides the PCAs in both row and column spaces. We compare SVD and PCA from the FDA view point, and extend the usual SVD to variations by considering different recenterings. A general scree plot is proposed to select an appropriate recentering in practice. Several matrix views of the SVD components are introduced to explore different features in data, including SVD surface plot, rotation movie, curve movie and image plot. These methods visualize both column and row information of a two-way matrix simultaneously, relate the matrix to relevant curves, show local variations and interactions between columns and rows. Several toy examples are designed to compare as well as reveal the different variations of SVD, and real data examples are used to illustrate the usefulness of the visualization methods.

Key words: Exploratory Data Analysis, Functional Data Analysis, Principal Component Analysis.

Introduction

Functional Data Analysis (FDA) is the study of curves (and more complex objects) as data [Ramsay and Silverman, 1997,Ramsay and Silverman, 2002]. Methods related to Principal Component Analysis (PCA) have provided many insights. Comparing to the PCA method, Singular Value Decomposition (SVD) is more fundamental, because SVD not only provides a direct approach to calculate the principal components (PCs), but also derives the PCAs in row and column spaces simultaneously. In this paper, we view a set of curves as a two-way data matrix, explore the connections and differences between SVD and PCA from a FDA view point, and propose several visualization methods for the SVD components.

SVD provides a useful factorization of the data matrix X, while PCA provides a nearly parallel factoring, via eigen-analysis of the covariance matrix, i.e. X^TX, when X is column centered at 0. The eigenvalues for X^TX are then the squares of the singular values for X, and the eigenvectors for X^TX are the singular rows for X. In this paper, we extend the usual (column centered) PCA method into a general SVD framework, and consider four types of SVDs based on different recenterings: SVD, column SVD (CSVD), row SVD (RSVD) and double SVD (DSVD) (cf. Section for details). Several criteria are discussed in this paper for model selection, i.e. selecting the appropriate type of SVD, including model simplicity, approximation performance, and interpretability etc. We introduce a general scree plot, which combines model simplicity and approximation performance together, and provides a visual aid for model selection. See Section for details. Several toy examples in Section are designed to illustrate the general scree plot and the differences of the four types of recenterings.

Visualization methods always help to find underlying features of a data set. In the context of PCA or SVD, common visualization methods include the biplot [Gabriel, 1971], scatter plots between singular columns or singular rows [Shen and Huang, 2005], etc. The biplot shows the relations between the rows and columns, and the scatter plot can be used to cluster the rows or columns. However these plots fail to show the functional curves.

In the FDA field, besides the above plots, it is common to plot singular columns or singular rows as curves [Ramsay and Silverman, 1997]. Marron et al., [2004] provided a visualization method for functional data (using functional PCA, basically CSVD in our paper), which shows the functional objects (curves), projections on the PCs and the residuals. These methods can also be applied in the SVD framework, but to understand all the structure in the data, they need to be applied twice, once for the rows and once for the columns. When the data set is a time series of curves, the method in Marron et al., [2004] uses different colors to show the time ordering. However, if the time series structure is complicated, color coding curves might not be enough to reveal the time effect.

In our paper, we propose several matrix views of the SVD components, which provide new underlying features of the data set. The major visualization tool is a set of surface plots. Related to the surface plot, we introduce SVD rotation movie, SVD curve movie, and SVD image plot. The surface plots for the SVD components show the functional curves of rows and columns simultaneously. SVD rotation movie shows the surface plots from different view angles, which makes it easier to find underlying information. In fact, the visualization method in Marron et al., [2004] (for example, the first column in Figure 3 of their paper) can be viewed as the surface plots from a special angle. Another special view angle, the overlook view of the surface plots, becomes the image plot, which can show the relative variation of the surface, and highlight the interaction between columns and rows. Motivated by the definition of SVD components (cf. Section 3), we introduce the SVD curve movie which can show the time varying features of the functions. One motivating example is an Internet traffic data set, which is discussed in Section 2 to illustrate the usefulness of the surface plot, the rotation movie and the curve movie. Two other real applications are reported in Section 6 as well. One chemometrics data set is used to show that a zoomed version of the surface plot and the SVD movies highlight local behaviors. A Spanish mortality data set is analyzed to illustrate that the image plot highlights the cohort effect (i.e., the interaction between age groups and years).

The rest of the paper is organized as follows. The motivating example in network traffic analysis is in Section 2. Section 3 gives a brief introduction of SVD, and compares it with PCA. Section 4 describes in detail about the generation of the plots. Section 5 shows several toy examples to illustrate the four types of recenterings. Two more real applications in chemometrics and demography are reported in Section 6.

Discussions

In this paper, we propose four types of recenterings for SVD, and describe a general scree plot to help decide which recentering is appropriate in practice. Various SVD visualization tools, such as the SVD surface plot, the image plot, the SVD curve movie and the SVD rotation movie, are proposed to find insightful features in a two way data matrix, especially a functional data set. These visualization methods simultaneously show the rows, columns, local variations, and interactions, which help to find underlying features. Furthermore, when the data set has time series structure in one direction, our method shows the time varying features clearly.

With noisy data, we notice that the SVD components contain a lot of noise. If one prefers, one can pre-smooth the data matrix using his favorite smoother, and then apply our programs to gain some insights, and then apply our program to get additional insights. We are working on incorporating some regularized SVD method [Huang et al., 2005] into our program, so that the exploration of noisy data will be easier.

Our notion of the two-way data matrix requires that the data are observed on common grid points. In cases with irregularly observed data points or missing data, one could preprocess the data using some smoothing or interpolation methods to obtain regular observed data matrices, before applying our visualization methods.

Acknowledgement

The authors gratefully acknowledge Dr. Andrés M. Alonso Fernández of Departamento de Estadística in Universidad Carlos III de Madrid, for providing the Spanish mortality data to us. Thanks also go to two referees and an Associate Editor for their helpful comments and constructive suggestions.

References

[Gabriel 1971]: Gabriel, K. R. (1971), "The biplot graphic display of matrices with application to principal component analysis," Biometrika, 58, 453-467.
[Huang et al. 2005]: Huang, J. Z., Shen, H., and Buja, A. (2005), Analysis of Two-Way Functional Data Using Regularized Singular Value Decomposition, working paper.
[Marron et al. 2004]: Marron, J. S., Wendelberger, J. R., and Kober, E. M. (2004), Time Series Functional Data Analysis, Los Alamos National Lab, no. LA-UR-04-3911.
[Ramsay and Silverman 1997]: Ramsay, J. O. and Silverman, B. W. (1997), Functional Data Analysis, Springer-Verlag.
[Ramsay and Silverman 2002]: - (2002), Applied Functional Data Analysis, Methods and Case Studies, Springer-Verlag.
[Shen and Huang 2005]: Shen, H. and Huang, J. Z. (2005), "Analysis of Call Center Arrival Data Using Singular Value Decomposition," Applied Stochastic Models in Business and Industry, 21, 251-263.

Footnotes:

¹Correspondence to Lingsong Zhang, Department of Statistics and Operations Research, University of North Carolina, Chapel Hill, NC, 27599-3260. Email: lszhang@email.unc.edu

File translated from T_EX by T_TH, version 3.67.
On 25 Apr 2006, 02:30.