%                                   302f16Assignment3.tex
\documentclass[12pt]{article} 
%\usepackage{amsbsy} % for \boldsymbol and \pmb 
\usepackage{graphicx} % To include pdf files!
\usepackage{amsmath}
\usepackage{amsbsy}
\usepackage{amsfonts}
\usepackage[colorlinks=true, pdfstartview=FitV, linkcolor=blue, citecolor=blue, urlcolor=blue]{hyperref} % For links
\usepackage{fullpage}
%\pagestyle{empty} % No page numbers


\begin{document}
%\enlargethispage*{1000 pt} 

\begin{center}   
{\Large \textbf{STA 302f16 Assignment Three}}\footnote{Copyright information is at the end of the last page.}
\vspace{1 mm}
\end{center}

\noindent
Please bring your R printout from Question~\ref{rmat} to the quiz; you may be asked to hand it in, or maybe not. The other problems are preparation for the quiz in tutorial, and are not to be handed in. 

This exercise set has an unusual feature. \emph{Some of the questions ask you to prove things that are false}. That is, they are not true in general. In such cases, just write ``The statement is false," and give a brief explanation to make it clear that you are not just guessing. The explanation is essential for full marks. Sometimes a small counter-example is desirable. 
\vspace{3mm}


\begin{enumerate} 

%%%%%%%%%%%%%%%% Text

\item Please read pages 11-19 in the textbook. In Section 1.7, the measure of model fit $R^2$ is presented without much explanation. The idea is that $\sum_{i=1}^n(y_i-\overline{y})^2$ represents the sum of squared vertical distances of the points on a scatterplot from a horizontal line with slope zero and intercept $\overline{y}$. Mathematically this could be the best fitting line, but in practice the line $y = b_0 + b_1 x$ is going to do better. That is, $\sum_{i=1}^n(y_i-\widehat{y}_i)^2 =  \sum_{i=1}^ne_i^2  \leq \sum_{i=1}^n(y_i-\overline{y})^2$, so that $0 \leq \frac{\sum_{i=1}^ne_i^2}{\sum_{i=1}^n(y_i-\overline{y})^2} \leq 1$. Small values of this ratio represent good performance of the least squares line relative to the horizontal line. It could be described as an index of ``lack of fit," because big values indicate relatively poor performance. Thus, $R^2 = 1 - \frac{\sum_{i=1}^ne_i^2}{\sum_{i=1}^n(y_i-\overline{y})^2}$ is a measure of good fit. Why it's something \emph{squared} will be taken up later. 
    \begin{enumerate}
        \item Prove that the least squares line must always pass through the point $(\overline{x},\overline{y})$, regardless of the data.
        \item Show the work leading to (1.28). Use the formula sheet. 
        \item Do Exercise 1.2.%   Prove (1.29) on p. 15.
        \item Do Exercise 1.4, but only the expected value, not the variance. The variance is much easier with matrices.
        \item Do Exercise 1.5. Answer all three parts of the question.
        \item Do both parts of Exercise 1.8. You don't have to do all the work of differentiating and solving again. Start with (1.11) and (1.14), and keep simplifying.
    \end{enumerate}

%%%%%%%%%%%%%%%%%%%%% Linear Algebra %%%%%%%%%%%%%%%%%
        
    \item Let     
\begin{tabular}{ccc}
$\mathbf{A} = 
     \left( \begin{array}{c c}
                 1 & 2 \\
                 2 & 4 
    \end{array} \right) $    &
$\mathbf{B} = 
     \left( \begin{array}{c c}
                 0 & 2 \\
                 2 & 1 
    \end{array} \right) $  &
$\mathbf{C} = 
     \left( \begin{array}{c c}
                 2 & 0 \\
                 1 & 2 
    \end{array} \right) $
\end{tabular}
    \begin{enumerate}
        \item Calculate $\mathbf{AB}$ and $\mathbf{AC}$
        \item Do we have $\mathbf{AB} = \mathbf{AC}$? Answer Yes or No.
        \item Prove $\mathbf{B} = \mathbf{C}$.. Show your work. 
    \end{enumerate}

\pagebreak

    \item \label{rmat} Make up a your own $4 \times 4$ symmetric matrix that is not singular (that is, the inverse exists), and is \emph{not a diagonal matrix}. If your first try is singular, try again. Call it $\mathbf{A}$. Enter it into R using \texttt{rbind} (see lecture slides). Make sure to display the input. Then, 
        \begin{enumerate}
            \item Calculate $|\mathbf{A}^{-1}|$ and $1/|\mathbf{A}|$, verifying that they are equal.
            \item Calculate $|\mathbf{A}^2|$ and $|\mathbf{A}|^2$, verifying that they are equal.
            \item Calculate the eigenvalues and eigenvectors of $\mathbf{A}$.
            \item Calculate $\mathbf{A}^{1/2}$.
            \item  Calculate $\mathbf{A}^{-1/2}$.
        \end{enumerate} 
Display both input and output for each part. Label the output with comments. Bring the printout to the quiz.  

    \item Recall the definition of linear independence. The columns of 
$\mathbf{X}$ are said to be \emph{linearly dependent} if there exists a $p \times 1$ vector $\mathbf{v} \neq \mathbf{0}$ with $\mathbf{Xv} = \mathbf{0}$. We will say that the columns of $\mathbf{X}$ are linearly \emph{independent} if $\mathbf{Xv} = \mathbf{0}$ implies $\mathbf{v} = \mathbf{0}$. Let $\mathbf{A}$ be a square matrix. Show that if the columns of $\mathbf{A}$ are linearly dependent, $\mathbf{A}^{-1}$ cannot exist. Hint: $\mathbf{v}$ cannot be both zero and not zero at the same time.
    
    \item  \label{ss} Let $\mathbf{a}$ be an $n \times 1$ matrix of real constants. How do you know $\mathbf{a}^\prime\mathbf{a}\geq 0$?

    \item  Recall the \emph{spectral decomposition} of a square symmetric matrix (For example, a variance-covariance matrix).  Any such matrix $\boldsymbol{\Sigma}$ can be written as 
            $\boldsymbol{\Sigma} = \mathbf{CD} \mathbf{C}^\prime$,
where $\mathbf{C}$ is a matrix whose columns are the (orthonormal) eigenvectors of $\boldsymbol{\Sigma}$, $\mathbf{D}$ is a diagonal matrix of the corresponding  eigenvalues, and $\mathbf{C}^\prime\mathbf{C} =~\mathbf{C}\mathbf{C}^\prime =~\mathbf{I}$. 

        \begin{enumerate}
           \item Let $\boldsymbol{\Sigma}$ be a square symmetric matrix with eigenvalues that are all strictly positive. 
                \begin{enumerate}
                    \item What is $\mathbf{D}^{-1}$?
                    \item Show 
            $\boldsymbol{\Sigma}^{-1} = \mathbf{C} \mathbf{D}^{-1} \mathbf{C}^\prime$
                \end{enumerate}

% \pagebreak

           \item Let $\boldsymbol{\Sigma}$ be a square symmetric matrix, and this time some of the eigenvalues might be zero. 
                \begin{enumerate}
                    \item What do you think $\mathbf{D}^{1/2}$ might be?
                    \item Define $\boldsymbol{\Sigma}^{1/2}$ as 
                    $\mathbf{CD}^{1/2} \mathbf{C}^\prime$.
                    Show $\boldsymbol{\Sigma}^{1/2}$ is symmetric.
                    \item Show $\boldsymbol{\Sigma}^{1/2}\boldsymbol{\Sigma}^{1/2} = \boldsymbol{\Sigma}$.
                \end{enumerate}

            \item Now return to the situation where the eigenvalues of the square symmetric matrix $\boldsymbol{\Sigma}$ are all strictly positive. Define $\boldsymbol{\Sigma}^{-1/2}$ as $\mathbf{CD}^{-1/2} \mathbf{C}^\prime$, where the elements of the diagonal matrix $\mathbf{D}^{-1/2}$ are the reciprocals of the corresponding elements of $\mathbf{D}^{1/2}$. 
                \begin{enumerate}
                    \item Show that the inverse of $\boldsymbol{\Sigma}^{1/2}$ is $\boldsymbol{\Sigma}^{-1/2}$, justifying the notation.
                    \item Show $\boldsymbol{\Sigma}^{-1/2} \boldsymbol{\Sigma}^{-1/2} = \boldsymbol{\Sigma}^{-1}$.
                \end{enumerate}

           \item The (square) matrix $\boldsymbol{\Sigma}$ is said to be \emph{positive definite} if
           $\mathbf{v}^\prime \boldsymbol{\Sigma} \mathbf{v} > 0$ for all vectors $\mathbf{v} \neq \mathbf{0}$. Show that the eigenvalues of a positive definite matrix are all strictly positive. 
           \item Let $\boldsymbol{\Sigma}$ be a symmetric, positive definite matrix. Putting together a couple of results you have proved above, establish that $\boldsymbol{\Sigma}^{-1}$ exists.
        \end{enumerate}

    \item Prove that the diagonal elements of a positive definite matrix must be positive.

    \item Using the Spectral Decomposition Theorem and $tr(\mathbf{AB})=tr(\mathbf{BA})$, prove that the trace is the sum of the eigenvalues for a symmetric matrix.

    \item Using the Spectral Decomposition Theorem and $|\mathbf{AB}| = |\mathbf{BA}|$, prove that the determinant of a symmetric matrix is the product of its eigenvalues.

%%%%%%%%%%%%%%%%%%%%% Random Vectors %%%%%%%%%%%%%%%%%

    \item Let $\mathbf{X} = [X_{ij}]$ be a random matrix. Show $E(\mathbf{X}^\prime) = E(\mathbf{X})^\prime$. 

    \item Let $\mathbf{X}$ be a random matrix, and $\mathbf{B}$ be a matrix of constants. Show 
    $E(\mathbf{XB})=E(\mathbf{X})\mathbf{B}$. Recall the definition 
    $\mathbf{AB}=[\sum_{k}a_{i,k}b_{k,j}]$.

    %Let $\mathbf{X}$ and $\mathbf{Y}$ be random matrices of the same dimensions. Show 
    %$E(\mathbf{X} + \mathbf{Y})=E(\mathbf{X})+E(\mathbf{Y})$. Recall the definition 
    %$E(\mathbf{Z})=[E(Z_{i,j})]$.
    
    \item Let the $p \times 1$ random vector $\mathbf{x}$ have expected value $\boldsymbol{\mu}$ and variance-covariance matrix $\mathbf{\Sigma}$, and let $\mathbf{A}$ be an $m \times p$ matrix of constants. Prove that the variance-covariance matrix of $\mathbf{Ax}$ is either 
    \begin{itemize}
        \item $\mathbf{A} \boldsymbol{\Sigma} \mathbf{A}^\prime$, or
        \item $\mathbf{A}^2 \boldsymbol{\Sigma}$..
    \end{itemize}
Pick one and prove it. Start with the definition of a variance-covariance matrix on the formula sheet.  

    \item  If the $p \times 1$ random vector $\mathbf{x}$ has mean $\boldsymbol{\mu}$ and variance-covariance matrix $\mathbf{\Sigma}$, show $\mathbf{\Sigma} = E(\mathbf{XX}^\prime) - \boldsymbol{\mu \mu}^\prime$.

    \item Let $\mathbf{x}$ be a $p \times 1$ random vector. Starting with the definition on the formula sheet, prove $cov(\mathbf{x})=\mathbf{0}$.. % FALSE

    \item Let the $p \times 1$ random vector $\mathbf{x}$ have mean $\boldsymbol{\mu}$ and variance-covariance matrix $\mathbf{\Sigma}$, let $\mathbf{A}$ be an $r \times p$ matrix of constants, and let $\mathbf{c}$ be an $r \times 1$ vector of constants. Find $cov(\mathbf{Ax}+\mathbf{c})$. Show your work. 

    \item Let the scalar random variable $y = \mathbf{v}^\prime \mathbf{x}$, where $\mathbf{x}$ is a $p \times 1$ random vector. What is $Var(y)$? Use this to prove that \emph{any} variance-covariance matrix must be positive semi-definite. 

\pagebreak

    \item The square matrix $\mathbf{A}$ has an eigenvalue equal to $\lambda$ with corresponding eigenvector  $\mathbf{x} \neq \mathbf{0}$ if $\mathbf{Ax} = \lambda\mathbf{x}$. 
        \begin{enumerate}
            \item Show that the eigenvalues of a variance-covariance matrix cannot be negative.
            \item How do you know that the determinant of a variance-covariance matrix must be greater than or equal to zero? The answer is one short sentence. 
            \item  Let $x$ and $y$ be scalar random variables. Recall $Corr(x,y) = \frac{Cov(x,y)}{\sqrt{Var(x)Var(y)}}$. Using what you have shown about the determinant, show $-1 \leq Corr(x,y) \leq 1$.  You have just proved the Cauchy-Schwarz inequality using probability tools.
        \end{enumerate}

    \item Let $\mathbf{x}$ be a $p \times 1$ random vector with mean $\boldsymbol{\mu}_x$ and variance-covariance matrix $\mathbf{\Sigma}_x$, and let $\mathbf{y}$ be a $q \times 1$ random vector with mean $\boldsymbol{\mu}_y$ and variance-covariance matrix $\mathbf{\Sigma}_y$. 
            \begin{enumerate}
                \item What is the $(i,j)$ element of $cov(\mathbf{x},\mathbf{y})$? See the definition on the formula sheet. 
                \item Find an expression for  $cov(\mathbf{x}+\mathbf{y})$ in terms of 
                $\mathbf{\Sigma}_x$, $\mathbf{\Sigma}_y$ and $cov(\mathbf{x},\mathbf{y})$. Show your work.
                \item Simplify further for the special case where $Cov(x_i,y_j)=0$ for all $i$ and $j$.

                \item Let $\mathbf{c}$ be a $p \times 1$ vector of constants and $\mathbf{d}$ be a $q \times 1$ vector of constants. Find $ cov(\mathbf{x}+\mathbf{c}, \mathbf{y}+\mathbf{d})$. Show your work.
                \item Starting with the definition on the formula sheet, show $cov(\mathbf{x,y})=cov(\mathbf{y,x})$.. % FALSE
                \item Starting with the definition on the formula sheet, show $cov(\mathbf{x,y})=\mathbf{0}$.. % FALSE
            \end{enumerate}

\end{enumerate}

\vspace{60mm}

\noindent
\begin{center}\begin{tabular}{l}
\hspace{6in} \\ \hline
\end{tabular}\end{center}
This assignment was prepared by  \href{http://www.utstat.toronto.edu/~brunner}{Jerry Brunner},
Department of Statistical Sciences, University of Toronto. It is licensed under a 
\href{http://creativecommons.org/licenses/by-sa/3.0/deed.en_US}
     {Creative Commons Attribution - ShareAlike 3.0 Unported License}. Use any part of it as you like and share the result freely. The \LaTeX~source code is available from the course website:
\href{http://www.utstat.toronto.edu/~brunner/oldclass/302f16} {\small\texttt{http://www.utstat.toronto.edu/$^\sim$brunner/oldclass/302f16}}

\end{document}