Commit 1b290f00 authored by Jung Alex's avatar Jung Alex

HA6 FeatureLearning

parent ffe47f44
\documentclass[article,11pt]{article}
\usepackage{fullpage}
\usepackage{url}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{bm, amssymb}
\usepackage{subfigure}
\renewcommand*{\thesection}{Problem~\arabic{section}:}
\renewcommand{\labelenumi}{(\alph{enumi})}
\newcommand{\cluster}{\mathcal{C}}
\input{../mlbp17_macros}
\title{CS-E3210- Machine Learning Basic Principles \\ Home Assignment 6 - ``Feature Learning''}
\begin{document}
\date{}
\maketitle
Your solutions to the following problems should be submitted as one single pdf which does not contain
any personal information (student ID or name). The only rule for the layout of your submission is that for each
problem there has to be exactly one separate page containing the answer to the problem. You are welcome to use the \LaTeX-file underlying this pdf,
available under \url{https://version.aalto.fi/gitlab/junga1/MLBP2017Public}, and fill in your solutions there.
\newpage
\section{The Principal Component}
Consider $\samplesize=20$ snapshots, available at \url{https://version.aalto.fi/gitlab/junga1/MLBP2017Public/tree/master/Clustering/images},
which are named according to the season when they have been taken, i.e., either ``winter??.jpeg'' or ``summer??.jpeg''.
We represent the $i$th snapshot, with $i=1,\ldots,\samplesize$, by the feature vector $\vx^{(\sampleidx)} \in \mathbb{R}^{d}$ with entries
representing the greyscale values of the image pixels belonging to the lower left square of size $40 \times 40$ pixels (this results in a feature length $d=40^2=1600$).
In order to speed up subsequent computations, we transform the original feature vector $\vx^{(\sampleidx)}$ into one single number $z^{(\sampleidx)} = \mathbf{w}^{T} \vx^{(\sampleidx)}$
using a normalized vector $\mathbf{w} \in \mathcal{S}^{d}$, with the unit sphere $\mathcal{S}^{d}= \{ \mathbf{u} \in \mathbb{R}^{d}: \| \mathbf{u} \|^{2}_{2} = 1 \}$ (which is the
set of all unit-norm vectors). % $\| \mathbf{w} \|_{2}^{2} = \mathbf{w}^{T} \mathbf{w} = 1$.
The vector $\mathbf{w}$ should be chosen such that we can accurately reconstruct the original feature vector using $\mathbf{v} z^{(\sampleidx)}$ with some normalized vector $\mathbf{v} \in \mathcal{S}^{d}$
(which might be different from $\mathbf{w}$). Let us measure the reconstruction error, when reconstructing $\vx^{(\sampleidx)}$ from $z^{(\sampleidx)}$, as
\begin{equation}
\emperror(\mathbf{v},\mathbf{w}| \dataset ) \defeq (1/\samplesize) \sum_{\sampleidx=1}^{\samplesize} \| \vx^{(\sampleidx)} - \mathbf{v} z^{(\sampleidx)} \|^{2}_{2} =
(1/\samplesize) \sum_{\sampleidx=1}^{\samplesize} \| \vx^{(\sampleidx)} - \mathbf{v} \mathbf{w}^{T} \vx^{(\sampleidx)} \|^{2}_{2}.
\end{equation}
We are interested in finding the vectors $\hat{\mathbf{v}},\hat{\mathbf{w}} \in \mathcal{S}^{d}$ which minimize the reconstruction error, i.e.,
\begin{equation}
\label{equ_optimality}
\emperror(\hat{\mathbf{v}},\hat{\mathbf{w}}| \dataset ) = \min_{\mathbf{v},\mathbf{w} \in \mathcal{S}^{d}}\emperror( \mathbf{v}, \mathbf{w}| \dataset ).
\end{equation}
What is the relation of the vectors $\hat{\mathbf{v}},\hat{\mathbf{w}} \in \mathcal{S}^{d}$, which satisfy \eqref{equ_optimality}, to the eigenvectors of
the matrix ${\bm \Sigma} = (1/\samplesize) \mathbf{X}^{T} \mathbf{X}$ with $\mathbf{X} = (\vx^{(1)},\ldots,\vx^{(\samplesize)})^{T} \in \mathbb{R}^{\samplesize \times d}$?
Using this relation, compute the optimal vectors $\hat{\mathbf{v}},\hat{\mathbf{w}} \in \mathcal{S}^{d}$ and the associated minimum reconstruction error $\emperror(\hat{\mathbf{v}},\hat{\mathbf{w}}| \dataset )$ for
the given dataset $\dataset=\{\vx^{(\sampleidx)}\}_{\sampleidx=1}^{\samplesize}$.
Illustrate the vectors $\hat{\mathbf{v}},\hat{\mathbf{w}} \in \mathcal{S}^{d}$ using grayscale plots \\(cf.\ \url{https://se.mathworks.com/help/images/ref/mat2gray.html}).
\noindent {\bf Answer.}
\newpage
\end{document}
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment