Feature Representation

Core idea:

Use a feature function to transform original data from ${\mathbb{R}}^d$ to some other ${\mathbb{R}}^D$

Basic example of this is to transform a Linear Classifier through the origin to one not through the origin.

Perceptron through origin transformation

We can even turn un-separable datasets into separable ones!

XOR data set transformation example

Original XOR dataset (1D version) does not have a linear separator:

Using the transformation $\varphi (x)=[x,x^2]$:

A linear separator in the $\varphi$ space is non-linear in the original space.

• For example, $x^2-1=0$ can be a separator in the $\varphi$ space, with the half-plane $x^2-1>0$ labeled as positive.
• The corresponding separator in the original space can then be found by considering what points are on the boundary of $x^2-1>0$ – obviously, the answer is $+1$ and $-1$;

Visualizing the the original separator:

• This is a very useful and generalizable strategy – it serves as the basis for kernel methods.

• One systematic strategy for constructing a new feature space is to use a polynomial basis.

• Features can also be improved through Feature Engineering.