Polynomial Basis

Systematic strategy for constructing a feature space if features are are naturally numerical

General idea

If you are using $k$th-order basis (where $k$ is a positive integer), you include a feature for every possible product of $k$ different dimensions in your original input.

Example: XOR problem using a polynomial basis as a feature transformation

XOR with feature transformation

Let’s say $k=2$ for the XOR problem in 2 dimensions.

The feature transform is: $\varphi ((x_1,x_2))=(1,x_1,x_2,x_1^2,x_1{x}_2,x_2^2)$

After training a perceptron for 4 iterations, we have: $\theta =(0,0,0,0,4,0)$ ${\theta }_0=0$ which corresponds to $0+0x_1+0x_2+0x_1^2+4x_1{x}_2+0x_2^2+0=0$

Plotting this, we can see the gray shaded region classified as negative and the white region

For more complex dataset, the perceptron can still find appropriate classifiers using this trick: