Population Coding Transformations

How can we use population coding to pass data between populations of neurons?

So far we’ve looked at interpreting the activity of a population of neurons, such as decoding encoded values to an imaginary readout node. But neurons send their output to other neurons.

Suppose we want $B$ to encode the same value encoded in population $A$. Can we just copy the neural activities? No, because $A$ and $B$ might have different numbers of neurons.

The correct approach is to decode the value from $A$ and then re-encode it using $B$‘s encoders.

Step by step transformation process

Encode $x$ into population $A$

$$A=\sigma (x{E}_A+{\beta }_A)$$

• $A\in {\mathbb{R}}^{1\times N},x\in {\mathbb{R}}^{1\times K},{\mathbb{E}}_{\mathbb{A}}\in {\mathbb{R}}^{\mathbb{K}\times \mathbb{N}}$

Decode $\hat{x}$ from $A$:

$$\hat{x}=A{D}_A$$

• $D_A\in {\mathbb{R}}^{N\times K}$

Re-encode $\hat{x}$ into population $B$:

$$\begin{array}{rl}B& =\sigma (\hat{x}{E}_B+{\beta }_B)\\ & =\sigma (A{D}_A{E}_B+{\beta }_B)\\ & =\sigma (AW+{\beta }_B)\end{array}$$

where $W=D_A{E}_B$.

• $E_B\in {\mathbb{R}}^{K\times M},{\beta }_B\in {\mathbb{R}}^{1\times M}$, $W\in {\mathbb{R}}^{N\times M}$

(Optional) Decode $\hat{\hat{x}}$ from$B$:

$$\hat{\hat{x}}=B{D}_B$$

Application

The reason we connect populations of neurons is because connections can perform transformations on the data.

For example, given $(x,y)$, we want to build a network that computes $x\cdot y$.

The decoder is trained by:

$$D_{xy}=\underset{\tilde{D}}{\mathrm{argmin}}||A\tilde{D}-xy||$$

Low-Dimensional Bottleneck

Even though we can form a full connection matrix $W$ from the product $DE$, it is actually a low-rank matrix.

In the example above,

$$D_{xy}\in {\mathbb{R}}^{N\times 1},E_B\in {\mathbb{R}}^{1\times M}⟹W=D_{xy}{E}_B\in {\mathbb{R}}^{N\times M}(\text{rank-1 matrix})$$

This might seem like a limitation, but it actually makes things more efficient.

How many FLOPs for AW?

$$A\in {\mathbb{R}}^N,W\in {\mathbb{R}}^{N\times M}⟹\text{FLOPs}=\mathcal{O}(NM)$$

How many FLOPs for $ADE$?

$$\begin{array}{r}A\in {\mathbb{R}}^N,D\in {\mathbb{R}}^{N\times 1},E\in {\mathbb{R}}^{1\times M}\\ AD\in {\mathbb{R}}^1,\text{FLOPs}=\mathcal{O}(N)\\ (AD)E\in {\mathbb{R}}^M,\text{FLOPS}=\mathcal{O}(M)\\ \text{Total FLOPs}=\mathcal{O}(N+M)\end{array}$$

This is why using low-rank representations (like $DE$) is not only compact but computationally efficient.