Loading...
A complete MLP-Mixer block combines token mixing and channel mixing with residual connections:
X1=X+TokenMix(X) X2=X1+ChannelMix(X1)
Where:
The residual connections ensure stable training and allow the network to learn identity mappings when beneficial.
Task: Implement a complete Mixer block with both mixing stages and residual connections.
X (2×2), W_token=zeros, W1=I, W2=I
X + 0 + GELU(X) = X + GELU(X)
Zero token weights mean no token mixing. Identity channel weights apply GELU directly. Final output includes both residuals.