ActNet: Breaking the Practicality Barrier of Kolmogorov-Arnold Networks

TL;DR

The Kolmogorov Superposition Theorem (KST) has long promised a way to represent complex multivariate functions as simple univariate ones, but its popular implementation, Kolmogorov-Arnold Networks (KANs), often struggles with efficiency. ActNet changes the game by leveraging a modern variant of KST (Laczkovich, 2021) to create a scalable, multi-head MLP-like architecture. It outperforms KANs and matches or beats SOTA MLPs in challenging physics-informed tasks (PINNs).

Background Positioning

In the landscape of function approximation, we have lived between the "approximation" world of MLPs (Cybenko, 1989) and the "exact representation" world of KST. While KANs recently revived interest in KST, they inherited the original formula's $O(d^2)$ complexity. ActNet sits at the intersection of rigorous mathematical theory and modern deep learning engineering, positioning itself as the most practical KST-based architecture to date.

The "Why": Why KANs Struggle and Why ActNet Works

The original 1957 Kolmogorov formula is mathematically elegant but computationally "pathological." It requires a massive number of unknown functions and offers little insight into their structure, often leading to non-differentiable "rugged" functions that are a nightmare for gradient descent.

The authors' core Insight is that not all KSTs are created equal. By switching to the Laczkovich (2021) formulation, ActNet gains:

• Linear Scaling: Number of inner functions scales with $O(d)$ instead of $O(d^2)$.
• Stability: A clever initialization scheme ensures that the variance of activations doesn't explode or vanish, regardless of how deep the network is.
• Practicality: It can be implemented in a single line of code using einsum.

Methodology: The ActLayer

The heart of ActNet is the ActLayer. Unlike a standard MLP layer that uses a fixed activation (like ReLU), the ActLayer learns the activation function itself by expanding it into a set of sinusoidal basis functions.

Figure 1: Comparison between an individual ActLayer (left) and the full ActNet pipeline (right).

Each "head" in the ActLayer applies a unique trainable activation function ${\varphi }_i$ to the input, followed by a linear projection. This makes it mathematically equivalent to a "Multi-Head MLP" where the non-linearity is customized for every channel.

Crushing the PINN Benchmarks

Physics-Informed Neural Networks (PINNs) are the ultimate stress test for these architectures because they require accurate derivatives. ActNet shines here because the derivative of an ActNet is itself another ActNet—meaning higher-order derivatives don't vanish.

Key Results:

1. Poisson and Helmholtz Equations: In highly oscillatory settings (frequency $w=32$), ActNet outperformed KANs and even Siren (the previous standard for periodic activations) by up to two orders of magnitude in error reduction.
2. Chaotic Systems: On the Kuramoto-Sivashinsky equation—a notoriously difficult chaotic PDE—ActNet nearly halved the error compared to the previous best results.

Figure 2: Visualizing predictions for the Helmholtz equation. ActNet captures the high-frequency peaks that KANs and MLPs miss.

Critical Analysis & Conclusion

Takeaway: ActNet demonstrates that the "Kolmogorov vs. MLP" debate isn't over. By moving away from the 1957 formulation toward more contemporary versions of the theorem, we can build models that have the theoretical depth of KST with the training efficiency of MLPs.

Limitations: While ActNet is much faster than KANs, it is still roughly 2-3x slower than a standard MLP due to the basis function expansion.

Future Outlook: The next frontier for ActNet is high-dimensional data (like ImageNet). If the efficiency gains seen in low-dimensional PDEs hold true in higher dimensions, ActNet could potentially replace the MLP blocks inside Transformers, leading to a new generation of "Kolmogorov-Transformer" hybrids.