Diffusion RNN

Watch a cloud forget its dimensions

We start with handwritten digits 0, 1, 2, and 7, represented in the three-dimensional latent space of a variational autoencoder (VAE). A second network learns to denoise these latent coordinates. The colored background points are digit embeddings; the red points are states of the recurrent denoiser.

Figure 1. Recomputed trajectories from the original trained model: 3,000 evolving states and 20,000 background embeddings. Drag the cloud to rotate it; scrub recurrent time to inspect the geometry. The labeled time intervals are visual guides, not measured phase transitions. Fractional times interpolate between integer model updates.

At early times the red cloud occupies a volume. Later, some directions have almost disappeared while others remain extended. This creates the appearance of a sheet, then a line, then a point. The phrase slow manifold describes the geometry we see: states approach it quickly, then evolve more slowly along it. Establishing a mathematically invariant slow manifold would require additional analysis.

In the talk, we also decoded states along the one-dimensional structures back into images. For the digit 0, traversing a line changes the shape from a vertically elongated loop to a horizontally elongated one. The endpoint resembles a prototype, while the line retains a family of writing styles. This motivates the distinction: a fixed point can represent a particular remembered pattern; an extended slow structure can represent organized variation among patterns.

Train a denoiser. Run it as an RNN.

Let X be a clean data point and let ε be an independent standard Gaussian vector. We corrupt the point with noise of standard deviation α, then train a network to reconstruct the clean input:

Training uses noisy–clean pairs, not trajectories. After training, we hold the parameters fixed and iterate

That is the entire recurrent rule. The same function is reused at every step. The time slider changes how many times it has been applied; it does not change the training noise or retrain the network.

The name “diffusion RNN” comes from the denoising motivation. The trajectories here are deterministic, at a fixed training noise scale. They are not a reverse-time diffusion sampler with a noise schedule, and we should not expect their long-time distribution to reproduce the training distribution. In fact, collapse toward simpler structures is precisely what we are examining.

The Gaussian case: nothing stretches, but some things shrink slowly

Before interpreting a neural network, it helps to solve a case that needs no network at all. Suppose the clean data are centered Gaussian with covariance Σ, and consider a linear denoiser f(y) = Jy. Expanding the mean squared error gives

Setting the derivative to zero yields the optimal map:

If Σ = Q diag(λ₁, λ₂, λ₃) Qᵀ, the learned map has exactly the same eigenvectors. Its gain along direction i is

For positive noise and positive finite variance, every gain lies strictly between zero and one. Every axis contracts. But larger-variance directions have larger gains, so they survive longer. The long axis gets longer relative to the short axis, even while both shorten in absolute terms.

Try α = 1 and advance time a few steps. The smallest-variance direction disappears first, leaving an approximate sheet. The second direction then fades, leaving an approximate line. Eventually even that line collapses. The characteristic contraction time is τᵢ = −1/log gᵢ. A useful hierarchy needs separated time scales, not merely three different eigenvalues.

There is a small but important qualification. With full-rank covariance, this linear map remains invertible at every finite time. A continuous three-dimensional cloud has not literally become two-dimensional after five iterations. Its thickness has become small relative to the other dimensions and to our observational resolution. “Dimensionality reduction” here is effective, rather than an instantaneous loss of mathematical dimension.

The limits are also informative. For small noise, the map approaches the identity on positive-variance directions. For α² much larger than the largest covariance eigenvalue, it approaches Σ/α². Repeated application then resembles a power iteration on the covariance, accompanied by overall contraction. Very large noise can make everything collapse too quickly to expose an interesting intermediate structure.

What survives beyond a Gaussian?

A nonlinear network can bend the preferred directions as it moves through state space. This is the intuition behind the talk’s phrase “nonlinear principal components”: local geometry can guide fast and slow motion without requiring one global set of straight axes.

But zooming in on arbitrary data does not automatically make its distribution Gaussian. The covariance calculation is exact for the Gaussian model; applying it patch by patch is a local approximation. A complementary statement is exact for any distribution with finite second moment under Gaussian corruption. With unlimited model capacity and population mean squared error, the optimal denoiser is the posterior mean:

where p_α is the clean distribution convolved with a Gaussian of variance α². To see this, differentiate the convolution: the Gaussian kernel contributes (X − y)/α²; dividing by the noisy density turns the weighted integral into a conditional expectation. Thus the denoiser’s displacement points along the score of a smoothed density. This connection belongs to the broader theory of denoising and score estimation, rather than being a new identity specific to this project [1].

Differentiating the posterior mean once more gives

For Gaussian data this reproduces the constant matrix above. For a nonlinear distribution it varies with position. Some eigenvalues can exceed one, so the ideal denoiser is not necessarily a global contraction. A trained finite network need not satisfy these identities exactly either. Both facts matter when interpreting complicated trajectories.

Near a fixed point, the Jacobian still tells us which perturbations decay quickly and which persist. Across a region, changing eigendirections can organize curved structures. This is a useful mechanism for the observed geometry, but it does not establish equivalence to a particular nonlinear PCA algorithm or prove that every denoising network has a hierarchy of slow manifolds.

Memory and abstraction as different resolutions

Noise determines which distinctions the reconstruction task asks a network to preserve. With very small corruption and a finite, well-separated training set, a noisy input can usually be assigned to one nearby example. Local attraction toward individual examples is then a plausible outcome. Larger corruption makes several examples compatible with the same noisy observation, so their conditional average matters.

This suggests a memory–abstraction trade-off. One regime preserves many local details; another emphasizes relationships shared across examples. The nonlinear experiments in the talk show this contrast through isolated attractors and extended transient structures. The Gaussian calculation explains one ingredient—unequal contraction rates—without proving the full empirical trade-off.

It would be too strong to equate small noise with memorization in every setting. For a smooth population distribution, the small-noise denoiser approaches the identity, not a collection of discrete memories. Likewise, arbitrarily large noise does not guarantee useful abstraction: it can wash out the very structure we hoped to preserve. Data spacing, network capacity, optimization, and observation time all enter the picture.

Can one network hold more than one geometry?

The talk’s second experiment adds a control input, so the recurrent rule becomes x_t+1 = f_θ(x_t, c). At c = −1, training examples come from a cylinder; at c = 1, they come from a Möbius strip. We then examine the learned dynamics at intermediate control values. This follows the idea of controlling learned recurrent structures through an additional input, inspired by Kim and colleagues [2].

The striking observation in the talk is a continuous-looking transformation between the two learned point-cloud geometries. This is a second possible sense of abstraction: a network can organize not only variation within a distribution, but variation between learned distributions.

There is no topological paradox here. A cylinder and a Möbius strip are not homeomorphic. A continuously changing neural output need not remain a smooth embedded surface, preserve rank, or avoid self-intersections throughout the interpolation. A cloud that appears to morph smoothly is not evidence of a topology-preserving deformation. Nor do endpoint training examples uniquely determine the intermediate geometry; architecture and optimization can influence it.

What would an abstraction capacity measure?

Associative-memory models invite us to count stable memories. The experiments here suggest a related question: how many useful slow structures can a recurrent network support, and how reliably can it select or manipulate them?

Counting visible sheets would not be enough. A capacity measure would have to specify robustness to perturbations, the separation of fast and slow time scales, the resolution at which structures are distinguished, and whether their coordinates capture meaningful variation in held-out data. These experiments do not provide that theory.

A practical next step would compare Jacobian spectra along trajectories, test the stability of the structures across training seeds, and measure decoded variation on held-out examples. For the controlled network, an especially useful comparison is a network trained on a cylinder alone: which intermediate features reflect the data, and which reflect the network’s own biases? The final backup slide of the talk poses exactly this concern.

The proposal is therefore modest but concrete: look beyond the final attractor. The route to it can carry structured information that a fixed-point description misses.

Sources and reproducibility

This article expands Diffusion RNN: a model of memory and abstraction, the APS March Meeting talk by Toni J. B. Liu and Jason Z. Kim (2024). The conference abstract appeared under the title Diffusion RNN: Extracting Low-Dimensional Structures in Data as Quasi-stable Manifolds; the talk was presented on March 6, 2024. The 41-slide deck and its speaker notes are the primary narrative source: slides 1–7 motivate the geometry, 8–23 present training and analysis, 24–29 develop the memory–abstraction interpretation, and 30–32 describe controlled manifolds. The mathematical exposition above makes the assumptions and finite-time qualifications explicit.

Figure 1 uses the original VAE and recurrent-network checkpoints from the research code at a pinned revision. No model was retrained for this article. The VAE has three latent coordinates; the denoiser uses training noise α = 1 and deterministic updates. We recomputed 80 updates from 3,000 Gaussian initial states, using seed 0, and display 5,000 latent embeddings per digit. The original plotting notebook did not fix the VAE sampling seed, so these are reproducible reconstructions of the experiment, not byte-identical frames from the talk.

Autoplay holds t = 3, 9, 18, and 80 for two seconds each, taking one second to transition between successive key frames, with interpolation between computed snapshots. Each regime occupies an equal quarter of the slider; its position is not linear in recurrent time. The four visual-guide intervals are 0–6 (3D), 6–14 (2D), 14–70 (1D), and 70–80 (0D). Manual scrubbing interpolates fractional times. All integer steps are available in the data. These display choices do not change the recurrent dynamics. Figure 2 is an analytical illustration with chosen eigenvalues, not a fitted neural-network experiment.

Figure 3 uses the original Model4_cylinder_mobius_t_0.5_-1_1_alpha_1 checkpoint at the same pinned revision. We evaluate 21 controls from −1 to +1 and 20 recurrent updates, from 1,500 common Gaussian initial states (standard deviation 5; NumPy default generator seed 0). This is a new, reproducible rendering of the trained model, rather than an extraction of the talk’s animation. Control-figure export script · Control-figure data.

Source and data checksums · MNIST export script · Trajectory data · Gaussian figure source · Research repository

Cite this article

Liu, T. J. B., & Kim, J. Z. (2026). Diffusion RNN: a model of memory and abstraction (Version 1.0). Research article. https://www.toni-liu.com/research/diffusion-rnn/

This is a web research article expanding the 2024 talk, not a peer-reviewed journal publication.

Download BibTeX

Related work

[1] Denoisers and density geometry. Alain and Bengio’s What Regularized Auto-Encoders Learn from the Data-Generating Distribution (JMLR 15, 3743–3773, 2014) connects regularized reconstruction to the score and local structure of the data density. This provides theoretical context for interpreting a denoiser geometrically. Our emphasis here is on the transient hierarchy produced by repeated deterministic application.

[2] Controlling recurrent representations. Kim, Lu, Nozari, Pappas, and Bassett’s Teaching recurrent neural networks to infer global temporal structure from local examples (Nature Machine Intelligence 3, 316–323, 2021) motivates using a control input to modify learned recurrent behavior. The cylinder–Möbius experiment applies that idea to denoising dynamics learned from static distributions.

[3] Score-based diffusion. Song and colleagues’ Score-Based Generative Modeling through Stochastic Differential Equations (ICLR 2021) develops generation through time-dependent scores and reverse-time dynamics. It is an important inspiration, while the autonomous fixed-noise recurrence studied here serves a different purpose: inspecting contraction and the structures retained along the way.