Portrait of Toni J.B. Liu

Toni J.B. Liu

I am a PhD candidate in physics at Cornell University, with an expected graduation date of May 2027. I use scientific methods to interpret, improve, and safeguard AI reasoning, and apply insights from AI to model physical systems. My first-author work has appeared at NeurIPS, ICLR, ICML, and EMNLP.

Before joining Cornell, I earned a B.S. in applied physics from Caltech and a B.A. in liberal arts from Wesleyan University.

Research Highlights

A curated selection, not a full publication list.

  1. Direction of Ignorance
  2. Jacobian Scopes for Causal Attribution
  3. LLMs think like a kernel machine
  4. Memorization and generalization in diffusion models
  5. In-context neural scaling law
  6. Hyperbolic Embedding and Attention

Direction of Ignorance

What does an LLM predict when context offers few clues? Across Llama, Qwen, Gemma, and Pythia, we find a single direction in the unembedding that encodes a unigram prior—the frequencies of individual tokens in training text. The model relies less on this prior as context becomes more informative; adjusting its strength at inference time can steer predictions and improve mathematical reasoning in Llama-3.2-3B-Instruct.

Panel a: Llama-3.2-3B token embeddings colored by frequency, with an arrow marking the prior direction.
Panel a: the token-frequency gradient in Llama-3.2-3B's unembedding. The arrow marks the prior direction.

Jacobian Scopes for Causal Attribution

When an LLM answers, which prompt tokens drove that decision? Jacobian Scopes attribute predictions—logits, full distributions, and uncertainty—to inputs via fast vector-Jacobian products, with strong performance on attribution benchmarks.

Original LLaMA illustration comparing token sensitivity for liberal and conservative predictions.
Original LLaMA illustration: different prediction targets highlight different prompt tokens. Data · Reproduce

LLMs think like a kernel machine

How do LLMs learn from in-context data? Using Intensive Principal Component Analysis, we visualize their density-estimation trajectories and find an inference process akin to kernel regression, with a kernel whose width and shape adapt as evidence accumulates.

Rotating inPCA comparison of LLaMA-2 70B, Gaussian KDE, and Bayesian histogram density-estimation trajectories. Legend for the density-estimation trajectories.
In-context density-estimation trajectories of LLaMA-2 70B, Gaussian KDE, and Bayesian histogram, shown in the first three inPCA dimensions.

Memorization and generalization in diffusion models

What separates abstract thoughts from concrete memories? Treating memories as fixed points, we cast abstraction as a hierarchy of slow manifolds and build a diffusion RNN that extracts them from data—with an analytic theory linking data distributions to the learned manifolds.

Diffusion RNN states converge toward slow manifolds among MNIST latent embeddings.
Diffusion RNN states (red) evolve toward slow manifolds among MNIST digits 0, 1, 2, and 7 in a three-dimensional VAE latent space.

In-context neural scaling law

Can a pretrained LLM make sense of an unseen synthetic language—say, symbols from a random Markov process? LLaMA 2 models in-context learn a range of stochastic systems, physical and symbolic, revealing an in-context neural scaling law.

Noisy logistic-map states, next-state prediction given the current state, and aggregate in-context loss.

Hyperbolic Embedding and Attention

Hyperbolic volume grows exponentially with radius—like the branching of a tree—making it a natural host for hierarchical data. Shadow Cones embed partial orders by casting nested shadows: a child lies in its parent’s cone, yielding a fast algorithm for embedding concept graphs.

Penumbral shadow cones in a Poincaré disk, with embeddings u, v, w, x, y, z and their partial-order relations.