Portrait of Toni J.B. Liu

Toni J.B. Liu

Research Highlights

A curated selection, not a full publication list.

  1. Jacobian Scopes
  2. LLMs think like a kernel machine
  3. Memory–abstraction trade-off in diffusion models
  4. In-context neural scaling law
  5. Embedding and attention in non-Euclidean spaces

Jacobian Scopes

When an LLM answers, which prompt tokens drove that decision? Jacobian Scopes attribute predictions—logits, full distributions, and uncertainty—to inputs via fast vector-Jacobian products, with strong performance on attribution benchmarks.

Jacobian Scope visualization: per-token influence on predicting liberal versus conservative from the same prompt.
Semantic Scope reveals the potential for political bias that may be present in models such as LLaMA3.2. The prediction for the subject being a “liberal” is attributed to the input token “Columbia”, while “conservative” to the token pairs for “the South”.

LLMs think like a kernel machine

How do LLMs learn from in-context data? Using Intensive Principal Component Analysis, we visualize their density-estimation trajectories and find an inference process akin to kernel regression, with a kernel whose width and shape adapt as evidence accumulates.

In-context DE trajectory of LLaMA 2-70b, compared to Gaussian KDE and Bayesian histogram.

Memory–abstraction trade-off in diffusion models

What separates abstract thoughts from concrete memories? Treating memories as fixed points, we cast abstraction as a hierarchy of slow manifolds and build a diffusion RNN that extracts them from data—with an analytic theory linking data distributions to the learned manifolds.

Diffusion RNN extracts a hierarchy of slow manifolds (red) from dimensionally reduced MNIST data.

In-context neural scaling law

Can a pretrained LLM make sense of an unseen synthetic language—say, symbols from a random Markov process? LLaMA 2 models in-context learn a range of stochastic systems, physical and symbolic, revealing an in-context neural scaling law.


Embedding and attention in non-Euclidean spaces

Hyperbolic volume grows exponentially with radius—like the branching of a tree—making it a natural host for hierarchical data. Shadow Cones embed partial orders by casting nested shadows: a child lies in its parent’s cone, yielding a fast algorithm for embedding concept graphs.