Research projects from physics to learning systems
How do large language models (LLMs) estimate probability densities from in-context data?
We leverage the Intensive Principal Component Analysis to visualize the density estimation (DE) process of LLMs. This geometric investigation leads to a natural interpretation of LLM's DE process as a kernel density estimator with adaptive kernel width and shape.
What makes abstract thoughts different from concrete memories? Considering memories as fixed points within the brain's dynamical system, we propose to understand abstraction as a hierarchy of slow manifolds, which serve as coarse-grained roadmaps to these fixed points.
We construct a diffusion RNN (Recurrent Neural Network), a neural-dynamical model that can perform memorization and abstraction, and demonstrate its ability to perform dimensionality reduction on the MNIST dataset.
Finally, we present an analytic theory that elucidates the relation between the distribution of data and the learned slow manifolds.
Could a pretrained LLM make sense of an unseen, synthetic language, say, a chain of symbols generated by a random Markov process? Yes. In fact, we show that LLaMA 2 models possess uncanny abilities to in-context learn a variety of stochstic systems, physical and symbolic. Our observation reveals an in-context version of neural scaling law.
In hyperbolic space, the volume of a ball grows exponentially with its radius, which mirrors the number of nodes in a tree. We provide a fast and intuitive embedding scheme by thinking of parent-child relations as subset relations between shadows formed by a light source and opaque objects.
How do plants anticipate external environemnts and regulate internal states?
We perturb marine algae with patterned illumination, and analyze the response dynamics using image analysis and reduced-dimension observables. Our experiment and analysis unveils wave-like patterns entrained by external illuminations, and coupled to a circadian internal oscillator.