"I have to constantly re-identify myself to myself, reactivate my own standards, my own convictions about what I’m doing and why." – Nina Simone
got my first angpao in the US from A. didn't exepct to see autoencoders mentioned in class. we were learning PCA i've been seeing autoencoders a lot lately, mainly the these two: sparse autoencoders and variational autoencoders.
at its core, autoencoders do the following:
- compress (encode) input data into a compact representation
- reconstruct (decode) original data from representation
- learn by minimizing the differences between input and output
Sparse autoencoders (SAE) achieves compression by enforcing sparsity in the hidden layer. adding a sparsity penalty to the loss function encourages most neurons to be inactive. the representation is deterministic but sparse. anthropic applies SAEs for mechanistic interpretability, where they want to understand how the mind of an LLMs works internally. The main challenge is individual neurons in an LLM respond to multiple unrelated concepts at once, i.e. a single neuron might process chinese text and pottery shapes, making it hard to understand what the neuron is actually "doing". with SAEs, they can decompose these mixed signals into clearer, more interpretabiliy features. so the goal here is less about compression but using sparsity to identify specific, monosemantic features of how a model processes information.
- Interpretability with Sparse Autoencoders (Colab exercises)
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
Variational autoencoder (VAE) achieves compression by learning a probability distribution in the latent space. a KL divergence term is added to the loss function to regularize the latent distribution. the representation is stochastic. the goal is to learn a continuous, meaningful latent space for generation. they can be used for gen ai (text, image, music generation), representation learning (text2image), anomaly detection, drug discovery, climate modeling, and any situation where you want to learn structured latent spaces that capture meaningful variation, and generate new samples with controlled properties.
- Variational Autoencoder - Model, ELBO, loss function and maths explained easily!
- Tutorial on Variational Autoencoders
- Variational Autoencoders are Beautiful
- Modern PyTorch Techniques for VAEs: A Comprehensive Tutorial
my classes are too short to go deep into any concept. teaching recommender systems in just two classes (2 hours each), you only scratch the surface. it's like my europe tour with my family where we travelled 5 countries in 11 days, you spend most of your time on buses, going from one famous landmark to the other.
homeworks are taking up so much time it's frustrating. i need to approach them with more curiosity. or they just seem like grunt work to me.