Clem Yang

0 followers · 317 views

on the atlas · 12

highlights · 23

What’s especially impressive is how cost-efficient it was to train—just $5.5M—thanks to architectural changes like Multi Token Prediction (MTP), Multi-Head Latent Attention (MLA) and a LOT (seriously, a lot) of hardware optimization.
Open-R1: a fully open reproduction of DeepSeek-R1
has a different reference diffusion, integration time and hence different regularization for the coupling and does not coincide with Brownian bridge used as ground truth.
Improving and generalizing flow-based generative models with minibatch optimal transport | OpenReview
code base does not use continuous time training but discrete
Improving and generalizing flow-based generative models with minibatch optimal transport | OpenReview
The results for these works are for arbitrary target / source measures though experimentally are only tried on Gaussian source measures.
Improving and generalizing flow-based generative models with minibatch optimal transport | OpenReview
The results are presented as if there was a static coupling however given minibatch couplings, it is not clear if any convergence results hold -
Improving and generalizing flow-based generative models with minibatch optimal transport | OpenReview
complexity in the regularized case with Sinkhorn, hence it is not practical to have high number of samples. It is far from clear how this method would be useful for computing OT beyond low dimensional toy examples other than as some sort of heuristic
Improving and generalizing flow-based generative models with minibatch optimal transport | OpenReview
complexity for linear solvers in the non regularized case and complexity in the regularized case with Sinkhorn, hence it is not practical to have high number of samples. It is far from clear how this method would be useful for computing OT beyond low dimensional toy examples other than as some sort of heuristic.
Improving and generalizing flow-based generative models with minibatch optimal transport | OpenReview
Minibatch OT suffers from complexity for linear solvers in the non regularized case and complexity in the regularized case with Sinkhorn, hence it is not practical to have high number of samples. It is far from clear how this method would be useful for computing OT beyond low dimensional toy examples other than as some sort of heuristic.
Improving and generalizing flow-based generative models with minibatch optimal transport | OpenReview
all collected without the use of external VLMs
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
how to build performant VLMs from scratch
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
distilling these closed VLMs into open ones
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
The brazil-build release command will build your package, run all unit tests, and perform style checks.
BT101: Make a code change — Learning about development at Amazon
using PyTorch’s Scaled Dot Product Attention (SDPA) achieves close to the same speed
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Once a VQ-VAE is fully trained, we can abandon the uniform prior imposed at training time and learn a new, updated prior p(z) over the latents.
Understanding VQ-VAE (DALL-E Explained Pt. 1)
codebook commitment loss
Understanding VQ-VAE (DALL-E Explained Pt. 1)
sg[x] stands for “stop gradient”,
Understanding VQ-VAE (DALL-E Explained Pt. 1)
codebook alignment los
Understanding VQ-VAE (DALL-E Explained Pt. 1)
he codebook vectors are learned via gradient descent
Understanding VQ-VAE (DALL-E Explained Pt. 1)
right inductive biases baked into the encoder (i.e. using a conv-net for images) and the right structure for the latent codes (i.e. a 32x32 grid for images
Understanding VQ-VAE (DALL-E Explained Pt. 1)
This argmin operation is a bit concerning, since it is non-differentiable with respect to the encoder. But in practice everything seems to work fine if you just pass the decoder gradient directly through this operation to the encoder (i.e. set its gradient to 1 wrt the encoder and the quantized codebook vector; and to 0 wrt all other codebook vectors)
Understanding VQ-VAE (DALL-E Explained Pt. 1)
VQ-VAE extends the standard autoencoder by adding a discrete codebook component to the network.
Understanding VQ-VAE (DALL-E Explained Pt. 1)
The posterior is being pulled towards the prior by the KL divergence, essentially regularizing the latent space towards the gaussian prior. This has the effect of keeping the latent distribution compactly distributed around 0
Understanding VQ-VAE (DALL-E Explained Pt. 1)
basics of variational autoencoders
Understanding VQ-VAE (DALL-E Explained Pt. 1)