Ali Ahmad
13 followers · 9 following · 343 views
on the atlas · 44
- mHC: Manifold-Constrained Hyper-Connections | alphaXiv1 savers
- Getting Caught Up to Modern LLM Research | Samarth Goel3 savers
- All the Transformer Math You Need to Know | How To Scale Your Model3 savers
- The Erotics of Intelligence - Helen Higgins1 savers
- Sparse Attention Post-Training for Mechanistic Interpretability1 savers
- UniTracker: Learning Universal Whole-Body Motion Tracker for Humanoid Robots1 savers
- BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion1 savers
- ravens' site3 savers
- The Principles of Diffusion Models1 savers
- robotic_world_model/scripts/environments/list_envs.py at master · leggedrobotics/robotic_world_model1 savers
- Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics1 savers
- alechelbling.com/UnderstandingIsomap/2 savers
- ACT-1: A Robot Foundation Model Trained on Zero Robot Data | Sunday | The helpful robotics company6 savers
- Chapter 1 An Informal Introduction ‣ Learning Deep Representations of Data Distributions1 savers
- Energy-Based Models1 savers
- The Practitioner’s Guide to the Maximal Update Parameterization - Cerebras2 savers
- Image Encoding - ueaj - Obsidian Publish1 savers
- All About Rooflines | How To Scale Your Model11 savers
- [M2L 2025] 5.2 Diffusion models - Sander Dieleman - YouTube1 savers
- π*0.6: a VLA That Learns From Experience1 savers
- Awesome tips2 savers
- DYNA-1 Research - DYNA Robotics1 savers
- All About PhD Applications | lucy lai5 savers
- Generative Models and Physics | Yi Zhuang You (UCSD) - YouTube1 savers
- MemER: Scaling Up Memory for Robot Control via Experience Retrieval1 savers
- MetaSDF: Meta-learning Signed Distance Functions1 savers
- NL.pdf7 savers
- PyTorch Cheatsheet1 savers
- The Reality Gap in Robotics1 savers
- A Proof of Learning Rate Transfer under $\mu$P2 savers
- Feature-wise transformations4 savers
- https://cs.stanford.edu/~jsteinhardt/ResearchasaStochasticDecisionProcess.html31 savers
- Tips for Empirical Alignment Research — AI Alignment Forum15 savers
- bell labs14 savers
- Diffusion is spectral autoregression – Sander Dieleman13 savers
- How to Keep Winning11 savers
- Functions are Vectors9 savers
- Generative modelling in latent space – Sander Dieleman6 savers
- Q-learning is not yet scalable6 savers
- Understanding Memorization via Loss Curvature5 savers
- The Continual Learning Problem5 savers
- torch.compile, the missing manual - Google Docs3 savers
- arxiv.org/pdf/2511.085442 savers
- DataRater: Meta-Learned Dataset Curation2 savers
highlights · 109
To some extent, the notion of interpolating between tasks using FiLM parameters can be applied even in the visual question-answering setting
Feature-wise transformationsany convex combination of FiLM parameters is likely to correspond to a meaningful parametrization of the FiLM-ed network
Feature-wise transformationsThese findings suggest that the affine transformation in FiLM layers is not modulated in a single, consistent way, i.e., using 𝛾 γ only, 𝛽 β only, or 𝛾 γ and 𝛽 β together in some specific way
Feature-wise transformationsFirst, FiLM parameters cluster by feature map in parameter space, and the cluster locations are not uniform across feature maps
Feature-wise transformationsOne way to visualize the FiLM parameter space is to plot 𝛾 γ against 𝛽 β, with each point corresponding to a specific task description and a specific feature map.
Feature-wise transformationsthere is nothing preventing us from considering a neural network’s activations themselves as conditioning informatio
Feature-wise transformationsstyle prediction network, trained jointly with the style transfer network to predict the conditioning parameters directly from a given style image.
Feature-wise transformationsLike conditional batch normalization discussed in the previous subsection, conditional instance normalization can be seen as an instance of FiLM where a FiLM layer replaces the post-normalization feature-wise affine transformation.
Feature-wise transformationsAs hinted earlier, conditional batch normalization can be viewed as an instance of FiLM where the post-normalization feature-wise affine transformation is replaced with a FiLM layer.
Feature-wise transformationsStrub et al. later on improved on the model by using an attention mechanism to alternate between attending to the language input and generating FiLM parameters layer by layer.
Feature-wise transformationsThe model’s linguistic pipeline is a FiLM generator which extracts a question representation that is linearly mapped to FiLM parameter values
Feature-wise transformationsview the concatenation of all FiLM scaling and shifting coefficients as both an instruction on how to modulate the conditioned network and a representation of the task at hand.
Feature-wise transformationsFrom the perspective of multi-task learning, we can view the conditioning signal as the task description
Feature-wise transformationsBy feature-wise, we mean that scaling and shifting are applied element-wise, or in the case of convolutional networks, feature map -wise
Feature-wise transformationsNote that the FiLM parameters are parameters in one network but predictions from another network, so they aren’t learnable parameters with fixed weights as in the fully traditional sense
Feature-wise transformationsThese layers are parametrized by some form of conditioning information, and the mapping from conditioning information to FiLM parameters (i.e., the shifting and scaling coefficients) is called the FiLM generator
Feature-wise transformationsLastly, these transformations only enforce a limited inductive bias and remain domain-agnostic
Feature-wise transformationsAlso, in practice, feature-wise transformations (often compounded across multiple layers) frequently have enough capacity to model complex phenomenon in various settings.
Feature-wise transformationsbut feature-wise interactions often strike a happy compromise between effectiveness and efficiency:
Feature-wise transformationshey leverage feature-wise interactions between the conditioning representation and the conditioned network.
Feature-wise transformationsadditive interactions is that they are more natural for applications that are less strongly dependent on the joint values of two inputs, like feature aggregation or feature detection
Feature-wise transformationsmultiplicative interactions is that they are useful in learning relationships between inputs, as these interactions naturally identify “matches”
Feature-wise transformationsIntuitively, this gating allows the conditioning information to select which features are passed forward and which are zeroed ou
Feature-wise transformationsA special instance of conditional scaling is feature-wise sigmoidal gating:
Feature-wise transformationsconditional scaling, i.e., scaling hidden layers based on the conditioning representation
Feature-wise transformationsInterestingly, conditional biasing can be thought of as another way to implement concatenation-based conditioning
Feature-wise transformationsvia conditional biasing, namely, by adding a bias to the hidden layers based on the conditioning representation.
Feature-wise transformationsIn this case, we would be forcing the model to carry this information around unaltered for many layers
Feature-wise transformationsHowever, this approach makes the implicit assumption that the input is where the model needs to use the conditioning information
Feature-wise transformationsTraining a NN as a surrogate optimization solver amounts to estimating a global solution function that maps varying problem input parameters to the corresponding optimal solutions
A Neural Network Framework for Discovering Closed-form Solutions to Quadratic Programs with Linear Constraintshave difficulty guaranteeing solution optimality and feasibility
A Neural Network Framework for Discovering Closed-form Solutions to Quadratic Programs with Linear ConstraintsA quick fix would be to concatenate a representation of the conditioning information to the noise vector and treat the result as the model’s input.
Feature-wise transformationsthe computation carried out by a model is conditioned or modulated by information extracted from an auxiliary input.
Feature-wise transformationsit often makes sense to process one source of information in the context of another
Feature-wise transformationsVQGAN and KL-regularised latents
Generative modelling in latent space – Sander Dielemanventure capitalists would look at our competitors and see that they were growing faster or making superficially more progress, and they'd ask us "why not just do this or that?" But it was never that simple. I wanted to build real technological depth. Something lasting. If it's easy, then there is no real lasting advantage
How to Keep WinningIf you've set your mind to something, then why ever quit? If you're not dead, then you're still in the game.
How to Keep WinningOnce you're deeply familiar with the death conditions, you can take extreme risks because you know you'll always come back when things go sideways.
How to Keep Winningan unscaled KL term has too strong an effect, imposing a stringent limit on latent capacity and thus severely degrading reconstruction quality.
Generative modelling in latent space – Sander DielemanYet almost invariably, the KL term used to regularise continuous latent spaces is scaled down significantly
Generative modelling in latent space – Sander Dielemanencourages the latents to follow the imposed prior distribution (usually Gaussian).
Generative modelling in latent space – Sander Dielemanbounds the likelihood from below and enables VAE training to tractably (but indirectly) maximise the likelihood of the data
Generative modelling in latent space – Sander DielemanKL regularisation, on the other hand, is a core part of the traditional VAE setup
Generative modelling in latent space – Sander Dielemanthis probably isn’t really necessary in most cases, because encoder expressivity is usually the limiting factor
Generative modelling in latent space – Sander Dielemanthe quantisation during training serves as a safeguard to ensure that the latents don’t end up encoding too much information.
Generative modelling in latent space – Sander DielemanRemove the quantisation step from the VQGAN recipe altogether, and replace it with a KL penalty, as in regular VAEs
Generative modelling in latent space – Sander Dielemanreinterpreting the quantisation step as part of the decoder, rather than the encoder, to get a continuous representation
Generative modelling in latent space – Sander Dielemanshaping the latent space
Generative modelling in latent space – Sander Dielemancurating the latent space
Generative modelling in latent space – Sander DielemanIt is just as important to control precisely which bits from the original input signals should be preserved in the latents, and how this information is presented
Generative modelling in latent space – Sander Dieleman