memset0
1 followers · 2 following · 248 views
on the atlas — 4
- (6 封私信 / 3 条消息) Speculative Speculative Decoding (SSD) - 知乎1 savers
- Forge: Scalable Agent RL Framework and Algorithm - MiniMax News | MiniMax1 savers
- (6 封私信 / 3 条消息) On-Policy Distillation 与 Self-Distillation：七篇论文的阅读笔记 - 知乎1 savers
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning23 savers
highlights — 12
标准 EAGLE 依赖验证后的目标模型激活值进行条件预测。 SSD-EAGLE-3 在目标激活值尚未产生时，使用草稿模型自身的激活值进行“自调节推测”（Self-conditioning）。
(6 封私信 / 3 条消息) Speculative Speculative Decoding (SSD) - 知乎提前准备：当验证器正在处理第 T 轮的令牌验证时，推测器并不闲置，而是提前预测第 T+1 轮可能的验证结果。 推测缓存 (Speculation Cache)：草稿模型为多种可能的验证路径（包括接受令牌数及后续产生的 Bonus Token）预先生成投机序列，并存入缓存
(6 封私信 / 3 条消息) Speculative Speculative Decoding (SSD) - 知乎利用一块额外的GPU，掩盖了draft model的推理，并通过fan-out cache预测target model的接受结果。文章还提到了和现在热门的eagle方案的兼容，取得了不错的效果。或许未来DT分离（Draft/Target分离，not David Tao）会成为新的LLM推理范式。
(6 封私信 / 3 条消息) Speculative Speculative Decoding (SSD) - 知乎This prompts the emergence of robust reasoning patterns that inherently prioritize "state-critical" tokens.
Forge: Scalable Agent RL Framework and Algorithm - MiniMax News | MiniMaxWe model CM as an explicit agent action, with context transitions naturally embedded within the environment's dynamics. The state transition from St to St+1 implicitly encapsulates the context-switching logic, effectively folding context adaptation directly into the model's training objective.
Forge: Scalable Agent RL Framework and Algorithm - MiniMax News | MiniMaxTo resolve this distribution shift and maintain reasoning fidelity, we integrate the CM mechanism directly into the RL interaction loop, effectively treating Context Management as a functional action that drives state transitions:
Forge: Scalable Agent RL Framework and Algorithm - MiniMax News | MiniMaxThis discrepancy forces the model to abruptly adapt to unexpected context transitions and process unfamiliar long-context structures on the fly, ultimately degrading its overall performance.
Forge: Scalable Agent RL Framework and Algorithm - MiniMax News | MiniMaxThe Challenge of Context Rot: As the number of interaction turns increases, the accumulation of intermediate reasoning steps and redundant observations creates an "attention dilution" effect. This accumulated noise causes the model to lose focus on critical information, even when operating strictly within its absolute context window limits.
Forge: Scalable Agent RL Framework and Algorithm - MiniMax News | MiniMaxIt serves as a standardized communication gateway that processes completion requests between the agent and the LLM. By utilizing common standard protocols, this server effectively isolates the complexities of the actual underlying model from the agent's high-level behavioral logic.
Forge: Scalable Agent RL Framework and Algorithm - MiniMax News | MiniMaxAgent Side: This layer abstracts the General Agent—comprising both white-box and black-box architectures—and its operational environment. It orchestrates recursive environmental interactions, allowing the Agent to function as a pure trajectory producer.
Forge: Scalable Agent RL Framework and Algorithm - MiniMax News | MiniMax。GKD 的理论起点正是把对齐位置从“数据/教师分布上的前缀”移动到“学生当前分布上的前缀”，即让学生先 roll out，再由教师给这些轨迹上的 token 提供监督，从而达到 on-policy 的效果
(6 封私信 / 3 条消息) On-Policy Distillation 与 Self-Distillation：七篇论文的阅读笔记 - 知乎七篇工作的共同主线可以表述为一句话：把“学生真实会经历的状态分布”与“教师提供的密集结构化信号”对齐。关键分歧在于是否忽略策略梯度、如何构造教师、以及是否允许对教师增益进行外推。GKD 强在统一与稳健，MiniLLM 强在目标忠实性，G-OPD 强在可调超越机制，OPSD/SDFT/SDPO 强在把外部教师需求转化为条件化 Self-Distillation
(6 封私信 / 3 条消息) On-Policy Distillation 与 Self-Distillation：七篇论文的阅读笔记 - 知乎