2 HN users recommended this explainer2 months2024–2025score 2
Years with an HN recommendation, 2007–2026
Recommended as a well-made, digestible introduction to scaling laws and why models cluster along a similar efficiency frontier.
A Welch Labs explainer on neural scaling laws and the compute-efficient frontier, asking why model loss seems to hit a line it cannot cross. It draws an analogy to an ideal gas law for AI.
“see eg for a well made/easily digestable intro” (HN, 2024)
Please someone correct me if I’m wrong, but my understanding of ML/LLMs is that this kind of hand crafting has been tried, but it is easier to train/less finicky to let behavior like this emerge from more data, see [1] “Bitter Lesson” and [2] “Scaling Laws”. MAMBA as an architecture claims to have some significant gains performance wise, but to my knowledge there haven't been any really large models (>~100B params) with open weights/leaked MAMBA architecture been disclosed other than this (7B). As mentioned by other comments, another dimension not to forget is the training data quality. Not only quantity but also quality really matters, is what we are learning more and more with LLMs.. [1] [link] [2] see eg [this video] for a well made/easily digestable intro
Well-made, accessible introduction to scaling laws.
I'm not deep enough in the ecosystem to ingest that paper but it's title reminded me of an interesting video I saw recently: "AI can't cross this line and we don't know why" [this video]
My attempt at a summary: authors characterize the data-compute pareto front (aka how bitter is the lesson, exactly?) For a different perspective, error vs compute, see [this video] and comments (I particularly liked the one about string theorists rediscovering a fundamental theorem in GR decades too late-- rediscovering how to integrate happens in every field, it's nothing to be ashamed of :)
Offers a different perspective on error versus compute.
Because they are hitting Compute Efficient Frontier. Models can't be much bigger, there is no more original data on the internet, so all models will eventually cluster to similar CEF as was described in this video 10 months ago [this video]
Describes the compute efficient frontier that models cluster toward.
Video list and Hacker News comment labels from HackerTube (prompted by @karpathy), released under ODC-By 1.0, built on the open-index/hacker-news archive. Hacker News data is provided by Y Combinator; comments remain the work of their authors. Video titles, channel names and thumbnails belong to their creators and are shown via YouTube. HackerTube Watch is an unofficial edition. Data export of 2026-10-31.
Plays via YouTube's privacy-enhanced embed
5 Hacker News comments
From the HN archive; up to 20 kept per video, favoring different commenters. Recommendations first.
Read more
Citations, criticism and jokes