Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
Sentiment Mix
Geography
Expert Signals
spidy__
author • 1 mention
Hacker News
source • 1 mention
AI-Generated Claims
Generated from linked receipts; click sources for full context.
Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB.
Supported by 1 story
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files.Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte.
Supported by 1 story
Those predictions are fed into an arithmetic coder to produce the compressed output.On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte).
Supported by 1 story
On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte).It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my AMD 7800XT).Checkout the repo - https://github.com/samyak112/pym-particles
Supported by 1 story
Related Events
Anthropic says Alibaba must be punished for largest Claude cloning attack - Ars Technica
LLMs • 6/26/2026
Google revamps AI coding team to catch Anthropic - Latest news from Azerbaijan
LLMs • 6/25/2026
'Bringing Claude Tag into Slack is about making AI multiplayer': You can now tag Claude directly in Slack - TechRadar
LLMs • 6/25/2026
Anthropic Urges Congress to Crack Down on AI Distillation By Chinese Rivals - Yahoo
LLMs • 6/25/2026
Meta Prediction Market App Puts Llama in Charge of Deciding What Is True - Tech Times
LLMs • 6/25/2026
Causality Chain
Preceded By