The Beautiful Mathematics Behind Every JPEG

1. A heretical idea about heat (1807)

In 1807 Joseph Fourier, studying how heat flows through a metal plate, presented a startling claim to the Institut de France: any function, even one with sharp corners, can be written as a sum of simple sine and cosine waves. Lagrange, among the greatest mathematicians alive, objected. It seemed absurd that smooth waves could add up to a jagged edge. Fourier was right, and his 1822 book Théorie analytique de la chaleur gave the world a new way to see.

The insight is this: a signal and its frequencies are two descriptions of the same thing. One says "what is the value here?" The other says "how much of each rhythm is present?" Nothing is lost in translation. But the second description can be far more economical, and economy is what compression is all about.

2. The 64 patterns that build every photograph

Applied to an 8 × 8 block of pixels, Fourier's idea becomes concrete. There are exactly 64 elementary patterns, made by multiplying a horizontal cosine wave by a vertical one. Here they are:

Here is the astonishing part. Every possible 8 × 8 image, whether a patch of sky, an eyelash or a letter "A", is a weighted mix of these 64 patterns. Exactly 64 weights, no more, no less. The DCT is simply the recipe that finds those weights.

The patterns are orthogonal: each one is perfectly independent of all the others, like the north, east and up axes of space, just in 64 dimensions. That is why the weights can be computed one at a time, and why the transform loses nothing: by Parseval's theorem, the total "energy" (sum of squares) is exactly the same before and after.

3. The formula

For a block of pixels f(x, y), the weight of pattern (u, v) is:

F(u,v) = ¼ · C(u) · C(v) · Σx Σy  f(x,y) · cos[(2x+1)uπ/16] · cos[(2y+1)vπ/16]

         where C(0) = 1/√2 and C(k) = 1 otherwise; x, y, u, v run from 0 to 7

Read it slowly and it is almost poetry. For each pattern, walk across the block, multiply each pixel by the pattern's value at that spot, and add everything up. If the block "resembles" the pattern, the sum is large. If not, positives and negatives cancel to nearly zero. The DCT is 64 questions of the form "how much do you look like this?"

4. Watch it happen on a real block

Below is an 8 × 8 block of brightness values from a photograph (0 = black, 255 = white). Before transforming, JPEG subtracts 128 so values are centred on zero.

Now the DCT. The shading shows the size of each weight:

Look where the energy went. The single top-left value, the DC coefficient (the block's average brightness), holds 86% of the block's total energy. Just 14 of the 64 coefficients carry 99% of it. The bottom-right region, the fast, fine-grained patterns, is a sea of small numbers.

This is energy compaction, and it is not luck. Neighbouring pixels in real photographs are strongly correlated: the pixel next to a sky-blue pixel is almost always sky blue. Smooth variation lives in low frequencies. The DCT turns that statistical fact into a lopsided list of numbers, and lopsided lists compress beautifully.

5. Where the loss happens and why you don't see it

The DCT itself loses nothing. Compression comes from the next step, quantization: dividing each coefficient by a number from a table and rounding. Here is the standard luminance table published with the JPEG standard:

Notice the shape. Small divisors in the top-left preserve the coarse structure carefully. Large divisors in the bottom-right crush fine detail. This table encodes a fact about human vision: our eyes are far less sensitive to rapid, fine-grained variation than to broad changes in brightness. Mathematics meets biology. Divide and round:

44 of the 64 values are now zero. Read in zig-zag order from the top-left, the long run of trailing zeros is stored as a single "end of block" symbol, and the remaining numbers are packed with Huffman coding. 64 pixel values shrink to a small fraction of their original size.

Reverse the process (multiply back, inverse DCT, add 128) and the decoded block differs from the original by an average of just 4.9 brightness levels out of 255 (largest single error: 15). Side by side:

6. The goosebumps moment: an image assembling itself

Now rebuild the block using only the first few coefficients, in zig-zag order from coarse to fine. Watch the picture emerge from nothing:

One number gives a grey square of exactly the right average brightness. Each further wave adds structure: the bright shape in the centre gathers itself, and by 21 coefficients, a third of the total, the average error has fallen to about 5 brightness levels out of 255. The remaining 43 coefficients add only fine texture. This is exactly what a progressive JPEG does as it loads over a slow connection: it sends the coarse waves first.

7. Why cosines, and not Fourier's full series?

Fourier's original transform treats a signal as if it repeats forever. Chop out an 8-pixel row whose left end is dark and right end is bright, repeat it, and you create an artificial cliff at every seam. Cliffs need many high-frequency waves to describe, which is precisely what we want to avoid.

The cosine transform quietly assumes something smarter: that the row is mirrored at its edges. A mirror image joins itself smoothly, with no cliff and no wasted waves. That one choice is why the DCT packs energy so tightly.

There is deeper beauty still. The theoretically perfect transform for compacting energy, the Karhunen–Loève transform, depends on the statistics of each image and is costly to compute. But for signals where neighbours are highly correlated, as in natural photographs, the fixed, universal cosine transform turns out to be almost indistinguishable from that optimum. A single elegant formula is nearly as good as the ideal custom solution for every photograph ever taken.

8. How compression evolved: a short history

It is a humbling arc. An idea born from the physics of heat waited 150 years for computers, was nearly dismissed as too simple, and then quietly became one of the most-used pieces of mathematics in human history. Billions of photographs a day are, underneath, sums of cosines.

9. What this means when you compress a photo

• The quality slider scales the quantization table. Lower quality means bigger divisors, more zeros and more crushed high frequencies.
• Busy images cost more. Foliage, hair and noise have energy in high frequencies that cannot be zeroed without visible damage.
• Blocking artifacts are 8 × 8 tiles whose fine coefficients were all quantized away, leaving each block a flat or gently shaded square.
• Resizing helps more than you expect. Fewer pixels means fewer blocks sharing the same byte budget. See our 20KB–200KB size guide.

For the full encoding pipeline (colour conversion, chroma subsampling and Huffman coding) read How JPEG Compression Works. Then open the compressor, drag the quality slider, and know that every movement is reshaping a quantization table and choosing which cosines survive.