How AI Sees an Image? Everything Is a Number

How does a computer that has never seen a cat in its life learn to recognise one? How does a machine without eyes “see” anything at all?

A few weeks ago the answer reached me from a completely different direction. I realised that music can be written down as a string of numbers. A digital recording is, at its core, sound samples turned into numbers; an MP3 file adds compression on top of that, but the foundation is the same. Nobody thinks twice about this today, because it is obvious.

But if music is numbers, why would an image be any different? A photo, a scanned invoice, a video frame, it is all the same trick, just delivered in a different form.

And that is exactly the mechanism behind how AI “sees”. No magic, just numbers and a few steps, which I will break down in a moment.

An Image Is a Table of Numbers

A photo on a computer is a grid of tiny squares, called pixels. To us, a pixel is a colour. To a computer, it is a set of numbers.

A pixel in a standard image has three numbers: how much red, how much green, how much blue. Each from 0 to 255.
A white pixel is (255, 255, 255). Black is (0, 0, 0). Pure red is (255, 0, 0).
An ordinary phone photo has over two million pixels. Multiply that by three numbers per pixel and you get several million numbers in a single file.

A computer never opens that photo and thinks “cat”. It gets a table of several million numbers and has to extract something from it.

A Sieve That Extracts Meaning

That table of numbers means nothing on its own. So a neural network passes it through a series of layers, like a set of sieves with progressively finer mesh.

The first sieve catches only the simplest things: places where a light pixel sits next to a dark one. That is enough to pick out edges, lines, contours. At this stage the network does not yet know whether it is looking at a cat, an invoice, or a bolt on a production line. It only sees where something changes in the table of numbers.

The second sieve takes those edges and assembles them into simpler shapes, arcs, angles, repeating patterns. The third combines shapes into larger structures, for example something resembling an ear, an eye, the corner of a document, or a scratch on a surface. The deeper the layer, the more abstract what the network detects becomes. The last sieve no longer looks at pixels at all, only at combinations of combinations.

This describes the classic mechanism, convolutional networks, on which most systems in everyday use have learned to recognise images for years. Newer models, including the ones behind today's AI assistants, arrive at a similar result by a different route, but they still start from a table of numbers and end with a pattern that can be recognised.

That is the whole trick. No single sieve “understands” the image, and the network as a whole does not understand anything in a human sense either. But the ability to tell patterns apart emerges from the fact that the output of one sieve becomes the input for the next.

A Fingerprint

After the series of sieves, something quite specific is left: a characteristic pattern of numbers, a bit like fingerprints that let you identify a person, even though they are not formally mathematically unique. The network does not, however, keep millions of photos in memory that it could search through for every decision. What it has learned is encoded in its weights, that is, in the settings of the sieves themselves, worked out during training.

A new pattern simply lands on one side or the other of the boundary those settings define. The network does not “know” that it is a cat. It only knows that this particular pattern landed on the side of the boundary that training associated with the label “cat”. Everything else, the decision, the classification, the label, is just a name given to which side of the boundary the pattern landed on.

It is worth remembering this, because if a document or photo leaves such a pattern in a system, the question of where it ends up and who has access to it becomes a matter for GDPR and the AI Act, not just a technical curiosity.

How the Network Knows What a Cat Is

Where do these patterns associated with the label “cat” even come from? In the simplest and most common case, someone labelled them by hand beforehand, though it also happens that a system learns without explicit labels, picking out regularities on its own from large amounts of data. Let us stick with the simpler scenario: you show the network thousands of photos captioned “cat” and thousands without that caption. At first the network guesses at random and gets it wrong almost every time.

After every mistake, the network slightly adjusts the settings of its sieves, that is, which edges, shapes, and patterns it pays attention to. It does not do this consciously. It is pure mathematics: check the error, nudge the settings in the direction that reduces that error, repeat. Millions of times.

After enough repetitions, the sieves themselves “learn” to pick out exactly the features that distinguish cats from the rest of the world, whiskers, ear shape, muzzle proportions. Nobody programmed this in advance. It emerged from the data and the corrections alone.

This also explains why a network sometimes gets things wrong in a way that seems absurd to a person, for example mistaking a muffin for a chihuahua. The network does not know the concept of “dog” or “cake”. It only knows patterns of numbers that statistically matched a label in the training data. If the data was too narrow or skewed, the pattern it learned will only work up to the first case that does not fit it.

The Same Trick, Everywhere

Exactly the same process happens when a system reads a scanned invoice or looks for a scratch on a part coming off a production line. A scanned invoice is also a table of numbers, the first sieve catches the edges of letters and table lines, the next assembles them into the shapes of digits and words, and the last recognises a pattern associated with the label “invoice number” or “amount due”. This is a simplification, because in practice it is usually several separate steps: reading the text, recognising the page layout, and only then assigning values to the right fields, but each of them works on the same principle.

Quality control on a production line works in a similar way, except the network learns the pattern that corresponds to a scratch, a dent, or a missing screw, instead of a letter or a cat. The input data changes, and so does what the network looks for. The mechanism stays the same.

Even video is no exception here, it just adds one dimension. A video frame is exactly the same table of numbers as a photo, except there are dozens of these tables every second, lined up one after another. The network therefore gets not a single set of numbers but an entire sequence, and can additionally notice how the pattern changes over time, for example that something is moving or disappearing from the frame.

What This Means in Practice

This mechanism has consequences if you are evaluating or deploying such a system, not just reading about it out of curiosity.

Training data quality is output quality. A network does not learn concepts, it learns patterns present in the examples it was shown. A narrow or biased dataset produces a narrow or biased model.
A model matches patterns, it does not understand content. That is why there is always an edge case, a muffin that looks like a dog, an invoice in an unusual format, a defect nobody labelled before. It is worth assuming from the start that such cases will turn up.
A result is a confidence score, not a guarantee. Systems return a number showing how “confident” the model is in its answer, and that confidence can be misleading, a model can be very convinced of a wrong answer. Where a mistake is costly, someone ultimately needs to be able to challenge the machine's decision.
A network is rarely trained from scratch. In practice it is more common to take a ready-made model that has already learned to recognise general shapes and patterns, and fine-tune it on your own, much smaller dataset. It is usually cheaper and faster, and the result is often just as good.

Summary

I am back at the starting point. Music is numbers, an image is numbers, a document is numbers. Once you see this, it is hard to go back to thinking of AI as something that “looks” or “understands” in a human sense. It is a very well trained way of sifting through numbers until a pattern is left that can be matched to something.

That does not make AI any less useful. It makes it something you can understand. And what you can understand stops being frightening.