Google released a tiny Gemma model recently, Gemma 3 270M. What’s interesting about the size of this model is that out of 270 million parameters, 170 million of them are embedding vectors, and only 100 million parameters are for transformer blocks. It is uncontroversial to say… a lot of information is encoded in these embedding vectors.

So, naturally, I want to take a quick look at these vectors from an interpretability perspective. The easy target is digit embeddings. Unlike Llama models, Gemma models simply split numbers into digits. For example, the number 123 is split into three tokens: 1, 2, and 3.

This makes things simple for us: we have 10 tokens to represent the 10 digits.

Here is how we get these 10 token embeddings:

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-270m")
model = AutoModelForCausalLM.from_pretrained("google/gemma-3-270m")

# Extract embeddings for digits 0-9
digits = "0123456789"
digit_tokens = tokenizer.encode(digits, add_special_tokens=False)
embeds = model.model.embed_tokens.weight
digit_embeddings = embeds[digit_tokens].detach().numpy()

Then, I do a simple PCA projection of these embeddings to a 3D space. Here are the results:

Click the button in the top left corner to see the plot in fullscreen.

Playing with the visualization a bit, we can see that the number zero is separate from the others (unsurprisingly), and the other numbers seem to form a circle in the natural order of the digits.

How about plotting the words zero, one, …, nine as well? I did that and here are the results:

Click the button in the top left corner to see the plot in fullscreen.

Notice how the word embeddings are (somewhat) parallel to the digit embeddings and follow the same structure as the digit embeddings.

The complete source code for generating these visualizations is available: