TokenMaxxing — Getting More Out of AI

// 00 — the foundation

It's just finding a function.

Before any of the neural net machinery makes sense, what is the underlying idea? In nature inputs give outputs they adhere to a pattern. We want to reverse-engineer the function that drives it.

The case you already know

Say you're characterising a temperature sensor on your gateway board. You know from physics it follows a polynomial. You collect a few (temperature, voltage) pairs and solve for the coefficients. Done.

// Known shape — solve analytically y = a x² + b x + c // data pairs (x₁,y₁), (x₂,y₂), (x₃,y₃) \to find a, b, c coefficients = weights

This works perfectly — when you know the shape upfront. The problem is, most real systems don't give you that luxury.

The case nobody can solve analytically

What's the function that maps pixels to "stop sign"? Nobody knows. You can't solve for coefficients if you don't know what kind of equation you're dealing with.

Universal Approximation Theorem

A neural network with enough neurons can approximate any continuous function, regardless of shape. You don't need to know the equation. You need data pairs and enough weights to tune. The weights are the coefficients of an equation you never had to write.

Two ways to fit the same data

function_approximation.demo — click to add data points

Click canvas to add points, then fit

Every ML method is a different guess about the function shape

That guess is called an inductive bias. Neural nets make the least opinionated guess — which is why they're the most flexible, and why they need the most data.

📐

Linear / Poly

Assumes a known shape. Solve analytically. Works when the assumption holds — sensor calibration, simple regressions.

y = mx + c

🌳

Tree / SVM

Assumes piecewise or linearly separable. Good for structured tabular data with clear decision boundaries.

if x > θ \to class A

🧠

Neural Net

Assumes very little. Stacks parameterised transformations. Learns the shape. Needs more data and compute.

f = σ(Wₙ\cdot\dotsσ(W₁x))

🔲

Transformer

Neural net with one extra assumption: any part of the input can relate to any other part. That's attention.

structure + learning

To connect all of this

A neural network is a polynomial with millions of terms and a smart structure, whose coefficients are found not by solving equations but by gradient descent — the same underlying idea, applied to functions too complex to write down.

// 01 — how it came about

The crash course: from one neuron to deep learning

Three ideas built on each other over 30 years. You don't need to know the math, seeing how they connect will change how you read model behaviour.

PART A — THE NEURON

A single neuron is one weighted sum plus a squish function. Every single thing in a large model is just this — repeated billions of times.

single_neuron.sim

What's happening

// Inputs (e.g. CAN frame features) x₁ = 0.8 x₂ = 0.5 // Weights — the coefficients being learned w₁ = 0.6 w₂ = -0.3 b = 0.1 // Weighted sum z = w₁\cdotx₁ + w₂\cdotx₂ + b = 0.43 // Activation: squish to a useful range output = σ(z) = 0.606

The activation function is what lets you stack layers without which, any number of linear layers collapses to one and we lose non-linearity.

Sigmoid σ

1 / (1 + e -z) \to 0 to 1 \cdot probabilities

ReLU

max(0, z) \to 0 to \infty \cdot fast, most used

Tanh

(e z -e -z) / (e z +e -z) \to -1 to 1 \cdot zero-centred

PART B — HOW IT LEARNS: BACKPROPAGATION

After a wrong prediction, the network works backwards through every connection asking: "how much did you contribute to this mistake?" Then nudges each weight a small amount. Repeat that a million times.

step 0 / 4

Ready

// Click Forward pass to begin // Watching a single neuron learn: input x=1.0, target y=1

One neuron. Input x=1.0. Target output=1.0. Starting weight=0.4. Let's watch it learn.

Loss function

Measures how wrong the prediction is. Like a score you're trying to drive to zero.

L = (ŷ - y)² // ŷ = predicted, y = actual

Gradient Descent

Step in the direction that reduces the loss. Learning rate η controls how big a step.

w = w - η \cdot \partial L/ \partial w // repeat millions of times on millions of examples

PART C — SCALE: THE DEEP NETWORK

Stack neurons into layers. Each layer learns a progressively more abstract representation of the input. While this is by design, nobody designs it explicitly.

deep_network.sim — fire a signal through layers

Input layer

Raw features. For a SocketCAN frame: message ID, DLC, the 8 payload bytes, timestamp delta, bus ID. Each value = one neuron.

Hidden layers

We define it, final state is learnt over training. Layer 1 might learn byte-level patterns, layer 2 sequences, layer 3 protocol-level abstractions. Training decides.

Output layer

The final answer. Classification (fault / normal / unknown) or regression (predict next value). Softmax turns it into probabilities.

Loss goes down as training runs

training_loss.plot

// 02 — what's new

LLMs: the same network, at a scale that changes everything

Claude, GPT-4, Gemini — all the same architecture underneath. A deep network. What changed is the data (text instead of labelled images), the scale (hundreds of billions of weights), and one key invention: attention.

tokenizer.demo — your text becomes numbers before the model sees it

The model doesn't read words. It reads tokens — subword pieces chosen by a compression algorithm run before training. Common words stay whole. Rare or compound words get split. This is why rare technical terms in your prompts often get fragmented — and why being specific still beats being vague.

From tokens to answer — one transformer pass

This happens once per token generated. In a 200-token response, this pipeline runs 200 times.

Tokenstext → integer IDs

EmbeddingsID → 4096-dim vector

Attentionwhich tokens matter?

FFNtransform each token

×96 layersrepeat deeply

Softmaxpredict next token

Attention — the part that makes transformers different

Before transformers, networks processed tokens in order — like reading one character at a time with no ability to look back. Attention lets every token look at every other token simultaneously and decide how relevant each one is. "CAN" can directly attend to "driver" and "Linux" in one step, regardless of distance.

attention_heatmap.viz — hover a row

              Hover a row to see what that token attends to — and why.
            

What this means for you

"CAN" and "FD" attend strongly to each other because the model has seen "CAN FD" as a compound concept billions of times in training. Your entire prompt is processed this way — all tokens attending to all others simultaneously. This is why a richer, more specific prompt gives the model more signal to attend to, not more noise to filter through.

The honest framing

The model doesn't understand your code the way you do. It has learned statistical co-occurrence patterns across trillions of tokens of code, docs, specs and discussion. It is an extremely powerful pattern completer. That's not a limitation — it's the mechanism. Knowing this is how you use it better.

Demo network (above)

parameters: ~50 layers: 3 training: seconds on CPU purpose: simple DNNs

GPT-2

parameters: 1.5B layers: 48 training: weeks on GPUs purpose: coherent language

Claude / GPT-4 class

parameters: ~100B+ layers: 100+ training: months, 10k+ GPUs purpose: general reasoning

// 03 — putting it to work

Using it better

Everything that follows is drawn from using these tools on actual SocketCAN driver work. The patterns that actually changed the output quality, or at least it seemed it did.

First: pick the right tool for the task

These tools all run LLMs under the hood. The real differences are context window size, how they integrate into your workflow, and how much autonomy they're given. Difference can be significant.

Tool	Model	Where it lives	Context	Best use in embedded/CAN work
GitHub Copilot	GPT-5 / Claude / Gemini (picker)	VSCode, JetBrains, CLI, web	64k–128k tokens	Chat agent boilerplate struct definitions, ioctl patterns, register macros. Fast, low friction. Model picker lets you swap to Claude or GPT-5 per task.
Cursor	Claude / GPT / Gemini / DeepSeek (picker)	Full IDE (VSCode fork)	up to 200k tokens (effective less)	refactor codebase-aware Reads your entire driver tree. "How does my CAN filter work?" and it actually knows. Background agents for longer autonomous tasks.
Claude (claude.ai)	Claude Sonnet / Opus	Chat / browser / desktop	~200k tokens	design review deep debugging Paste entire .c files. Strong on SocketCAN API questions, kernel patch review, architecture.
Claude Code	Claude Sonnet / Opus	Terminal + desktop app	up to 1M tokens	autonomous multi-file edits Runs bash, edits files, cross-compiles. "Add CANFD BRS support to this driver" — it does it. Parallel sub-agents for larger tasks.
OpenAI Codex	GPT-5.x-Codex series	CLI, desktop app, web, IDEs	~192k tokens	code gen test harness gen Solid at generating kunit test scaffolding for protocol state logic. Parallel cloud sandboxes per task.

Patterns that actually change the output

1. Give it a project context block

The model doesn't know your toolchain, your kernel version, your hardware, or your team's conventions unless you tell it. A context block pasted at the start of a session pays for itself in the first response. Write it once, save it somewhere, paste it in.

// Project context — paste this at the start of any session
Target: ARM Cortex-A53, Linux 6.1, Yocto
Hardware: TJA1042 transceiver on SPI, dual CAN bus (can0 / can1)
Stack: SocketCAN, socket(AF_CAN, SOCK_RAW, CAN_RAW)
Frame type: CANFD with BRS enabled, 64-byte payload
Conventions: kernel coding style, no dynamic alloc in ISR paths
Build: cross-compile aarch64-linux-gnu-gcc, -Wall -Wextra clean

Every answer that follows is now grounded in your actual environment. Without this, the model interpolates from the average of all Linux driver code it has seen — which might be kernel 5.4 on x86.

2. Front-load what matters, cut filler

Remember: the model is doing weighted attention over your tokens. Filler words dilute the signal. Nouns and verbs — especially domain-specific ones — carry the weight. You don't have to write like a "caveman", but you do want the high-information tokens early.

❌ generic output

"could you help me write a CAN driver please"

✅ actionable output

"SocketCAN CANFD driver, Linux 6.1, TJA1042 SPI transceiver, ARM Cortex-A53. can_rx_handler() drops frames above 8 bytes. [paste code]. Root cause and fix."

Second prompt gives the model hardware, kernel version, exact symptom, and code. The attention mechanism has real signal to work with. Output is precise.

3. Give it something wrong to fix, not a blank canvas

Asking for code from scratch produces average code. Giving the model a flawed implementation and asking it to fix it produces better output — it has something concrete to reason against. This works especially well for ISR safety and error paths.

// Instead of "write me a safe CAN ISR"
"This ISR has a race condition in the flag handling. Here's the code: [paste]. Find it and fix it. Use the appropriate kernel synchronisation primitive for this context."

4. Ask it to reason before it answers

For multi-step debugging and architecture questions, adding "think through this step by step" measurably improves correctness. For simple codegen it adds noise — use it when the problem has hidden steps.

"The bitrate timing values in this CANFD setup look wrong but it compiles. Think through the calculation step by step — nominal 500kbit/s, data 2Mbit/s, 80MHz clock — then tell me where the error is."

5. Ask separately "what am I missing?"

Codegen skews optimistic — it produces the happy path well. Error handling, edge cases, and resource cleanup are almost always underspecified unless you ask for them explicitly. Make it a habit to follow every implementation with a second prompt.

"What error paths or edge cases did you not handle in that implementation? List them. Then fix them."

You will almost always get at least two things the first pass missed — null pointer handling, missing error returns, unhandled -EAGAIN from socket ops.

6. Ask for the test before the code

Describe the behaviour you want. Ask for a kunit test or a user-space mock first. Then ask for the implementation that passes it. This forces the model to make the spec concrete before generating, which surfaces ambiguities early.

"Write a kunit test for a CAN filter function that should: accept standard 11-bit frames matching mask 0x7FF, reject extended 29-bit frames, handle zero-length DLC correctly. Then write the implementation that passes it."

When evaluating a new tool — three questions

🔍

Does it know the domain?

⚡

Does it fit your workflow?

🛡️

Can you verify what it produces?

Mental model

This model has read everything on the internet, It does not think. It predicts the most statistically likely next token given everything in its context window. The richer, more specific, more grounded your context — the more that prediction looks like what an expert would hopefully write. Context quality is the lever. Everything else is technique.