Every time someone in an architecture committee says "let's throw AI at it," the first thing I ask is: what specific problem does it solve, and what happens when it gets it wrong? It's not a rhetorical question. It's what separates a serious architecture decision from a passing trend.
You don't need to be a machine learning engineer to take part in that conversation. But you do need to grasp a handful of concepts that, in my experience, almost nobody explains assuming you don't know what they're talking about. Here they are, no formulas.
A model doesn't "know" anything, it calculates probabilities
A trained model is a function with a huge number of parameters adjusted from data. When you give it an input, it doesn't "think" or "remember" in the human sense: it calculates the most probable output based on the patterns it saw during training. This matters for architecture because it changes the question you should be asking yourself. It's not "is the model intelligent?", it's "how representative were the training data relative to the cases I'm going to face in production?". A model trained with 2023 data doesn't know that a new policy you published yesterday exists, and there's no way for it to intuit that.
Training vs. inference: two completely different costs
Training a model is expensive, slow, and happens (almost always) once or on a periodic basis. Inference is running the already-trained model against a new input, and it's what happens every time a user makes a query. As an architect, the cost that's going to hurt you every month is the inference one: latency, compute, and in the case of external providers, the price per token or per call. Designing for AI without modeling that recurring cost is the same mistake as designing an architecture without modeling I/O cost — it looks great in the prototype and hurts on the production bill.
LLMs in particular: tokens, context, and why they hallucinate
A language model doesn't process words, it processes tokens (text fragments). It has a limited context window: everything that doesn't fit in that window, the model simply doesn't "see." And when it doesn't have the information it needs to answer, it doesn't always stop to say "I don't know" — it generates the statistically most plausible continuation, whether it's correct or not. That's a hallucination: it's not an isolated bug, it's a direct consequence of how the mechanism works. Any system that puts an LLM in the critical path of a decision (financial, medical, legal) without an external verification mechanism is gambling, not architecting.
RAG: the pragmatic answer to outdated knowledge
Retrieval-Augmented Generation is, in essence, search first and generate later: instead of relying on what the model memorized during training, you inject relevant, up-to-date information (pulled from your database, your vector search engine, your documents) into the prompt so it uses it as context. It doesn't make the model smarter, but it reduces the "this is no longer true" problem and gives you something much more valuable from an architecture standpoint: traceability. You can show where the answer came from.
What an architect should decide (and what they shouldn't)
It's not your job to tune hyperparameters or choose the neural network architecture — that's what the ML team is for, if you have one, or the model provider if you're consuming it as a service. What you are responsible for, and nobody else will do for you, is defining the system's boundaries: what happens when the model fails or answers poorly, how a decision that involved AI gets audited, what data goes into the prompt and what should never go in, and whether the use case can tolerate a 2%, 10%, or flat-out zero error rate.
AI doesn't change the fundamentals of software architecture. It's still about designing for failure, defining clear boundaries between components, and being honest about what the system can and can't guarantee. It's just that now one of those components, instead of failing deterministically, fails probabilistically — and that is, perhaps, the only genuinely new concept you actually need to internalize.
