Building Amit.AI, Part 2: The Backend

Part 1 was the browser. This is the other half: a small FastAPI service that answers questions about me, as me, from facts I wrote down.

Same rules as before: no keys, no limits, no secrets, just the interesting decisions.

Here is the whole trip of one question again. This post covers everything from the BACKEND box down:

1. Python, for the TypeScript brain

I write Python and TypeScript, and this service sits right on the overlap. If you come from JS/TS, here's the map I'd hand you for the pieces that show up in this project:

FastAPI plus Pydantic is a great fit for this kind of service. A route is a typed function, request validation reads almost like a Zod schema, and async streaming is a generator that yields chunks. The framework does the rest.

2. No vector database

Every AI tutorial wants you to chunk your documents, embed them and store them in a vector DB. I came very close to building all of it.

Then I counted. My whole professional background, as Markdown, comes to about 3.3k tokens. That fits in a prompt with room to spare.

So the "retrieval" step is: read every file at startup, glue them together, and put them in the system prompt on every request. The model sees everything, which makes it more accurate than retrieval would be, because there are no retrieval misses.

knowledge/
├── profile.md
├── experience.md
├── projects.md
├── skills.md
└── education.md

A few details I'm happy with:

• Files are loaded once at startup, not per request.
• Years of experience aren't hardcoded. The files contain a placeholder like {years_since:YYYY-MM} that gets resolved to a number when the server starts, so my "years of experience" never goes stale.
• The loader sits behind a small interface, so swapping in real retrieval later changes one file. The trigger for that is clear: when I start writing long blog posts and want the bot to know them. Writing these posts is slowly walking me towards the thing I skipped.

Cutting scope felt like cheating at first. It wasn't. Every component had to earn its place.

3. The prompt is where the work is

The system prompt does the job a lot of code usually does. The knowledge goes first in a delimited block, then the rules. Without quoting it, here is what it covers:

• Identity. It speaks as me, in first person, but admits it's a virtual version if asked directly. It must never pretend a human is typing live.
• Pronouns. "What's his stack?" means "your stack", because on this site there is only one "he".
• Grounding. Answer from the knowledge block only. If it isn't there, say so plainly and point to the contact page.
• No promotions. This is my favourite rule. The model must not upgrade my verbs. If my notes say I "contributed", it can't say I "led". LLMs love flattering their subject, and I don't want a chatbot with a better résumé than mine.
• Scope. Only my professional background. No homework, no trivia, no "write me a poem about React hooks".
• Style. Short, plain text, no markdown, no emojis, answer first.

Two lessons from tuning it:

1. Rules need examples. "Say you don't know" made the model too eager to give up on questions it could answer. Telling it to check related facts first (for a relocation question, answer from my location and job preferences) fixed that.
2. Format instructions can sneak around scope. If someone asks for something out of scope "as a table", the model may decide the table makes it fine. So the prompt says outright that a requested format never widens the scope.

4. Don't trust the prompt alone

A prompt is a polite request. I'd rather not build security on politeness.

The server runs its own checks before the model is ever called:

• Strict request validation. Message count, message size and total size are capped, and the roles a client may send are limited, so nobody can sneak in a "system" message. Pydantic validators make this almost boring.
• Guards that run before the LLM. Obvious jailbreak phrasing and requests for private details get a canned reply, and the model never sees them. Cheaper, faster and more reliable than asking it nicely to refuse.
• Rate limiting, an origin allowlist and a budget mindset, because anything that calls a paid or free-tier model needs a seatbelt.

None of this is exotic. The point is layering: input checks, then guards, then the prompt, with the model as the last line of defence rather than the only one.

5. A gateway for free-tier reality

I'm running on free tiers. The thing nobody tells you is that the daily token limit, not the request limit, is what you hit first.

The math was sobering. Each request carries the full knowledge base plus instructions plus history, so it's a few thousand tokens. A free model's daily token allowance divided by that is a surprisingly small number of conversations per day. This is also why Tier 0 matters so much: every FAQ answer served locally is a call I didn't spend.

So I built a small gateway that holds an ordered list of models and tries them in sequence:

async def stream(self, messages):
    for provider in self.providers:
        if self.is_cooling(provider):
            continue

        started = False
        try:
            async for chunk in provider.stream(messages):
                if not started:
                    started = True
                yield chunk
            return
        except UpstreamError as error:
            if started:
                raise  # tokens already reached the user, can't switch now
            self.record_failure(provider, error)

    raise UpstreamError("All LLM providers failed or are cooling down")

Three ideas in there:

• Fall back only before the first token. Once a word has streamed to the visitor, you can't swap models mid-sentence. After that, the stream just ends with an error.
• Cooldowns. When a model answers "slow down" (HTTP 429), it goes on a skip list for as long as it asked, so I stop hammering it.
• Every provider speaks the same dialect. They're all OpenAI-compatible endpoints, so one small class talks to all of them, and changing a model is a config change, not a code change.

Two surprises worth sharing:

• Hidden "thinking" counts against your token cap. One fallback model burned its entire output allowance on invisible reasoning and stopped after a few words. Those models now get more headroom, and the answers stay short because the prompt asks for it.
• A truncated answer is an error, not a result. If the model gets cut off by the token limit, that counts as a failure. It is never cached, and before the first token it triggers the next model.

6. Caching, but only where it's safe

Repeat questions are common ("what's your stack?"), so there's a small in-memory cache. It's a bounded LRU with an expiry, keyed on the lowercased, whitespace-collapsed question.

The rule that matters: only first-turn questions are cached. Later turns depend on the conversation, so "tell me more" must never get a stale answer from someone else's chat. Only complete answers are stored, and guard replies and errors never are.

One tiny detail I like: the answer is stored before the final "done" event is sent. If a client disconnects right after the last token, the generator can be closed before the next line runs, and the cache write would be lost.

7. Seeing what happens

Each request ends with a single structured log line: outcome, which model answered, total latency, time to first token. Visitor IPs are hashed, so I can spot one noisy client without storing who they are.

Which model answered a given request is tracked with a ContextVar. The gateway is shared by every request, so it can't just store that on itself. It's the same problem AsyncLocalStorage solves in Node, so if you've met one, you've met the other.

What I'd tell past me

• Count your tokens before you reach for a vector database. Mine fit in a prompt.
• Defer instead of guessing, on both the matcher and the model.
• Put cheap checks in front of expensive calls. Guards, the matcher and the cache all exist to save a model call.
• Free tiers are a design constraint. The gateway, the cache and the matcher exist because of them, and the architecture got better for it.
• Write the "what I deliberately did not build" list. Mine has a written trigger for each item. It stops you gold-plating.

Still on the list: evals for the answers, tracing, and shared rate limits. Probably the next post.

Go try it. Ask it something about me. Then try to break it. I'd genuinely like to know what you find.