IT HootClub — AI Community Newsletter

Over the last two weeks, we settled two things. Week 1: what a model actually "knows" is whatever was in its training data, compressed into billions of weights — it doesn't have a filing cabinet, and anything outside that training is a gap. Week 2: on a consumer laptop you can run models for inference, but you can't train them. Training wants a GPU. Inference doesn't.

That brings us to Week 3. We know what a model is, and we know what our hardware can and can't do. Now we need a plan — and we need to know what "better" would even look like before we spend hours trying to get there. This week is about both.

The project, stated plainly

We're going to build a model that actually understands the Milwaukee tech ecosystem — local colleges and their programs, the organizations holding it together, the events, the companies, the people doing the work. A model that, if you asked it what's happening in Milwaukee tech, could answer like someone who actually lives here.

That's a gap in every major model today. Gemini, Claude, GPT — they know Milwaukee exists and can repeat generalities. They don't know MKE Tech Hub Coalition is running Catalyze. They don't know Global AI Milwaukee meets monthly at WCTC. They can't tell you which community college has an AI program and which doesn't. We're going to fix that — on a local, open model, using data we collect ourselves.

The phases

I'm sketching six phases, roughly. Not all of them will go as planned. That's part of the honest story of the series. Some phases may take more than one edition — when that happens, Building Intelligence may skip a week with an explicit note about where we are and what's next.

1. Baseline — measure what every model (cloud and local) produces today for a Milwaukee-specific question. That's this week.
2. Data collection — build the Milwaukee tech corpus. Sources, scraping, cleaning, formatting. Where the work actually is.
3. RAG phase — stand up a retrieval-augmented system on one of the local models. Fast to build, easy to update, no training required. Re-baseline.
4. Fine-tuning phase — take a local model into the WCTC lab, fine-tune on the corpus, use the GPU. Re-baseline.
5. Hybrid — fine-tuned model with RAG layered on top. Final evaluation.
6. Publish — expose it somewhere readers can try. Document what it cost, what it took, and where it fell short.

RAG and fine-tuning aren't competing paths here — they're complementary phases. Fine-tuning shapes how a model thinks and writes about a domain. RAG handles what's new and specific, the things that would otherwise require retraining every week. The real question isn't "which one" — it's "in what order, and why." I'll defend the sequencing as we go.

This week: the bar

Before we can claim any model "got better" at Milwaukee tech, we need to know where every model starts. That's the baseline — and it's this week's Fireside Chat.

We're asking the same question to all six models on the table: the three cloud frontier models (Gemini, Claude, GPT) and the three local models currently on my laptop (Llama 3, Qwen 3, Mistral 3). The question is deliberately grounded and checkable:

"I'm a new software engineer relocating to Milwaukee. Name five local tech events, meetups, or community organizations I should know about in 2026, and briefly describe what each one is for."

We can grade these. Real organizations exist (MKE Tech Hub Coalition, Global AI Milwaukee, Madison AI, WiscoREIA, TechRise Milwaukee, and more). Fake ones will show themselves. Vague ones will read vague. This is our bar. Every future edition of Building Intelligence will be measured against these responses.

Read the responses in this week's Fireside Chat below. My initial read — honestly — was "tough to beat." The local models in particular sounded credible, specific, and useful. If those responses hold up under scrutiny, this whole project might be harder to justify than I thought going in.

But that's the question, isn't it? Do they hold up?

Under the Hood this week introduces the rubric we'll use to find out. Next week, we apply it to every response — cloud and local — and see what's actually there. Until then, read the six responses below and form your own first impression. We'll compare notes next week.

How to Measure a Baseline

The hardest part of any AI project isn't building the thing. It's proving the thing actually works better than what you started with. That requires a baseline — and a rubric honest enough to grade the baseline fairly.

Here's the four-dimension rubric I'm using across this series. It applies to any AI response, not just the six we'll look at this week. Keep it. Use it on every AI output you see.

1. Accuracy

Is what it said actually true? The most basic test, and the one most often skipped. For the Milwaukee question, we can check: does MKE Tech Hub Coalition exist? (Yes.) Does the model name it correctly? Does it describe what the organization actually does, or something adjacent? Inaccuracy can be subtle — a real organization described with the wrong mission is a more insidious error than a fully invented one, because it passes a casual read.

Score 0–3: 0 = fabricated or wrong, 1 = mostly wrong, 2 = mostly right, 3 = accurate and verifiable.

2. Specificity

Is the answer grounded in real detail, or does it hide behind generalities? "Milwaukee has a growing tech scene with several meetup groups" is specifically nothing. A good answer names names, dates, venues, focus areas. Specificity is the difference between an answer that helps a new engineer get their feet under them and one that sounds like a press release.

Score 0–3: 0 = entirely generic, 1 = a few specifics surrounded by filler, 2 = mostly specific, 3 = sharp, concrete, useful.

3. Hallucination rate

How much of this was made up? This is the most important test in this project, because hallucination is how language models fail most often — confidently. A model that invents one fake organization among four real ones has a 20% hallucination rate, and that single fake entry is enough to make the whole answer untrustworthy. Measure hallucination as a percentage of claims that don't check out.

Score 0–3: 0 = half or more hallucinated, 1 = significant invention, 2 = mostly real with minor errors, 3 = fully verifiable.

4. Practical usefulness

Could a real person actually use this? An answer can be accurate, specific, and free of hallucination — and still be useless because it's the wrong shape for the question. For the Milwaukee question: would someone relocating here actually know what to do next after reading the response? Could they find the events? Know who to reach out to? If the answer leaves them with more questions than they started with, this dimension drops.

Score 0–3: 0 = unusable, 1 = partial, 2 = actionable with effort, 3 = immediately useful.

Using the rubric

Total score out of 12. That's the baseline number we'll track across every model for this project. When we re-measure after RAG, after fine-tuning, after the hybrid — this is the number we're trying to move.

One honest caveat: AI responses are non-deterministic. The same prompt produces different answers across runs. For a rigorous baseline you'd want to run the prompt 3–5 times per model and average the scores. This week's edition captures one run per model. That's the baseline we're committing to. Future editions will re-run it and report variance.

I'm a new software engineer relocating to Milwaukee. Name five local tech events, meetups, or community organizations I should know about in 2026, and briefly describe what each one is for.