In the business world, a moat is a quality of a business that makes it difficult for competitors to take that company’s profits. With how hard it is to train models for large language models (LLMs) and generative AI, it might seem like Anthropic, Open AI, and other LLM companies would have huge moats given the amount of compute it takes to build models. But open source models are quickly draining that moat, and now the only thing standing in the way of a customer using one of these models on their own hardware instead one from the larger companies is physical computing resources. [TerminalBytes] demonstrates a few of these models on personally owned computers to show the current state of the art.
[TerminalBytes] started off running the 27B version of the Qwen3.8 on a Mac Studio with 256 GB of unified RAM, which is plenty for this task. But it’s also enough to benchmark a few different models. Qwen3.6 is compared to 3.8, and then the different quants of each model are also compared. Quants are compressed versions of models that need fewer bits to store weights, meaning that the same models can run in less memory with smaller losses in fidelity. Many of these quants run on machines with 32 GB of RAM or less, encompassing many average gaming PCs. There’s even a 1-bit quant that [TerminalBytes] tested which can easily run on a machine with 16 GB, although with mixed results.
Keep in mind that this is just the current state of affairs with open LLMs. Future versions of these models are likely to optimize the number of tokens produced per unit time, or otherwise increase quality of responses while requiring less computer resources. We don’t really think that the ease of running local models will be the sole reason that the AI bubble pops, though. The fact that not every computer user is running Linux is proof enough of that.
IBM has rolled out the newest models in its family of open-weight large language models designed to be downloaded and self-hosted. The newly launched Granite 4.2 comes in 3B, 8B, and 30B parameter variants.
Like previous versions, IBM is taking a decoder-only approach here. These new releases offer a 128,000-token context window natively. The 8B and 30B variants (not the 3B one) also go through an agentic reinforcement-learning block; they were trained for expanded capabilities like using the terminal, searching the web, or using external tools. The 3B model supports tools too, but without the same level of specialized training.
Definitions for the AI era. (GPT-5.6 Sol Illustration, Click for larger image.)
Jargon stinks. What do the terms open weights, RAG, and agent mean exactly? Here’s a plain English, slightly snarky glossary of befuddling AI terminology with references for further reading.
AI is a broad name for the technology. Machine learning is the part where a system learns from data instead of following rules somebody wrote, a neural network is the structure that does the learning, and deep learning just means a neural network with a lot of layers.
Here’s the nitty-gritty: the terms that get used loosely, and the distinctions the loose usage hides.
1. Model, LLM, frontier model
ChatGPT is the app you open; an LLM, or large language model, is the AI running inside it.
“Frontier” isn’t a technical category at all. It means the handful of biggest and most capable models at any given moment, so the trophy keeps changing hands.
Everyone says “LLM” and hardly anyone could define it on the spot. “Frontier model” is worse. It’s a ranking, announced by the people being ranked.
Further reading:How ChatGPT Works: A Non-Technical Primer (MIT Sloan). Rama Ramakrishnan walks through the predict-the-next-word mechanism everything else is built on.
2. Prompts, tokens, parameters
A prompt is the thought, question, or instructions you provide to the LLM (plus whatever the app added before it without telling you). The LLM takes the prompt and generates words, both in its internal “thinking” process and in the answer it shows you.
Tokens are (roughly) the words going in and coming out. The model chops your prompt into tokens, then produces more of them as it answers, and they’re what the industry charges by.
Parameters, also called weights, are the numbers inside the model. A frontier model has hundreds of billions of them and the biggest now run to trillions, and nobody can tell you what any single one does.
Parameter counts get quoted like horsepower. The number nobody advertises is how many tokens it takes to answer your question, and that’s the one that shows up on the bill.
Further reading: The only AI glossary you’ll need this year (TechCrunch, July 2026). Its entries on tokens and weights are the clearest short treatment of the building blocks.
3. Pre-training, post-training, fine-tuning
Pre-training is feeding the model most of the internet, so it learns to predict the next word in a sentence. That’s the expensive part, and it produces something that knows a great deal but can’t follow an instruction.
Post-training is where people rank its answers and it learns to give more of what ranked well. Fine-tuning is post-training done by you, to somebody else’s model, on your data.
Pre-training costs hundreds of millions and gets you a model that won’t answer a question well. Post-training is what gets you the product.
From scratch, you buy (or rent) the computers and do the work to build and train a model. Distillation trains a cheap model on an expensive model’s outputs, so it inherits the behavior without the bill. Distillation is against most AI companies’ terms of service.
OpenAI accused DeepSeek of distilling its models, which is a bold position for a company that trained on the whole internet without asking. Learning from other people’s work is fine right up until the other people are you.
Training is how you build a model. Inference is what happens every time it answers: the model runs and produces a result.
Training is a one-time cost. Inference is a cost you’ll pay forever. Training runs for months and costs hundreds of millions; one inference, meaning one answer, costs a fraction of a cent, and it happens billions of times a day.
Training costs get announced. Inference costs get discovered. Only one of them shows up in a press release.
We typically use LLMs by accessing an app like ChatGPT, Claude, or Gemini. But experts often want the model itself, not just an app wrapped around it. Open weights means that an AI expert can download the model and run it on a server. You don’t get the data or the code that made it.
Open source means data and software that experts can use and modify, which almost no major model offers (AI2’s Olmo is a rare exception).
API-only means you can’t have the model at all. You send your text to the company’s computers, the answer comes back, and you pay for every use, which is also what’s happening when you use ChatGPT or Claude through an ordinary account.
Open weights is how you claim the open-source mantle without giving much away. Open washing, basically.
The context window is how much text the model can hold in mind at once, including your question and everything pasted into the conversation.
Memory is a feature that saves facts about you and slips them back into the context window later.
RAG, short for retrieval-augmented generation, searches a document collection and drops the relevant passages into the context window before the model answers.
Nothing in the model remembers you. The app keeps a file on you and pastes it in before every conversation, and that’s a less charming way to describe the same feature.
Further reading:Glossary of Terms: Generative AI Basics (MIT Sloan Teaching & Learning Technologies). Defines context window and RAG in plain language, and is careful to put the model’s “memory” in quotation marks.
8. Chatbot, workflow, agent
A chatbot answers and stops. A workflow runs the steps you defined, in your order. An agent receives a goal instead of steps, and works out for itself what to do, calling out to other software and checking the results until it’s done or stuck.
Ask about a delayed flight and a chatbot quotes you the policy; a workflow uploads the refund form you built; an agent rebooks you.
Useful test: if it decides its own next step, it’s an agent. If you decided the steps, it’s a workflow.
Further reading:Building effective agents (Anthropic, December 2024). The source of the distinction: workflows run predefined code paths, agents direct their own.
9. Hallucination, AI slop, AI cream
A hallucination is a confident falsehood, like a citation to a paper that doesn’t exist. The model isn’t lying; it has no notion of truth to violate. It’s producing text that looks like the right kind of answer.
AI slop is a different failure: accurate, fluent, and worthless. Think of the LinkedIn post that says nothing in 300 fluent words.
AI cream is the third case and the rare one: superb writing authored with the help of AI.
Nobody sets out to make slop. Everyone believes they’re making cream.
Further reading: 2025 Word of the Year: Slop (Merriam-Webster, December 2025). The dictionary definition turns on quantity: low-quality content “produced usually in quantity” by AI.
Why language models hallucinate (OpenAI, September 2025). Argues that hallucinations persist because benchmarks score accuracy alone, so guessing beats admitting ignorance.
10. Alignment, guardrails, censorship
Alignment is the research problem of getting a model to do what people want when nobody’s watching. Guardrails are the rules behind its refusals: “no, I won’t tell you how to make a bio weapon.” Censorship is a guardrail that blocked something you wanted.
The same refusal is “safety” in the press release, “guardrails” in the documentation, and “censorship” on X.
Further reading: Model Spec (OpenAI, updated December 2025). A published rulebook for what one model will and won’t do, which makes refusals arguable rather than mysterious.
I snuck in one novel term that’s been sorely absent from the field. Can you tell which one?
Further reading: other glossaries
Five general AI glossaries, listed roughly from most opinionated to most technical.
Glossary of Terms: Generative AI Basics (MIT Sloan Teaching & Learning Technologies). Twenty-odd entries aimed at people who use the tools rather than build them.
Machine Learning Glossary (Google for Developers). Hundreds of technical entries, and the only glossary here that defines “AI slop” a few lines away from several hundred pieces of real math.
Login
