Don't Put PyTorch, Hugging Face and Ollama on the Same Shelf — AI Frameworks Come in Three Layers
What does the math, what loads the model, what runs it fast
"If Transformer is like MVC, then just like there's Django besides Rails, there must be something besides Hugging Face, right?"
Good instinct, but the answer doesn't fit in one line. On the AI side there are three layers, not one. Put PyTorch, Hugging Face transformers and Ollama on one shelf and the comparison keeps falling apart. After this post, when a new AI tool name shows up, you'll be able to place it in a layer first and then choose within that layer.
The three layers
Base framework: the matrix math and training engine. PyTorch, TensorFlow, JAX. You use these to build or train a model from scratch.
Model library: loads existing models so you can use them. Hugging Face transformers (Python), transformers.js (JavaScript). Under the hood they call a base framework (or ONNX Runtime).
Inference engine: focused only on running a trained model fast. llama.cpp, vLLM, Ollama, MLX.
In Rails terms: the base framework is Ruby and Rack, the model library is Rails, and the inference engine is roughly Puma. "Which is better, Rails or Puma?" is a strange question. So is "Ollama or PyTorch?"
Layer by layer
| Name | Language | Layer | Good at | Weak at |
|---|---|---|---|---|
| PyTorch (Meta) | Python (C++ core) | Base | Most used in research and industry | Lots of code if you write it yourself |
| TensorFlow / Keras (Google) | Python | Base | Mobile deployment tooling (TFLite) | New model code often shows up for PyTorch first |
| JAX (Google) | Python | Base | TPUs, used to train Gemini | Takes time to learn |
| HF transformers | Python | Model | Number of public models, fine-tuning too | Slower inference than dedicated engines |
| transformers.js | JavaScript | Model | Runs ONNX in the browser and Node | Large models are out of reach |
| llama.cpp | C/C++ | Inference | GGUF, LLMs on CPU and Mac, quantization | No training |
| Ollama | Go | Inference | One line: ollama run |
Hard to fine-tune the details |
| vLLM | Python | Inference | Throughput under concurrent requests | Needs a GPU server |
| MLX (Apple) | Python / Swift | Inference + training | Fast on Apple Silicon | Mac only |
Ollama's README lists llama.cpp as a supported backend. So "Ollama or llama.cpp?" is a within-layer question: do you want convenience or direct control?
The language is basically Python
Python - everything (training, fine-tuning, inference)
JS/TS - transformers.js (ONNX). Running in the browser is the big win
C/C++ - llama.cpp, ONNX Runtime
Go - Ollama is the main one. No training
Ruby - few options. Gems wrapping ONNX Runtime, roughly
The stack behind this blog is Rails, so I kept running into that last line. Using a local model from a Rails project comes down to two options: run a separate Python server and call it over HTTP, or convert the model to ONNX and call that. If the job can run in the browser, transformers.js lets you skip the server entirely.
How to choose
Training or fine-tuning: PyTorch at the base, HF transformers on top. Using someone else's model as-is: a model library. Running a fixed model fast and cheap: an inference engine. On your own Mac, Ollama. On a shared GPU server with many concurrent users, vLLM.
What HF transformers' pipeline() does inside is covered in How the Hugging Face pipeline works. Repacking the same weights for the browser is in ONNX is a package format.
If you're trying to attach a local model to a non-Python stack like Rails and can't decide which layer goes where, feel free to ask negabaro. We're working through this exact setup ourselves.
Key Concepts
When a tool name comes up, place it in a layer first: base, model library or inference engine
Compare only within a layer. Ollama vs vLLM makes sense, Ollama vs PyTorch does not
Training: PyTorch + HF transformers. Just running: an inference engine
From Rails, run a separate Python server or call through ONNX