Open-Source AI: Local Models and Data Paths

At a glance

Local inference is one hop, not the whole boundary. The article's reproducible check: run the complete task offline and inventory every configured endpoint.

Sources

1. 1. docs.ollama.com · faq
2. 2. lmstudio.ai · offline
3. 3. docs.continue.dev · embeddings
4. 4. modelcontextprotocol.io · transports

Data behind this diagram

The article's reproducible check: run the complete task offline and inventory every configured endpoint.

Outbound path	Destination	Control
model pulls + updates	registries / hugging face	pin immutable revision, record digest
extension telemetry	vendor	disable explicitly (continue, cline)
cloud embeddings / rerank	remote provider	code chunks are sent — use local roles
remote mcp server	tool host	server receives tool arguments
cloud fallback models	hosted api	ollama_no_cloud=1, test offline

Scroll →

Open-source AI combines three separate questions. The first is where inference runs. The second is what the licence permits. The third is whether the downloaded artifact can be traced to the publisher that produced it. Treating any one of these as proof of the other two gives a false result.

Map the inference stack, not only the model

A local generation endpoint plus a cloud embedding endpoint is not a local system. Continue's own embedding documentation shows both an in-extension transformers.js option and remote provider configurations. Code chunks must be sent to a remote embedding service for that service to compute their vectors.

The same rule applies to tools. The Model Context Protocol transport specification distinguishes local standard-input servers from HTTP transports. A remote MCP server receives the tool request and any arguments required to execute it. A model running on the same laptop does not change that transfer.

Classify the licence at system level

The Open Source Initiative's Open Source AI Definition 1.0 applies the freedoms to use, study, modify and share an AI system. Its preferred form for modification includes model parameters plus sufficient information about training data and code. A licence badge on a weight file answers only part of that test.

Model catalogues should retain these tiers instead of flattening them into one open-source field. The classification concerns legal and technical autonomy. Privacy remains a property of the deployed data path.

Uncensored and abliterated models are in scope

Uncensored models, fine-tunes with different refusal policies and abliterated checkpoints are relevant when licence and operator autonomy are being compared. Abliteration changes what a model will answer by editing or suppressing refusal-related directions in its representations or weights. It is not a data-flow protection. The runner, extension, embedding service and tools see the same data unless those components are changed separately.

The original refusal-direction work, Arditi et al., arXiv:2406.11717, found a direction that mediated refusal across several instruction-tuned models. Later evaluations show that collateral effects depend on model family and ablation strength. A model name containing abliterated is therefore not a quality measurement.

Verify weight provenance before loading

A third-party quant from a pseudonymous uploader is an unsigned artifact unless the publisher supplies a verifiable signature or attestation. A familiar username, download count or filename is not cryptographic provenance. SafeTensors limits executable deserialization behaviour, but a safe container does not prove who produced the tensor values.

1. Start with the original publisher record. Record the organisation, repository, licence, base model and announced files.
2. Pin an immutable revision. Hugging Face's download client accepts a commit hash as the revision. A moving main branch is not a durable identifier.
3. Record the artifact digest. Hash the downloaded file before import and retain the digest beside the model configuration.
4. Compare supplied hashes or signatures. A matching self-published checksum detects transfer changes. It establishes identity only when the checksum came through an independently trusted publisher channel.
5. Prefer non-executable formats. Hugging Face's security documentation explains the arbitrary-code risk of pickle files and recommends SafeTensors when available.
6. Retain the conversion chain. For GGUF or other quants, record the source commit, conversion tool commit, command, quantization type and output digest. Without that chain, the quant is a separate unverified release.

publisher: ai2
repository: allenai/OLMo-2-1124-7B-Instruct
revision: <full immutable commit hash>
artifact: model-00001-of-00004.safetensors
sha256: <locally computed digest>
format: safetensors
converter: none
license_source: <publisher licence URL>
reviewed_at: 2026-08-09

Hardware tiers are memory budgets, not speed promises

A 4-bit weight-only floor is approximately half a byte per parameter. That arithmetic puts 8B, 14B, 32B, 70B and 600B parameters at about 3.7, 6.5, 14.9, 32.6 and 279.4 GiB respectively. Real use needs additional memory for quantization metadata, the key-value cache, context, runtime buffers, the operating system and other processes. Mixture-of-experts models also separate total stored parameters from parameters used for each token.

The table is a capacity reference, not an efficacy or throughput claim. Runner support, memory bandwidth, GPU offload, prompt length and concurrent sessions determine whether a model is usable for a particular workload.

A reproducible local-only check

# Disable Ollama cloud features, then restart the app or service.
export OLLAMA_NO_CLOUD=1

# Pull the chosen model before the offline test.
ollama pull <publisher/model:tag>

# Confirm the API is bound to localhost.
lsof -nP -iTCP:11434 -sTCP:LISTEN

# Disconnect the network, then verify local inference.
curl http://127.0.0.1:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{"model":"<publisher/model:tag>","prompt":"local test","stream":false}'

1. Inventory every configured endpoint. Include chat, autocomplete, embeddings, reranking, speech, search, MCP and crash reporting.
2. Disable cloud fallbacks. Failure of the local runner should remain visible rather than select a remote model.
3. Run the complete task without internet access. Test indexing, retrieval, generation, tools and history, not only one prompt.
4. Inspect local storage. Chat databases, document caches, vector stores, temporary files and synced folders remain part of the boundary.
5. Record exceptions. For each required remote service, name the endpoint, fields sent, retention policy and approval condition.

Sources

For an example of how two apps can get different answers from the same downloaded model, see Qwen3.8-27B OBLITERATED: Why the Same Model Gives Different Answers.

• Ollama FAQ: local prompts, network binding and cloud controls
• LM Studio offline operation and desktop app privacy policy
• Continue telemetry documentation and embedding roles
• Cline telemetry documentation
• MCP transports and remote servers
• Open Source AI Definition 1.0
• Refusal in Language Models Is Mediated by a Single Direction, arXiv:2406.11717
• Investigating the Impact of Quantization Methods on Safety and Reliability, arXiv:2502.15799
• Measuring and Eliminating Refusals in Military Large Language Models, arXiv:2603.10012
• Different Paths to Harmful Compliance, arXiv:2604.18510
• Hugging Face Hub immutable revisions and repository security