OpenViking: Inside the Context Database Architecture

OpenViking starts from a plain problem: useful data exists, but agents still struggle to use it. A model also needs a surface it can act through and a substrate that keeps context; otherwise every task falls back to stuffing material into the prompt.

A model that writes SQL fluently still has to know which tables to query and how they join. Many agent failures have the same shape. Reasoning is not the weak point; the access plan is: where to look, how far to search, which memory belongs to whom, and whether a write is safe. OpenViking calls the system that answers those questions a context database.

Why A Filesystem-Shaped Interface

Most agent context is not born as clean relational records. It is code, documents, PDFs, images, tickets, meetings, chat logs, and memories. Using it is closer to search and recommendation than to transaction processing: first shrink a noisy corpus into a plausible scope, then rank, read, and refine.

Relational databases remain the right tool for metadata, billing, jobs, and structured state. They are a poor primary interface for agents, because the agent must discover schemas, tables, joins, and valid predicates before it can ask for anything. A path is a cheaper control primitive: pick this project, this user's memory space, this document subtree, then search inside it.

These are layers, not rivals. OpenViking stacks them in one query: tenant fields and a path narrow the scope, vector search runs inside it, and an optional rerank orders the result. The rest of the architecture follows from that order.

The Shape Of The System

The implementation is deliberately polyglot. Python owns the server because parsing, document processing, multimodal understanding, model SDKs, and AI dependencies live in that ecosystem, and OpenViking is IO- and pipeline-heavy long before it is CPU-bound. Rust owns the surfaces where startup time, binary delivery, and IO throughput matter: the ov CLI and RAGFS, the content filesystem. C++ carries the embedded vector engine that came from VikingDB, so the project reuses mature indexing code instead of rewriting the hardest part.

RAGFS started life as AGFS, a separate Go server the Python process talked to. It has since been rewritten in Rust and is loaded into the server process as a Python extension, which removes a hop from every file operation.

The split defines each layer's contract. Agents speak in commands and URIs. The server enforces identity and runs jobs. VikingFS and RAGFS give context a traversable shape. The vector index and file storage decide what can be retrieved and what persists. Clients reach the server over HTTP; the CLI, SDKs, MCP endpoint, and Skills all use the same API, and VikingBot, the reference agent shipped in the repository, reads and writes through it too.

The two paths explain a common surprise. A plain add-resource returns once content is parsed and placed, while summaries and vectors are still in the queue; a Git repository import returns even earlier, before the clone finishes. The file becomes readable before it becomes searchable. Workflows that search right after writing should pass --wait or poll the returned task.

Directory Semantics Are The Addressing Layer

Vector search has a scaling problem that hurts RAG more than recommendation. A recommender can recall thousands of candidates through several channels and rely on coarse and fine ranking. An agent cannot pass thousands of chunks downstream. The final context window may hold only tens of chunks, and filling it too full weakens the model before it starts reasoning.

Scalar filters are the first answer: tenant, owner, time, level, and source type should prune the search space. Directory retrieval is the more general answer. A lot of useful context is already a tree: code, calendars, wikis, books, service trees, category taxonomies, geographies. VikingDB turned that observation into a path-aware index, and OpenViking exposes it through viking:// URIs.

The detail that matters: the path is not a text field. Every record OpenViking writes to the vector index has a uri field of type path. The engine keeps directory bitmaps for it, so a query can take a whole subtree, or exactly one level of it, by prefix and depth instead of matching path strings row by row. That is why directory semantics lower the cost of generating filters: one path plus a depth is far easier for an agent to produce than a hand-built predicate over an unknown schema.

Multiple Roots Mean Multiple Logical Views

A multi-root tree is not several physical copies of one file. The same object would be indexed under several logical trees, and each tree is a different way to narrow retrieval before vector search. A document could live in the project resource tree, appear in a calendar tree by creation time, and be reachable through a category or geography tree when the domain needs it.

This is where the architecture is headed, not what the API exposes today. The public top-level namespaces are resources, user, and agent. Time is organized as directories inside the resource tree: an image added without an explicit target lands under viking://resources/images/YYYY/MM/DD/. Independent roots such as calendar, geo, or category are not callable yet.

{
  "op": "must",
  "field": "uri",
  "conds": ["/user/alice/memories"],
  "para": "-d=1"
}

• d=-1 searches the whole subtree under the directory.
• d=0 matches the node itself.
• d=x searches downward by x levels.

Agents never write this DSL by hand. The target directory passed to find or search is compiled on the server into a d=-1 path scope and merged with account, user, ACL, context-type, and level conditions into a single vector query. Exact URI lookups use d=0.

Progressive Disclosure For Context

L0 and L1 are directory-level sidecars. File summaries roll up into the directory's overview, and the abstract is taken from the overview. A parent summarizes its children, but large directories use a stable sample of at most 32 direct children by default, and an overview can lag behind recent changes. Absence from a summary does not mean the source is absent. The layers control how much an agent reads; they do not force a search to descend level by level. One query can hit a record at any level directly.

Files, Virtual URIs, And Multimodal Objects

viking:// is a logical database namespace, not a physical storage path. The source path can be kept as provenance, while the physical key stays internal: viking://resources/docs/auth is stored under /local/{account_id}/resources/docs/auth, with the account prefix added by the server. The visible URI comes from the upload command, a parent the user names, or OpenViking defaults, and it links the stored object to its rows in the vector index.

With one logical view, the canonical URI and the matched URI are the same. Once multiple roots exist, retrieval should show the matched view so the agent knows why an item appeared, while reads and writes still target the canonical URI.

Multimodality is a separate axis from directory semantics. Text, code, PDFs, and images all benefit from path-scoped retrieval. Images make the difference easy to see: a query may hit the text summary the VLM wrote for the image, the image embedding itself, or both. The directory decides where to search; the embeddings decide what is similar inside that scope. Image-to-image search needs the embedding model configured as multimodal; a text-only model still indexes the image's summary but cannot take an image as the query.

ov add-resource ./docs/images/demo.png --wait
# Without --to, images land in date directories:
# viking://resources/images/2026/05/09/...

ov find "architecture diagram with three storage layers" --uri viking://resources/images
ov find --image ./query.png --uri viking://resources/images

Distributed By Decoupling Storage

The open-source distribution starts as a single-machine service, but the architecture is pointed at managed deployment. The key move is to separate the instance from the data. The vector index can be the in-process engine, a remote HTTP service, or Volcengine VikingDB; file storage can be local disk or S3-compatible storage. With both on remote services, an instance holds no authoritative data of its own.

The open-source build also avoids mandatory dependencies such as Redis and Kafka. Temporary working directories, task records, work queues, and path locks sit behind the same filesystem abstraction: queues persist in SQLite on top of it, and locks are lock files by default. When several processes must coordinate locks, the lock provider can switch to Redis. That is an option, not a prerequisite for starting the server.

Running several instances adds two concerns. Uploaded files land on the instance that received the request by default, so replicas need the shared upload mode. Path locks need a provider every process can see. Primary/backup storage can replicate writes to backup backends, synchronously or asynchronously, but it never promotes a backup on its own.

Beyond that, there are two ways to lay out instances. Both are architectural tradeoffs to design for, not deployment modes that ship ready to use:

One principle orders everything else: files are the source of truth and the vector index is derived from them. An index can be rebuilt from retained files; lost files can only come back from a backup. So OpenViking prefers a missing search result to a wrong one. Deleting removes index records first and files second, so search never returns a file that is gone; if the second step fails, the file is still there and a retry finishes the job. Moving copies the content, updates the index, and then removes the source.

Identity: Treat Agents As Database Users

The hardest multi-tenant question is not accounts. It is whether an agent is subordinate to a human user, owns data by itself, or should be treated as a peer. OpenViking went through all three designs and settled on the third.

Local multi-tenancy starts with a root API key and explicit user registration. The hosted service does not expose a root key; the console issues user keys directly. The product surface differs, but the invariant holds: every read and write carries a real identity before it touches private context, and the root key manages accounts and users but cannot read tenant data in API-key mode.

This is a privacy decision as much as a modeling one. A customer-service agent may keep memories about visitors who are not registered users and hold no API keys. Forcing those visitors into the User abstraction would make the authorization graph less true and less safe. Separating agents into their own identity type had a similar appeal, letting any agent reach a user's global memory, and it fails the same case.

What landed separates the data owner from the interaction object. The user is the data owner, a person or an agent service holding its own key. The objects that user serves, such as a visitor, a group member, or a code repository, become peers under it: viking://user/support-bot/peers/customer-alice/memories. A peer narrows retrieval and reads inside one user; it gets no key and does not create a tenant. The User / Peer post walks through the model, and the coding agent plugins use it to keep one memory per repository.

# server ov.conf: set server.root_api_key before startup
# client ovcli.conf: use the same root key for admin commands
ov admin register-user default alice
# write the returned user key back to ovcli.conf for everyday reads and writes

Performance Is A Pipeline Problem

Once storage can be distributed, capacity is mostly a deployment choice. Performance is harder. Reads, such as RAG queries and memory recall, scale mainly by adding instances. Writes cross parsing, splitting, VLM calls, embedding, summarization, memory extraction, IO movement, and locks.

Optimization directions and progress

1. Queue model calls under concurrency limits. This is now configuration: embedding.max_concurrent defaults to 10 and vlm.max_concurrent to 32.
2. Replace the Go AGFS server with embedded Rust. Done: RAGFS runs inside the server process.
3. Parallelize tree operations such as find and tree, which recurse over many directories.
4. Reduce copies between the receiving, working, and visible directories during upload. On object storage a move is a copy, so this cost shows up directly in end-to-end latency.

For observation, ov status gives a summary, ov observer breaks it down by queue, models, retrieval, and filesystem, and the server exposes /metrics for Prometheus-compatible collectors.

Privacy: Context Is Plaintext

A context database stores the material an agent reasons with, and that material is often sensitive by definition. OpenViking handles this with key-based identity, a root key limited to administration, user-scoped visibility enforced in the index, optional file encryption, and privacy configs for Skill secrets.

Encryption is implemented, and it is not free. It uses envelope encryption with a root key, a key per account, and a key per file, so tenants are cryptographically separated and the blast radius of a leak shrinks. The root key can stay in a local file, in Vault, or behind Volcengine KMS. Authorized reads still return plaintext, enabling encryption does not rewrite existing files, and it covers file storage: retrieval text in the vector index needs its own protection. Remote storage has to be decrypted before operations such as grep, so privacy controls affect latency and operator ergonomics, not only compliance.

ov privacy categories
ov privacy list skill
ov privacy skill search-web
ov privacy upsert skill search-web \
  --values-json '{"api_key":"secret-2","base_url":"https://example.com"}'
ov privacy activate skill search-web 2

Secret extraction from a Skill relies on a model and only replaces values it recognizes and matches. It reduces plaintext secrets in shared Skills; it does not guarantee that every secret is found.

What To Remember

The core architectural judgment is that context is not a blob. It has paths, scopes, identities, consistency constraints, performance budgets, and privacy boundaries. OpenViking is useful because it lets agents consume those properties through an interface they already know how to navigate.

The architecture is still moving from concept to product construction. Open source has brought enough usage, issues, and feedback to make capacity and performance the next hard problems, and some items from earlier plans, such as the Rust filesystem and the peer identity model, have already landed. The design is useful because it names the database properties a context system must expose before agents can depend on it, and it keeps the unfinished consistency and latency work in plain view. For why context engineering becomes a database problem in the first place, read the database paradigm post.

Thanks to everyone who has contributed code, ideas, data, and use cases. The remaining questions are not slideware questions; they are the ones that appear when real agents, real data, and real users start sharing one context substrate.