OpenShell/architecture/object-metadata.md

Object Metadata Convention

Overview

OpenShell adopts a Kubernetes-style object metadata convention for all top-level domain objects. This standardizes how resources are identified, labeled, and queried across the platform. All resources that users interact with directly (Sandbox, Provider, SshSession, InferenceRoute) follow this convention.

Core Principles

1. Uniform Metadata Structure

All top-level objects embed a common ObjectMeta message containing:

• Stable ID: Server-generated UUID that never changes
• Human-readable name: User-friendly identifier (unique per object type)
• Creation timestamp: Milliseconds since Unix epoch
• Labels: Key-value pairs for filtering and organization

2. Trait-Based Access

Rather than accessing metadata fields directly (e.g., sandbox.metadata.as_ref().unwrap().id), code uses trait methods from openshell_core::metadata:

use openshell_core::{ObjectId, ObjectName, ObjectLabels};

let id = sandbox.object_id();        // Returns &str
let name = sandbox.object_name();    // Returns &str
let labels = sandbox.object_labels(); // Returns Option<HashMap<String, String>>

This provides:

• Uniform API across all object types
• Graceful fallback (returns empty string if metadata is None)
• Reduced boilerplate in code that works with multiple object types

3. Labels for Organization and Filtering

Labels are key-value metadata attached to objects for:

• Grouping related resources (e.g., all dev environment sandboxes)
• Filtering in list operations (e.g., show only sandboxes with team=backend)
• Automation and selection in scripts

Implementation Pattern

Protobuf Definition

Define ObjectMeta once in proto/datamodel.proto:

message ObjectMeta {
  string id = 1;
  string name = 2;
  int64 created_at_ms = 3;
  map<string, string> labels = 4;
}

Embed it in top-level objects:

message Sandbox {
  ObjectMeta metadata = 1;
  SandboxSpec spec = 2;
  SandboxStatus status = 3;
  int32 phase = 4;
  int32 current_policy_version = 5;
}

Migration: When adding metadata to an existing object, shift field numbers to make room for metadata = 1. This maintains backward compatibility if done before release.

Trait Implementation

Implement the three traits for each object in crates/openshell-core/src/metadata.rs:

impl ObjectId for Sandbox {
    fn object_id(&self) -> &str {
        self.metadata.as_ref().map(|m| m.id.as_str()).unwrap_or("")
    }
}

impl ObjectName for Sandbox {
    fn object_name(&self) -> &str {
        self.metadata.as_ref().map(|m| m.name.as_str()).unwrap_or("")
    }
}

impl ObjectLabels for Sandbox {
    fn object_labels(&self) -> Option<HashMap<String, String>> {
        self.metadata.as_ref().map(|m| m.labels.clone())
    }
}

Pattern: Always return empty string for missing metadata rather than panicking. This makes code resilient to malformed data.

Persistence Layer

The Store trait in crates/openshell-server/src/persistence/mod.rs provides three methods for working with objects:

// Store/retrieve by stable ID
async fn put_message<T: ObjectType + Message + ObjectId + ObjectName>(
    &self,
    message: &T,
) -> Result<(), String>;

async fn get<T: ObjectType + Message>(&self, object_type: &str, id: &str) 
    -> Result<Option<ObjectRecord>, String>;

// Retrieve by human-readable name
async fn get_message_by_name<T: ObjectType + Message + ObjectId + ObjectName>(
    &self,
    name: &str,
) -> Result<Option<T>, String>;

Database schema pattern: Each object type has:

• id column (TEXT PRIMARY KEY) — stable UUID
• name column (TEXT UNIQUE NOT NULL) — user-facing name
• payload column (BLOB) — serialized protobuf
• created_at_ms column (INTEGER) — denormalized from metadata for indexing
• updated_at_ms column (INTEGER) — last modification time

Label Filtering

Label selectors follow Kubernetes conventions:

Format: key1=value1,key2=value2 (comma-separated, AND logic)

Implementation:

1. Parse selector into key-value pairs
2. For each object, check that ALL selector labels match
3. Return only objects where every label in the selector exists with the exact value

SQL pattern (PostgreSQL with JSONB):

WHERE labels @> '{"env": "dev", "team": "backend"}'::jsonb

SQL pattern (SQLite):

WHERE json_extract(labels, '$.env') = 'dev'
  AND json_extract(labels, '$.team') = 'backend'

The list_with_selector method on Store handles this transparently.

Validation Rules

Labels must follow Kubernetes naming conventions (enforced in crates/openshell-server/src/grpc/validation.rs):

Label keys:

• Optional prefix + / + name (e.g., example.com/app or app)
• Prefix: DNS subdomain (lowercase alphanumeric, -, ., max 253 chars)
• Name: alphanumeric + -, _, ., max 63 chars
• Cannot start or end with - or .

Label values:

• Alphanumeric + -, _, .
• Max 63 characters
• Can be empty string

Validation functions:

validate_label_key(key: &str) -> Result<(), Status>
validate_label_value(value: &str) -> Result<(), Status>
validate_labels(labels: &HashMap<String, String>) -> Result<(), Status>

Validation timing: Validate at API ingress (gRPC handlers) before persisting. Reject invalid labels immediately rather than storing and failing later.

CLI Integration

Creating Objects with Labels

openshell sandbox create --label env=dev --label team=backend
openshell provider create openai --label project=research

Pattern: Repeatable --label key=value flags parsed into HashMap<String, String>.

Listing with Selectors

openshell sandbox list --selector env=dev
openshell sandbox list --selector env=dev,team=backend

Display: Show labels in tabular output when present, or in detail views.

Testing Requirements

Unit Tests

Test validation logic for:

• Valid label keys (with and without prefix)
• Invalid keys (bad characters, too long, empty segments)
• Valid label values
• Invalid values (non-alphanumeric, too long)
• Selector parsing

Integration Tests

Test persistence layer:

• Store object with labels
• Retrieve by name and verify labels present
• Filter with single-label selector
• Filter with multi-label selector (AND logic)
• Empty results for non-matching selector

E2E Tests

Test full workflow through CLI:

• Create multiple objects with different labels
• List all objects
• Filter by single label
• Filter by multiple labels
• Verify labels persist across gateway restarts

Location: e2e/rust/tests/sandbox_labels.rs (or equivalent for each object type)

Migration Checklist

When adding object metadata to a new resource type:

1. Proto changes:

   • Add ObjectMeta metadata = 1; field
   • Shift existing field numbers if needed
   • Update any references to old id/name fields

2. Trait implementations:

   • Implement ObjectId trait
   • Implement ObjectName trait
   • Implement ObjectLabels trait
   • Add to crates/openshell-core/src/metadata.rs

3. Persistence:

   • Add database migration (SQLite + PostgreSQL)
   • Create labels column (JSON/JSONB type)
   • Migrate existing id/name to ObjectMeta
   • Update ObjectType implementation
   • Update create/read operations to use new structure

4. Validation:

   • Add label validation in gRPC handlers
   • Validate on create and update operations
   • Test validation with unit tests

5. API updates:

   • Add label_selector parameter to List RPC
   • Implement selector filtering in persistence layer
   • Add labels field to Create/Update RPCs

6. CLI updates:

   • Add --label flag to create command
   • Add --selector flag to list command
   • Update completion for label keys (if applicable)
   • Display labels in list and get output

7. Tests:

   • Unit tests for validation
   • Integration tests for persistence
   • E2E tests for CLI workflow

8. Documentation:

   • Update user-facing docs for new flags
   • Add examples with labels to guides

Common Patterns

Creating Objects with Metadata

use crate::persistence::current_time_ms;

let now_ms = current_time_ms()
    .map_err(|e| Status::internal(format!("get current time: {e}")))?;

let sandbox = Sandbox {
    metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta {
        id: uuid::Uuid::new_v4().to_string(),
        name: user_provided_name,
        created_at_ms: now_ms,
        labels: request.labels,
    }),
    spec: Some(spec),
    status: None,
    phase: SandboxPhase::Provisioning as i32,
    current_policy_version: 0,
};

// Validate before persisting
validate_object_metadata(sandbox.metadata.as_ref(), "sandbox")?;
store.put_message(&sandbox).await?;

Filtering by Labels

let sandboxes = if request.label_selector.is_empty() {
    store.list(Sandbox::object_type(), limit, offset).await?
} else {
    validate_label_selector(&request.label_selector)?;
    store.list_with_selector(
        Sandbox::object_type(),
        &request.label_selector,
        limit,
        offset,
    ).await?
};

Accessing Metadata Fields

use openshell_core::{ObjectId, ObjectName};

// Good: trait-based access
let sandbox_id = sandbox.object_id();
let sandbox_name = sandbox.object_name();

// Avoid: direct field access
let sandbox_id = sandbox.metadata.as_ref().unwrap().id.as_str(); // Don't do this

Anti-Patterns to Avoid

❌ Bypassing Validation

// Bad: storing labels without validation
store.put_message(&sandbox).await?;

// Good: validate before storing
validate_labels(&sandbox.metadata.as_ref().unwrap().labels)?;
store.put_message(&sandbox).await?;

❌ Direct Field Access

// Bad: fragile to missing metadata
let id = sandbox.metadata.as_ref().unwrap().id.clone();

// Good: trait-based with fallback
let id = sandbox.object_id().to_string();

❌ Inconsistent Object Construction

// Bad: forgetting created_at_ms or labels
let sandbox = Sandbox {
    metadata: Some(ObjectMeta {
        id: uuid::Uuid::new_v4().to_string(),
        name: "test".to_string(),
        ..Default::default()  // Silently sets created_at_ms=0, labels=empty
    }),
    ..Default::default()
};

// Good: explicit fields
let sandbox = Sandbox {
    metadata: Some(ObjectMeta {
        id: uuid::Uuid::new_v4().to_string(),
        name: "test".to_string(),
        created_at_ms: current_time_ms()?,
        labels: request.labels,
    }),
    ..Default::default()
};

❌ Client-Side ID Generation

// Bad: letting clients specify IDs
let sandbox = Sandbox {
    metadata: Some(ObjectMeta {
        id: request.id,  // Never trust client-provided IDs
        ..
    }),
    ..
};

// Good: server generates stable IDs
let sandbox = Sandbox {
    metadata: Some(ObjectMeta {
        id: uuid::Uuid::new_v4().to_string(),
        ..
    }),
    ..
};

References

• Kubernetes API Conventions: https://kubernetes.io/docs/reference/using-api/api-concepts/
• Label Syntax: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
• Proto definition: proto/datamodel.proto
• Trait implementations: crates/openshell-core/src/metadata.rs
• Persistence layer: crates/openshell-server/src/persistence/mod.rs
• Validation logic: crates/openshell-server/src/grpc/validation.rs
• E2E tests: e2e/rust/tests/sandbox_labels.rs