All procedures
planuai-explorerredacted view

Explorer 2.0 Redaction And Embedding Lanes

Synced UAI Explorer plan documentation from docs/plans/explorer-2.0-redaction-embedding-lanes.md.

Updated 2026-06-21·Freshness: Reference·No secret-like patterns were observed in this view.

Explorer 2.0 Redaction And Embedding Lanes

Created: 2026-06-20

Redacted Reader Goal

The redacted view should feel intentional, professional, and trustworthy.

Users should understand:

  • what document they are allowed to see,
  • which access boundary was applied,
  • whether sensitive material was hidden,
  • which secret categories were detected,
  • what to do next if they need deeper access.

Users should never see:

  • raw API keys,
  • bearer tokens,
  • private keys,
  • credential URLs,
  • secret-bearing environment values,
  • hidden filesystem paths outside the curated catalog.

Current Slice

The reader now shows:

  • a redaction state pill,
  • a redaction brief,
  • category counts for matched redaction types,
  • highlighted redacted placeholders in the document,
  • an access boundary rail with surface, sensitivity, tags, code gate, and audit event.

The API now treats redaction as a server-side boundary:

  • catalog content is redacted before it is sent to the browser,
  • context notes are redacted before human and AI delivery,
  • payloads state redaction_boundary=server-side,
  • payloads state raw_available_to_client=false,
  • legacy filesystem, raw download, and upload APIs are refused in public-safe mode,
  • API responses use no-store cache headers,
  • API responses state X-UAI-Redaction-Boundary=server-side and X-UAI-Raw-Content=not-sent,
  • browser security headers restrict injected scripts, framing, and unsafe resource loading.

Client-side controls are usability hints only. They are not the security boundary.

Air-Gapped Redaction Recommendation

For a public deployment, use a two-container or two-volume model:

  1. raw-survey-zone: device survey exports, raw source files, old project archives, and unclassified material. This zone is operator-only and not mounted by the public Explorer web app.
  2. redaction-worker: reads raw material, applies classifiers and redaction, writes sanitized documents, summaries, metadata, hashes, and review queues.
  3. public-explorer-zone: serves only curated catalog records, sanitized documents, approved summaries, context notes, and safe embedding indexes.

The public web container should have no filesystem path to raw survey material. If someone edits frontend JavaScript, disables CSS, or tampers with the DOM, there is still no raw content available to reveal.

Minimum production rule:

  • mount sanitized output read-only into Explorer,
  • do not mount raw device survey folders into Explorer,
  • keep ALLOW_ADMIN_RAW_READ=false,
  • keep PUBLIC_SAFE_MODE=true,
  • verify /api/tree, /api/file, /api/raw, /api/upload, and /api/uploads return 403 after authentication,
  • verify readiness reports raw_survey_zone_not_mounted=pass, and that mounting raw-survey-zone under the served data root blocks readiness,
  • allow restricted/code lanes only through reviewed catalog entries,
  • record hashes for sanitized artifacts, not raw secret-bearing content.

For MacBook, Devine, and Cadence imports, treat the source material as raw survey input until a redaction worker creates approved summaries, standards extracts, lineage metadata, and catalog entries. The public Explorer container should see the approved outputs only.

If conversion tooling such as MarkItDown is used for PDFs, Office documents, images, archives, or exported notes, run it only in the raw survey/redaction zone. Conversion output is still untrusted until redaction, review, and catalog tagging pass.

Embedding Lane Recommendation

Do not start with raw whole-device embeddings.

Keep two separate retrieval worlds:

  • Public/AI retrieval: content an authorized AI user can retrieve through /api/ai/context/{slug}.
  • Private operator metadata: survey inventories, review queues, and redaction planning artifacts used only inside governed private workflows.

Start with safer public/AI lanes:

  1. Survey lane: file paths, project names, modified dates, and hashes.
  2. Summary lane: AI-generated or human-written summaries after redaction.
  3. Idea lane: extracted ideas, version lineage, and promotion state.
  4. Reader lane: approved catalog text and context notes.
  5. Restricted lane: code-aware embeddings only for explicitly code-approved material.

Expose these lanes as context tiers before building full retrieval:

  • L0: START_HERE_AI.md and current operating truth.
  • L1: project brief and canonical status.
  • L2: approved runbooks and evidence.
  • L3: sanitized retrieval index.
  • L4: archive references.
  • R: raw survey zone, not mounted publicly.

The first local agents should classify and summarize; they should not decide authorization.

MCP servers and local AI adapters must not index raw survey folders, unapproved sanitized exports, selected-redaction inputs, direct DATA_ROOT reads, SQLite, upload folders, .env, .ssh, or arbitrary device paths. They may retrieve only approved AI-visible catalog packets after /ready confirms public-safe mode, server-side redaction, disabled raw/file/upload APIs, and no raw survey mount.

Private operator metadata can still be embedded for local planning later, but that requires a separate restricted-lane runbook and must not be confused with the public or AI-facing Explorer context index.

Metadata-Only Export Worker

tools/sanitize_survey_export.py creates a safer bridge between raw device survey inventories and outside-agent review. It reads a survey manifest and inventory CSV, rejects raw-content columns, strips file aliases and raw source hashes, and writes:

  • manifest.json: export boundary and omitted-field contract,
  • topic-summary.json: grouped counts, tags, destinations, and lanes,
  • review-queue.json: row-indexed private review references without file aliases,
  • catalog-candidates.json: non-publishable catalog planning records,
  • agent-brief.md: outside-agent instructions and topic table.

Generated metadata exports are not public catalog content. They are review inputs until a separate redaction worker creates approved summaries or excerpts.

Selected Artifact Redaction Worker

tools/redact_selected_artifacts.py is the first private-lane redaction worker for explicit raw-file selections. It reads a selection JSON, resolves each source_path under a private --source-root, rejects traversal outside that root, refuses binary input, and writes sanitized outputs to --out.

The worker writes:

  • manifest.json: export boundary and review contract,
  • redaction-report.json: row-level redaction state without raw paths,
  • catalog-import.json: non-publishable catalog candidates,
  • artifacts/*.md: sanitized markdown artifacts with JSON front matter.

The output contract is deliberately conservative:

  • raw_content_included=false,
  • raw_source_path_included=false,
  • raw_source_hash_included=false,
  • public_publishable=false,
  • human_visible=false,
  • ai_visible=false,
  • review_required=true,
  • redaction boundary: worker-server-side.

This worker is not the public security boundary by itself. It is a bridge from operator-only raw material into a review queue. Public UAI Explorer should see only the approved sanitized artifacts after review, tagging, and catalog import.

Zai Or Spark Agent Prompt

Use this prompt for an outside agent only with redacted or non-secret survey exports:

You are helping design UAI Explorer 2.0, a public-safe documentation and
knowledge-lineage system. Review the provided redacted survey export. Do not
request or infer secrets. Produce:

1. device inventory observations,
2. project/version lineage candidates,
3. stale-but-valuable material candidates,
4. idea salvage categories,
5. recommended catalog tags,
6. redaction or access concerns,
7. a simple embedding-lane proposal that separates public, internal, code, and
   restricted material.

Keep the UX simple. Prefer practical workflows over complex dashboards. Treat
out-of-sync data as provenance, not trash.

Spark 5.3 Option Set

If Spark 5.3 is used, send three jobs:

  • survey-classifier: classify device/project/version/freshness from redacted survey manifests.
  • idea-salvage: extract reusable ideas and assign new, needs_review, parked, promoted, or superseded.
  • embedding-policy: propose safe embedding lanes from approved catalog and redacted summaries only.

Do not give Spark raw secret-bearing files until Explorer has a reviewed restricted-lane policy.