¶ HashiCorp Vault Architecture
Centralized secrets management, host-level Vault Agent authentication, secure .NET runtime configuration, Docker Swarm delivery, and immutable Jenkins deployments.
¶ 1. Purpose, Scope & Status
This document describes the architecture implemented for secret storage and controlled runtime distribution in the Dream Consistency environment. The verified implementation baseline is the Hestino DEV migration of Hestino.Admin.Api, Hestino.Management.Api and DreamConsistency.AccessControl.Api. The same pattern can be evaluated for other products and environments, but their deployment must not be assumed.
DC-VAULT-ARCH-001, version 1.0 (24 October 2026). Wiki edition prepared 26 October 2026. Audience: Software Architecture, DevOps, Service Owners. Classification: Internal engineering documentation.| Evidence label | Meaning |
|---|---|
| CONFIRMED | Observed in source code, pipeline configuration, host configuration or deployment evidence. |
| REFERENCE PATTERN | Architecture to follow when onboarding another service, subject to its own security review. |
| TO VERIFY | Operational setting that was not established by the source evidence and must not be treated as implemented. |
¶ In scope
Vault OSS and KV v2, AppRole, a host-level Vault Agent, policy/path isolation, file permissions, ASP.NET Core configuration, Docker Swarm, the separation of Jenkins build/deploy pipelines, secret rotation, monitoring, recovery, and migration lessons.
¶ Non-goals
This design is not dynamic database-user issuance on every request, Kubernetes Vault Injector, or a requirement for each .NET service to use a Vault SDK. Neither browser applications nor the Hestino digital menu receive Vault credentials or server-side secrets.
¶ 2. Architectural Principles
¶ 3. Confirmed Infrastructure Inventory
| Layer | Observed Hestino DEV implementation |
|---|---|
| Vault server | HashiCorp Vault OSS 2.1.1; internal endpoint vault.dreamconsistency.internal:8200, private address 10.10.0.4; separate operator UI at vault.dreamconsistency.com. |
| Secrets engine | KV v2 at mount kv/. |
| Authentication / ACL | AppRole hestino-dev; actual current policy name and complete HCL permissions must be checked in Vault. |
| Deployment host | DEV-Hestino-16GB; Docker Swarm network hestino-dev. |
| Agent | Systemd service vault-agent.service; config /etc/vault-agent.d/vault-agent.hcl; templates /etc/vault-agent.d/templates/. |
| Rendered files | /run/hestino-secrets/<service>/appsettings.secrets.json. |
| Service consumers | Admin.Api, Management.Api, AccessControl.Api; read-only mount in container at /run/secrets. |
| CI/CD | Separate Core Docker Release (build, test, audit, push) and Hestino DEV Deploy (pull, verify, deploy) jobs. |
¶ 4. Runtime Topology & Trust Flow
Figure 1. Confirmed logical layout for Hestino DEV. Replica counts are from the October 2026 reference snapshot, not a live service-status query.
Vault acts as the central security authority. The Agent authenticates through AppRole, reads only authorized KV v2 paths, and materializes JSON on the host. Swarm services only consume their respective local files. The underlying Vault connection and TLS verification settings must be confirmed in the actual Agent configuration.
Figure 2. Secret and trust flow. No Vault authentication material is injected into application containers.
¶ The three security boundaries
| Boundary | Required control / exposure |
|---|---|
| Vault storage and administration | Vault authentication, narrowly scoped policies, access control, auditing, protected backups, and restricted operator access. |
| Host and Agent | The Agent can handle plaintext secrets in memory and restricted files. Host root and privileged Docker operators remain within the trusted administrator boundary. |
| Container consumption | Each service receives only its own read-only bind mount; no broad mount of the parent secrets directory and no Agent credentials inside the container. |
¶ 5. Identity, Policies & KV v2 Namespaces
For the confirmed Hestino DEV host, hestino-dev is the AppRole for the product/environment and the shared host Agent. Individual microservices do not require a separate AppRole under this approved host trust boundary. The trade-off is the blast radius of a compromised host or role: all ACL-authorized secret paths may be exposed. Expand policies by reviewed, explicit, read-only paths.
The data/ segment matters: policy HCL and Agent templates use the KV v2 API path, while the logical CLI path does not include it.
hestino-dev policy with only the example block. Do not give read/list/delete permissions beyond the actual service requirement. Metadata/list privileges are not necessary for ordinary template reads.¶ 6. Configuration Ownership
| Configuration class | Source | Examples / rationale |
|---|---|---|
| Secrets | Vault KV v2, rendered into secrets JSON | Realm DB connection strings, API keys, passwords. Not in the Jenkins environment, Swarm spec, or source repository. |
| Public environment overrides | Docker environment or controlled public runtime JSON | ServiceInfo, non-secret service identifiers, logging format and mode, selected whitelist values. |
| Image defaults | appsettings.json | Non-secret defaults and array indexes 0–3 in the observed whitelist configuration. |
In the Hestino DEV implementation, MessageTransportSettings__ServiceSendersWhiteList__4 supplies the fifth whitelist entry through non-secret runtime configuration. A one-item JSON array added through Vault would typically override index 0 rather than append index 4. Array indexing must be verified against the effective .NET configuration.
¶ 7. Vault Agent, Templates & File Permissions
On DEV-Hestino-16GB, the systemd-managed vault-agent.service uses /etc/vault-agent.d/vault-agent.hcl. Each onboarded service adds a separate template stanza and destination directory. The template renders the entire KV v2 object so Jenkins does not need to map secret keys.
Observed AccessControl template: /etc/vault-agent.d/templates/hestino-access-control-api-config.ctmpl. Observed rendered destination: /run/hestino-secrets/access-control-api/appsettings.secrets.json.
| Component | Observed permissions / identity | Impact |
|---|---|---|
| Agent process | UID 996, GID 986 (vault-agent) | Authorized host process reads its authentication material and writes rendered files. |
| Parent directory | /run/hestino-secrets restrictive traversal | A deploy SSH identity can be unable to test -d even though the service can read through the approved mount. |
| Service directories | Mode 2750; group 1654 for the non-root services | Setgid group propagation and limited traversal. |
| Rendered JSON | Mode 0640; group 1654 | Approved UID/GID 1654:1654 can read, other identities cannot. |
| Container mount | Read-only bind to /run/secrets | The application must not be able to write the secret source. |
| Admin.Api exception | Observed running as root inside its container | Do not assume all current service runtime identities match the non-root example. |
/run/hestino-secrets to world-readable for Jenkins. Test file readability using an ephemeral container with the same actual UID/GID and read-only bind mount as the service.¶ 8. ASP.NET Core Configuration Pipeline
The .NET application starts with WebApplication.CreateBuilder(args), which registers default configuration providers. Dream Consistency explicitly loads the Vault-rendered JSON before creating the DreamConsistency_ServiceDescriptor. When a key exists in multiple providers, the later Vault JSON provider takes precedence over earlier providers. DC_VAULT_REQUIRED=true enforces a required secrets file.
Figure 3. Configuration precedence and lifecycle. JSON file reload does not by itself prove refresh of singleton-held values or DB connections.
reloadOnChange: true allows the configuration provider to reread the file. It does not prove that DreamConsistency_ServiceDescriptor or existing connection pools rebind automatically. Use a controlled rollout or verified refresh mechanism with readiness checks after DB credential rotation.¶ 9. Jenkins & Docker Swarm Architecture
The reference implementation uses two separate pipelines. The Core Docker Release builds an immutable image from Bitbucket master using restore, security audits, build, 1,374 tests and additional contract tests before pushing the versioned image. The realm-specific Hestino DEV Deploy accepts IMAGE_TAG, pulls the prebuilt artifact, checks runtime prerequisites, generates a Swarm Stack specification and performs a rolling update.
Figure 4. CI and CD are deliberately separate. The host Agent, not Jenkins, retrieves runtime application secrets. The functional readiness gate is a recommendation, not an already-confirmed green build requirement.
¶ Confirmed deployment checks
| Phase | Verified check / action |
|---|---|
| Preflight | Validate immutable image tag X.Y.Z-<12 SHA>; ensure manager mode, hestino-dev overlay network and active Agent. |
| Artifact and file checks | Pull the exact registry image; read the specific secrets mount in an ephemeral container using the actual non-root service identity. |
| Non-secret state | Initialize runtime-public.json from existing running configuration once; thereafter keep it as the controlled source for public configuration. |
| Deployment | Generate Stack YAML with read-only bind, DC_VAULT_REQUIRED=true, two replicas where specified, rolling update and rollback settings. |
| Post-deploy | Check deployed image, expected replica state, no DB connection secrets in Swarm environment, read-only mount and equality of expected non-secret runtime keys. |
2/2 check must not be treated as functional readiness. Add a safe DB-readiness and sign-in/transaction smoke gate before declaring a release ready.¶ 10. Secret Lifecycle & Rotation
Authorized operators create or change the service configuration in KV v2, which maintains versions. The Agent periodically re-reads static secret values and renders the latest JSON. Subsequent service processes load the new file, but an already-running process can retain earlier singleton or connection-pool state.
| Step | Recommended rotation procedure |
|---|---|
| 1. Prepare | Announce the change, assign an owner and prepare overlap between old and new DB credentials when feasible. |
| 2. Write | Update KV v2 through an authorized session; verify the new version using metadata only, without printing payloads. |
| 3. Verify render | Check Agent health, file presence, permissions and freshness without exposing secret values. |
| 4. Rotate consumers | Perform a controlled replica-by-replica restart/rollout and verify application/database readiness. |
| 5. Close | Only after successful functional verification, revoke the old credentials and record version and audit evidence. |
reloadOnChange guarantees application-level singleton or connection-pool refresh.¶ 11. Security Threat Model & Protective Controls
| Threat | Controls and caveats |
|---|---|
| Compromised host / AppRole | Restrict AppRole read permissions to its approved service paths; accept and document the per-host blast radius. Root and Docker admins remain privileged. |
| Secrets in CI logs | Do not output JSON, passwords, tokens or rendered file content; never enable set -x around secret-bearing operations. Tests report only safe status. |
| Secrets in Docker spec | Remove DB connection strings and API credentials from docker service inspect environment exposure; use read-only bind mounts. |
| Registry credential persistence | A Docker warning was observed for plaintext ~/.docker/config.json. An ephemeral DOCKER_CONFIG or credential helper is a recommended improvement. |
| Policy expansion | Prefer explicit read on KV v2 data/ paths; grant list/delete only when separately justified. |
| Operator UI exposure | Treat Vault UI as an administrative surface; review network restrictions, operator permissions and available MFA/SSO capabilities. |
| Broken UNIX permissions | Keep restrictive directory traversal and mode 0640 files, no world-readable secrets, and read-only service mounts. |
| Unverified TLS settings | Confirm actual Agent Vault address, CA/certificate trust chain and TLS verification settings; the architecture document does not establish these details. |
Never record: actual Vault tokens, root tokens, RoleID, SecretID, database passwords or secret-bearing JSON in Wiki.js, Git, Slack, Jenkins workspace artifacts or CI logs. Status-only checks are preferred; hashes of low-entropy data can still leak information.
¶ 12. Monitoring, Backup & Incident Response
¶ Monitoring signals
Track Vault seal/health state, failed AppRole logins, denied KV requests, policy changes, Agent service/journal errors, successful template renders, secret-file presence/permissions/freshness, Swarm restarts and update failures, DB readiness, sign-in smoke results, and the Git SHA → registry digest → deployed image relationship.
¶ Backup and disaster recovery
The actual Vault persistent storage backend and its backup schedule have not been verified. If it is integrated Raft, use the documented vault operator raft snapshot procedures and rehearse recovery in isolation; otherwise use the storage backend-specific backup process. Protect backups as sensitive assets, keep secure copies away from the primary host, and confirm recovery of seal/unseal material and audit evidence.
| Incident symptom | Safe initial verification | Response |
|---|---|---|
| Agent not running | systemctl is-active vault-agent.service | Review systemd journal and authorized Agent configuration; do not display authentication material. |
| Rendered file missing | Privileged stat, directory traversal and permissions | Validate KV path, policy, AppRole, Agent template and destination directory. |
| Swarm reports 2/2; sign-in fails | DB readiness and sanitized application error logs | Fail functional release; investigate service behavior rather than treating replica count as health. |
| CLI SELECT works; .NET times out | Inspect effective descriptor, image version and connection pooling | Isolate application connector/pool/runtime behavior; do not presume a verified cause. |
| Secret changes but process uses old values | File modification time and singleton lifecycle | Controlled rolling restart and functional validation. |
| Deploy SSH claims directory is missing | Read-only mount test as actual container UID/GID | Preserve secure host directory permissions. |
¶ 13. Lessons from the Hestino DEV Migration
During the 23–24 October 2026 AccessControl migration, the Agent correctly rendered the file, yet a host-side SSH directory check failed because the deploy user lacked UNIX directory traversal rights. The container identity could read through the intended mount. Additional Jenkins embedded Python quoting and validation of an empty ServiceInstance_UUID were corrected.
A green deploy was followed by MySqlConnector Connect Timeout expired during sign-in. Non-secret values, the DB connection string, network/TCP, handshake and MariaDB SELECT 1 were checked. Temporary status-only application probes reported ConfigMatchesVault=True and DescriptorMatchesVault=True on both replicas. Functionality returned after another image/deploy cycle, but the cause of the earlier timeout was not conclusively proven. A refreshed connection-pool state is a hypothesis, not a confirmed root cause.
| Rule | Operational implication |
|---|---|
| Avoid compounded changes | Do not combine a secret-source migration with unrelated shared NuGet package version changes when separable. |
| Preserve legitimate runtime semantics | Do not invent a ServiceInstance_UUID if an empty value is legitimate in that environment. |
| Test arrays by effective key | Do not relocate indexed whitelist entries blindly into a new Vault JSON array. |
| Use remote commit identity | Immutable image tags derive from the remote master commit, not necessarily the local working tree. |
| Keep rollback until sign-off | Do not remove the old .env until functional verification and approved cleanup. |
| Remove diagnostics from release images | Temporary config-matching probes are troubleshooting aids, not intended permanent output. |
¶ 14. Decisions, Open Items & Improvement Plan
| Decision / action | Status | Next action |
|---|---|---|
| Shared host Agent and AppRole per product/environment | CONFIRMED for Hestino DEV | Reassess trust scope before using on another host or product. |
| One KV v2 JSON config per service | CONFIRMED | Retain separate templates and restricted destination directories. |
| Actual Vault audit device and retention | TO VERIFY | Inventory configured audit sinks, retention and responsibility. |
| Storage, HA, seal/unseal and backup | TO VERIFY | Record actual topology and exercise a storage-appropriate restore. |
| AppRole token and SecretID lifetimes | TO VERIFY | Document rotation and recovery without service interruption. |
| Database readiness and login smoke gate | RECOMMENDED | Add non-secret, safe functional checks to the deploy acceptance gate. |
| AccessControl health endpoint currently not DB-aware | CONFIRMED in reference | Introduce dedicated readiness or functional verification. |
| Operational secret rotation workflow | RECOMMENDED | Define owner, cadence/SLAs, credential overlap and rollout approvals. |
| Non-root container identity standardization | RECOMMENDED | Prefer 1654:1654 where appropriate; treat current Admin exception separately. |
| Secure registry login handling | RECOMMENDED | Avoid persisting plaintext Docker credentials on deploy identities. |
¶ 15. Related Guides & References
¶ Authoritative sources
| Reference | Description |
|---|---|
DC-VAULT-ARCH-001 · v1.0 · 2026-10-24 | Internal architecture reference for Hestino DEV; basis for this Wiki edition. |
DC-VAULT-RUNBOOK-001 · v1.0 · 2026-10-24 | Internal service onboarding and operational implementation guide. |
| Vault Agent AppRole auto-auth | Official Vault documentation. |
| Vault Agent templates | Official Vault documentation. |
| KV secrets engine v2 | Official Vault documentation. |
| Vault Raft operator commands | Only applicable once integrated Raft storage has been confirmed. |
¶ 16. Change Log
| Date | Change | Author / source |
|---|---|---|
| 2026-10-26 | Created English Wiki.js architecture page from the verified Hestino DEV architecture document; added dedicated diagrams, trust boundaries and linked runbooks. | Dream Consistency Engineering / documentation draft |
| 2026-10-24 | Published baseline internal architecture reference, version 1.0. | DC-VAULT-ARCH-001 |