deepnet/dit-releases

Watch

Fork 0

tenant-converge DNS phase: cold-start chicken-and-egg + non-idempotent record creation #7

New issue

Closed

opened 2026-08-11 20:07:01 +00:00 by jsutter · 1 comment

jsutter commented

2026-08-11 20:07:01 +00:00

Owner

Problem

Two related bugs in dito tenant-converge DNS phase prevent cold-start convergence of a new tenant with dns.provider = "cloudflare":

1. Cold-start chicken-and-egg

The DNS phase reads the Cloudflare API token from the tenant OpenBao at https://keys.<domain>. But keys.<domain> only resolves after the DNS phase creates the A record for it. Without DNS records, the converge can't reach OpenBao to read the CF token to create DNS records.

Workaround: manually create DNS records via the Cloudflare API, then ack GATE-CF-TOKEN to defer the DNS phase. But this leads to bug #2.

2. Non-idempotent DNS record creation

When DNS records already exist (created manually or from a prior partial converge), the DNS phase tries to CREATE them again via the Cloudflare API. Cloudflare returns 400 "An A, AAAA, or CNAME record with that host already exists" (error code 81053). The converge treats this as a fatal error and stops.

The DNS phase should check for existing records before creating (upsert, not create). Or use the Cloudflare API's upsert behavior.

Reproduction

dito tenant-init sjc ... (new tenant with cloudflare DNS)
dito tenant-converge sjc --ack GATE-REBUILD-CONFIRM — DNS phase gates (GATE-CF-TOKEN) because OpenBao unreachable (no DNS records)
Manually create DNS A records for keys/forge/auth/*.apps via Cloudflare API
dito tenant-converge sjc --ack GATE-REBUILD-CONFIRM — DNS phase runs, tries to CREATE records, gets 400 (already exists), fatal error

Suggested fixes

Cold-start: the DNS phase should fall back to connecting to the OpenBao via the container's internal IP (e.g., http://10.77.0.13:8200) when the public URL (https://keys.<domain>) is unreachable. Or support an OPENBAO_ADDR env var override.
Idempotency: the DNS phase should list existing DNS records before creating. If a record already exists with the same type+name+content, skip it. If it exists with different content, update it (upsert). Cloudflare's API supports listing records by name — use that to check before creating.

Environment

dito v1.5.4
Observed during SJC tenant cold-start (tenants/sjc.nix, domain sonomajetcenter.com)
Required manual DNS record creation + GATE-CF-TOKEN deferral to complete convergence
The tenant-init output also has a bug: sealConfig one-liner is missing a trailing semicolon (Nix syntax error)

## Problem Two related bugs in `dito tenant-converge` DNS phase prevent cold-start convergence of a new tenant with `dns.provider = "cloudflare"`: ### 1. Cold-start chicken-and-egg The DNS phase reads the Cloudflare API token from the tenant OpenBao at `https://keys.<domain>`. But `keys.<domain>` only resolves after the DNS phase creates the A record for it. Without DNS records, the converge can't reach OpenBao to read the CF token to create DNS records. **Workaround:** manually create DNS records via the Cloudflare API, then ack `GATE-CF-TOKEN` to defer the DNS phase. But this leads to bug #2. ### 2. Non-idempotent DNS record creation When DNS records already exist (created manually or from a prior partial converge), the DNS phase tries to CREATE them again via the Cloudflare API. Cloudflare returns 400 `"An A, AAAA, or CNAME record with that host already exists"` (error code 81053). The converge treats this as a fatal error and stops. The DNS phase should check for existing records before creating (upsert, not create). Or use the Cloudflare API's upsert behavior. ## Reproduction 1. `dito tenant-init sjc ...` (new tenant with cloudflare DNS) 2. `dito tenant-converge sjc --ack GATE-REBUILD-CONFIRM` — DNS phase gates (GATE-CF-TOKEN) because OpenBao unreachable (no DNS records) 3. Manually create DNS A records for keys/forge/auth/*.apps via Cloudflare API 4. `dito tenant-converge sjc --ack GATE-REBUILD-CONFIRM` — DNS phase runs, tries to CREATE records, gets 400 (already exists), fatal error ## Suggested fixes 1. **Cold-start:** the DNS phase should fall back to connecting to the OpenBao via the container's internal IP (e.g., `http://10.77.0.13:8200`) when the public URL (`https://keys.<domain>`) is unreachable. Or support an `OPENBAO_ADDR` env var override. 2. **Idempotency:** the DNS phase should list existing DNS records before creating. If a record already exists with the same type+name+content, skip it. If it exists with different content, update it (upsert). Cloudflare's API supports listing records by name — use that to check before creating. ## Environment - dito v1.5.4 - Observed during SJC tenant cold-start (tenants/sjc.nix, domain sonomajetcenter.com) - Required manual DNS record creation + GATE-CF-TOKEN deferral to complete convergence - The `tenant-init` output also has a bug: `sealConfig` one-liner is missing a trailing semicolon (Nix syntax error)

jsutter commented

2026-08-14 20:14:17 +00:00

Author

Owner

Both bugs are already fixed in the current codebase (v1.9.0).

Bug 1 (cold-start chicken-and-egg): The DNS phase now falls back to an SSH port-forward to the container's internal IP when the public OpenBao URL is unreachable. See src/phases/dns.rs:245-251:

let container_ip = ctx.config.container_ip();
if !container_ip.is_empty() {
    // ... port_forward(LOCAL_PORT, container_ip, 8200)
}

There is also a cf_token_unreachable_gate (dns.rs:68) that handles the unreachable case gracefully with an actionable gate message.

Bug 2 (non-idempotent DNS creation): The DNS phase now uses converge_cf_upsert (dns.rs:424) which calls cf.ensure_record — an idempotent upsert that treats already-present records as success (line 482). A GATE-DNS-OVERWRITE gate handles records with different content before overwriting.

Both fixes shipped between v1.5.4 and v1.9.0. Closing as already-fixed.

Both bugs are already fixed in the current codebase (v1.9.0). **Bug 1 (cold-start chicken-and-egg):** The DNS phase now falls back to an SSH port-forward to the container's internal IP when the public OpenBao URL is unreachable. See `src/phases/dns.rs:245-251`: ```rust let container_ip = ctx.config.container_ip(); if !container_ip.is_empty() { // ... port_forward(LOCAL_PORT, container_ip, 8200) } ``` There is also a `cf_token_unreachable_gate` (`dns.rs:68`) that handles the unreachable case gracefully with an actionable gate message. **Bug 2 (non-idempotent DNS creation):** The DNS phase now uses `converge_cf_upsert` (`dns.rs:424`) which calls `cf.ensure_record` — an idempotent upsert that treats already-present records as success (line 482). A `GATE-DNS-OVERWRITE` gate handles records with different content before overwriting. Both fixes shipped between v1.5.4 and v1.9.0. Closing as already-fixed.

jsutter closed this issue

2026-08-14 20:14:17 +00:00

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

deepnet/dit-releases#7

No description provided.

Rows
Columns