Connections, Part 2: The Secret Lifecycle — Masking, Merging, Refreshing

There is one invariant our entire credential system is built around, and it fits in a sentence: a plaintext secret crosses the wire exactly once — at creation — and the browser never sees it again. Not on the edit form, not in an API response, not in a "reveal" endpoint. After the moment a user types a secret and hits save, the only thing the client ever receives back is a sentinel that means "a value is set."

That sounds obvious. Almost nobody does it. Open the settings page of your favorite SaaS tool, pop the network tab, and edit an API key. You will be surprised how often the stored secret comes back to prefill the form.

This is part 2 of a two-part series on our connections model:

1. Part 1 covered the data model: one blueprint table, one instance table, and declarative auth application.
2. This post is the secret's lifetime: how it is stored, how it is edited without ever being read back, and how OAuth tokens stay fresh without stampedes.

Auxx.ai is open source, so every claim here is checkable in the repo.

The box

Secrets live in a single encryptedSecrets text column on Credential, as an AES-256-GCM box with a versioned format:

// packages/credentials/src/crypto/secret-box.ts
/**
 * Encrypt a secrets object to the versioned format:
 * `v2:` + base64(iv(12) ‖ authTag(16) ‖ ciphertext).
 */
export function encryptSecrets(secrets: Record<string, unknown>): string {
  const iv = crypto.randomBytes(IV_LENGTH)
  const cipher = crypto.createCipheriv('aes-256-gcm', getKey(), iv)
  const ciphertext = Buffer.concat([
    cipher.update(JSON.stringify(secrets), 'utf8'),
    cipher.final(),
  ])
  const authTag = cipher.getAuthTag()
  return V2_PREFIX + Buffer.concat([iv, authTag, ciphertext]).toString('base64')
}

The v2: prefix is doing real work: it makes the format self-describing, which lets us run two decryption policies against the same primitives.

Credential secrets decrypt strictly — decryptSecrets throws on anything without the v2: prefix, because an unencrypted credential secret is a bug, full stop. But ConnectionDefinition columns like oauth2ClientSecret decrypt leniently: decryptValue returns a non-v2: payload unchanged. That one policy difference is what made encrypting those columns deployable at all — you can ship the code that reads encrypted values before the backfill that encrypts existing rows, and plaintext dev seeds keep working. Deploy-then-backfill, no flag day.

Inside the box, the JSON has a small conventional shape: OAuth tokens at accessToken / refreshToken, a bare API key at secret, and multi-field connection variables nested under fields (host, password, client_secret, …). That nesting matters enormously in a minute.

The sentinel: what the browser actually sees

When a user opens the edit form for an existing connection, the server has to seed the form with something for each secret field. It sends this:

// packages/credentials/src/crypto/client.ts
/**
 * Sentinel a form submits in place of an unchanged masked secret.
 * The server strips it instead of persisting it.
 */
export const HIDDEN_VALUE = '__HIDDEN__'

export function maskForEdit(fields: MaskField[], stored: Record<string, unknown>) {
  const values: Record<string, string> = {}
  for (const field of fields) {
    const raw = stored[field.key]
    if (field.secret) {
      values[field.key] = raw != null && raw !== '' ? HIDDEN_VALUE : ''
    } else {
      values[field.key] = raw == null ? '' : String(raw)
    }
  }
  return values
}

A secret field with a stored value becomes the literal string __HIDDEN__ — an "is set" marker, nothing more. A secret field with nothing stored becomes '', so the form knows to render it as required. Plain fields come back real, because they were never secrets.

Note the loop: only declared fields are emitted. The projection iterates the definition's field list, not the stored bag, so system-provisioned keys — accessToken, client_id, anything an app wrote programmatically — are structurally excluded from the response. There is no denylist to keep updated; keys the form didn't declare simply never exist in the projection.

The save path is the mirror image. The client submits the form bag back, and any secret field still carrying the sentinel is dropped before the write:

// packages/credentials/src/crypto/client.ts — resolveForWrite
if (field.secret) {
  if (isMasked(value)) continue // unchanged → let the store merge keep existing
  secrets[field.key] = value
}

isMasked accepts two shapes: the exact HIDDEN_VALUE sentinel, and anything matching the mask-shape regex /^.{2,4}\*+.{2,4}$/ — a few characters, a run of asterisks, a few characters. The second case is defensive: some display surfaces show a maskValue-style preview like sk-a****3xyz, and if a client ever echoes one of those back, persisting it would silently corrupt the stored secret. Both shapes mean "the user didn't touch this," and both are stripped.

One more detail from maskValue itself: secrets shorter than ten characters return a fixed-length full mask (********) rather than a proportional one. A mask that shrinks with the secret leaks the secret's length, and "this API key is six characters" is information an attacker shouldn't get for free.

The whole module — sentinel, regex, projection, split — lives in packages/credentials/src/crypto/client.ts, and it is client-safe by construction: no Node crypto, no database imports, nothing but pure functions. The browser bundle can import the exact code the server uses to decide what is masked, so the two sides can never disagree about what the sentinel is.

Merge, never replace

Dropping the sentinel creates an obligation: if the submitted bag no longer contains client_secret, the write path must keep the stored client_secret, not delete it. Every credential write is therefore a merge, and the merge has to be nesting-aware.

Recall the stored shape — multi-field secrets live under fields, as siblings of the tokens:

{
  "accessToken": "ya29...",
  "refreshToken": "1//0g...",
  "fields": { "client_id": "...", "client_secret": "..." }
}

A naive flat merge — { ...existing, ...partial } where partial is { client_secret: 'new' } — writes client_secret at the top level and leaves the real one under fields untouched. Worse, a flat merge of { fields: { client_secret: 'new' } } replaces the whole fields object, silently wiping client_id. We know because an early version did exactly that. The fix is a dedicated store function that merges one level down:

// packages/credentials/src/store/merge-secret-fields.ts
const existingFields = (existing.fields ?? {}) as Record<string, unknown>
const mergedFields = { ...existingFields }
for (const [key, value] of Object.entries(partial)) {
  if (value === undefined || value === '') continue // keep existing
  mergedFields[key] = value
}
const merged = { ...existing, fields: mergedFields }

Three properties, each one a bug we no longer have: blank or absent fields keep their stored value (the "leave it empty to keep it" edit semantics); edited fields overwrite only themselves; and sibling keys — accessToken, refreshToken, secret — are untouched by the ...existing spread, so editing a connection variable can never clobber a live token.

The edit surface funnels everything through one function, mergeManualConnectionEdit in packages/lib/src/connections/merge-manual-edit.ts: multi-field secrets go through mergeSecretFields, a bare API key merges at secrets.secret, and plain variables read-modify-write into metadata.connectionVariables. Both the platform reconnect path and the app reconnect path call it, so there is exactly one place where "editing a connection" is defined.

Put the last two sections together and you get the full loop. Read: decrypt server-side, project through maskForEdit, send sentinels. Write: strip sentinels through resolveForWrite, merge the survivors into the encrypted bag. The plaintext secret existed in transit once, on creation, and never again.

Refreshing without stampedes

OAuth access tokens expire, and the naive strategy — refresh when a request 401s — means every expiry costs a failed request. So we refresh ahead of expiry. The obvious implementation is a fixed window: refresh whenever less than two minutes remain.

The fixed window has a failure mode we hit in production. Some providers issue tokens with very short lifetimes — MCP servers in particular hand out tokens living only a few minutes. If the token's whole lifetime is at or under your refresh-ahead window, the token is always "expiring": every single call triggers a refresh, and since many providers rotate the refresh token on each use, you churn through rotations as fast as you make requests. The window has to be proportional:

// packages/lib/src/credentials/ensure-fresh-credential-token.ts
/**
 * Refresh-ahead window: `min(120s, 25% of token lifetime)`. A fixed window
 * would make any token with a TTL at or under it permanently "expiring".
 */
function expirySkewMs(input: { expiresAt: Date; lastRefreshAt?: Date | null; createdAt?: Date }) {
  const issuedAt = input.lastRefreshAt ?? input.createdAt
  if (!issuedAt) return MAX_SKEW_MS
  const lifetimeMs = input.expiresAt.getTime() - issuedAt.getTime()
  if (lifetimeMs <= 0) return MAX_SKEW_MS
  return Math.min(MAX_SKEW_MS, lifetimeMs * SKEW_LIFETIME_FRACTION)
}

A one-hour Google token refreshes two minutes early. A five-minute MCP token refreshes seventy-five seconds early. Neither is ever permanently expiring.

The second production problem is concurrency. A busy org has workers, web requests, and scheduled jobs all resolving the same credential; when its token nears expiry, they all decide to refresh at once. With rotate-on-use refresh tokens this is not merely wasteful — it is destructive: two concurrent refreshes race, and the loser persists a refresh token the provider has already invalidated. The credential is now dead and the user has to reconnect.

So refresh is single-flight, per credential, via a Redis SET NX lock:

const acquired = await redis.set(lockKey(credentialId), '1', 'EX', 30, 'NX')
if (!acquired) {
  // Someone else is refreshing — wait for the lock to clear, then let the
  // caller re-read whatever the winner persisted.
  for (let i = 0; i < LOCK_POLLS; i++) {
    await new Promise((resolve) => setTimeout(resolve, LOCK_POLL_DELAY_MS))
    if (!(await redis.get(lockKey(credentialId)))) break
  }
  return true
}

Losers don't fail and don't refresh — they poll briefly for the lock to clear, then return true, which tells the caller "the stored secrets may have changed, re-read them." The 30-second TTL bounds a crashed holder.

Two policy decisions in this function are worth stating out loud, because both invert the instinct to fail safe.

First: if Redis is down, refresh anyway, without the lock. The comment in the code says it in five words — "correctness beats stampede protection." A missed lock risks a redundant refresh; refusing to refresh guarantees requests go out with an expired token. We take the race.

Second: the function never throws. A failed refresh logs a warning and leaves the stored token in place. Maybe the token still works; maybe the provider is having a bad minute; either way the caller's 401 retry path — which calls back in with force: true, skipping the expiry check entirely — owns the live failure. Refresh is an optimization layered on a correct slow path, and an optimization that can take down the request it was optimizing is not one.

Two grants, one seam, inverted triggers

Part 1 made the claim that client-credentials is "downstream just a bearer connection." The token-production seam is where the two grants actually differ, and the difference is a beautiful little inversion.

Both route through ensureFreshCredentialToken. For the refresh_token grant, a credential with no expiresAt means we can't tell when this token dies — so do nothing, and let the 401 path handle it if it's dead. For client-credentials, the same missing expiresAt means the opposite: no token has ever been minted — so mint one right now, before the first request:

if (grant === 'refresh_token' && !hasRefreshToken) return false
if (!force) {
  if (!expiresAt) {
    // refresh: can't tell → 401 path owns it. client-credentials: no token yet → mint now.
    if (grant === 'refresh_token') return false
  }
  // ...else: shared skew-window check
}

Identical state, opposite meanings, one if. From there the grants diverge into refreshCredentialTokens and mintClientCredentialToken in packages/lib/src/connections/oauth2-token-grants.ts — but both POST through a single shared postOAuth2TokenRequest helper that handles basic-auth vs. request-body client authentication identically, so the two grants can never drift on header handling. And a rotation detail that guards against a common provider behavior: refreshToken: tokenData.refresh_token || secrets.refreshToken — providers that don't rotate the refresh token simply omit it from the response, and overwriting the stored one with undefined would orphan the credential.

Two failure surfaces, deliberately separate

When refresh fails, two different audiences need two different answers.

The retry machinery needs "how many times in a row has this failed?" That's consecutiveRefreshFailures, a plain counter on the credential — a circuit breaker that stops the scheduled refresh job from hammering a provider. A recognizably permanent failure (an invalid refresh token) jumps the counter straight to the open threshold instead of incrementing, because retrying an invalid grant five times is just five more log lines.

The user needs "do I have to reconnect?" That's a separate classified layer: lastAuthError (the classified error type, e.g. invalid_grant), lastAuthErrorAt, and a requiresReauth boolean that the UI renders as the reconnect banner. Mixing these up gives you either a scary "reconnect your account" banner for a transient network blip, or a silent breaker that never tells the user their revoked token needs attention.

The recovery path ties them together. A successful refresh doesn't just store the new token — it clears everything:

// packages/credentials/src/store/record-refresh.ts — recordRefreshSuccess
.set({
  consecutiveRefreshFailures: 0,
  requiresReauth: false,
  lastAuthError: null,
  lastAuthErrorAt: null,
  lastRefreshAt: now,
  expiresAt: options.expiresAt,
})

A fresh token is the proof of health. Without this, a credential that recovered on its own keeps showing "Auth required" forever — a bug we shipped once and now enforce against by making recordRefreshSuccess and recordRefreshFailure the only writers of the breaker fields.

No fallbacks at the edges

The last piece is a policy about failure at the consumption edge. Our email channels — Gmail, Outlook — get their access tokens through this whole stack via getChannelAccessToken in packages/lib/src/providers/channel-token-accessor.ts, and that function has a property we defend in review: there is no stored-token fallback.

// Reads go ONLY through the resolver — there is no stored-token fallback: an
// unlinked credential or a resolver failure returns null so the caller fails
// loudly rather than silently serving a stale/unrefreshed token.

The tempting version reads the credential row directly when the resolver fails — after all, there's probably a token sitting right there. But a fallback like that converts every resolver bug into a silent one: the channel keeps limping along on a stale token that works until it doesn't, at 3 a.m., with no error pointing at the actual cause. Returning null means an unlinked credential fails on the first send with a clear log line, while the credential is still fresh in someone's memory. Loud and early beats quiet and eventual.

The whole lifecycle, in one paragraph

A secret is typed once and encrypted into a versioned AES-256-GCM box. Every read after that is a server-side reveal; the browser only ever sees __HIDDEN__. Edits strip the sentinel and merge — nesting-aware, blank-keeps-existing — so no field can clobber its siblings. OAuth tokens refresh ahead of expiry through a proportional window, single-flight behind a Redis lock, with failures counted by a breaker for the machines and classified into a reconnect signal for the humans. And when any of it fails, it fails loudly at the edge instead of silently serving yesterday's token.

That's the series. Part 1 was the model — one blueprint table, one instance table, auth as data. This part was the lifecycle that keeps the model trustworthy. Together they're why adding a new provider to Auxx.ai is a row of declarative data, and why our AI keys, app connections, MCP servers, and email channels all get masking, merging, and refresh for free.

Entry points, if you want to dig in:

• packages/credentials/src/crypto/client.ts, the client-safe sentinel/mask/split core
• packages/credentials/src/crypto/secret-box.ts, the encryption box and both decrypt policies
• packages/credentials/src/store/merge-secret-fields.ts, the nesting-aware merge
• packages/lib/src/credentials/ensure-fresh-credential-token.ts, single-flight refresh
• packages/lib/src/connections/oauth2-token-grants.ts, the two grants

Auxx.ai is open source. PRs welcome.