Configuration and Secrets for Cloud-Native .NET: Layering, Mounted Secrets, Key Vault and Validation

Learning outcome

You can predict which configuration source wins in ASP.NET Core, read Kubernetes Secrets mounted as files with the key-per-file provider, explain why a Kubernetes Secret is not encrypted by being base64, choose between environment variables, mounted files and Azure Key Vault with workload identity, and make a bad setting stop the app at startup instead of at the first request.

Intuition

Configuration is a stack of transparent sheets. Each source writes some keys; a later sheet covers the same key on an earlier one. Secrets are just keys that must never be printed, never be baked into an image, and should be rotated without a rebuild.

flowchart BT
    A["appsettings.json"] --> B["appsettings.Production.json"] --> C["environment variables (ConfigMap / Secret env)"] --> D["command line"] --> E["mounted secret files / Key Vault (added after builder creation)"]
    E --> R["IConfiguration: last writer wins"]

Deep dive

• Default order in WebApplication.CreateBuilder: appsettings.json, appsettings.{Environment}.json, user secrets (Development only), environment variables, command-line arguments. Later sources win. Anything you add after the builder is created (Key Vault, key-per-file) goes on top.
• Kubernetes ConfigMap and Secret: both can be injected as environment variables (envFrom) or mounted as files, one file per key. The name of each file becomes the key, and the key-per-file provider maps __ in a file name to :.
• Env vars or files?
  • Env vars are read once at process start; changing the Secret has no effect until the pod restarts.
  • Mounted Secret files are refreshed by the kubelet when the Secret changes (not when mounted with subPath), so the app can pick up a rotated value if the provider reloads.
  • Env vars leak more easily: crash dumps, /proc, diagnostics endpoints and child processes all see them.
• A Secret is base64, not encryption. Anyone who can read the Secret object can decode it. Protect it with Kubernetes RBAC, and use encryption at rest for etcd (on AKS, KMS etcd encryption with an Azure Key Vault key is available).
• Azure Key Vault on AKS, two common patterns:
  • Secrets Store CSI driver (the AKS "Azure Key Vault provider" add-on) mounts Key Vault secrets as files in the pod; read them with key-per-file. It can also sync them into a Kubernetes Secret if you need env vars.
  • Configuration provider: builder.Configuration.AddAzureKeyVault(vaultUri, new DefaultAzureCredential()) (package Azure.Extensions.AspNetCore.Configuration.Secrets). On AKS the credential comes from workload identity (Lesson 8), so no client secret exists anywhere. Key Vault names use -- where configuration uses :.
• Validate on start: AddOptions<T>().BindConfiguration("Section").ValidateDataAnnotations().ValidateOnStart() makes the host throw OptionsValidationException during StartAsync. In Kubernetes the pod crashes before it becomes ready, the rollout stops, and the old pods keep serving (Lesson 9). Without it, the failure appears on the first request that touches the options.
• Never log secrets. Log whether a value is present, its length, or a redacted prefix. OpenTelemetry and structured logs make accidental capture easier, not harder.

Code example

// L02: configuration layering, environment variables, Kubernetes-style secret files (key-per-file),
// options validation at startup, and keeping secrets out of logs.
using System.ComponentModel.DataAnnotations;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;

// 1. Layering: later providers win. Default host order: appsettings.json, appsettings.{Env}.json,
//    user secrets (Development only), environment variables, command line.
Environment.SetEnvironmentVariable("Payments__TimeoutSeconds", "5");                 // injected by the platform
IConfiguration cfg = new ConfigurationBuilder()
    .AddInMemoryCollection(new Dictionary<string, string?> { ["Payments:TimeoutSeconds"] = "30", ["Payments:Region"] = "westeurope" })   // stands in for appsettings.json
    .AddInMemoryCollection(new Dictionary<string, string?> { ["Payments:TimeoutSeconds"] = "10" })                                       // stands in for appsettings.Production.json
    .AddEnvironmentVariables()
    .AddCommandLine(["--Payments:Region=centralindia"])
    .Build();
Console.WriteLine($"Payments:TimeoutSeconds = {cfg["Payments:TimeoutSeconds"]} (env var Payments__TimeoutSeconds wins over both JSON layers)");
Console.WriteLine($"Payments:Region = {cfg["Payments:Region"]} (command line wins)");

// 2. Kubernetes mounts a Secret as a directory: one file per key. KeyPerFile reads it; "__" in a file name maps to ":".
var secretDir = Path.Combine(Path.GetTempPath(), "tin-secrets-demo");
Directory.CreateDirectory(secretDir);
File.WriteAllText(Path.Combine(secretDir, "Payments__ApiKey"), "sk_live_DEMO_ONLY_9f3a");
File.WriteAllText(Path.Combine(secretDir, "ConnectionStrings__orders"), "Host=pg;Database=orders;Password=DEMO_ONLY");
var withSecrets = new ConfigurationBuilder().AddConfiguration(cfg).AddKeyPerFile(secretDir, optional: false).Build();
Console.WriteLine($"secret file -> Payments:ApiKey present: {withSecrets["Payments:ApiKey"] is { Length: > 0 }}, length {withSecrets["Payments:ApiKey"]!.Length}");
Console.WriteLine($"secret file -> ConnectionStrings:orders starts with: {withSecrets.GetConnectionString("orders")![..7]}");

// A Kubernetes Secret's data field is base64: an encoding, not encryption.
string b64 = Convert.ToBase64String("DEMO_ONLY"u8);
Console.WriteLine($"base64 '{b64}' decodes to '{System.Text.Encoding.UTF8.GetString(Convert.FromBase64String(b64))}' with no key");

// 3. Options validation at startup: a bad value stops the host before it takes traffic.
async Task<string> StartWith(string timeout)
{
    var b = Host.CreateApplicationBuilder();
    b.Logging.ClearProviders();                    // keep the demo output short
    b.Configuration.AddInMemoryCollection(new Dictionary<string, string?> { ["Payments:TimeoutSeconds"] = timeout, ["Payments:Region"] = "westeurope", ["Payments:ApiKey"] = "x" });
    b.Services.AddOptions<PaymentsOptions>().BindConfiguration("Payments").ValidateDataAnnotations().ValidateOnStart();
    using var host = b.Build();
    try { await host.StartAsync(); await host.StopAsync(); return "started"; }
    catch (OptionsValidationException ex) { return "refused to start: " + ex.Failures.First(); }
}
Console.WriteLine($"TimeoutSeconds=5   -> {await StartWith("5")}");
Console.WriteLine($"TimeoutSeconds=500 -> {await StartWith("500")}");

// 4. Never log secrets: log presence/length or a redacted form.
static string Redact(string s) => s.Length <= 4 ? "****" : s[..3] + new string('*', s.Length - 3);
Console.WriteLine($"log line: ApiKey={Redact(withSecrets["Payments:ApiKey"]!)}");
Directory.Delete(secretDir, true);

public sealed class PaymentsOptions
{
    [Range(1, 60)] public int TimeoutSeconds { get; set; }
    [Required] public string Region { get; set; } = "";
    [Required] public string ApiKey { get; set; } = "";
}

Output (.NET 10, Release)

Payments:TimeoutSeconds = 5 (env var Payments__TimeoutSeconds wins over both JSON layers)
Payments:Region = centralindia (command line wins)
secret file -> Payments:ApiKey present: True, length 22
secret file -> ConnectionStrings:orders starts with: Host=pg
base64 'REVNT19PTkxZ' decodes to 'DEMO_ONLY' with no key
TimeoutSeconds=5   -> started
TimeoutSeconds=500 -> refused to start: DataAnnotation validation failed for 'PaymentsOptions' members: 'TimeoutSeconds' with the error: 'The field TimeoutSeconds must be between 1 and 60.'.
log line: ApiKey=sk_*******************

Code walkthrough

• Payments:TimeoutSeconds was 30 in the base layer and 10 in the "Production" layer; the environment variable Payments__TimeoutSeconds=5 won, because environment variables are added later.
• Payments:Region was westeurope in JSON and centralindia on the command line; the command line wins over everything in the default order.
• Two files in a temporary directory stand in for a mounted Secret volume. Key-per-file turned Payments__ApiKey into Payments:ApiKey (length 22) and ConnectionStrings__orders into the orders connection string.
• REVNT19PTkxZ decoded to DEMO_ONLY with one call and no key: that is all a Kubernetes Secret's data field protects.
• With TimeoutSeconds=500 the host refused to start: the [Range(1, 60)] attribute failed inside StartAsync because of ValidateOnStart.
• The log line shows sk_ followed by asterisks: enough to tell which kind of key is configured, nothing usable.

Failure modes

• Secrets in appsettings.json committed to Git or copied into the image.
• Treating base64 in a Secret manifest as protection, and committing that manifest.
• Reading config into a static field at startup and expecting a rotated secret to take effect.
• Missing validation: the app starts "healthy" and fails on the first payment.
• Using env vars for large or multi-line values (certificates); mount files instead.
• A : in an environment variable name on Linux shells; use __.

Interview angle

"How do you manage secrets for a .NET service on AKS?" Answer: "Secrets live in Azure Key Vault. The pod uses workload identity, so there is no credential to store: either the Secrets Store CSI driver mounts the secrets as files that I read with key-per-file, or the app reads Key Vault through the configuration provider with DefaultAzureCredential. Non-secret settings come from a ConfigMap as environment variables. I validate options with ValidateOnStart so a bad value fails the rollout, and I never log secret values. Kubernetes Secrets are only base64, so I limit who can read them with RBAC."

Practice questions

1. An env var and appsettings.Production.json set the same key. Which wins?

Answer: The env var; it is added after the JSON files.

2. You rotate a Kubernetes Secret used through envFrom. When does the app see the new value?

Answer: After the pod restarts; env vars are fixed at process start. Mounted files (without subPath) are refreshed.

3. What does ValidateOnStart change in a Kubernetes rollout?

Answer: A bad value crashes the new pod at startup, it never becomes ready, and the rollout stalls while old pods keep serving.

4. What is the Key Vault name for ConnectionStrings:orders?

Answer: ConnectionStrings--orders.

Revision checklist

• Order: JSON, env-specific JSON, user secrets (Dev), env vars, command line; later wins.
• __ in env vars and secret file names maps to :.
• Secrets: Key Vault + workload identity; CSI files or the config provider.
• base64 is encoding; protect Secrets with RBAC and encryption at rest.
• ValidateOnStart; redact in logs.

Related on TechInterviewNotes

• Lesson: Identity, security and configuration on Azure
• Lesson: Security, configuration, health checks and production delivery
• Lesson: Debugging: is it running? Startup, logs and configuration

References

• Configuration in ASP.NET Core
• Kubernetes Secrets
• Options pattern: validation
• Azure Key Vault configuration provider in ASP.NET Core
• Use the Azure Key Vault provider for Secrets Store CSI Driver in AKS