Configuration reference
Every setting of /etc/kbf/kbf.toml, by table. A key is the long name of a flag (--storage-listen is storage-listen), and a flag on the command line overrides the key. A secret is never a value: the key holds the path of the file that has it. The file must not be writable by group or other. The modes are frontend, storage, daemon, storage-and-daemon, all-in-one.
The source of truth is the templates in crates/kbf-cli/tests/golden, which a test holds equal to the flag definitions in kbf-cli. Where this page and a template differ, the template is right.
Tables: containers daemon handover history identity listener log macos memory outputs scheduler storage-client storage-node stores tls transfer
[containers]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
actions-memory-max-gib | text ACTIONS_MEMORY_MAX_GIB | none | daemon storage-and-daemon all-in-one | Written to the actions cgroup's memory.max at start (container): the most memory all leases together may use, in GiB, so the rest stays for what else runs on the node. The node reports the lowest memory.max above its leases, or its memory when that is less, and the CPUs of their cpuset |
cgroup-parent | text CGROUP_PARENT | none | daemon storage-and-daemon all-in-one | The cgroup lease cgroups are made under, from the cgroup root (container). Left out, the daemon sets up its own cgroup, which its systemd unit must delegate (Delegate=yes): it moves itself into a supervisor leaf there and makes actions beside it, with cpu, memory and pids enabled. Given, it must already enable cpu, memory and pids for its children |
cgroup-root | text | /sys/fs/cgroup | daemon storage-and-daemon all-in-one | Where cgroup v2 is mounted (container). For tests of the cgroup setup only (Not shown in --help.) |
container-nofile | number | 65536 | daemon storage-and-daemon all-in-one | Each container's open-file limit, soft and hard (container). Rootless, it cannot exceed the daemon's own hard limit |
container-nproc | number | 32768 | daemon storage-and-daemon all-in-one | The process limit, soft and hard, of each container's user (container). Every container's root is the same host id, so it bounds all the node's actions together |
container-pids-limit | number | 8192 | daemon storage-and-daemon all-in-one | Each container's pids.max: its tasks, threads included (container) |
container-shm-mib | number | 64 | daemon storage-and-daemon all-in-one | The size of each container's /dev/shm, in MiB (container) |
id-files | text ID_FILES | none | daemon storage-and-daemon all-in-one | A directory holding passwd, subuid and subgid that the container driver's startup check reads instead of /etc's. For tests of that check only (Not shown in --help.) |
lease-placement | text | none | daemon storage-and-daemon all-in-one | Where each lease runs among the node's CPUs (container, Linux): none, the default, lets a lease use any CPU the daemon may; numa-node puts each lease on one NUMA node, its CPUs (cpuset.cpus) and memory (cpuset.mems), the node with the least CPU booked by running leases (the lowest id on a tie). --reserve-cpus is a quota on all leases together, so it still applies; it removes no CPU from a node. Needs the cpuset controller delegated to the daemon (the start fails and says how when it is not); a host with fewer than two NUMA nodes that hold CPUs runs as none, which the start logs. Refused with a driver other than container (macOS has no such driver) |
lease-swap-kill-mib | number | 512 | daemon storage-and-daemon all-in-one | A lease at its own memory cap is killed, and reported out of memory, once it has pushed more than this many MiB into swap at its cap... (container) |
lease-swap-kill-percent | number | 25 | daemon storage-and-daemon all-in-one | ...or this percentage of its booked memory, whichever is more. The kernel caps each lease's RAM at its booking x 1.5 + 512 MiB and lets it swap without limit; a lease is killed by this rule only while its own cap keeps being hit (its memory.events max grew since the last sample) and its swap rose since the last sample, and only swap that rose so counts, never swap that host pressure moved out of it |
lease-swap-poll-ms | number | 1000 | daemon storage-and-daemon all-in-one | How often each lease's max events and swap are sampled, in milliseconds (container) |
numa-dir | text | /sys/devices/system/node | daemon storage-and-daemon all-in-one | Where the kernel lists NUMA nodes (container). For tests of the placement only (Not shown in --help.) |
oci-runtime | text | /usr/bin/crun | daemon storage-and-daemon all-in-one | The OCI runtime that runs each action's container (container): crun, run directly. Each action gets a fresh rootless sandbox. It needs the daemon user's subordinate ids in one range, newuidmap and newgidmap (package uidmap) on PATH, and every directory down to --data-dir searchable by others |
psi-memory-file | text | /proc/pressure/memory | daemon storage-and-daemon all-in-one | Where the kernel reports memory pressure stall information. For tests only (Not shown in --help.) |
self-cgroup | text | /proc/self/cgroup | daemon storage-and-daemon all-in-one | The file naming the daemon's own cgroup (container). For tests of the cgroup setup only (Not shown in --help.) |
supervisor-memory-min-mib | number | 256 | daemon storage-and-daemon all-in-one | Written to the daemon's supervisor leaf's memory.min at start, in MiB (container, without --cgroup-parent): memory the kernel does not reclaim from the daemon while it uses no more, so builds short of memory never take it. The kernel protects no more than the unit's and its slice's MemoryMin= allow; the daemon warns when one is lower. 0 protects nothing |
[daemon]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
async-threads | text N | none | all | Worker threads of the async runtime, which run every task between its waits. Default: every CPU the process may use on the server; max(2, reserve/4), at most the physical cores, on the daemon |
blocking-threads | text N | none | all | The most threads the runtime's blocking pool runs at once, for calls that wait on a disk or a process (a thread is made when a call needs one and ends when idle). Default: 512 (tokio's) for the server, 64 for the daemon |
cpu-threads | text N | none | all | Threads for CPU-bound work, such as hashing, which never waits and so is held to the CPUs. Default: every CPU the process may use on the server; max(2, reserve/2), at most the physical cores, on the daemon |
daemon-user | text | _kbf | daemon storage-and-daemon all-in-one | This daemon's own user, which keeps read and remove access to what a whole-machine lease's user writes in its lease directory (native-whole-machine) |
data-dir | text DIR | none | daemon storage-and-daemon all-in-one | The directory the daemon keeps everything in, on a local disk: leases/ (lease directories, the container and native drivers), and under the container driver (and the native driver on macOS) cache/ (the input cache, docs/design/worker-cache.md) and trees/ (shared read-only input trees, docs/design/fast-sandbox.md#4-shared-trees). Made if missing; absolute. The cache and trees are on, sized from the filesystem: half and a quarter of its size less what it keeps free (5%, at least 1 GiB), and the cache evicts entries no running lease links to keep that much free. A directory another daemon owns, or that cannot be used, is left alone: the daemon warns and runs uncached |
data-max-bytes | text BYTES | none | daemon storage-and-daemon all-in-one | Lowers the size the cache and the trees are budgeted from (half and a quarter of it), under the filesystem's automatic one |
driver | list | none | daemon storage-and-daemon all-in-one | How leases run; once per runtime. The daemon runs each lease on the driver that serves its kind and refuses to start when two serve the same kind. The node's own driver (native or container) is set up first, whatever the order given |
image | list | none | daemon storage-and-daemon all-in-one | A container image this node is pinned to, as an action names it: docker://<repo>@sha256:<digest>, one architecture's manifest digest (container; repeatable). At start the daemon makes sure each is in the node's image store, as the prepare step of every lease does: it fetches the image from its registry by that digest, checks every byte against a digest, and unpacks it once. It exits with status 2 if one cannot be fetched, fails a check, or names an image index. The node's status lists them (GET /v1/nodes, software.container_images). No work is placed by image yet: an action may still name another image |
reconnect-max-ms | number | 30000 | daemon storage-and-daemon all-in-one | The longest wait between rounds of connection attempts, in milliseconds. Each wait is jittered down by up to half |
reconnect-ms | number | 1000 | daemon storage-and-daemon all-in-one | The wait, in milliseconds, after a session ends and after the first round of connection attempts that all failed. It doubles with each further failed round |
reserve-cpus | text N | none | daemon storage-and-daemon all-in-one | Logical CPUs the farm's own processes keep, in CPUs (hyperthreads count one each). The daemon gives its leases every CPU it may use but these, with no pinning, and sizes its own pools from them. Default: 8 (half the CPUs on a machine of fewer than 16). 0 is allowed; a value that leaves no CPU for leases, or is more than the machine has, is refused. On a host that also runs the server, this covers both; the server has no reserve flag |
reserve-share | text N | none | daemon storage-and-daemon all-in-one | How many of the farm's processes on this host share the reserve: the daemon sizes its pools from reserve / share CPUs. Default 1 |
role | text | all | all-in-one | The roles to run |
update-dir | text DIR | none | daemon storage-and-daemon all-in-one | The root installer's directory: the daemon reads its status.json there. The frontend's orders are written to orders/ beneath it unless --orders-dir says otherwise. Left out, the node updates by hand: it says so and touches no directory. Not taken by the all-in-one binary yet |
[handover]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
envoy-check-interval-ms | number | 1000 | frontend | The time between two of Envoy's health checks of a frontend (the cluster's interval). |
envoy-check-timeout-ms | number | 1000 | frontend | How long Envoy's health check waits for an answer (the cluster's timeout). |
envoy-unhealthy-threshold | number | 2 | frontend | How many of Envoy's checks in a row must fail before it stops choosing a frontend (unhealthy_threshold). The three give the settle: a frontend waits interval x threshold + timeout + interval before it goes not ready (Envoy has then seen the successor) and again before it exits (Envoy then sends it nothing). |
handover-follow-interval-ms | number | 200 | frontend | How often a frontend that has handed over looks in the state store for the results of the Execute and WaitExecution streams it kept open. |
handover-peer-name | text DNS_NAME | none | frontend | The DNS name the REAPI certificate of the other instance carries. A frontend whose REAPI listener has a certificate forwards the calls it still receives after a handover over TLS to the successor, verified against --storage-ca and this name; without it such a frontend refuses a handover. |
handover-quiesce-secs | number | 10 | frontend | The longest a frontend that hands over waits for the Execute calls between their request and their queueing, and the results being taken, before it stands down anyway. |
handover-socket | text PATH | none | frontend | The control socket of a handover (mode 0600, this user only). A frontend serving binds it; one started while another serves on the same --state-store (and the other --instance) binds its own listeners, asks the other to hand over its scheduler state and its daemons, and takes them without failing a build. The other goes not ready once Envoy has seen the new one, forwards the calls Envoy still sends it, and exits when it holds nothing. |
handover-watchdog-secs | number | 15 | frontend | How long after it has handed over a frontend waits for its successor to serve. If the successor does not (or its control connection closes), the frontend goes not ready itself, so Envoy answers cleanly instead of choosing a frontend that can start nothing new. |
instance | text a|b | none | frontend | Which of the two instances this is. Instance a listens where --listen, --worker-listen and --api-listen say, instance b ten ports above each (port 0 stays 0): the two blocks Envoy's configuration names. A handover goes to the other instance; without --instance a frontend refuses one. |
[history]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
history-index-root | text ROOT | none | frontend all-in-one | Where history's index rows live in the state store: ui/, or <prefix>ui/ for a prefix that ends in /, so that one state store can hold several farms' history apart. Left out, it is ui/. Needs --history-objects |
history-objects | text URI | none | frontend all-in-one | Where build history is kept: an object store, s3://bucket/prefix?endpoint=http://host:port (the key pair is the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY of the environment) or file:///absolute/path (a directory this process owns, not the cold store's). The attempts and cache hits of each invocation a client named are recorded under <prefix>history/ after their results are sent, never before and never delaying one; the index rows go to --state-store, which this flag needs. A full queue, a slow object store or a slow state store drops records and counts them (kbf_history_dropped_total on /metrics), and never fails or delays a call. The writer reads and writes only keys under history/. Left out, nothing is recorded |
invocation-memory | number | 268435456 | frontend all-in-one | The memory, in bytes, the invocation records may take together (docs/design/invocations.md, section 3.5). At the bound, records of a minted id go first, then idle ones, least recently seen first; with only open builds left, the largest stops taking rows. Records are kept in memory only and lost on restart. Their heap was measured at up to 1.08 times this at 200 kB and above, and 1.12 at 50 kB (section 3.5) |
invocations-open | switch | false | all-in-one | Serves the build pages (GET /v1/invocations/{id} and below) and the blob route (GET /v1/blobs/{digest}) on the operator API to every caller the REAPI policy accepts, with no invocations or blobs key in the policy file (docs/design/invocations.md section 4.3): it fills in allow for whichever of the two keys the file leaves out. Without it, and without the keys, those routes answer 404. Pages and blobs hold build outputs and logs: give it only for a listener that only trusted people reach |
[identity]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
ca-cert | text CA_CERT | none | daemon storage-and-daemon all-in-one | PEM file of the CA certificates that sign server certificates |
cas | text CAS | none | daemon storage-and-daemon all-in-one | The server's worker listener, an https:// URL (normally --server's), that the container and native drivers read and write blobs through, over mutual TLS with this daemon's --ca-cert, --cert, --key and --tls-server-name. Plain http:// is refused: the worker listener admits each blob call by the daemon's certificate. Every driver's blobs go through one transfer engine, over one connection, within the --transfer-* limits |
cert | text CERT | none | daemon storage-and-daemon all-in-one | PEM file of this daemon's client certificate chain |
frontend | text URL | none | storage storage-and-daemon | Registers with this frontend (http:// or https://) and heartbeats to it. Without it the node is a one-member map of itself. |
key | text KEY | none | daemon storage-and-daemon all-in-one | PEM file of this daemon's private key |
label | list | none | daemon storage-and-daemon all-in-one | A node label, key=value, reported as label.<key>; repeatable. A Mac in komira's pool runs with --label pool=darwin-sized |
node-id | text NODE_ID | none | daemon storage-and-daemon all-in-one | A stable name for this node, unique within the cell |
server | list | none | daemon storage-and-daemon all-in-one | A kbf-server worker listener to connect to, as an https://host:port URL; repeatable, or several separated by commas. The host may be a DNS name with a record per server: every round of attempts resolves it again and tries each address. The daemon never stops trying (see --reconnect-max-ms) |
tls-server-name | text TLS_SERVER_NAME | none | daemon storage-and-daemon all-in-one | The name to verify in the server certificate, when it is not the URL's host |
[listener]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
advertise-addr | text HOST:PORT | none | storage storage-and-daemon | The address callers dial, where --storage-listen is not routable. Default: the address the node bound. |
api-listen | text API_LISTEN | none | frontend all-in-one | The operator API listener (HTTP/JSON under /v1). Off unless given. Reads are open to whoever reaches it: bind it where only operators do |
http2-keepalive-interval-secs | number | 30 | frontend all-in-one | How often, in seconds, the REAPI and worker listeners send an HTTP/2 PING on a quiet connection. 0 sends none (and --http2-keepalive-timeout-secs is unused) |
http2-keepalive-timeout-secs | number | 20 | frontend all-in-one | How long, in seconds, a PING may go unanswered before the connection is closed |
listen | text | 127.0.0.1:8980 | frontend all-in-one | The REAPI listener. Without --reapi-tls-cert and --reapi-tls-key it serves plain text, on any address; off loopback the start line carries a warning |
reapi-max-streams | number | 10000 | frontend all-in-one | The most HTTP/2 streams one connection to the REAPI listener may have open. buck2 keeps one Execute stream open per outstanding action, idle until the action finishes, so this bounds how large a backlog one buck2 invocation can show the farm. The connection's flow-control window is this many streams' windows; above about 2000 streams each stream's window is smaller than 1 MiB so that the product stays under HTTP/2's largest window. At least 1, at most 32767 |
releases-dir | text DIR | none | frontend all-in-one | The release store (docs/design/pull-updates.md, section 6): a directory where the tarballs of a release and their build attestations are kept, uploaded with PUT /v1/releases/{commit}/{platform}/{name} on the operator API (which needs the token) and fetched by the nodes through ReleaseService.Fetch on the worker listener, behind its mutual TLS. Made if missing. Without it every update endpoint answers 503 saying so |
shutdown-timeout-secs | number | 0 | frontend all-in-one | How long, in seconds, a stop (SIGINT or SIGTERM) waits for REAPI clients to finish: it refuses new Execute calls (UNAVAILABLE, which Buck2 retries) and sends GOAWAY, and every open Execute and WaitExecution stream stays open until its action is done. Buck2 never recovers a stream that breaks mid-action, so the default, 0, is no deadline; set one at least as long as the longest action. When it runs out, the open streams end UNAVAILABLE (a client still uploading is cut off) |
storage-listen | text | 127.0.0.1:8982 | storage storage-and-daemon | Where the node serves gRPC. Without the TLS flags it serves plain text, and the address must be loopback. |
tls-reload-interval-secs | number | 30 | frontend storage storage-and-daemon all-in-one | How often, in seconds, a TLS listener looks at its certificate, key and client CA files (the --*-tls-cert, --*-tls-key and --*-client-ca flags). A change is applied to the next TLS handshake, with no restart and without ending a connection already open: renew a certificate by replacing the files. Files that do not make a working configuration (unreadable, not PEM, a key that is not the certificate's, an expired certificate, a renewal caught between its two writes) are logged and counted in kbf_tls_reloads_total{outcome="rejected"}, and the previous files keep serving; the next look finds them whole and applies them. The files must be good at start |
worker-listen | text | 127.0.0.1:8981 | frontend all-in-one | The kbf.worker.v1 listener. Without mutual TLS (--worker-tls-cert, --worker-tls-key, --worker-client-ca) it serves plain text, on any address; off loopback the start line carries a warning |
worker-max-streams | number | 200 | frontend all-in-one | The most HTTP/2 streams one connection to the worker listener may have open: a daemon's persistent session stream and its short transfer streams. At least 1, at most 32767 |
[log]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
log-dir | text DIR | none | frontend storage storage-and-daemon all-in-one | The directory of the durable log file, kbf-server.log and its rotated kbf-server.log.1, .2, ... (created if missing): every event stderr gets, in the same format, kept across restarts. Left out, it is <hot store directory>/log when the hot store is a directory; with neither, there is no file and the server logs to stderr alone (the start logs which). A directory or file that cannot be written never stops the server ([crate::log_file]) |
log-files | number | 8 | frontend storage storage-and-daemon all-in-one | How many log files are kept, the current one included; rotation drops the oldest. The log never exceeds --log-max-bytes times this, plus one line |
log-max-bytes | number | 67108864 | frontend storage storage-and-daemon all-in-one | The most bytes one log file holds before it is rotated; a line longer than this goes alone to a fresh file. With --log-files it bounds the log's size on disk |
[macos]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
mac-session-socket | text | /var/run/kbf-mac-session/socket | daemon storage-and-daemon all-in-one | The socket kbf-mac-session serves (native-whole-machine) |
pool-first-uid | text POOL_FIRST_UID | none | daemon storage-and-daemon all-in-one | The uid of kbf-pool-0, and the group id of all the pool users (pool_first_uid in kbf-mac-provision's profile); at least 600 |
pool-users | text POOL_USERS | none | daemon storage-and-daemon all-in-one | How many lease users the pool has (native, macOS, a daemon run as root): every action runs as one of kbf-pool-0 .. kbf-pool-<N-1>, one lease per user at a time, as kbf-mac-provision makes them (pool_users). Required of a root daemon, which runs no action as root; refused for any other |
require-metal-toolchain | switch | false | daemon storage-and-daemon all-in-one | This node's actions use Metal (native): an Xcode whose Metal toolchain is not installed (xcodebuild -showComponent MetalToolchain) is not ready |
xcode-apps | text | /Applications | daemon storage-and-daemon all-in-one | The directory searched for Xcode*.app (native). Each Xcode whose own xcodebuild answers -version, -license check and -checkFirstLaunchStatus, and for which xcrun --find clang does (each within a minute, under the actions' sandbox), is ready: it is reported as an xcode entry, and an action that names its build runs with it as DEVELOPER_DIR. One that is not is listed in the node's status with why and the command that fixes it, and logged at WARN. The daemon says Hello without waiting for its first survey, which runs in the background: until it ends, each Xcode is listed as not surveyed yet and none is reported |
xcode-recheck-secs | number | 180 | daemon storage-and-daemon all-in-one | How often, in seconds, the Xcodes are asked again (native), so one fixed while the daemon runs becomes ready without a restart |
[memory]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
memory-headroom-mib | number | 512 | daemon storage-and-daemon all-in-one | ...plus this many MiB (native). A lease that booked nothing has no limit |
memory-ladder-gib | list | [4, 16, 32] | frontend all-in-one | The memory rungs of an action, in GiB, ascending and each at least 1. The first is what an action that names no size (kbf-book-mem-gib) books. An action that passes its own memory limit runs again on the next rung above what it booked, and after the last on a whole host: the memory of the largest node that could run it, so that no other lease fits there. A rung not below that node's memory is skipped for the whole host. An action that does finish keeps its rung: later runs of it start there |
memory-limit-percent | number | 150 | daemon storage-and-daemon all-in-one | A lease's processes are killed past this percentage of its booked memory... |
memory-poll-ms | number | 250 | daemon storage-and-daemon all-in-one | How often a lease's memory is measured, in milliseconds (native) |
memory-pressure-pause-percent | number | 10 | daemon storage-and-daemon all-in-one | The daemon takes no new lease while the node's memory stall time (the some line's avg10 in /proc/pressure/memory, as a percentage) is above this and it runs a lease: it says so to the server, which places nothing on it until the stall falls, and the leases it runs go on. With no lease it serves whatever the stall, so it is never left idle for good. Off where the kernel reports no pressure stall information (macOS, a kernel built without it) |
[outputs]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
output-max-bytes | number | 17179869184 | daemon storage-and-daemon all-in-one | The most bytes one action's output files may hold together |
output-max-depth | number | 512 | daemon storage-and-daemon all-in-one | The deepest directory an output directory may hold, in levels below it (its own entries are level 1) |
output-max-entries | number | 1000000 | daemon storage-and-daemon all-in-one | The most entries (files, directories, symlinks and anything else) one action's outputs may hold together, the declared outputs included |
output-max-stdio-bytes | number | 1073741824 | daemon storage-and-daemon all-in-one | The most bytes each of the action's stdout and stderr may hold |
output-upload-deadline-secs | number | 1800 | daemon storage-and-daemon all-in-one | How long one lease's whole output upload (its output files, stdout and stderr) may take, in seconds, before the lease fails with a reason naming the upload and the time it ran (container and native drivers). Without it, a store that stops answering holds the lease, and its slot, with no error |
segment-upload-idle-secs | number | 60 | storage storage-and-daemon all-in-one | How long, in seconds, a segment upload may go without a call before it is dropped and its memory returned. An upload is also dropped when the connection that began it closes; one that is committing is kept until its commit ends |
segment-upload-max-bytes | number | 34603008 | storage storage-and-daemon all-in-one | The most bytes, records and footers together, one segment upload on the worker listener may hold in memory (docs/design/write-path.md, section 4.2): the segments it declares that are not written to the hot store as they arrive (all of them, without one). Not a limit on a blob's size: a segment held in memory holds no record over 16 MiB (kbf_front::MAX_UNSTAGED_BLOB_BYTES), and an upload with a larger one and no room on the hot store is answered FAILED_PRECONDITION, so the daemon sends its records by ByteStream, which streams a large blob to the cold store. Those ranges are held until the upload commits, admitted under --write-bytes-in-flight with every other upload: a Begin waits while that is spent. At least 33 MiB (kbf_front::MIN_SEGMENT_UPLOAD_BYTES), a daemon's upload of small files with room to spare, and at most --write-bytes-in-flight, so any legal Begin can be admitted; the server refuses to start otherwise. At most kbf_front::MAX_ADMISSION_BYTES, the most one admission counts, so a Begin never holds more than it is counted for under the bound |
segment-upload-streams | number | 16 | daemon storage-and-daemon all-in-one | How many PutRange calls one segment upload of an action's outputs has in flight at once, as HTTP/2 streams on the one connection to --cas (docs/design/write-path.md, section 4.1); also the parallel ReadRange calls of a segment-range read |
segment-uploads-in-flight | number | 4 | daemon storage-and-daemon all-in-one | How many segment uploads of one action's outputs are in flight at once: the server checks and stages one upload while the next are sending (docs/design/write-path.md, section 4.1). Their PutRange calls still take permits of --transfer-in-flight's uploads' share |
[scheduler]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
finished-retention-secs | number | 60 | frontend all-in-one | How long, in seconds, a finished operation is kept after its callers are answered. Within it, WaitExecution on the operation's name streams its result (a client whose Execute stream broke gets a result the action cache does not keep, such as a failure); after it, the name is NOT_FOUND. Every operation finished within it is held in memory |
heartbeat-interval-ms | number | 5000 | frontend all-in-one | The heartbeat interval daemons are asked for, in milliseconds, at most half the window in which a daemon may act on a Start (each Start names the newest heartbeat the server took, so a longer interval would leave it too little) |
input-tree-cache-bytes | number | 268435456 | all-in-one | The most bytes of decoded Directories the input check keeps (each costs its digests, 40 bytes apiece, and 160 bytes) |
input-tree-cache-entries | number | 1048576 | all-in-one | The most decoded Directories an Execute's input check keeps in memory (docs/design/storage.md#input-tree-check). Least recently used go first |
input-tree-reads-in-flight | number | 64 | all-in-one | The most Directory blobs the input check reads from the store at once, across all Executes |
progress-interval-secs | number | 30 | frontend all-in-one | How often, in seconds, every open Execute and WaitExecution stream is sent its operation again (same name, not done, current stage), so a load balancer's idle timeout never cuts a long action. 0 sends none |
readyz-store-timeout-ms | number | 2000 | frontend all-in-one | How long, in milliseconds, GET /readyz on the operator API and the REAPI listener's grpc.health.v1 service wait for the object store to answer their read-only probe before they answer not ready |
state-store | text URI | none | all-in-one | Where the scheduler keeps its state, written before each change is acknowledged and replayed at start: sqlite:///absolute/path, a file on a local disk that this process owns (created, with its directory, if missing; the file is 0600 and a directory it makes 0700; every start is a new epoch, so the daemon kills the runs of the leases it held before). Left out, it is <data-dir>/state.db. Any other value, postgres:// included, is refused (Postgres is not wired yet) |
state-store | text URI | none | frontend | Where the scheduler keeps its state, written before each change is acknowledged and replayed at start: sqlite:///absolute/path, a file on a local disk that this process owns. Every start is a new epoch, so daemons kill the runs of the leases they held before. The file is created, with its directory, if missing (0600, and 0700 for a directory it makes) and held by one process: a second frontend on it refuses to start. Any other value, postgres:// included, is refused (Postgres is not wired yet). Left out, the scheduler's state is in memory and a restart loses it |
unservable-wait-secs | number | 300 | frontend all-in-one | How long, in seconds, queued work waits while no connected daemon can run it (none satisfies its platform, or none that does is large enough) before it is refused FAILED_PRECONDITION. The wait restarts whenever one can |
[storage-client]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
storage | text URL | none | frontend all-in-one | Proxy mode (docs/design/architecture-v2.md, section 7): the URL of a storage node (kbf storage), such as http://127.0.0.1:8990. The REAPI listener then answers a client's CAS, ByteStream and action-cache calls from that node, passing ByteStream through without holding a blob. The local stores are still opened: Execution and the worker listener use them (kbf frontend is this mode without them: Execution reads the node too). Plain text (http://) only here; kbf frontend takes the flags of the mutual TLS to the node. Without it, the server answers those calls itself |
storage-ca | text PATH | none | frontend | The CA the storage node's certificate chains to; the node must present a certificate for the host in the --storage URL. |
storage-client-cert | text PATH | none | frontend | The frontend's certificate toward the storage node (PEM); it must name the role frontend (a URI subjectAltName urn:kbf:role:frontend). |
storage-client-key | text PATH | none | frontend | The private key of --storage-client-cert (PEM). |
storage-dial-timeout-ms | number | 10000 | daemon storage-and-daemon all-in-one | The longest, in milliseconds, that dialling a storage node may take: for its TCP connection and for its TLS handshake each. Must be 1 to 3600000 (one hour) |
storage-ping-interval-ms | number | 10000 | daemon storage-and-daemon all-in-one | How often, in milliseconds, a connection to a storage node with a call on it is pinged (HTTP/2 PING). Must be 1 to 3600000 (one hour) |
storage-ping-timeout-ms | number | 10000 | daemon storage-and-daemon all-in-one | How long, in milliseconds, a ping to a storage node may go unanswered before the connection is dropped and the calls on it fail over to the next owner. Must be 1 to 3600000 (one hour) and no more than --storage-ping-interval-ms |
[storage-node]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
member-dwell-secs | number | 300 | frontend all-in-one | How long, in seconds, a down storage member stays in the map before it leaves and its blobs move to other owners |
member-ttl-secs | number | 15 | frontend all-in-one | How long, in seconds, a storage node's membership row lives without a heartbeat before the state store expires it and the member goes down in the storage map. Nodes are asked to heartbeat three times as often |
storage-weight | text N | none | storage storage-and-daemon | The node's share of the digests relative to the others. Default: 1. |
storage-zone | text LABEL | none | storage storage-and-daemon | The zone label to register with. Default: none. |
[stores]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
cold-store | text URI | none | storage storage-and-daemon all-in-one | The durable object store, where every blob ends up: file:///absolute/path (a directory, kbf_objstore::FsStore: locked while the server runs, a second server on it is refused; a write that would leave its filesystem with less free than it keeps, 5% of its size and at least 1 GiB, is refused and wakes the evictor, which forgets the least recently used objects (never one used in the last minute) and deletes their files while less than twice that is free) or s3://bucket/prefix?endpoint=http://host:port (MinIO, RustFS; the key pair is AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY; query keys region, conditional-put=true for a store that refuses to overwrite a key, and min-put-rate=BYTES_PER_SEC, the slowest rate a PUT body may reach the store at: a PUT is cut after 60 seconds plus its body's length at that rate, 1 MiB/s by default). Objects are keyed under <prefix><log id>/ (kbf/ for a directory and for an S3 URI without a path), the id of the metadata log. A directory is checked against the index by a start-up pass in the background; a lost or damaged object is also found when it is read, or before an action-cache hit that needs it is served (a miss). There is no memory: store: a file:// path on a tmpfs is one |
cold-store-evict-grace-ms | number | 60000 | storage storage-and-daemon | For tests of eviction only: how long after its last use an object in a file:// cold store is never evicted, in milliseconds |
cold-store-evict-grace-ms | number | 60000 | all-in-one | For tests of eviction only: how long after its last use an object in a file:// cold store is never evicted, in milliseconds (Not shown in --help.) |
cold-store-floor-bytes | text BYTES | none | storage storage-and-daemon | For tests only: the free bytes a file:// cold store's filesystem must keep, in place of 5% of it, at least 1 GiB (a test's disk would decide what is evicted) |
cold-store-floor-bytes | text BYTES | none | all-in-one | For tests only: the free bytes a file:// cold store's filesystem must keep, in place of 5% of it, at least 1 GiB (a test's disk would decide what is evicted) (Not shown in --help.) |
cold-store-size-bytes | text BYTES | none | storage storage-and-daemon | For tests of eviction only: a file:// cold store's filesystem is taken to be this many bytes, holding nothing but the store's object files |
cold-store-size-bytes | text BYTES | none | all-in-one | For tests of eviction only: a file:// cold store's filesystem is taken to be this many bytes, holding nothing but the store's object files (Not shown in --help.) |
hot-store | text URI | none | storage storage-and-daemon all-in-one | The fast tier in front of the cold store: file:///absolute/path, a directory on an SSD (or a tmpfs). Uploads land there and are flushed to the cold store in the background; reads are served from it while a blob is still there. Its budget is not a flag: 80% of the filesystem's free space less what it keeps free (5% of its size, at least 1 GiB, and never more than half of what is free), at most --hot-store-max-bytes; at the budget, writes take the direct path to the cold store. Without it, uploads go straight to the cold store. A daemon's outputs (a segment upload, or the in-process plane) and a ByteStream Write over 16 MiB are written through to it as they arrive, so a large one is never held in memory. Without it, or when it has no room, a daemon's blob over 16 MiB (kbf_front::MAX_UNSTAGED_BLOB_BYTES) is streamed to the cold store as it arrives (a ByteStream Write, or the in-process plane's own stream), never held whole and never refused for its size. The first start with a hot store raises the metadata log to machine version 2, once; a binary that runs only version 1 no longer reads it. The metadata log lives in <dir>/meta, unless --meta-dir says otherwise |
hot-store-abandon | text ID | none | storage storage-and-daemon all-in-one | Abandons this server's hot store tier <id> at start: every object staged on it is marked unreachable as missing (its blobs answer Unavailable until uploaded again; nothing is answered NOT_FOUND), and the directory becomes a new tier. For a lost or replaced disk, after a start was refused because the directory's TIER file is missing or names another tier. The id is the one the refusal names |
hot-store-max-bytes | text BYTES | none | storage storage-and-daemon all-in-one | The most bytes the hot store's block files may hold before they are flushed, lower than its automatic budget |
meta-dir | text DIR | none | storage storage-and-daemon all-in-one | The directory of the metadata log (the CAS index and the action cache, a single-voter Raft log), on a local disk; created if missing. An empty directory gets a new log with a new id; every later start over it reads the id from the directory. Left out, it is <hot store directory>/meta; required when there is no file:// hot store. Given, it wins over the hot store's directory |
store-put-attempts | number | 3 | storage storage-and-daemon all-in-one | How many times a segment PUT is tried. Only a PUT that did not complete (connect, reset, timeout) or that the store answered 5xx is tried again, after 100 ms and then four times longer each time, under a new object key; if every attempt fails, the segment's blobs are answered UNAVAILABLE |
write-bytes-in-flight | number | 2147483648 | storage storage-and-daemon all-in-one | The most bytes uploads may hold at once. A ByteStream Write is admitted before it reads past its first message, for its blob's size up to 16 MiB (kbf_front::MAX_UNSTAGED_BLOB_BYTES) and for 16 MiB above that (a larger blob is streamed onto the hot store, or to the cold store without room there, not held); a BatchUpdateBlobs for its total; a segment upload's Begin for what it holds in memory. A caller over the bound waits, nothing is refused for load. At least --segment-upload-max-bytes (by default 33 MiB, under the default 2 GiB). Not covered: each HTTP/2 stream's message as tonic decodes it, before the handler runs, up to 17 MiB per stream (about 3.4 GiB for 200 streams), times the streams one connection may open on each listener |
write-linger | number | 2 | storage storage-and-daemon all-in-one | How long, in milliseconds, the oldest blob of an open segment waits for concurrent uploads to join it before the segment is sealed and written (docs/design/write-path.md, section 3.1). No caller is answered before its segment has committed, so this is added to an upload's latency at most |
write-own-segment-bytes | number | 16777216 | storage storage-and-daemon all-in-one | Blobs of this many bytes or more are written as a segment of their own, past the group, so they neither wait for the linger nor hold up small writes |
write-puts-in-flight | number | 4 | storage storage-and-daemon all-in-one | The most segment PUTs, each with its footer read-back, in flight at once. A segment sealed while all are busy waits; new blobs go to a new open segment |
[tls]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
api-token-file | text API_TOKEN_FILE | none | frontend all-in-one | A file holding the token operator API writes (cordon, drain, uncordon) must present as Authorization: Bearer <token>. It must be owned by the server's user with mode 0600 or 0400, or the server refuses to start; run the server as a user other than the daemon's, whose builds could read it. Without it, writes are refused |
reapi-auth-policy | text REAPI_AUTH_POLICY | none | frontend all-in-one | A JSON file with the REAPI listener's authentication policy and authorizers (docs/reapi-auth.md), read once at start; a file that is not a valid policy stops the server. Without it every REAPI call is accepted and allowed. The worker listener does not read it; on the operator API only the build pages and the blob route do, and they are off unless it names their authorizers |
reapi-tls-cert | text REAPI_TLS_CERT | none | frontend all-in-one | PEM certificate (chain) of the REAPI listener. With --reapi-tls-key it serves TLS with this server certificate only; clients present no certificate. It is meant for a certificate from the farm's own internal CA, trusted by the front's proxy (its hop to this server) and by clients that dial this server directly; TLS for the farm's client-facing name still ends at the front. --reapi-auth-policy applies over TLS as over plain text |
reapi-tls-key | text REAPI_TLS_KEY | none | frontend all-in-one | PEM private key of the REAPI listener; it must match --reapi-tls-cert, or the server refuses to start |
storage-client-ca | text PATH | none | storage storage-and-daemon | The CA callers' certificates chain to; each must name a role (frontend, storage, daemon). |
storage-tls-cert | text PATH | none | storage storage-and-daemon | The node's server certificate (PEM). |
storage-tls-key | text PATH | none | storage storage-and-daemon | The server certificate's private key (PEM). |
worker-client-ca | text WORKER_CLIENT_CA | none | frontend all-in-one | PEM CA that client certificates must chain to. Each must also name the role the method it calls takes: the daemon role (urn:kbf:role:daemon) for the worker session and blob calls, the storage role (urn:kbf:role:storage) for kbf.map.v1.Membership, with DNS or IP names that cover the host of the instance id and of the address it registers; a frontend certificate may call nothing here |
worker-deny-list | text WORKER_DENY_LIST | none | frontend all-in-one | The worker listener's deny list: certificate serials, public keys and node ids it refuses (format in the identity module docs). Read at start, then again at every check (a daemon's Hello, Heartbeat, Result, NodeStatus and blob calls, a storage node's Register and Heartbeat), so an edit takes effect without a restart. A node entry also refuses a storage certificate with that DNS name. A list that cannot be read after start refuses all of those calls. Needs mutual TLS |
worker-tls-cert | text WORKER_TLS_CERT | none | frontend all-in-one | PEM certificate of the worker listener. With --worker-tls-key and --worker-client-ca it serves mutual TLS; without all three, plain text |
worker-tls-key | text WORKER_TLS_KEY | none | frontend all-in-one | PEM private key of the worker listener |
[transfer]
| Key | Type | Default | Modes | Description |
|---|---|---|---|---|
transfer-bytes-in-flight | number | 268435456 | daemon storage-and-daemon all-in-one | The most bytes of blobs read into memory in flight at once, for the whole daemon. A blob larger than this is fetched alone |
transfer-chunk-bytes | number | 1048576 | daemon storage-and-daemon all-in-one | The most bytes one ByteStream write message carries |
transfer-find-missing-chunk | number | 1000 | daemon storage-and-daemon all-in-one | The most digests one FindMissing call asks about |
transfer-in-flight | number | 64 | daemon storage-and-daemon all-in-one | The most blob requests in flight at once, for the whole daemon: ByteStream reads and writes, segment ranges and FindMissing calls, over its one connection to --cas. Uploads hold at most all but a quarter of them |
transfer-memory-blob-bytes | number | 1048576 | daemon storage-and-daemon all-in-one | The largest input file fetched through memory, where every lease asking for it at once shares one fetch; a larger one is streamed into its file |
transfer-retries | number | 3 | daemon storage-and-daemon all-in-one | How many times a blob request is retried after UNAVAILABLE or a broken connection |
transfer-stream-window-bytes | number | 8388608 | daemon storage-and-daemon all-in-one | The HTTP/2 window of each stream to --cas; the connection's is eight times it |