DOCS·ENGINE·CLUSTERING
04 · ENGINE

Clustering

XERJ ships an embedded Raft implementation — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. Where the ring stands today, plainly: leader election, heartbeats, and the authenticated inter-node transport are real. Index metadata (create, delete, mapping, and settings) and the primary shard plan for that index are proposed through the Raft log and applied on every member. A document written with PUT, POST /{index}/_doc, PUT /{index}/_create, POST /_bulk, POST /{index}/_bulk, or POST /{index}/_update/{id} is stored on the member that owns its shard, whichever node accepted the call. POST /{index}/_update_by_query runs on every primary of that index and each primary copies the documents it updated. POST /{index}/_delete_by_query runs on every primary and each primary copies the deletes. POST /v1/indices/{name}/docs, DELETE /v1/indices/{name}/docs/{id}, and POST /v1/indices/{name}/docs/_bulk run on the primary for that id and copy the document or the delete. GET and DELETE /{index}/_doc/{id} follow the same owner. GET or POST /{index}/_search fans out to every owner and merges hits. index.number_of_replicas set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A restarted member, and a member that starts with an empty log or behind a compacted prefix, installs committed metadata from the on-disk Raft log or from InstallSnapshot. A replica store that is missing the primary's live documents then copies them and rejoins the in-sync set (public tracking: xerj-org/xerj#1170).

What cluster mode does not give you yet

Replicas follow the create setting, not a later changeindex.number_of_replicas on PUT /{index} places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later PUT /{index}/_settings stores a new number and does not move copies. There is no wait_for_active_shards. A write is acknowledged when a quorum of the copies has confirmed: copies are the primary plus the replicas placed at create, and a quorum is copies / 2 + 1. The primary already counts, so with two replicas one confirmation is enough and the caller still succeeds. The replica that did not confirm is removed from the in-sync set by a Raft entry, and later reads do not use it until it has copied the primary's live documents and that enlarged set has committed. If fewer replicas confirm than that quorum, the response is 503 replica_not_acknowledged: the primary has the write, and a retry of the index copies it to the replicas still in sync. number_of_replicas 0 or 1 does not loosen this: there is no replica to wait for, or that single replica must confirm.
Some native ingest stays on the receiverPOST /v1/indices/{name}/docs, DELETE /v1/indices/{name}/docs/{id}, and POST /v1/indices/{name}/docs/_bulk run on the shard primary and copy to the in-sync replicas. The response stays the native envelope. Logs, OTLP, syslog, turbo, and pipeline ingest still write on the node that received them. POST /{index}/_delete_by_query and POST /{index}/_update_by_query run on every primary. Each primary deletes or updates only the documents in its own shard and copies that change to the in-sync replicas. max_docs counts across those primaries, in node-id order. A primary that does not answer refuses the request before any document is written. A copy that misses quorum is 503 replica_not_acknowledged: the primary has the delete or the update. ?pipeline= on an update-by-query runs on the primary. Pipelines are not replicated, so a pipeline that is not on that primary is 400 and writes nothing.
Search merges hits, not every clauseA query, including match_all, is run on each owner and the hits are merged by the per-shard score (not a global IDF), then from/size is applied. Aggregations, suggest, collapse, rescore, profile, search_after, and scroll are refused with 400 when the index has more than one owner — the response is not a partial merge. /_search with no index, and a comma-separated or wildcard index list, still search only the node that received them. The per-shard window is capped at 10,000 hits.
No replica promotionA restarted member reopens {data_dir}/_raft and applies the snapshot plus the log suffix before it serves, so the shard plan and the in-sync set come back. After 64 applied metadata entries the prefix is snapshotted and a member whose log does not overlap receives InstallSnapshot. The snapshot is the metadata commands, not the documents. A placed replica that is missing those documents copies them from the primary and is published back into the in-sync set; a write still needs the primary, and there is no promotion. A metadata history bigger than 8 MiB stays in the log instead of being snapshotted. Changing membership does not move copies.
No node-lifecycle APIsThere is no drain / activate / remove endpoint and no region split-merge. Earlier revisions of this page documented them; the binary has never shipped them.

With number_of_replicas at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to an in-sync copy when the primary does not answer. A copy that missed an acknowledged write is not in that set until it has copied the primary's live documents. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log. A restart replays the prefix that member had already stored. A new metadata change still needs a leader who can commit.

What you get

Bootstrapping a 3-node cluster

Start three nodes, each with its own data dir, and point them at each other. On first startup they hold a Raft election and one becomes leader:

# /etc/xerj/node-a.toml  (run on 10.0.0.11)
[server]
data_dir     = "/var/lib/xerj"
bind_address = "10.0.0.11"

[tls]                                     # required off loopback (or terminate TLS in front
enabled   = true                          # and set server.allow_insecure_network_bind = true)
cert_path = "/etc/xerj/certs/node-a.crt"
key_path  = "/etc/xerj/certs/node-a.key"
allow_insecure_grpc_h2c = true            # gRPC is cleartext h2c; required off loopback with TLS on

[cluster]
enabled     = true
port        = 9300                        # intra-cluster Raft + search port
auth_secret = "<the same secret, 16+ chars, on every node>"   # or XERJ_CLUSTER_AUTH_SECRET
peers       = [
  "10.0.0.11:9300=10.0.0.11:9300",       # this node
  "10.0.0.12:9300=10.0.0.12:9300",
  "10.0.0.13:9300=10.0.0.13:9300",
]
tick_ms = 50
# /etc/xerj/node-b.toml  (run on 10.0.0.12)
[server]
data_dir     = "/var/lib/xerj"
bind_address = "10.0.0.12"

[tls]                                     # required off loopback (or terminate TLS in front
enabled   = true                          # and set server.allow_insecure_network_bind = true)
cert_path = "/etc/xerj/certs/node-b.crt"
key_path  = "/etc/xerj/certs/node-b.key"
allow_insecure_grpc_h2c = true            # gRPC is cleartext h2c; required off loopback with TLS on

[cluster]
enabled     = true
port        = 9300                        # intra-cluster Raft + search port
auth_secret = "<the same secret, 16+ chars, on every node>"   # or XERJ_CLUSTER_AUTH_SECRET
peers       = [
  "10.0.0.11:9300=10.0.0.11:9300",
  "10.0.0.12:9300=10.0.0.12:9300",       # this node
  "10.0.0.13:9300=10.0.0.13:9300",
]
tick_ms = 50

Each peers entry is "<host>:<port>=<host>:<port>": node ids are socket addresses, and the node refuses to start on an entry whose label differs from its address (#1171 — a friendly label made that node's vote replies undeliverable). The current node is the entry whose address matches its own bind_address and cluster.port. Three more startup checks apply to this example and each refuses to boot when unmet: cluster.auth_secret (or XERJ_CLUSTER_AUTH_SECRET) must be set; a non-loopback bind_address needs [tls] or server.allow_insecure_network_bind = true; and with TLS on off loopback, the cleartext gRPC listener needs tls.allow_insecure_grpc_h2c = true. Start all three boxes:

$ xerj --config /etc/xerj/node-a.toml     # on 10.0.0.11
$ xerj --config /etc/xerj/node-b.toml     # on 10.0.0.12
$ xerj --config /etc/xerj/node-c.toml     # on 10.0.0.13

Within a second the ring elects a leader. The log is still the place to see who won. GET /_cluster/health reports the configured membership (this node's address plus distinct peers), not a liveness probe and not a count of nodes holding your documents:

$ journalctl -u xerj -f | grep xerj_cluster
INFO xerj_cluster::transport: TCP transport listening (authenticated) node=10.0.0.11:9300
INFO xerj_cluster::raft: Starting election node=10.0.0.12:9300 term=1
INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1
$ curl -sH "Authorization: ApiKey $XERJ_API_KEY" \
    http://10.0.0.11:9200/_cluster/health | jq '.status, .number_of_nodes'
"green"
3

What replicates, and what stays on one node

PUT /orders on any member — leader or follower — creates orders on every member, including its mapping, the settings and aliases in that request, and a primary on each member. GET /orders and GET /orders/_mapping on another member succeed after the call returns. DELETE /orders removes it on every member. A settings or mapping change is one log entry per concrete index. A follower that cannot reach a leader returns 503; it does not create the index locally and report success.

PUT /orders/_doc/1, POST /orders/_update/1, and each action in POST /orders/_bulk on any member stores the document on the primary for that id and on enough in-sync replicas to make a quorum of the copies. POST /orders/_update_by_query does the same for every document it changes. POST /orders/_delete_by_query deletes on every primary and copies each delete to the in-sync replicas. POST /v1/indices/orders/docs, DELETE /v1/indices/orders/docs/1, and POST /v1/indices/orders/docs/_bulk do the same for one document: the primary stores it, or deletes it, and copies that to the in-sync replicas. The caller still receives the native envelope. An action whose primary is another member is not written into the accepting node's index. With number_of_replicas 1 that is the one replica. With 2, one of the two is enough; the other is left out of the in-sync set until it copies the primary's live documents and the enlarged set commits, and a later read does not use it before that. GET /orders/_doc/1 from any other member returns it, from the primary while that member answers and from an in-sync replica when it does not. POST /orders/_search from any member returns documents stored on the primaries, using an in-sync replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and number_of_replicas 0, then routes. With no elected leader that write is 503 and the index is not created. [cluster] enabled = false does not place replicas: the setting is stored, and the document stays on the node that received it.

Several things around that path stay local, on purpose for this step:

Failure modes

Leader crashFollowers time out and elect a new leader — measured ~250 ms on a 3-node loopback ring, including one split-vote retry. The ring stays stable with the dead member absent. Metadata already applied on the survivors stays. The dead member's log is under its data directory; a restart replays it.
Hung or dead peerOne WARN on the down transition, then quiet: retries back off 100 ms → 6.4 s, and messages to live peers are never delayed behind a dead one. A peer that answers again logs "peer reachable again" at INFO.
Minority partitionThe minority side cannot reach quorum and its elections never conclude; the majority side keeps its leader. Raft safety holds — at most one leader per term.
Node rejoinRestart with the same data directory reloads the snapshot, the log suffix, and the shard plan. A metadata entry committed while this node was down is installed from a peer that still has it, or from that peer's snapshot once the log prefix has been compacted. A replica store that does not have the primary's live documents copies them after that metadata is applied. A wiped data directory catches metadata up the same way and then runs that copy (#1170).

When not to cluster

When you need a replica to accept writes after the primary is gone, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, acknowledges a document write (PUT/POST _doc, _bulk, _update, _update_by_query, _delete_by_query, or native /v1 document index, delete, and _bulk) once a quorum of its copies has it, copies the primary's live documents onto a replica that joined late or missed a write, and reloads metadata when a member restarts. It does not promote a replica, and adding or removing a member does not move copies. One node avoids the forwarding hop and is trivial to back up. The default config ships with [cluster] enabled = false.

Source · engine/crates/xerj-cluster/src/raft.rs · engine/crates/xerj-cluster/src/transport.rs · engine/crates/xerj-cluster/src/runner.rs