Backup and recovery

DigiFIR's record lives on the ledger, which every member institution holds a copy of. Most failures are therefore repaired by rebuilding a node from its peers. Backups protect what exists in only one place: a CA's database and keys, private data held only by a state's own peers and the judiciary's, and the backend's identity vault.

What to back up

Data	Where	Why	How
Peer ledger: blocks, private data store, history, transient data	Each peer's volume	Private data exists only on the member peers: a state's narratives are on its own peers and the judiciary's, nowhere else	Offline copy of the volume, one peer at a time
CouchDB of each peer	Its own volume	Saves rebuilding the world state from the blocks	With its peer, stopped after the peer, so the CouchDB copy is never newer than the peer's
Orderers: ledger, Raft log and snapshots	Each orderer's volume	The backup point of a consensus-type migration	Every orderer stopped at once, in maintenance mode
Fabric CA: database and keys	The CA's home; in the pilot PostgreSQL and the HSM	Registrations, enrolment history and revocations live only here	Offline copy
Configuration	Channel configuration blocks, collection files, configtx.yaml	Rebuilding nodes and repeating definitions	In git and in the release archive
Backend read models	PostgreSQL schema rm	None needed: rebuilt from the ledger at any time with projector --rebuild, swapping schemas with no downtime
Identity vault, operational data, audit log	PostgreSQL schemas vault, ops, audit	Exist only in the backend	The database operator's backups; vault values are encrypted, so a dump reveals nothing without the HSM
Encrypted files and documents	Object storage	Evidence ciphertext	Versioning on; object-lock legal holds on evidence while its keys exist
Audit log exports	Write-once storage and the CERT-In log store	Evidence of access	Exported every night

WARNING

Backups hold private data, including victims' identities, and CA keys. Encrypt them with an institution-held key, store them in India and keep them for a bounded time. An erasure under a court order does not reach backups, so after a purge, re-encrypt or expire the backups that predate it.

CAUTION

Never keep chaincode packages in git or a release archive. The connection.json inside them holds the peers' client key for the chaincode servers.

Every backup directory has a MANIFEST with the target, time, channel height, versions and the SHA-256 of every archive. Record the MANIFEST's own SHA-256 outside the backup. In the pilot, restore refuses a backup unless that recorded value matches.

Restore a peer

1. Run ./scripts/network.sh backup peer0.police-b. The script stops the peer, then its CouchDB, archives both volumes, and starts them again.
2. To restore, run ./scripts/network.sh restore peer0.police-b <backup directory>. The script checks the archives against the MANIFEST, refuses a backup of another node, replaces the volumes and starts the containers.
3. The peer catches up from the orderers and the other peers.

In the pilot the same is a snapshot of the node's persistent volumes with its pod scaled to zero.

A lost CouchDB

A peer rebuilds its CouchDB from its own block store. Replace the lost database with an empty one and restart the peer: it recommits the missing blocks with their private data. A peer that joined from a snapshot is the exception; it is rebuilt from a newer snapshot instead.

A lost peer

Join a new, empty peer to the channel, or join it from a ledger snapshot of the institution's own peer (snapshot and joinBySnapshot). A peer joined from a snapshot pulls the private data it is entitled to from the other member peers.

NOTE

A snapshot carries the whole public state but no private data. Trust a snapshot only from the institution's own peer, or compare its hash with a snapshot another institution's peer took at the same block.

A lost orderer

With three Raft orderers, one lost orderer leaves a quorum and ordering continues. Two lost orderers stop ordering until one is back; peers still answer queries.

An orderer whose ledger is lost must not return as the same Raft member with an empty log, or it crashes. Bring it back as a new member instead, with ./scripts/network.sh rejoinOrderer <orderer>:

1. Remove it from the consenter set; the other two keep the quorum.
2. Bring the node back with its identity, TLS certificate, host name and an empty ledger.
3. Add it back to the consenter set. Raft gives it a new ID.
4. Join it with the latest configuration block. It replicates the chain from the others.

Each step is a configuration update signed by the ordering organisation's administrator.

The ordering service as a whole

Orderers are backed up only all together, and only for one purpose: the backup point of a change of consensus type, such as the move to SmartBFT.

./scripts/network.sh maintenance on      # no transaction is ordered; peers cannot pull past this point
./scripts/network.sh backup orderers     # every orderer stopped at once and archived
# ... the configuration updates; to abort, before leaving maintenance mode:
./scripts/network.sh restore orderers <backup directory>
./scripts/network.sh maintenance off     # a transaction commits and every node reaches the same height

Ordering stops for 5 to 10 seconds while the orderers are copied. Losing every orderer, or a majority, outside such a window has no restore path that keeps the peers' newer blocks. That case is prevented instead: the three orderers sit in three failure domains, and a lost one is rejoined while a majority runs.

The backend

• Read models are never restored. The projector rebuilds them from the ledger, and its checkpoint is written in the same transaction as each block, so a crash in the middle of a block is safe.
• The relay watchdog resubmits signed actions the ordering service dropped, from their stored envelopes.
• PostgreSQL backups of the vault, operational data and audit log follow the database operator's procedures; a point-in-time-recovery runbook for the backend's database is still to be written for the pilot.

Drills

Every runbook was drilled on the development network on 1 November 2026, on the HSM profile: certificate renewal, onboarding a third state, chaincode upgrade and rollback, backup and restore, lost nodes, and revocation. The final drill took 31 minutes and ended with 55 smoke checks passing on five organisations and eight peers. In the pilot, restore drills run every quarter: rebuild a peer from nothing, restore a CA, and replace a lost orderer.

Sources: digifir-network docs/OPERATIONS.md ("Backups and restore", "An orderer or a peer is lost", "Verification log"), docs/KUBERNETES.md ("Storage and backups"), docs/SECURITY_REVIEW.md (M9, L18, TRUST-14); digifir-backend docs/architecture/BACKEND.md sections 4.4, 5.4, 6.3 and 9.