Operations runbooks

These runbooks keep the network and backend running. Each one has a pilot procedure and a development-network command, and every one was drilled on the development network. The full procedures are in docs/OPERATIONS.md and docs/KUBERNETES.md of the network repository; this page summarises them.

Who signs which change

Change	Signatures needed
Add or remove an organisation; Application capabilities or policies	The judiciary's administrator and two of the other three members' administrators
An organisation's anchor peers or MSP (root certificates, revocation list)	That organisation's administrator only
Consenters, batch size and timeout, maintenance mode, consensus type	The ordering organisation's administrator
Channel capabilities	The Application administrators as above, and the ordering organisation's administrator
A chaincode definition (version, sequence, policy, collections)	Approvals of the judiciary and two of the other three organisations, then one commit

Administrators sign channel updates themselves. The backend never holds an administrator key: the chaincodes' system transactions, which the backend prepares, are signed with tools/admin-sign and a smart card. Kubernetes jobs that need an administrator run only in an operations window, with a dated administrator certificate that is revoked when the window closes.

Renew certificates

Certificate	Lifetime	Renew
Node enrolment and TLS (peers, orderers)	1 year	30 days before expiry
User and gateway enrolment and TLS	1 year	30 days before expiry
CA root certificates	15 years	2 years before expiry, with the same key and start date
Fabric CA server's HTTPS certificate	1 year	With the CA's yearly maintenance

The certificate exporter alerts at 30 and 7 days before expiry.

./scripts/network.sh renew peer0.police-a tls        # a node's certificate; restarts the node
./scripts/network.sh renew orderer1 tls              # an orderer: rotated in the consenter set first
./scripts/network.sh renewCA police-a                # a CA root, same key and start date
./scripts/network.sh renew police-a/gateway msp --stage-only   # a gateway, registered before the switch

CAUTION

An expired gateway certificate stops the backend from submitting for that institution. An expired TLS certificate cuts the node off, and an orderer drops out of Raft. Renew on the 30-day alert, not the 7-day one.

Revoke a compromised identity

Revocation has two independent layers, and a compromised gateway needs both.

1. Fabric: revoke the certificate at the organisation's CA and put the new revocation list into the organisation's MSP in the channel configuration, signed by the organisation's administrator. From that configuration block on, every peer and orderer refuses the certificate. Revoking only at the CA changes nothing on the network.

   ./scripts/network.sh revoke police-a/<name> keycompromise

2. DigiFIR contracts: revoke the signing key in identity-registry with key.revoke by a registrar, or AdminRevokeKey by the MSP administrator when no registrar can sign. From the next block the contracts refuse actions signed with it.

For a lost officer's phone, the registrar ends the officer's sessions and revokes the key. See the registrar manual.

In the pilot, the revocation list is first committed to the repository and checked by a second person; the hourly channel audit fails while a committed revocation is missing from the channel.

Upgrade a chaincode

A chaincode upgrade is a new definition: a new version and package, the next sequence, the same name. With chaincode as a service, each organisation runs a server of the new version beside the running one, so the upgrade is a rollout of servers followed by one commit.

1. CI builds the image from the tagged source and publishes it by digest.
2. Each organisation starts a server of the new version, installs a package pointing at it, and approves the new definition. The running version keeps serving meanwhile.
3. Once the lifecycle policy is satisfied, one organisation commits. From that block, every peer uses its organisation's new package.
4. Check that every peer shows the new version and sequence, run a transaction through it, then stop the old servers.

make -C ../digifir-chaincode deploy-dev VERSION=1.1 SEQUENCE=2
./scripts/network.sh status        # committed version, sequence, policy and collections

The chaincode specification fixes the order of upgrades: a release that adds fields to identity-registry records is committed for the three case chaincodes first.

WARNING

A sequence never goes back. To roll back, commit the next sequence with the previous version's package. Once a record holding a field added by the new release has been written, older code refuses it, so the release can only be rolled forward. Run the release's check queries before deciding.

Maintenance mode

./scripts/network.sh maintenance on stops ordering and stops peers from pulling blocks, for a consensus migration or an orderer backup. maintenance off restarts the peers and checks that a transaction commits. See Backup and recovery.

Compact CouchDB after an erasure

After a court-ordered erasure, run ./scripts/network.sh compact (or the same per-database compaction in the pilot). It asks each CouchDB to compact the channel's private-data databases, waits, and fails if any old revision of a document can still be read.

The backend's pilot checklist

Before the backend's first start in the pilot, and again for each institution that joins:

1. The backend's own HSM token, and one token per institution in FABRIC_GATEWAYS, each with its own PIN. The backend refuses to start otherwise.
2. The backend's own keys on its token: the session-token signing key and the vault's keys.
3. The 32-byte HMAC key for service action IDs on the backend's token, checked with pkcs11-tool because the backend only notices it is missing when the worker first uses it.
4. Each institution's gateway key and document key-encryption key on that institution's token.
5. FABRIC_GENESIS_HASH, the value every institution computed from its own copy of block 0 and compared.
6. deploy/k8s/k8s.sh backend-check <values.yaml> --live passes on the backend's cluster.

Moving to Fabric 3.x and SmartBFT

The channel is prepared for the move: channel participation API, per-organisation orderer endpoints, P-256 identities only, and no Fabric 3.1-only chaincode APIs. The move makes sense once each institution runs its own orderer; with four, a SmartBFT ordering service tolerates one faulty orderer. The switch is a channel configuration change in maintenance mode that needs a rehearsal with four orderers in staging first.

Sources: digifir-network docs/OPERATIONS.md, docs/KUBERNETES.md, README.md; digifir-chaincode docs/SPEC.md section 1.15; digifir-backend docs/architecture/DECISIONS.md ADR-03.