Monitoring and logs

DigiFIR is monitored with Prometheus, Alertmanager and Grafana. They scrape the operations endpoints of every Fabric node, every CA, every CouchDB, the backend's three processes, and a certificate exporter. On the development network, ./scripts/network.sh monitor up starts the whole stack. In the pilot, the same rules and dashboards run in kube-prometheus-stack in each institution's cluster.

What is watched

Job	Targets	What it measures
fabric-orderer	The three orderers	Raft leadership and quorum, leader changes, transaction submission, block cutting, Raft disk latency
fabric-peer	Every peer	Ledger height and how far peers diverge, validation codes, endorsement latency and refusals, block commit time, gossip membership, state database latency
fabric-ca	Every CA server	Enrolment, registration and revocation traffic, refused requests
couchdb	Every CouchDB	Availability, errors, latency, reads and writes, rich queries without an index
digifir-api, digifir-projector, digifir-worker	The backend's processes	HTTP latency and errors per route, the signed-action relay, projector lag, jobs and the outbox; 36 metrics in all, on the internal port 9464
cert-exporter	One exporter	The expiry of every certificate of the network and of each TLS endpoint, which Fabric does not export itself
prometheus, alertmanager, grafana	The monitoring itself

Prometheus scrapes every 15 seconds and keeps 15 days of data by default.

Dashboards

Grafana has five dashboards, in a folder named digiFIR:

Dashboard	Shows
Network overview	Height per peer and the divergence between peers, validation codes, endorsement, commit, gossip
Orderers	Raft leader, cluster size, leader changes, broadcast, block cutting
CouchDB	Availability, errors and latency of each state database
Fabric CAs and certificates	CA traffic and the days left on every certificate
Backend	Requests per second, server errors, relay p95, projector lag, open jobs, outbox backlog, HTTP latency per route against the 300 ms read target

Alerts

44 alert rules, each with a promtool unit test that proves it fires on the failure it describes and stays silent when healthy.

Group	Alerts
Fabric availability	FabricPeerDown, FabricOrdererDown, FabricCADown, CouchDBDown
Fabric consensus	FabricRaftNoLeader, FabricRaftFrequentLeaderChanges, FabricRaftQuorumDegraded, FabricOrdererBehind, FabricOrdererChannelNotActive, FabricBroadcastFailures, FabricOrdererSlowDisk
Ledger and endorsement	FabricPeerBehind (more than 10 blocks), FabricEndorsementFailuresRising, FabricInvalidTransactionsCommitted, FabricMVCCConflictsHigh, FabricBlockCommitSlow, FabricPeerGossipMembershipIncomplete, FabricPeerStateDBSlow
CAs and CouchDB	FabricCAServerErrors, FabricCAAuthenticationFailures, CouchDBServerErrors
Security	FabricDebugLogging, DigifirChannelAuditFailed, DigifirChannelAuditMissing
Backend	DigifirBackendDown, DigifirApiErrors, DigifirApiSlowReads, DigifirRelaySlow, DigifirRelayInvalidations, DigifirLedgerAlert, DigifirProjectorLagging, DigifirProjectorStalled, DigifirJobFailing, DigifirOutboxBacklog
Certificates	DigifirCertificateExpiringSoon (30 days), DigifirCertificateExpiring (7 days), DigifirCertificateExpired, DigifirCertificateFilesUnreadable, DigifirCertificateEndpointUnreachable
Monitoring itself	Watchdog, MonitoringTargetDown, PrometheusRuleEvaluationFailures, PrometheusAlertmanagerUnreachable, AlertmanagerNotificationsFailing

DigifirLedgerAlert covers the backend's own ledger checks: a broken hash chain, an invalid orderer signature, a peer that disagrees with another institution's, read views that diverge from world state, and held or unknown transactions. DigifirChannelAuditFailed fires when the hourly channel audit finds the channel configuration different from what the institutions committed.

Routing

Receiver	For
digifir-page	Critical alerts: the on-call rota's paging, through a webhook to the national NOC's incident tool or SMS through the NIC gateway
digifir-ticket	Warnings: the operations team's ticket queue or email through the NIC or state mail relay
digifir-noc	Anything without a route
heartbeat	A dead man's switch the NOC expects every minute

The receivers in the repository are placeholders until the pilot's paging, ticketing and NOC endpoints are known. A critical security incident also starts the CERT-In report, which is due within 6 hours.

Health endpoints

Component	Endpoint
Peers, orderers	/healthz and /metrics on their operations ports
Fabric CAs	/healthz and /metrics on their operations ports
Backend	/healthz (liveness), /readyz (readiness, which reports each institution's peer), /v1/status (public status); a stalled projector does not fail readiness
Network status	./scripts/network.sh status: containers, health, Raft leader, heights, committed chaincodes and policies

While the backend's read views are 5 seconds or more behind the ledger, its responses carry an X-Digifir-Read-Lag header.

Logs

Source	Format and retention
Backend processes	JSON logs (pino), with tokens, private payloads, OTPs and signatures redacted; OpenTelemetry traces over OTLP (HTTPS in the pilot)
Backend access logs	Time, request ID, principal kind, key ID, route, status, latency, client address and user agent, sent to an India-hosted log store that becomes write-once after 24 hours; kept 180 days (CERT-In)
Backend audit log	Hash-chained rows of security events: refused signed actions, private reads, downloads, sessions, enrolments, admin changes; exported nightly to write-once storage and the CERT-In log store
Fabric nodes	FABRIC_LOGGING_SPEC; peers log their command configuration at warning so no password is logged; debug logging raises the FabricDebugLogging alert

Check the monitoring setup

./scripts/network.sh monitor check runs offline: promtool check config, check rules and test rules over the 53 rules (alerts and recording rules), amtool check-config and routing, the dashboards' JSON, and the certificate exporter's tests.

Sources: digifir-network docs/MONITORING.md, README.md ("Monitoring"), monitoring/; digifir-backend docs/architecture/BACKEND.md sections 15.3, 15.4 and 16.