Monitoring and Observability

Monitoring Rust blockchain applications in production.

Advanced⏱️ 50 min📚 Prerequisites: 1

Monitoring and Observability

Observability is crucial for production blockchain applications. It includes logging, metrics, and tracing.

Three Pillars of Observability

Logs: What happened and when
Metrics: How much, how often
Traces: Request flow through system

Logging

Structured Logging with tracing


TOML
[dependencies]
tracing = "0.1"
tracing-subscriber = "0.3"


RUST
use tracing::{info, error, warn, debug};

fn main() {
    tracing_subscriber::fmt::init();
    
    info!(target: "blockchain", "Node starting");
    info!(height = 1000, "Blockchain synced");
    
    error!(target: "blockchain", error = ?err, "Sync failed");
    warn!(target: "network", "Peer disconnected");
}

Log Levels

ERROR: Critical failures
WARN: Potential issues
INFO: Important events
DEBUG: Detailed debugging
TRACE: Very detailed

Metrics

Prometheus Integration


TOML
[dependencies]
prometheus = "0.13"


RUST
use prometheus::{Counter, Histogram, Registry};

lazy_static! {
    static ref BLOCKS_MINED: Counter = Counter::new(
        "blocks_mined_total",
        "Total blocks mined"
    ).unwrap();
    
    static ref BLOCK_TIME: Histogram = Histogram::with_opts(
        HistogramOpts::new("block_time_seconds", "Time to mine block")
    ).unwrap();
}

fn mine_block() {
    let timer = BLOCK_TIME.start_timer();
    // ... mining logic ...
    BLOCKS_MINED.inc();
    timer.observe_duration();
}

Key Metrics for Blockchain

Block height: Current chain height
Transaction throughput: TPS
Peer count: Connected peers
Sync status: Sync percentage
Memory usage: RAM consumption
CPU usage: Processing load
Network I/O: Bytes sent/received

Distributed Tracing


RUST
use tracing::{instrument, Span};

#[instrument]
async fn process_transaction(tx: Transaction) -> Result<(), Error> {
    // Function automatically creates span
    // Logs entry, exit, and errors
}

Alerting

Critical Alerts

Node down: No health check response
Sync stalled: Height not increasing
High error rate: Many failed operations
Memory leak: Continuously growing memory
Disk full: Running out of storage

Alert Configuration


YAML
# Prometheus alert rules
alert: BlockchainNodeDown
expr: up{job="blockchain-node"} == 0
for: 5m
annotations:
  summary: "Blockchain node is down"

Dashboards

Grafana Dashboards

Visualize:

Block production rate
Transaction volume
Network activity
Resource usage
Error rates

Log Aggregation

ELK Stack: Elasticsearch, Logstash, Kibana
Loki: Log aggregation by Grafana
CloudWatch: AWS logging
Stackdriver: GCP logging

Best Practices

Structured logs: Use JSON format
Context: Include request IDs, user IDs
Sampling: Don't log everything at DEBUG
Rotation: Rotate log files
Retention: Keep logs for compliance
Privacy: Don't log sensitive data

Code Examples

Structured Logging

Using tracing for structured logs

RUST

struct BlockchainNode {
    height: u64,
    peer_count: u32,
}
impl BlockchainNode {
    fn log_status(&self) {
        println!("Height: {}, Peers: {}", self.height, self.peer_count);
    }
    fn log_error(&self, error: &str) {
        println!("ERROR: {} at height {}", error, self.height);
    }
}
fn main() {
    let node = BlockchainNode {
        height: 1000,
        peer_count: 5,
    };
    node.log_status();
    node.log_error("Sync failed");
}

Explanation:

Structured logging with fields makes logs searchable and analyzable. Tools like ELK or Loki can parse structured logs for better insights.

Metrics Collection

Collecting blockchain metrics

RUST

struct BlockchainMetrics {
    blocks_mined: u64,
    transactions_processed: u64,
    current_height: u64,
    peer_count: u32,
    sync_percentage: f64,
}
impl BlockchainMetrics {
    fn new() -> Self {
        BlockchainMetrics {
            blocks_mined: 0,
            transactions_processed: 0,
            current_height: 0,
            peer_count: 0,
            sync_percentage: 0.0,
        }
    }
    fn update(&mut self, height: u64, peers: u32) {
        self.current_height = height;
        self.peer_count = peers;
        self.sync_percentage = (height as f64 / 10000.0) * 100.0;
    }
    fn display(&self) {
        println!("Metrics:");
        println!("  Blocks mined: {}", self.blocks_mined);
        println!("  Transactions: {}", self.transactions_processed);
        println!("  Height: {}", self.current_height);
        println!("  Peers: {}", self.peer_count);
        println!("  Sync: {:.1}%", self.sync_percentage);
    }
}
fn main() {
    let mut metrics = BlockchainMetrics::new();
    metrics.update(5000, 10);
    metrics.blocks_mined = 100;
    metrics.transactions_processed = 50000;
    metrics.display();
}

Explanation:

Metrics provide quantitative data about blockchain performance. They're essential for monitoring health, performance, and detecting issues early.

Exercises

Metrics Structure

Create a metrics structure for tracking blockchain stats!

Easy

Starter Code:

RUST

struct Metrics {
    blocks: u64,
    transactions: u64,
}
fn main() {
    let mut metrics = Metrics::new();
    metrics.increment_blocks();
    metrics.display();
}

Previous Lesson

← 57 ci cd pipeline

Next Lesson

59 rust project structure →