Example Chia monitoring stack, using:

• mtail to collect metrics from Chia logs
• chia_exporter to collect metrics from the Chia node
• node_exporter or windows exporter to collect system metrics
• prometheus to store metrics
• promtail and loki to collect and store logs from the Chia node and plotters (and system too if desired)
• grafana to display everything

This includes a docker-compose configuration to run everything, but this is primarily intended for development and testing.

WARNING this is NOT a one-click install, expect to need to do some work setting everything up for your machine. PLEASE read the notes below and understand what all the services are, what they do, and how they work together.

The mtail program is in mtail/chialog.mtail. Currently it only collects harvester metrics:

• chia_harvester_blocks_total: cumulative number of block challenges attempted
• chia_harvester_plots_total: current number of plots
• chia_harvester_plots_eligible: cumulative number of plots that passed filter
• chia_harvester_proofs_total: cumulative number of proofs won
• chia_harvester_search_time: histogram of proof search times

NOTE you need to set log_level to INFO in your Chia config.yaml to get harvester metrics.

The chia_exporter is used to collect metrics from the Chia node RPC API.

The example Grafana dashboard is in grafana/dashboards/Chia.json. It defines a number of variables that will be auto-populated from the node metrics. Grafana dashboards are easily customized to show what you're interested in seeing, in the way you find best; this dashboard is just meant to demonstrate what can be done.

The docker-compose file will mount the Chia log from $HOME/.chia/mainnet/log/debug.log, verify that this location is correct and set the log level to INFO in the Chia configuration (usually at $HOME/.chia/mainnet/config/config.yaml).

Run:

docker-compose up -d

This will do the following:

• Build container image with configuration for mtail from source
• Build container image for chia_exporter from source
• Download other images from docker hub
• Run containers in the background, attached to the host network (this makes it easy to communicate with native services, but has some trade-offs. See notes.)

The grafana service provisions the prometheus and loki datasources and a basic dashboard that displays harvester and node metrics.

Access Grafana at http://localhost:3000 and login with the default admin/admin username and password (you'll be prompted to change the password).

• It's highly encouraged to run the node exporter natively rather than in docker - see the discussion in the node_exporter docs. On Ubuntu you can run sudo apt install prometheus-node-exporter, which includes disk SMART monitoring (disk temperatures, etc) as well. If you do run it in Docker, you'll need to bind-mount in any other volumes you want to monitor (add them to the volumes list in docker-compose.yml, e.g. - '/scratch:/scratch'). See issue #3.

• The docker-compose file uses the $HOME environment variable for the Chia log paths. Verify that these paths are correct, and if you run the docker-compose commands with sudo then you'll have to replace $HOME with the actual path (since root's home is not your home!). Even better, add your user to the docker group so you don't have to use sudo:

    sudo usermod -a -G docker username
  

• On Mac you'll need to run node_exporter natively, not under Docker: brew install node_exporter. You'll probably need to change the networking setup too, since Docker on Mac runs in a VM. See the windows docker-compose and prometheus configs.

The node exporter does not work on Windows; instead you need to use the Windows exporter for system metrics. Modified config and example dashboard are in the windows branch. You may also want to review the discussion in issue #2.

These steps will get you to a working setup (but aren't the only way):

• Install Docker Desktop
• Install Visual Studio Code
• Install git
• Install Windows exporter
• Clone the chiamon repository with VSCode
• Modify docker-compose.yml:
  • Change volume paths to point to your home directory.
• Run services. In VSCode with docker extension you can just right-click on docker-compose.yml and select "Compose Up"
• Check target status in Prometheus at http://localhost:9090/targets
• Access Grafana at http://localhost:3000 (admin/admin).

To monitor multiple nodes (e.g. multiple harvesters), you just need to run the appropriate exporters (e.g. node_exporter and mtail for a harvester) and add them as targets in the prometheus config, for example:

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100', 'harvester1:9100', 'harvester2:9100']
  - job_name: 'mtail'
    static_configs:
      - targets: ['localhost:3903', 'harvester1:3903', 'harvester2:3903']

If you're also running Loki to collect logs, you'll also want to run promtail on every node, and configure it to push logs to Loki on your monitoring node:

clients:
  - url: http://loki-server:3100/loki/api/v1/push

Copyright 2021 Kevin Retzke

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

See LICENSE.txt