Skip to main content

8.7 Grafana

Grafana is an analytics and interactive visualization tool that connects to the previously configured Prometheus tool to display real-time metrics, logs, and alerting dashboards. This guide walks you through the process of installing Grafana and configuring it to run as a dedicated background service.

tip

Further details about node analytics can be found on the Monitoring Tools page in the 🧠 Theory section.

info

The following steps are performed on your 📟 node server.

1. Create System User​

Running services as a system user with minimal privileges is a best practice, limiting damage if compromised. The Blackbox Exporter user will only be able to read and execute service-specific files. Use the system's user creation tool to add a new one.

sudo adduser --system grafana-server-worker --group --no-create-home
Full Command Explanation
FlagDescription
--system Creates a system user, used to run services and daemons rather than for people to log in with.
--group Creates a new group with the same name as the user.
--no-create-home Prevents creation of a home directory since the service does not need one.

If you want to confirm that the user has been created, you can search for it within the password file, housing all essential information for each user account. Using the search tool grep, we can check if the user exists within the file.

grep "grafana-server-worker" /etc/passwd

The output should look similar to this:

grafana-server-worker:x:117:123::/home/grafana-server-worker:/usr/sbin/nologin

2. Install Grafana​

To add the Grafana server to your node, you have to install it's package. In the case of Grafana, the setup is more extensive than downloading a single executable and adding it to the binary tree yourself. We therefore use the package manager APT, which is also used for regular Linux or Ubuntu software. To verify the package's authenticity, we must first add the Grafana GPG software key to our system's keyring folder.

tip

Further information about package management can be found on the Utility Tools page in the 🧠 Theory section.

2.1 Add GPG Key: Download and save Grafana's signing key to the system’s keyring.

cd
curl -fsSL https://packages.grafana.com/gpg.key|sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/grafana.gpg
Full Command Explanation
FlagDescription
-fFails silently on errors to ensure the scripts does not continue on broken responses.
-sSilent mode to suppress progress output.
-SShows errors even in silent mode.
-LFollows redirects to the updated download location.
|Sends the output of curl directly into the gpg command.
--dearmorConverts the ASCII key to binary so it can be used by APT.
-oDefines the output location and stores the binary key in the trusted GPG folder.

2.2 Key Verification: Verify if the Grafana Key was added using the gpg command.

gpg --no-default-keyring --keyring /etc/apt/trusted.gpg.d/grafana.gpg --list-keys
Full Command Explanation
FlagDescription
--no-default-keyringAvoids loading the user’s default keyring in ~/.gnupg.
--keyringExplicitly defines which keyring to use for listing trusted GPG keys.
--list-keysDisplays all public keys of a specified keyring for verification.

You should find an entry similar to this:

/etc/apt/trusted.gpg.d/grafana.gpg
----------------------------------
pub   rsa3072 2023-01-06 [SC] [expires: DATE]
      0E22EB88E39E12277A7760AE9E439B102CF3C0C6
uid           [ unknown] Grafana Labs <engineering@grafana.com>
sub   rsa3072 2023-01-06 [E] [expires: DATE]

2.3 List Repository: Add Grafana to the trusted repository list, to allow it's installation.

sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"

Press Enter to continue fetching all the necessary packages.

info

The add-apt-repository is the standard way to add additional repositories to your sources in Ubuntu and many other Debian-based systems. However, it requires additional software packages that were installed during the Software Preparation. Ensure you've completed the previous steps.

2.4 Update Package List: Update the package list using the previous Grafana GPG key.

sudo apt update

2.5 List Grafana Versions: Check available Grafana versions that can be installed.

apt list -a grafana

2.6 Install Grafana: Download the latest supported Grafana build from the package list.

sudo apt install grafana

2.7 Freeze Updates: Put Grafana Updates on hold so dashboards dont break during regular package updates.

sudo apt-mark hold grafana

2.8 Set Binary Permissions: Set the correct owner and access rights for the executables.

sudo chown -R grafana-server-worker:grafana-server-worker /usr/sbin/grafana-server
sudo chmod 755 /usr/sbin/grafana-server

2.9 Set File Permissions: Set the correct owner and access rights for the configuration and log files.

sudo chown -R grafana-server-worker:grafana-server-worker /etc/grafana
sudo chown -R grafana-server-worker:grafana-server-worker /var/lib/grafana
sudo chown -R grafana-server-worker:grafana-server-worker /var/log/grafana
sudo chmod 755 /var/lib/grafana

3. Service Configuration​

Once the Grafana software was installed with the correct permissions, we can update it's default service configuration, that automatically starts Grafana during boot and restarts during crashes. The updated configuration will also check for logging and an internet connection before it starts up and uses the previously created user.

tip

Most of the default configuration remains untouched. Compare the content and only add missing or updated properties.

4.1 Create Service File: Create a system service file using your preferred text editor.

sudo vim /lib/systemd/system/grafana-server.service

4.2 Add Configuration: Paste the following content using your preferred logging tool, then save and exit the file.

tip

Further details on Journal and System Logging can be found on the Utility Tools page in the the 🧠 Theory section.

[Unit]
Description=Grafana instance
Documentation=http://docs.grafana.org
Wants=network-online.target
After=network.target network-online.target postgresql.service mariadb.service mysql.service influxdb.service

[Service]
EnvironmentFile=/etc/default/grafana-server
User=grafana-server-worker
Group=grafana-server-worker
Type=simple
Restart=on-failure
WorkingDirectory=/usr/share/grafana
RuntimeDirectory=grafana
RuntimeDirectoryMode=0750
ExecStart=/usr/share/grafana/bin/grafana server                                     \
                            --config=${CONF_FILE}                                   \
                            --pidfile=${PID_FILE_DIR}/grafana-server.pid            \
                            --packaging=deb                                         \
                            cfg:default.paths.logs=${LOG_DIR}                       \
                            cfg:default.paths.data=${DATA_DIR}                      \
                            cfg:default.paths.plugins=${PLUGINS_DIR}                \
                            cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}

LimitNOFILE=10000
TimeoutStopSec=20
CapabilityBoundingSet=
DeviceAllow=
LockPersonality=true
MemoryDenyWriteExecute=false
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=full
RemoveIPC=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
SystemCallArchitectures=native
UMask=0027
RestartSec=5
SyslogIdentifier=grafana-server
StandardOutput=journal
StandardError=journal
PrivateUsers=true

[Install]
WantedBy=multi-user.target
Full Property Explanation
PropertyDescription
DescriptionA human-readable description of the service shown in systemctl status.
DocumentationLink to documentation of the software that is being used for this service file.
WantsThe service should try to start network-online.target before starting Grafana.
After- network.target: Ensures networking setup is enabled before the service is started.
- network-online.target: Waits for the network to be fully online.
- postgresql.service: Ensures the PostgreSQL database service is started.
- mariadb.service: Waits for the MariaDB backend database to launch.
- mysql.service: Ensures the MySQL server is running for data sourcing.
- influxdb.service: Waits for the InfluxDB database as dashboard dependency.
EnvironmentFileSpecifies a file containing the service's environment variables used during startup.
UserExecutes the service as the grafana-server-worker user.
GroupExecutes the service under the grafana-server-worker group.
TypeIndicates running at a simple service in the foreground without forking.
RestartRestarts the service on-failure for a variety of reasons.
WorkingDirectorySets the working directory for the executed process.
RuntimeDirectorySets the runtime directory before start, removed when the service stops.
RuntimeDirectoryModeSets the file mode permissions for the runtime directory.
ExecStartLink to binary at /usr/share/grafana/bin/grafana, started with the command.
LimitNOFILEControls the maximum number of file descriptors the service can open.
TimeoutStopSecSpecifies the maximum time the service should stop before being forcibly terminated.
LockPersonalityThe service's kernel personality setting will be locked to prevent changes.
MemoryDenyWriteExecuteDetermines whether the service can create writable and executable memory mappings.
NoNewPrivilegesPrevents privilege escalation which processes can be apply for.
PrivateDevicesThe service will not have access to any physical devices.
PrivateTmpCreates an isolated /tmp directory for the service.
ProtectClockThe service cannot change the system clock if enabled.
ProtectControlGroupsPrevents the service from altering process resource limits and accounting settings.
ProtectHomeUsed to prevent the service from accessing the user's home directories.
ProtectHostnameThe service cannot change the system's hostname if enabled.
ProtectKernelLogsPrevents the service to access sensitive kernel log ring data.
ProtectKernelModulesPrevents the service from altering the system's hardware capabilities.
ProtectKernelTunablesDenies modifying kernel variables, restricting its ability to alter the system's behavior.
ProtectProcRestricts the visibility of other processes in /proc.
ProtectSystemSpecifies where the service can write files to the disk.
RemoveIPCIf enabled, it ill remove all connection objects when logging out.
RestrictAddressFamiliesRestricts the socket address families the service can use.
RestrictNamespacesLimits the types of Linux namespaces the process can access.
RestrictRealtimeBlocks acquiring real-time scheduling of the operation system.
RestrictSUIDSGIDRestricts creation or use of privilege files.
SystemCallArchitecturesRestricts the system calls the service can execute to a specific architecture.
UMaskSets the default file creation permissions for the service.
WantedByBinds the service to the multi-user.target, so it starts during all boot processes.
RestartSecDelay in seconds before restarting the service.
SyslogIdentifierTags logs from the service with grafana-server to help distinguish them.
StandardOutputSends regular service logs to the journal or syslog system.
StandardErrorSends error service logs to the journal or syslog system.
PrivateUsersRuns the service in its own namespace mapped to nobody and nogroup.
warning

If you renamed the user, make sure to update both User and Group values to prevent running the service as root.

5. Start the Grafana Service​

After setting up the service, you can enable and start the system service.

5.1 Reload Daemon: Reload the system daemon to include the new service.

sudo systemctl daemon-reload

5.2 Start Service: Start the Grafana service using the system control.

sudo systemctl start grafana-server

5.3 Enable Autostart on Boot: Enable the service to start automatically during boot.

sudo systemctl enable grafana-server

The output should look similar to this:

Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable grafana-server
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /lib/systemd/system/grafana-server.service.

6. Check Service Status​

You can fetch the current status from the system control to check if the Grafana service is running and configured correctly. The command will display whether it is active, enabled, or disabled and show recent log entries.

sudo systemctl status grafana-server

The output should look similar to this:

● grafana-server.service - Grafana instance
     Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendo>
     Active: active (running) since [DATE] UTC; [TIME] ago
       Docs: http://docs.grafana.org
   Main PID: 84000 (grafana)
      Tasks: 17 (limit: 38043)
     Memory: 65.7M
        CPU: 3.855s
     CGroup: /system.slice/grafana-server.service
             └─84000 /usr/share/grafana/bin/grafana server --config=/etc/grafan>

[DATE] [TIME] [USER] grafana[84000]: logger=modules t=2023-05-18T15:09:2>...
...

7. Adjusting the Time Zone​

To solve issues when having different timezones set on your node and your personal computer, but also ensure sure that metrics have the correct timestamp, you can update the default time settings which Grafana is applying under the hood.

7.1 Verify Timezone: Check if the wanted timezone is already set on your node.

timedatectl

7.2 Update Timezone: If there is an time offset, update your node's timezone.

sudo timedatectl set-timezone <your-timezone>
info

Exchange <your-timezone> with one of the Available Timezone Names.

tip

Timestamp changes take immediate effect for all services and log files.

Maintenance​

Proper maintenance ensures that all the components are working as intended and can be updated on the fly. Logging: Check the latest status of the system service.

sudo journalctl -f -u grafana-server

Restarting: If you made any changes or updates to configuration, reload the system daemon and start the exporter.

sudo systemctl daemon-reload
sudo systemctl restart grafana-server

Stopping: You can stop the exporter using the system control.

sudo systemctl stop grafana-server
tip

Further information about system control or logging can be found on the Utility Tools page in the 🧠 Theory section.

Revert Setup​

If something went wrong, you can remove the user or delete the service and related files all together.

1. Stop and Disable Service: Stop the tool and remove it's service link from the system's boot.

sudo systemctl stop grafana-server
sudo systemctl disable grafana-server

2. Change Binary Ownership: Change the owner of the Grafana and Grafana Server executables.

sudo chown -R root:root /usr/sbin/grafana
sudo chown -R root:root /usr/sbin/grafana-server

3. Change Folder Ownership: Change the owner of the configuration and database folers.

sudo chown -R root:root /etc/grafana
sudo chown -R root:root /var/lib/grafana
sudo chown -R root:root /var/log/grafana

4. Remove the Service and Config Files: Delete all service files and reload the system daemon.

sudo rm /lib/systemd/system/grafana-server.service
sudo rm -rf /etc/grafana
sudo rm -rf /var/lib/grafana
sudo rm -rf /var/log/grafana
sudo systemctl daemon-reload

5. Delete Binary: Remove the executable Grafana and Grafana Server from your system.

sudo rm -rf /usr/sbin/grafana-server
sudo rm -rf /usr/sbin/grafana

6. Remove User and Group: Prune the user and all it's cached configurations.

sudo deluser --remove-all-files grafana-server-worker
sudo delgroup grafana-server-worker