11.3 Problem Scanning

This guide shows how to identify synchronization or configuration problems, to help solve problems in advance.

info

The following steps are performed on your 📟 node server.

Check the Node Status

The LUKSO CLI comes with a built-in status command to check the health of execution, consensus, and validator clients.

lukso status

Listen to Live Logs

The LUKSO CLI comes with its own logging functionality, directly printed to the terminal and stored in the node directory.

Mainnet

# Check execution logs
lukso logs execution

# Check consensus logs
lukso logs consensus

Testnet

# Check execution logs
lukso logs execution --testnet

# Check consensus logs
lukso logs consensus --testnet

Search Log Files

The file logging system of the LUKSO CLI also enables keyword-based filtering for log file contents.

info

You can search the extending log files using the grep tool for global expression search.

Mainnet

# Fetch all execution warnings
lukso logs execution | grep "warning"

# Fetch all validator warnings
lukso logs validator | grep "warning"

# Fetch all consensus warnings
lukso logs consensus | grep "warning"

# Fetch all execution errors
lukso logs execution | grep "error"

# Fetch all validator errors
lukso logs validator | grep "error"

# Fetch all consensus errors
lukso logs consensus | grep "error"

Testnet

# Fetch all execution warnings
lukso logs execution --testnet | grep "warning"

# Fetch all validator warnings
lukso logs validator --testnet | grep "warning"

# Fetch all consensus warnings
lukso logs consensus --testnet | grep "warning"

# Fetch all execution errors
lukso logs execution --testnet | grep "error"

# Fetch all validator errors
lukso logs validator --testnet | grep "error"

# Fetch all consensus errors
lukso logs consensus --testnet | grep "error"

tip

After executing the command, the terminal is waiting for an input. You will have to press the ENTER key to see the outputs.

warning

If you run the LUKSO CLI through service automation, ensure to execute all lukso commands with sudo permissions.

Attach Execution Clients

Execution clients have JSON-RPC endpoints allowing users to request information and access the blockchain state of their local nodes directly. Erigon and Geth even have built-in JavaScript consoles linked with such endpoints for programming capabilities. Both tools are helpful for manual checking of syncing progress, peers, blocks, and balances.

JSON RPC

JSON-RPC is a lightweight communication protocol encoded in JSON, allowing calls to be sent to a service or server. Each execution client exposes a related interface at port 8545 to retreive calls to interact with the Ethereum network though the local node. The RPC allows to query network stats, node metadata, blockchain data, or even send transactions and interact with smart contracts.

tip

By default, the LUKSO Network Configuration allows a variety of JSON RPC modules for the Nethermind and Besu client, while restricting their use for Erigon and Geth clients due to their JavaScript console. If you're using Geth or Erigon, you will first have to adjust the client-configuration to allow the regular data packages from being accessed.

Geth and Erigon

1. Stop Node Operation: Depending on your setup method, there are different ways to stop your node to adjust configurations.

LUKSO CLI

cd <lukso-working-directory>
lukso stop

info

Exchange <lukso-working-directory> with the path of the node folder.

Service Automation

sudo systemctl stop lukso-validator

Force Client Shutdown

Geth

sudo pkill geth

Erigon

sudo pkill erigon

Teku

sudo pkill teku

Nimbus-Eth2

sudo pkill nimbus_beacon_node
sudo pkill nimbus_validator_client

Lighthouse

sudo pkill lighthouse

tip

The Lighthouse client uses a single binary for both the consensus and validator processes.

Prysm

sudo pkill prysm
sudo pkill validator

2. Enable Full RPC Data: Modify the default network configuration of your execution client.

Geth

Use your preferred editor to modify the client configuration file.

Vim

cd <lukso-working-directory>/configs/<network>
vim geth/geth.toml

Nano

cd <lukso-working-directory>/configs/<network>
nano geth/geth.toml

info

Exchange <lukso-working-directory> and <network> with the path of the node folder and the network type.

Change the included module list to include the default execution client commands.

# Default Value
HTTPModules = ["net"] # ["net", "eth", "web3", "debug", "engine", "txpool"]

# Updated Value
HTTPModules = ["net", "engine", "eth", "web3"] # ["net", "eth", "web3", "debug", "engine", "txpool"]

Erigon

Use your preferred editor to modify the client configuration file.

Vim

cd <lukso-working-directory>/configs/<network>
vim erigon/erigon.toml

Nano

cd <lukso-working-directory>/configs/<network>
nano erigon/erigon.toml

info

Exchange <lukso-working-directory> and <network> with the path of the node folder and the network type.

Change the included module list to include the default execution client commands.

# Default Value
"http.api" = "erigon,engine" # "eth,erigon,engine,debug,trace"

# Updated Value
"http.api" = "erigon,engine,eth,net,web3" # "eth,erigon,engine,debug,trace"

warning

Ensure there are no missing characters or additional spaces within the configuration file.

Full Module Explanation

Module	Description	Potential Risks
ETH	Core blockchain operations.	None. Enabled by default without configuration.
NET	Retrieve network status and connections.	None. Enabled by default without configuration.
WEB3	Blockchain helpers and client metadata.	None. Enabled by default without configuration.
ENGINE	Engine API used by consensus client.	None. Enabled by default for validator nodes.
TXPOOL	Inspect pending and queued transaction pool.	Exposes pool and consumes computing power.
ADMIN	Low-level node management for peers and RPCs.	Lets attackers shut down RPCs, or export chain data.
DEBUG	Deep execution tracing and state inspection.	Exposes call stacks, storage slots and pre-images.
TRACE	Block and transaction replay traces.	Heavy EVM-execution calls can freeze or crash a node.

Save the file and exit the editor.

3. Restart the Node: Depending on your setup, there are different ways to start your node with the updated configuration.

LUKSO CLI Node

cd <lukso-working-directory>
lukso start --checkpoint-sync

info

Exchange <lukso-working-directory> with the path of the node folder.

LUKSO CLI Validator

cd <lukso-working-directory>
lukso start --validator --transaction-fee-recipient "<your-fee-recipient-address>" --checkpoint-sync

info

The following properties need to be exchanged:

• <lukso-working-directory> with the path of the node folder
• <your-fee-recipient-address> with the wallet address receiving staking profits

Service Automation

sudo systemctl start lukso-validator

After the clients were started, verify that their services are still up.

sudo lukso status

4. Connect to Interface: While the node is running, connect to the interface of the specific execution client.

Geth

geth attach http://localhost:8545

Erigon

erigon attach http://localhost:8545

5. Retrieve Data: If you are listening to the port, you can check your connections and interact with the execution client.

# Check Available Commands
> eth
> net
> web3

# Check Current Block Height
> parseInt(eth.blockNumber, 16)

# Check Syncing Status
# -- Syncing: Object
# -- Synced: False
> eth.syncing

# Retrieve Execution Peers
> parseInt(net.peerCount, 16)

# Get Current Gas Price
> parseInt(eth.gasPrice)

# Check External Connections
> net.listening

# Verify Network Chain ID
> parseInt(eth.chainId, 16)

# Verify Client Version
> web3.clientVersion

# Quit Port Connection
> exit

info

The parseInt() function will convert the hexadecimal output to a human-readable decimal number.

Nethermind and Besu

1. Install Querying Tool: Install the JSON query service for data processing from the RPC endpoint.

sudo apt install jq

2. Call Endpoints: While the node is running, call the JSON RPC of the execution client.

# Check Current Block Height
curl -s -X POST -H "content-type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
  http://localhost:8545 | jq -r '.result' | xargs printf '%d\n'

# Check Syncing Status
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' \
  http://localhost:8545 | jq -r '.result'

# Retrieve Execution Peers
curl -s -X POST -H "content-type: application/json" \
  --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' \
  http://localhost:8545 | jq -r '.result' | xargs printf '%d\n'

# Get Current Gas Price
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_gasPrice","params":[],"id":1}' \
  http://localhost:8545 | jq -r '.result' | xargs printf '%d\n'

# Check External Connections
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"net_listening","params":[],"id":1}' \
  http://localhost:8545 | jq -r '.result'

# Verify Network Chain ID
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' \
  http://localhost:8545 | jq -r '.result' | xargs printf '%d\n'

# Verify Client Version
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"web3_clientVersion","params":[],"id":1}' \
  http://localhost:8545 | jq -r '.result'

Full Command Explanation

Command	Description
curl -s -X POST -H <text> --data <text> <port>	Executes -X a silent -s HTTP POST request to the given <port>, while attaching custom header -H and --data content as <text> payload.
jq -r '.result'	Filters the raw -r JSON object and extracts the .result value of the data object.
xargs printf '%d\n'	Converts the hexadecimal result to a human-readable decimal %d number.

Fetch Consensus API

Access to the consensus endpoint allows for fetching validator status, health checks, peer counts, and metadata of the beacon chain. The feature can help verify client performance and check synchronization health directly from the beacon node's HTTP port.

REST API

A REST API is a web-based interface for querying structured data. Ethereum consensus clients expose their status and internal metrics on various ports like 3500, 5051, or 5052, to allow users to access information such as synchronization progress, network stability, node metadata, or the current chain head.

tip

By default, the LUKSO Network Configuration opens the REST API ports for all consensus clients.

1. Install Querying Tool: Install the JSON query service for data processing from the REST API endpoint.

sudo apt install jq

2. Call Endpoints: While the node is running, call the REST endpoint of the consensus client.

Teku

# Check Syncronization Status
curl -s http://localhost:5051/eth/v1/node/syncing | jq

# Check Overall Health
curl -s http://localhost:5051/eth/v1/node/health | jq

# Check Node Identity
curl -s http://localhost:5051/eth/v1/node/identity | jq

# Check Number of Peers
curl -s http://localhost:5051/eth/v1/node/peer_count | jq

# Fetch Latest Block Header
curl -s http://localhost:5051/eth/v1/beacon/headers/head | jq

Prysm

# Check Syncronization Status
curl -s http://localhost:3500/eth/v1/node/syncing | jq

# Check Overall Health
curl -s http://localhost:3500/eth/v1/node/health | jq

# Check Node Identity
curl -s http://localhost:3500/eth/v1/node/identity | jq

# Check Number of Peers
curl -s http://localhost:3500/eth/v1/node/peer_count | jq

# Fetch Latest Block Header
curl -s http://localhost:3500/eth/v1/beacon/headers/head | jq

Lighthouse and Nimbus-Eth2

# Check Syncronization Status
curl -s http://localhost:5052/eth/v1/node/syncing | jq

# Check Overall Health
curl -s http://localhost:5052/eth/v1/node/health | jq

# Check Node Identity
curl -s http://localhost:5052/eth/v1/node/identity | jq

# Check Number of Peers
curl -s http://localhost:5052/eth/v1/node/peer_count | jq

# Fetch Latest Block Header
curl -s http://localhost:5052/eth/v1/beacon/headers/head | jq