> ## Documentation Index
> Fetch the complete documentation index at: https://seilabs-docs-bridge-release-v6-7-0.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Detailed guide for troubleshooting node problems.

export const RandomPeers = ({format = 'bash', network = 'mainnet', count = 5}) => {
  const PEERS = {
    mainnet: ['d53ab7681ed0df3d3249fc0132df7c3a131c9c1a@45.250.253.40:51656', '4ee8aea5bb58e9038d72e74322e3bba755287398@202.8.8.183:11956', '81409623ae4da3ec7b400cf640dea0b0999a964b@57.128.230.96:51556', '61a5be64b5215786fe5da712584678d0626636b5@87.249.137.71:51556', 'f83c536f43df9a5d900cd3c2f702c04d7dddc7b5@136.243.67.45:11956', 'b8d600a2f568576b5a2df5ee649cf9b809389064@162.19.62.176:26756', '57860b18ed3e1bbe8901ba73f2e63c7e6fe8b3d3@57.129.54.81:26656', '04fc6bba6c5c33034811612dca31c7adda24b299@91.134.60.37:16856', 'de64b779c7f4091e6f1765f5ca4c46f9d3011732@65.108.70.106:46656', '3be6b24cf86a5938cce7d48f44fb6598465a9924@p2p.state-sync.pacific-1.seinetwork.io:26656', '70e0c91b83b5ed1beaca798267f2debdf97dac10@18.156.6.83:26656', 'dd6b1ae002a15c1c8a38e05660f49a93c75d4159@148.251.181.225:26656'],
    testnet: ['71beea83970431f55816eee5f066a611a1dc80f7@p2p.state-sync.atlantic-2.seinetwork.io:26656', '65c257f9275beb1b99ca169ef89743c034b15db0@3.76.192.224:26656', '33588592e477c5238ff2a5d8dc765f85790ef853@23.109.47.225:26656', 'babc3f3f7804933265ec9c40ad94f4da8e9e0017@testnet-seed.rhinostake.com:11956', '8542cd7e6bf9d260fef543bc49e59be5a3fa9074@seed.publicnode.com:56656']
  };
  const pickRandom = (arr, n) => {
    const shuffled = [...arr];
    for (let i = shuffled.length - 1; i > 0; i--) {
      const j = Math.floor(Math.random() * (i + 1));
      [shuffled[i], shuffled[j]] = [shuffled[j], shuffled[i]];
    }
    return shuffled.slice(0, Math.min(n, arr.length));
  };
  const CopyIcon = ({className}) => <svg role="img" aria-label="Copy" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
      <path d="M7 7m0 2.667a2.667 2.667 0 0 1 2.667 -2.667h8.666a2.667 2.667 0 0 1 2.667 2.667v8.666a2.667 2.667 0 0 1 -2.667 2.667h-8.666a2.667 2.667 0 0 1 -2.667 -2.667z" />
      <path d="M4.012 16.737a2.005 2.005 0 0 1 -1.012 -1.737v-10c0 -1.1 .9 -2 2 -2h10c.75 0 1.158 .385 1.5 1" />
    </svg>;
  const CheckIcon = ({className}) => <svg role="img" aria-label="Copied" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
      <path d="M5 12l5 5l10 -10" />
    </svg>;
  const ShuffleIcon = ({className}) => <svg role="img" aria-label="Shuffle" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
      <path d="M16 3h5v5" />
      <path d="M4 20l16.2 -16.2" />
      <path d="M21 16v5h-5" />
      <path d="M15 15l6 6" />
      <path d="M4 4l5 5" />
    </svg>;
  const pool = PEERS[network] ?? [];
  const [peers, setPeers] = useState(() => pool.slice(0, count));
  const [copied, setCopied] = useState(false);
  const shuffle = () => {
    setPeers(pickRandom(pool, count));
  };
  useEffect(() => {
    shuffle();
  }, [network, count]);
  if (pool.length === 0) {
    return <div className="not-prose w-full">
        <pre className="m-0 p-3 rounded-md bg-neutral-100 dark:bg-neutral-800 text-sm opacity-70" style={{
      fontFamily: 'var(--sei-font-mono)'
    }}>
          No peers configured for network "{network}".
        </pre>
      </div>;
  }
  const peerString = peers.join(',');
  const displayText = format === 'toml' ? `persistent_peers = "${peerString}"` : `PEERS="${peerString}"`;
  const handleCopy = async () => {
    try {
      await navigator.clipboard.writeText(displayText);
      setCopied(true);
      setTimeout(() => setCopied(false), 2000);
    } catch {}
  };
  return <div className="not-prose w-full">
      <div className="relative">
        <pre className="m-0 px-4 py-3 pr-28 rounded-lg bg-neutral-100 dark:bg-neutral-900 text-[12.5px] leading-[1.55] whitespace-pre-wrap break-all border border-neutral-200 dark:border-neutral-700 text-neutral-900 dark:text-neutral-100" style={{
    fontFamily: 'var(--sei-font-mono)'
  }}>
          <code style={{
    fontFamily: 'var(--sei-font-mono)'
  }}>{displayText}</code>
        </pre>
        <div className="absolute top-2 right-2 flex items-center gap-1.5">
          <button type="button" onClick={shuffle} aria-label="Shuffle peers" title="Shuffle" className="inline-flex items-center gap-1 px-2 py-1 rounded-md text-[11px] font-semibold border border-neutral-300 dark:border-neutral-600 bg-white/80 dark:bg-neutral-800/80 text-neutral-700 dark:text-neutral-200 hover:bg-white dark:hover:bg-neutral-800 transition-colors">
            <ShuffleIcon />
            Shuffle
          </button>
          <button type="button" onClick={handleCopy} aria-label="Copy peers" title="Copy to clipboard" className="inline-flex items-center gap-1 px-2 py-1 rounded-md text-[11px] font-semibold border border-neutral-300 dark:border-neutral-600 bg-white/80 dark:bg-neutral-800/80 text-neutral-700 dark:text-neutral-200 hover:bg-white dark:hover:bg-neutral-800 transition-colors">
            {copied ? <>
                <CheckIcon className="text-green-600 dark:text-green-400" />
                Copied
              </> : <>
                <CopyIcon />
                Copy
              </>}
          </button>
        </div>
      </div>
    </div>;
};

Knowing common errors and their solutions helps keep your node healthy.

## Common error codes

These are the most frequent errors and their solutions:

### Consensus errors

If you get a consensus error, act quickly and appropriately:

```text theme={"dark"}
Error: "Consensus failure - height halted"
Solution: Check for network upgrades or chain halts
Command: seid status
```

```text theme={"dark"}
Error: "Private validator file not found"
Solution: Restore validator key or check file permissions
Location: $HOME/.sei/config/priv_validator_key.json
```

```text theme={"dark"}
Error: "Duplicate signature"
Solution: IMMEDIATELY STOP NODE - potential double signing risk
Action: Check validator operation on other machines
```

### Network errors

Network errors can prevent your node from participating in consensus:

```text theme={"dark"}
Error: "Dial tcp connection refused"
Solution: Check network connectivity and firewall rules
Commands:
  - netstat -tulpn | grep seid
  - ufw status
```

```text theme={"dark"}
Error: "No peers available"
Solution: Verify peer connections and network config
Commands:

- curl localhost:26657/net_info
```

### Database errors

Database corruption can require immediate attention:

```text theme={"dark"}
Error: "Database is corrupted"
Solution: Reset database or restore from backup
Commands:
  - seid tendermint unsafe-reset-all
  - cp -r backup/data $HOME/.sei/
```

### Diagnostic commands

These commands help you investigate problems and monitor your node:

```bash theme={"dark"}
# Check node synchronization
seid status

# Check validator status
seid query staking validator $(seid tendermint show-validator)

# Monitor real-time logs
journalctl -fu seid -o cat

# View system resource usage
top -p $(pgrep seid)
```

## AppHash mismatch errors

If you get an AppHash mismatch, capture the state to compare it with a known-good version:

```bash theme={"dark"}
# For SeiDB (all supported nodes):
git clone https://github.com/sei-protocol/sei-db.git
cd sei-db/tools
make install
systemctl stop seid
seidb dump-iavl -d $HOME/.sei/data/committer.db -o /home/ubuntu/iavl-dump
systemctl restart seid
```

<Note>As with FlatKV, the memIAVL store has two possible locations. Nodes created before the layout change keep the legacy `$HOME/.sei/data/committer.db` path shown above (it takes precedence when present). New nodes use `$HOME/.sei/data/state_commit/memiavl`. Point `-d` at whichever path exists on your node.</Note>

On Giga Storage nodes, EVM state lives in a FlatKV store instead of the memIAVL trees. An AppHash comparison there also requires a FlatKV state dump. Use the `dump-flatkv` command to iterate and dump the physical `(key, value)` pairs into per-bucket files. The files match the `dump-iavl` format, so the same diff tooling works on both:

```bash theme={"dark"}
# For FlatKV (Giga Storage nodes hold EVM state in FlatKV):
systemctl stop seid
seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump
systemctl restart seid
```

<Note>Nodes created before the storage layout change keep the FlatKV store at the legacy path `$HOME/.sei/data/flatkv` instead of `$HOME/.sei/data/state_commit/flatkv`. Pass whichever directory exists on your node to `--db-dir`.</Note>

The `dump-flatkv` command accepts these flags:

* `--db-dir` (`-d`): The FlatKV database directory.
* `--output-dir` (`-o`): The output directory, with one file for each bucket. Required unless `--lthash-only` is set.
* `--height`: The FlatKV target version. The default, `0`, selects the latest available version.
* `--bucket` (`-b`): Restrict the dump to a single bucket (`account`, `code`, `storage`, or `legacy`). The default is all buckets. This only filters which files are written; the full keyspace is always scanned, and the LtHash always covers all four buckets.
* `--lthash`: Also compute the per-bucket and total LtHash (lattice hash) over the scanned state, and verify the total against the committed root recorded in snapshot metadata. The default is `true`. A mismatch exits non-zero.
* `--lthash-only`: Compute and verify the LtHash without writing any bucket dump files. Requires `--lthash=true`, and does not require `--output-dir`. The default is `false`.
* `--read-limit-mb`: Throttle the scan to at most this many MiB/s of (key+value) bytes read, so a dump against a running node does not starve the chain of disk bandwidth. The default is `64`. Set `0` for unlimited. Keep it at the default or lower on a shared/live node; raise it only for offline runs on idle disks.

Because `dump-flatkv` opens an independent read-only clone of the store (the snapshot is hard-linked and the changelog WAL is replayed into a temp directory), it is safe to run against a live, block-producing node, and the `--read-limit-mb` throttle prevents it from starving the node of disk bandwidth.

For example, to dump only the `storage` bucket at a specific version:

```bash theme={"dark"}
seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump --height 12345678 --bucket storage
```

To verify the FlatKV lattice hash against the committed snapshot metadata without writing any key/value dump files:

```bash theme={"dark"}
seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --lthash-only
```

For an offline run on an idle disk, go full speed and skip the LtHash:

```bash theme={"dark"}
seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump --read-limit-mb 0 --lthash=false
```

With `--lthash`, the output prints a per-bucket count and checksum plus a `TOTAL` LtHash, followed by a `LtHash verification vs snapshot metadata` PASS/FAIL line. Verification is skipped (rather than failing) when the selected snapshot predates LtHash metadata, since the committed hash then covers only replayed WAL deltas rather than full state.

### Comparing EVM state between memIAVL and FlatKV

When you debug an AppHash mismatch that involves EVM state, a byte-for-byte physical dump can diverge between backends, even when the underlying state is identical. This happens because every FlatKV value embeds a per-key block-height stamp (the height at which the key was last written or migrated). On a freshly migrated node, this stamp differs from the memIAVL leaf versions.

The `evm-logical-digest` command works around this with a backend-independent digest of the EVM *logical* state. On both sides, it strips the serialization-version and block-height header. Then it digests only the logical payload: account balance, nonce, and code hash, plus bytecode and storage words. With this digest, you can compare a memIAVL node and a FlatKV node at the same chain height:

```bash theme={"dark"}
# FlatKV digest at a height (WAL-replays to it):
seidb evm-logical-digest --backend flatkv \
    --db-dir $HOME/.sei/data/state_commit/flatkv --height 213200000

# memIAVL digest at the same height (0 = current symlink):
seidb evm-logical-digest --backend memiavl \
    --db-dir $HOME/.sei/data/state_commit/memiavl --height 213200000
```

The `--backend` flag accepts `flatkv`, `memiavl`, and `composite`.

For memIAVL, `--memiavl-open-mode` controls how leaves are read:

* `snapshot` (the default) sequentially scans the completed snapshot `kvs` file at `snapshot-<height>/evm`. This is the fast path and requires an on-disk snapshot at that exact height (or `--height 0` for the current symlink). Prefer it whenever your target height matches an existing snapshot boundary.
* `replay` opens a read-only DB, replays the changelog up to `--height`, then walks the in-memory/mmap tree. It is roughly an order of magnitude slower than `snapshot`, so use it only when no snapshot exists at the target height (for example, when a node's snapshot rewrite lags the tip):

```bash theme={"dark"}
# memIAVL digest for a height with no on-disk snapshot (slower):
seidb evm-logical-digest --backend memiavl --memiavl-open-mode replay \
    --db-dir $HOME/.sei/data/state_commit/memiavl --height 213205000
```

On a node that is mid-migration, EVM state is split between the two backends: FlatKV holds the rows already migrated, while memIAVL still holds the rows not yet past the migration boundary. Use `--backend composite` to digest the union of both, so a migrating node can be compared against a memIAVL-only node at the same height. In `composite` mode, `--db-dir` is not required; instead provide the two backend directories with `--flatkv-dir` and `--memiavl-dir`. Because a live migrating node usually keeps memIAVL snapshots at heights outside FlatKV's retained window, pass `--memiavl-open-mode replay`:

```bash theme={"dark"}
# Mid-migration node: digest the flatkv + memiavl union at one height:
seidb evm-logical-digest --backend composite --memiavl-open-mode replay \
    --flatkv-dir $HOME/.sei/data/state_commit/flatkv \
    --memiavl-dir $HOME/.sei/data/state_commit/memiavl --height 213200000
```

Each run prints per-bucket `bucket_digest` values and a single `FINAL_DIGEST` line that covers the `account`, `code`, `storage`, and `legacy` buckets. Compare the `FINAL_DIGEST` lines from both backends at the same height. They should match. FlatKV can contain FlatKV-only migration marker rows that a memIAVL-only node never owns: the migration-version marker (present once a migration completes) and the migration-boundary cursor (present only while a migration is in flight). The command automatically omits both rows from the final result, so memIAVL, mid-migration, and completed nodes all produce comparable digests.

For targeted debugging, the command can also inspect a single normalized bucket instead of printing the global digest. The [seidb tooling section of the technical reference](/node/technical-reference#comparing-evm-state-across-backends) has the full flag reference, including inspect mode, sharding, and `--find-hash`. These examples list the first 50 `account` rows with version metadata, and shard the `storage` bucket under a key prefix by the next 2 bytes:

```bash theme={"dark"}
seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \
    --inspect-bucket account --list --list-limit 50 --details

seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \
    --inspect-bucket storage --key-prefix 03 --shard-next-bytes 2
```

<Warning>
  **Enable SeiDB State Commit (SC).** Set `sc-enable = true` in the `[state-commit]` section of `app.toml`. The legacy IAVL backend was fully removed, and SC is now mandatory. If SC is not enabled, the node no longer falls back to IAVL. It panics at startup with this error:

  ```text theme={"dark"}
  SeiDB state-commit (SC) must be enabled; IAVL backend has been fully deprecated
  ```

  The `seid debug dump-iavl` command was also removed with the IAVL backend. To inspect state, use the `seidb dump-iavl` tool shown above.
</Warning>

When you report a problem, always include the app hash, commit hash, and block height from your logs.

### Identifying AppHash errors

In logs, AppHash errors usually look like this:

```text theme={"dark"}
ERR wrong Block.Header.AppHash. Expected [EXPECTED_HASH], got [ACTUAL_HASH]
block_id={"hash":"...","parts":{"hash":"...","total":1}} height=[HEIGHT]
```

**Common causes:**

* Using an incorrect node version during sync (make sure that you run the latest version)
* Corrupted or incorrectly applied snapshots
* Database inconsistencies from improper shutdowns
* Syncing with outdated or incompatible peers

**Resolution steps:**

1. **Stop the node immediately.**

2. **Try a node rollback first.** See [Node rollback](/node/troubleshooting#node-rollback).

3. **If the rollback fails, restore from a fresh snapshot:**

   * Download a recent snapshot from trusted providers (Polkachu, PublicNode)
   * Make sure that you use the correct node version
   * Verify that peer configurations are up to date

4. **Restart the node and monitor the logs for continued errors.**

### Peer connection issues as AppHash red herrings

**Important:** Peer connection failures are often symptoms of underlying AppHash errors, not the root cause.

If you see many peer connection errors like these:

```text theme={"dark"}
ERR failed to handshake with peer
ERR failed to send request for peers
ERR peer handshake failed endpoint={} err=EOF
```

**Do not focus only on fixing peer connections first.** Instead:

1. **Scan your logs carefully** for AppHash errors that may appear intermittently
2. **Look for the actual error pattern:**
   ```text theme={"dark"}
   ERR wrong Block.Header.AppHash. Expected [HASH], got [HASH]
   ```
3. **Check whether your node is stuck** at a specific height despite peer connection attempts

**Why this happens:**

* AppHash mismatches prevent proper block validation
* The node cannot advance to new blocks because of a state inconsistency
* Peers may reject connections from nodes with corrupted state
* The network appears to be the problem, but the cause is a local state issue

**Debugging approach:**

1. **First, check for AppHash errors** in your logs (search for "wrong Block.Header.AppHash")
2. **If you find AppHash errors**, treat them as the primary issue
3. **Focus on peer connection fixes** only if no AppHash errors exist

This approach targets the root cause, not the symptoms, and can save hours of debugging time.

### Peer connection and handshake issues

**Identifying peer issues:**

Look for these error patterns in your logs:

```text theme={"dark"}
ERR failed to handshake with peer err="expected to connect with peer \"[EXPECTED_ID]\", got \"[ACTUAL_ID]\""
ERR failed to send request for peers err="no available peers to send a PEX request to (retrying)"
ERR peer handshake failed endpoint={} err=EOF module=p2p
```

**Common causes:**

* Outdated peer configurations with mismatched node IDs
* Network infrastructure changes on the peer side
* A firewall that blocks connections on port 26656
* DNS resolution issues

**Resolution steps:**

1. **Update peer configurations** with current node IDs:

   <RandomPeers network="mainnet" format="toml" />

2. **Verify network connectivity:**

   ```bash theme={"dark"}
   # Test connection to peer endpoints
   nc -zv p2p.state-sync-0.pacific-1.seinetwork.io 26656

   # Check if port 26656 is open for inbound connections
   netstat -tulpn | grep :26656
   ```

3. **Check the current peer status:**
   ```bash theme={"dark"}
   curl http://localhost:26657/net_info | jq '.result.peers | length'
   curl http://localhost:26657/lag_status | jq .
   ```

### Sync performance issues

**Identifying sync problems:**

Monitor these indicators:

```bash theme={"dark"}
# Check sync status and lag
curl http://localhost:26657/lag_status | jq .

# Monitor if height is progressing
curl http://localhost:26657/status | jq '.result.sync_info'
```

**Common solutions:**

1. **Increase the packet payload size** for large block processing:

   ```toml theme={"dark"}
   # In config.toml [p2p] section
   max-packet-msg-payload-size = 1024000  # Increase from default 102400
   ```

2. **Optimize the mempool settings** in `config.toml`:

   ```toml theme={"dark"}
   # In [mempool] section
   keep-invalid-txs-in-cache = true
   ttl-duration = "5s"
   ttl-num-blocks = 5
   ```

3. **If the node gets stuck at a specific height:**
   * Try restarting the node
   * If that does not help, perform a rollback
   * Consider taking a fresh snapshot

**Warning signs to watch for:**

* The current height does not increase over time
* Increasing lag between the current height and the max peer height
* Repeated timeout errors in the logs
* Mempool size that consistently reaches its limits

## Crash and panic debugging

For crashes, panics, or nil pointer exceptions:

* Capture at least 1,000 lines of logs before the crash or 15 minutes of log data, whichever gives more context
* Include the full stack trace, if it is available

### Logging configuration

Proper logging configuration is essential for debugging and monitoring:

```toml theme={"dark"}
# In config.toml
# Set appropriate log level
log_level = "debug"  # Use "trace" for maximum detail

# Choose log format
log_format = "json"  # Use "plain" for human-readable logs
```

Configure log rotation to manage storage:

```bash theme={"dark"}
# Example logrotate configuration
sudo tee /etc/logrotate.d/seid << EOF
/var/log/seid/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 sei sei
    sharedscripts
    postrotate
        systemctl reload seid
    endscript
}
EOF
```

Enable core dumps for crash analysis:

```bash theme={"dark"}
# Set unlimited core dump size
ulimit -c unlimited

# Configure core dump location
echo "/tmp/core.%e.%p" > /proc/sys/kernel/core_pattern
```

## Other common issues and fixes

1. **Sync problems**

   * Check available disk space (`df -h`)
   * Make sure that peer connections work (`curl http://localhost:26657/net_info`)
   * Check that the firewall allows port 26656

2. **Performance issues**

   * Monitor system resources (`htop` or `iotop`)
   * Check disk I/O performance (`iostat`)
   * Analyze network traffic (`iftop`)

3. **Database issues**

   * Run database integrity checks:

     ```bash theme={"dark"}
     seid debug dump-db | grep -i error
     ```

     If you find errors, consider restoring from a recent backup.

   * To keep less historical data, lower `ss-keep-recent` in `app.toml`.

   * To rebuild the node with a smaller database, reset it. Then resync from a
     [snapshot](/node/snapshot) or with [state sync](/node/statesync). The
     reset deletes all chain data, so it is not a pruning tool. Before you
     reset, back up `priv_validator_key.json` and `priv_validator_state.json`,
     as described in [Clean up](/node/statesync#clean-up):

     ```bash theme={"dark"}
     seid tendermint unsafe-reset-all --home $HOME/.sei --keep-addr-book
     ```

     Alternatively, remove old state snapshots manually to free disk space:

     ```bash theme={"dark"}
     rm -rf $HOME/.sei/data/snapshots/*
     ```

## Node rollback

To roll back a node from an AppHash mismatch, first stop the node in your
preferred way.

Next, roll back the node:

```bash theme={"dark"}
seid rollback
```

Then, restart the node.

If you see this error when you try to roll back:

```bash theme={"dark"}
failed to initialize database: resource temporarily unavailable
```

This means that you did not shut down the node properly. In that case, try to shut down or kill the `seid` process directly. If this does not help, restart your machine.

Then try the rollback steps again.

## HashVault app-hash equivocation panic

Autobahn validators and fullnodes run an app-hash equivocation guard called HashVault, enabled by default. It stores the app hashes the node has committed in a durable database under `<PersistentStateDir>/hashvault`. If the node is ever about to commit a different app hash for a height it has already finalized, HashVault halts the node to prevent it from "changing its mind" about a finalized block.

When the guard trips, you will see a panic in the logs similar to:

```text theme={"dark"}
Hashvault detected app hash mismatch; node attempted to change its mind. DO NOT RESTART WITHOUT HUMAN INVESTIGATION. ...
blockHeight=[HEIGHT] existingHex=[...] incomingHex=[...] hashVaultDir=[/path/to/hashvault]
```

<Warning>
  **Do not restart the node without human investigation.** A HashVault mismatch can indicate a genuine equivocation. Restarting blindly, or removing the guard, can cause your validator to externalize a conflicting hash for an already-finalized height, which risks **slashing**.
</Warning>

**Investigate first.** Determine whether the mismatch reflects a real equivocation (for example, the same validator key running on more than one machine, or committed state that diverged from what the vault recorded) or a benign situation such as an out-of-band rollback or restore that left the committed app state inconsistent with the vault's history.

### Recovery (only if you are certain there is no real equivocation)

If, and only if, you are certain the stored hashes are wrong and there is no real equivocation, you can bypass the guard:

1. **Stop the node.**
2. **Delete the HashVault data directory** shown as `hashVaultDir` in the panic message (by default `<PersistentStateDir>/hashvault`).
3. **Restart the node.** It starts with an empty equivocation history and re-commits hashes as it re-executes blocks.

Deleting this directory removes equivocation protection for the heights it covered. If the node then commits a conflicting hash for a height it has already finalized, the validator may be slashed.

### Last-resort: disabling HashVault

If you repeatedly hit the same panic on new blocks and are very sure the stored hashes are totally wrong, you can run the node with the guard disabled by setting the top-level `hash-vault-disabled-unsafe` field in `config.toml`:

```toml theme={"dark"}
# config.toml (top-level, before any [section] header)
hash-vault-disabled-unsafe = true
```

This runs the node **without any app-hash equivocation protection** and logs error-level warnings on every startup. The default is `false`, and you should re-enable the guard (set it back to `false` or remove the line) as soon as the underlying issue is resolved.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.