Solana Mainnet-Beta Cluster Restart 1 June 2022

CLUSTER HAS RESTARTED

THESE INSTRUCTIONS ARE NOW OBSOLETE

PLEASE REMOVE THE –wait-for-supermajority AND –expected-bank-hash ARGUMENTS FROM YOUR START SCRIPT NOW

INSTALL THE LATEST SOLANA RELEASE AND ENSURE YOU HAVE THE UPDATED EXPECTED SHRED VERSION:

--expected-shred-version 51382 \

No need to restart your validator once removed, just remove them so when you next restart you won’t forget to do so.

Identify your highest optimistic slot:

Highest optimistically confirmed slot: 135986379

^ Find this by grepping your logs for “optimistic_slot”:

grep "optimistic" /path/to/log/file | tail

The highest slot might not be the last to appear in your logs, identify the highest of those printed.

Note: If your last confirmed slot is lower than that the one listed above, this is likely your node crashed before it was able to observe the latest supermajority. In this case, update according to step 2 then proceed to the Appendix.

Important: DO NOT delete your ledger directory.

Step 1: Create a snapshot at slot 135986379

You need to stop your validator process if it is still running.

This document assumes your ledger directory is called ledger/. If not then adjust the following commands accordingly.

Use the ledger tool to create a new snapshot at slot 135986379, replacing the two instances of <ledger path> to your actual ledger path:

$ solana-ledger-tool --ledger <ledger path> create-snapshot 135986379 <ledger path> --hard-fork 135986379

Add --snapshot-archive-path <PATH_TO_SNAPSHOTS> before “create-snapshot” if you have a separate snapshot dir, and --accounts <PATH_TO_ACCOUNTS> before –hard-fork if you have a separate accounts dir

The final line of output should be “Shred version: 51382”, and this snapshot file should now exist:

ledger/snapshot-135986379-6d5RCuQ3zvdwmepN75xuuQJf1Aa8jMxpbhisqWETgCkJ.tar.zst

Check your ledger/ directory to ensure that you have no snapshot newer than ledger/snapshot-135986379-6d5RCuQ3zvdwmepN75xuuQJf1Aa8jMxpbhisqWETgCkJ.tar.zst This is very unlikely, but if found should be removed - please post on Discord if you were to find a newer snapshot! Snapshots older than ledger/snapshot-135986379-6d5RCuQ3zvdwmepN75xuuQJf1Aa8jMxpbhisqWETgCkJ.tar.zst should not be removed.

NOTE: If you receive “Error: Slot 135986379 is not available”, please see appendix

Step 2: Update your validator software. The restart requires a patch

If you are on 1.9.x upgrade to 1.9.28

If you are on 1.10.x upgrade to 1.10.23

Step 3: Adjust your validator command-line arguments, temporarily for this restart to include:

(--known-validators aren’t needed if you have your own local snapshot and have set –no-genesis-fetch as your validator won’t be downloading anything, you can omit those arguments in this case)

--wait-for-supermajority 135986379 \

--no-snapshot-fetch \

--no-genesis-fetch \

--expected-bank-hash DfRg2DQzWVQjRTBSXwTaYgHDPZbQ85ebLrfayJmMENtp \

--expected-shred-version 51382 \

--known-validator PUmpKiNnSVAZ3w4KaFX6jKSjXUNHFShGkXbERo54xjb \

--known-validator SerGoB2ZUyi9A1uBFTRpGxxaaMtrFwbwBpRytHefSWZ \

--known-validator FLVgaCPvSGFguumN9ao188izB4K4rxSWzkHneQMtkwQJ \

--known-validator qZMH9GWnnBkx7aM1h98iKSv2Lz5N78nwNSocAxDQrbP \

--known-validator GiYSnFRrXrmkJMC54A1j3K4xT6ZMfx1NSThEe5X2WpDe \

--known-validator LA1NEzryoih6CQW3gwQqJQffK2mKgnXcjSQZSRpM3wc \

--known-validator Certusm1sa411sMpV9FPqU5dXAYhmmhygvxJ23S6hJ24 \

--known-validator 9bkyxgYxRrysC1ijd6iByp9idn112CnYTw243fdH2Uvr \

--known-validator 12ashmTiFStQ8RGUpi1BTCinJakVyDKWjRL6SWhnbxbT \

--known-validator FdaysQ2BZWUGBy8nqFgiudnrhzJp4xChQ8B4zJdc2JZB \

(Remove the previous value of “--expected-shred-version“ if present).

Once the cluster restarts and normal operation resumes, remember to remove --wait-for-supermajority and --expected-bank-hash before the next update or restart. They are only required for the restart. You can also go back to your old known-validators at that point.

(see next page for step 4)

Step 4: Start your validator

As it boots, it will load the snapshot for slot 135986379 and wait for 80% of the stake to come online before producing/validating new blocks.

To confirm your restarted validator is correctly waiting for 80% stake, look for this periodic log message to confirm it is waiting:

INFO solana_core::validator] Waiting for 80% of activated stake at slot 135986379 to be in gossip...

And if you have RPC enabled, ask it repeated for the current slot:

$ solana --url http://127.0.0.1:8899 slot

Any number other than 135986379 means you did not complete the steps correctly.

Once started you should see log entries for “Activate stake” visible in gossip and “waiting for 80% of stake” to be visible. You can track these to see how stake progresses.

If you couldn’t produce your snapshot locally follow appendix on next page below

Appendix: Resolution if you did not preserve your ledger or your last optimistically confirmed slot is below 135986379

NOT RECOMMENDED - this resolution should only be attempted if your ledger/ directory is unavailable or you are unable to produce a snapshot for 135986379.

ONLY IF your ledger history is corrupt or otherwise unavailable and your last confirmed slot is lower than 135986379, follow these instructions to get a new snapshot:

Your validator will need to download a new snapshot from one of the known validators. Alternative snapshot download methods are also provided further below. A snapshot will be verified as valid by the bank hash in the arguments below.

For this you need to remove –no-snapshot-fetch if present.

Add these arguments to restart:
--wait-for-supermajority 135986379\