Handling DKG Failure
Handling DKG failure
While the DKG process has been tested and validated against many different configuration instances, it can still encounter issues which might result in failure.
Our DKG is designed in a way that doesn't allow for inconsistent results: either it finishes correctly for every peer, or it fails.
This is a safety feature: you don't want to deposit an Ethereum distributed validator that not every operator is able to participate in.
The most common source of issues lies in the network stack: if any of the peers' Internet connection glitches substantially, the DKG will fail. If you are attempting to run the dkg command in two places at once, or you have a charon run command with the same charon-enr-private-key as you are trying to DKG with, these may also disrupt a key generation ceremony.
Charon's DKG doesn't allow peer reconnection once the process is started, but it does allow for re-connections before that.
When you see the following message:
this means your Charon instance is waiting for all the other cluster peers to start their DKG process: at this stage, peers can disconnect and reconnect at will, the DKG process will still continue.
A log line will confirm the connection of a new peer:
As soon as all the peers are connected, this message will be shown:
Past this stage no disconnections are allowed, and all peers must leave their terminals open in order for the DKG process to complete: this is a synchronous phase, and every peer is required in order to reach completion.
If for some reason the DKG process fails, you would see error logs that resemble this:
As the error message suggests, the DKG process needs to be retried.
Cleaning up the .charon directory
.charon directoryOne cannot simply retry the DKG process: Charon refuses to overwrite any runtime file in order to avoid inconsistencies and private key loss.
When attempting to re-run a DKG with an unclean data directory - which is either .charon or what was specified with the --data-dir CLI parameter - this is the error that will be shown:
The disallowed_entity field lists all the files that Charon refuses to overwrite, while data-dir is the full path of the runtime directory the DKG process is using.
In order to retry the DKG process one must delete the following entities, if present:
validator_keysdirectorycluster-lock.jsonfiledeposit-data.jsonfile
The charon-enr-private-key file must be preserved, failure to do so requires the DKG process to be restarted from the beginning by creating a new cluster definition.
If you're doing a DKG with a custom cluster definition - for example, create with charon create dkg, rather than the Obol Launchpad - you can re-use the same file.
Once this process has been completed, the cluster operators can retry a DKG.
Further debugging
If you are trying to create an extremely large, geographically diverse cluster, there is a chance the process could be timing out. Consider adding the flags --timeout=5m --shutdown-delay=60s to allow more time for the ceremony to complete and safely shut down across all nodes.
In order for the Obol team to debug your issue as quickly and precisely as possible, please provide full logs in text form, not through screenshots or display photos.
Providing complete debug logs from all peers is particularly important, since it allows the team to reconstruct precisely what happened throughout the ceremony.
Last updated