Skip to main content

Backup and Disaster Recovery - Custody

This document outlines the recommended setup for backup and disaster recovery (BDR) of Sodot MPC Infra in custodial settings.

The objective is to ensure that private keys remain accessible, confidential, and recoverable under any failure or incident scenario.

Backup​

Overview​

Each MPC Infra custodial deployment is recommended to consist of a 3-node Vertex cluster. Each Vertex is an MPC signer node operating within a trusted execution environment (TEE). Every node holds only a single share of each private key, preventing a single point of compromise.

Key shares can be exported as encrypted backups based on a public backup encryption key and a private backup decryption key for disaster recovery. Encryption is based on a hybrid scheme combining RSA-OAEP-SHA256 for key encapsulation and ChaCha20-Poly1305 for data encryption and authentication.

For full technical details regarding private and public backup key generation and use for recovery can be found here, including sample API requests and code examples.

Backup Key Generation​

The backup key should be generated in a secure environment that is independent of the environment which the Vertex MPC cluster is deployed in.
Concretely, it is recommended to perform key generation on an offline machine or using an HSM. The private backup decryption key should remain offline, until a disaster recovery process takes place.

Security Consideration

Each Vertex node should have a unique backup private key. These keys can be generated either in separate secure environments or together if strict isolation is not required. Using a single backup key for all nodes must be avoided, as it would undermine the distributed security model, allowing one node to decrypt the others’ shares during a disaster recovery process.

Configuration​

During deployment, each Vertex node is configured with backup encryption and decryption keys using deployment configuration parameters which include:

  • Backup public encryption key
    • --backup-public-key
    • --backup-public-key-scheme="RSA_OAEP_SHA256_CHACHA20_POLY1305"
  • Backup private decryption key
    • --import-private-key
    • --import-private-key-scheme="RSA_OAEP_SHA256_CHACHA20_POLY1305"

Full Terraform documentation found here.

Encrypted key share export - In case a backup public key is configured, then the node will be able to export encrypted key shares. In order to access this API, it is required that the dedicated KeysExporter role will be assigned for the user performing the request.

Encrypted key share import (recovery) - In case an import private key is configured, then the node will be able to import encrypted backups created with the above export mechanism. The encrypted key share will be decrypted within the TEE and loaded into the Vertex state, allowing seamless use of the imported key share. This should only be configured in the case of a new Vertex node being deployed as part of a disaster recovery process.

Note for the Disaster Recovery Process

During a disaster recovery process, it is recommended to generate a fresh private key in a secure environment and use the fresh backup public key as the backup public key of the newly deployed Vertex node.

Backup Key and Encrypted Key Material Storage​

Encrypted key share backups should be stored in immutable (WORM) object storage such as AWS S3 Object Lock or GCP Bucket Lock. These backups are append-only, ensuring resilience against accidental deletions or overwrite operations.

Recommended best practices include:

  • Event-driven backups immediately after key generation or import.
  • Scheduled backups at regular intervals to minimize recovery point objectives (RPO) and preventing event-driven backup drift.
  • Regular "break the glass" emergency procedures to ensure all backup measures are functioning properly.

As mentioned above, the import private key required to decrypt backup shares should be stored separately (ideally offline or within an HSM) and only injected during redeployment for Vertex node restoration purposes.

Disaster Recovery Scenarios​

Total Cloud Environment Outage – Loss of All Vertices​

Impact: Signing becomes unavailable in the affected region.
Objective: Deploy a new Vertex cluster in a secondary region and restore encrypted shares.

Procedure:

  1. Deploy a new Vertex cluster using Terraform in the disaster recovery region with --import-private-key enabled.
  2. Import the encrypted server shares via the /backup/import-key-share API endpoint.
  3. Redeploy any rule servers used by the lost deployment and reconfigure policies.
  4. Reattach policies using /admin/policies/attach-policy-to-key.
  5. Redirect trading systems to the new cluster endpoints and validate signature integrity.

Expected Recovery Time (RTO): depends on access time required for the import private key.
Recovery Point (RPO): depends on regular backup cadence.

Database Loss (Vertex)​

Each Vertex uses a managed PostgreSQL database with replication. In the event of database loss, the node can restore from snapshots or re-import its key shares from encrypted backups in case snapshot recovery fails. The cluster also remains operational in cases where quorum is maintained.

Relay Outage​

If the Relay service fails, existing signing sessions will fail, new signing sessions are temporarily paused while existing WebSocket or FIX sessions continue.
Since the Relay does not use any persistent state, restoration is achieved by redeploying the service and confirming signing integrity before resuming operations.

The Relay deployment utilizes HPA with kubernetes for scalability, which will redeploy Relay pods in case of failure. Therefore the above scenario should only occur due to lower level infrastructure issues at the kubernetes level.

Offline Recovery​

In case there is a need to recover the full keys offline, it is possible to use the Sodot SDKs for Node, Go or Rust (technical details can be found here) in an offline environment in order to recover the full keys from decrypted key shares from all 3 Vertex nodes. This is not a recommended option; it is, however, provided as an exit mechanism from using Sodot software.

Runbooks​

Below are references to copy-paste snippets for each relevant operation for DR.

Export Encrypted Share​

Used to generate an encrypted backup of a Vertex key share.
Reference here.

Import Encrypted Share (Recovery)​

Used to import an encrypted key share to a fresh Vertex node during recovery.
Reference here.

Decrypting Backup Shares​

For decrypting encrypted key share backup outside of a Vertex node, can be used for offline recovery purposes.
Reference here.