# Azure Sodot MPC Vertex is a self-hosted deployable service that represents an MPC party for MPC operations (such as keygen, sign, refresh etc.). It takes care of storing the secret share, as well as running the MPC operations themselves. It can be used seamlessly with all Sodot MPC SDKs. The Vertex exposes a REST API, documented [here](https://docs.sodot.dev/vertex-api-reference), that allows creating and managing secret shares easily. ## Architecture The self-hosted solution comes in two forms: a dedicated [Terraform module](https://developer.hashicorp.com/terraform/intro) and a [Helm chart](https://helm.sh/docs/topics/charts/) (`vertex-azure-aci`, which the Terraform module uses internally). Both should be run on your organization's computing infrastructure. We recommend you first look at our example Terraform module and modify it according to your organization's needs. The module runs the Vertex as confidential containers on [Azure Container Instances](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-confidential-overview) (ACI), scheduled from an AKS cluster through [virtual nodes](https://github.com/microsoft/virtualnodesOnAzureContainerInstances). It creates: * an AKS cluster, with two virtual nodes that place the Vertex pods on ACI; * a virtual network, with a subnet delegated to ACI that reaches the internet through a NAT gateway; * a PostgreSQL flexible server with private access only; * a managed identity for the Vertex, which is also its database login; * an Ingress that terminates TLS with your certificate. It is the Vertex's only public entry. The Vertex encrypts its database with a key in your Managed HSM, which releases it only to the Vertex running in an attested confidential container (see [Secure Key Release](#setting-up-skr)). ## Requirements Before starting the installation process, make sure you have the following installed: 1. The `terraform` [CLI](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli), version 1.14 or later. 2. The `az` CLI tool for Azure, with the `confcom` and `attestation` extensions (see [Generating the Confidential Computing Policy](#generating-the-policy)), and `bash` and `jq`, which the module's scripts use. 3. Make sure you are **logged in** (i.e `az login`). 4. Make sure your account has permissions to **create resources** (such as clusters, load balancers, etc...) and to assign roles. In addition, this guide assumes you're running the provided example Terraform module as the root module (although most steps are applicable when running it as a submodule as well). ## Installation Process The installation process is made of the following steps: 1. Collecting installation parameters 2. Obtaining a TLS certificate 3. Obtaining the Terraform module 4. Setting up Secure Key Release 5. Applying the Terraform module to your cloud environment 6. Setting the DNS Record for `URL_ENDPOINT` to point to the Vertex ### 1. Collecting Parameters At this step you should collect the following pieces of information, listed down below. At the end of each bullet we put in parentheses a capitalized name which will be used to refer to the value being set throughout the rest of this guide. 1. The endpoint URL that you wish to host your Vertex at (`URL_ENDPOINT`). 2. The Sodot Docker registry user and token which provides access to all relevant Helm charts, Docker images and Terraform modules (`ARTIFACTORY_USER` and `ARTIFACTORY_TOKEN`). You can obtain these by following the [Obtaining Credentials](/vertex/deployment/obtaining-artifacts-credentials) guide. 3. The product you wish to deploy over the Vertex (`PRODUCT`). Currently, the supported products are `mpc-infra` and `exchange-api-vault`. ### 2. Obtaining a TLS Certificate You should obtain a TLS certificate for the `URL_ENDPOINT` domain. The certificate should be configured inside an instance of [Azure Key Vault](https://azure.microsoft.com/en-us/products/key-vault) (`KEY_VAULT_NAME`). To use this certificate for the Vertex, make sure you pass the relevant variables when performing the `terraform apply` operation (more on that specifically below). ### 3. Obtaining the Terraform Module You can consume the **Vertex** Terraform module from the Sodot private registry by downloading it as a zip file by its version. 1. Fetch the desired release (replace `` with a tag such as `v3.0.0`): ```bash curl -u ":" \ -O "https://repo.sodot.dev/artifactory/sodot-terraform-modules/sodot/vertex/azure/.zip" ``` Alternatively, you can use the Artifactory GUI instead.

2. Unzip and make it your working directory: ```bash unzip .zip -d vertex cd vertex ``` ### 4. Secure Key Release (SKR) with Azure Confidential Workloads on ACI :::note[Secure Key Release Feature] Secure Key Release (SKR) is a functionality of Azure Key Vault (AKV) Managed HSM. Secure key release enables the release of an HSM protected key from AKV to an attested Trusted Execution Environment (TEE), such as the Vertex confidential workload on ACI. SKR adds another layer of access protection to your data decryption/encryption keys where you can let only a specific application + TEE runtime environment with known configuration get access to the key material (in our case, only the Vertex will be able to access the DB encryption key). The SKR policies defined at the time of the KEK (key encryption key) creation govern the access to these keys. ::: In order to use the SKR feature, you will need to [spawn and activate a new Managed HSM instance](https://learn.microsoft.com/en-us/azure/key-vault/managed-hsm/quick-create-cli), with you as the Managed HSM Administrator. After activating the Managed HSM, give the identity that runs Terraform the `Managed HSM Crypto Officer` role (using the `Local RBAC` tab of the mHSM resource). Crypto Officer manages roles but can't create keys, so the module also grants that identity `Managed HSM Crypto User` on keys, and waits for the grant before it creates the Vertex's key. The module only assigns the roles that identity doesn't already hold, so assignments you made by hand are reused. Several deployments can use the same Managed HSM. #### Generating the Confidential Computing Policy To run the Vertex as a confidential workload on ACI, the Terraform module generates a confidential computing enforcement (CCE) policy, which is specific to your environment and so has to be generated locally. `terraform plan` generates the policy without changing the HSM key; `terraform apply` adds the policy to the key's release policy before it rolls out the Vertex. As the policy generation requires access to the Azure CLI, and specifically to the `az confcom` extension, there are two options to perform this step: **Option A**: If you are running on a Linux machine, you can just install the `confcom` extension and run the terraform commands directly from your machine. **Option B**: If you are running on MacOS (as the `confcom` extension is not supported), or do not wish to install the `confcom` extension on your machine, you can use the provided Dockerfile to build a container that will run the terraform commands for you. #### Linux with confcom Extension If you are running on a Linux machine, you can install the `confcom` extension by running: ```bash az extension add --name confcom az extension add --name attestation ``` #### MacOS Builder Image Inside the downloaded terraform code directory, build the builder image: ```bash cd policy_builder # build the image docker build --platform linux/amd64 . -t sodot-policy-terraform cd .. ``` Set `ARTIFACTORY_USER` and `ARTIFACTORY_TOKEN` in your shell, then prepare the builder with all required credentials: ```bash # Create a docker volume to store your az credentials docker volume create azure_creds # Create a docker volume to store your artifactory credentials docker volume create docker_creds # First, login to az cli and store the creds docker run --platform linux/amd64 -v azure_creds:/root/.azure -v docker_creds:/root/.docker -it sodot-policy-terraform az login # Store Artifactory credentials for confcom to pull the Vertex and SKR images printf '{"auths":{"repo.sodot.dev":{"auth":"%s"}}}' "$(printf '%s:%s' "$ARTIFACTORY_USER" "$ARTIFACTORY_TOKEN" | base64)" | docker run --platform linux/amd64 -v docker_creds:/root/.docker -i sodot-policy-terraform \ sh -c 'umask 077 && cat > /root/.docker/config.json' ``` For an existing deployment, follow [Upgrading the Module](#upgrading-the-module) before the first plan or apply. Finally, run the terraform commands inside the builder image from the **root** directory of the repo: ```bash # Run these commands in the terraform code directory docker run --platform linux/amd64 -v azure_creds:/root/.azure -v docker_creds:/root/.docker -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD:/work" -it sodot-policy-terraform terraform init docker run --platform linux/amd64 -v azure_creds:/root/.azure -v docker_creds:/root/.docker -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD:/work" -it sodot-policy-terraform terraform apply # You can use the same command to perform other tf commands (e.g. destroy) ``` ### 5. Applying the Terraform Module to Your Cloud Environment Setting up the relevant infrastructure is performed by running `terraform init` followed by `terraform apply`. Where you run those commands depends on the option you chose in [Generating the Confidential Computing Policy](#generating-the-policy). #### Configuring Terraform The module’s inputs are declared in `variables.tf` and have sensible defaults where appropriate. You **must** supply the following variables when invoking **both** `terraform apply` and `terraform destroy`: * `azure_subscription_id` - the ID of the Azure subscription where the cluster will be provisioned. * `location` (default `westus`) - the Azure region of the cluster and the rest of the resources. It must offer confidential containers on ACI and virtual nodes; the module accepts the regions [listed by Azure](https://github.com/microsoft/virtualnodesOnAzureContainerInstances?tab=readme-ov-file#region-deployments). Changing it replaces every resource. * `artifactory_user=` - provides Terraform with permissions to pull the Helm chart and the Vertex Docker image. * `artifactory_token=` - provides Terraform with permissions to pull the Helm chart and the Vertex Docker image. * `product_name=` - the Sodot product to deploy over the Vertex (e.g. `mpc-infra` or `exchange-api-vault`). * `sodot_relay_api_key=` - an API key for accessing the Sodot Relay Server. * `sodot_relay_url=` - the URL for your organization's deployed Relay Server. * `vertex_admin_token=` - the secret admin token that will be used to set up all other users of the Vertex. * `tls_cert_key_vault_name=` - configure the name of the key vault holding the certificate. * `tls_cert_rg_name=` - configure which Resource Group the key vault is associated with. * NOTE: This is **not** the Resource Group that the Vertex cluster will be created under. * `tls_cert_uri=` - the ID (URI) of the certificate that will be used for TLS termination, without its version. * `vertex_dns_address=` - equal to `URL_ENDPOINT`. * `secure_key_release_encryption_hsm_name=` - the name of the Managed HSM instance that will be used for SKR (see above requirements). * `secure_key_release_encryption_hsm_rg_name=` - the name of the Resource Group that the Managed HSM instance is associated with. The rest of the variables have sensible defaults, but it is recommended to go over them and customize them for your own needs. :::tip[Backup Settings] If you wish to backup your key shares manually, consult the [Backing Up Key Shares](/vertex/backup_key_shares#configuring-backup-settings-for-the-vertex) guide for more Terraform configurations and info. ::: Example for **option A** (inside the unzipped directory): ```bash terraform init terraform apply \ -var "azure_subscription_id=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" \ -var "location=westus" \ -var "tls_cert_uri=https://keyvault-xx.vault.azure.net/secrets/cert-xx" \ -var "tls_cert_key_vault_name=keyvault-xx" \ -var "tls_cert_rg_name=DefaultResourceGroup-XXX" \ -var "artifactory_user=my-artifactory-user" \ -var "artifactory_token=my-artifactory-token" \ -var "product_name=mpc-infra" \ -var "sodot_relay_api_key=ABEiM0RVZneImaq7zN3u8g==" \ -var "sodot_relay_url=my-relay.XXXXX.com" \ -var "vertex_admin_token=XXXXXXXXXXXXXXXXXX" \ -var "vertex_dns_address=my-vertex.XXXXX.com" \ -var "secure_key_release_encryption_hsm_name=my-hsm-name" \ -var "secure_key_release_encryption_hsm_rg_name=my-hsm-resource-group" ``` :::warning[Terraform state holds secrets] The Terraform state contains your registry credentials, the admin token and the Relay API key. Keep it in an encrypted backend with restricted access, and keep a backup. ::: #### Chart and Image Versions * `vertex_chart_version` (default `1.18.5`) - the exact version of the `vertex-azure-aci` chart, used both to generate the policy and to deploy. Each chart pins the Vertex image and its SKR sidecar by digest, and both images are signed by Sodot. To upgrade the Vertex, raise this version; an apply never upgrades the chart by itself. * `vertex_image_tag` - leave unset to use the image the chart pins. Setting it (for example to roll back to a version you tested) replaces the chart's digest pin with that tag. #### Sizing and Availability * `vertex_min_replicas` and `vertex_max_replicas` (both default `2`) - the number of Vertex pods. Each pod requests 1 vCPU and 2 GiB, which sizes its container group. A PodDisruptionBudget keeps one pod serving while pods are evicted, and probes on `/health` keep a rollout from removing old pods before a new one has released its key and started. * `database_sku_name` (default `B_Standard_B1ms`) - the PostgreSQL server's SKU. Each Vertex keeps up to about 25 PostgreSQL connections under load, and a Vertex that can't get a connection at startup never starts serving. Size the server for `vertex_max_replicas` pods plus one extra pod during a rollout. `B_Standard_B1ms` allows about 50 connections (2 Vertex pods) and its CPU is burstable, so use a General Purpose SKU, such as `GP_Standard_D2ds_v5`, in production, and raise it before raising `vertex_max_replicas`. * `aks_node_vm_size` (default `Standard_D4ads_v5`) - the VM size of the AKS nodes. Each virtual node takes about 3 vCPUs and 12 GB on an AKS node, so use at least 4 vCPUs and 16 GiB. * `virtual_node_replicas` (default `2`) and `virtual_node_admission_controller_replicas` (default `2`) - the number of virtual nodes and of their admission controllers, each with a PodDisruptionBudget. Vertex pods are spread over the virtual nodes, and each virtual node serves up to 200 pods. Scaling down leaves the removed virtual nodes registered; delete them with `kubectl delete node `. #### Networking * Virtual nodes require [Azure CNI Node Subnet](https://learn.microsoft.com/en-us/azure/aks/concepts-network-azure-cni-pod-subnet), which new clusters use (`aks_network_plugin_mode`, default unset). Set `aks_network_plugin_mode = "overlay"` only for a cluster that already runs Azure CNI Overlay (see [Upgrading the Module](#upgrading-the-module)). * `azure_vnet_*_address_prefixes` - the address ranges of the virtual network and its subnets. The ACI subnet (`azure_vnet_cg_subnet_address_prefixes`) is delegated to ACI. * The Vertex's Kubernetes Service is `ClusterIP`. The Ingress is its only public entry. #### Database Safety The database holds the Vertex's key shares, so the module protects it: * Terraform refuses any plan that replaces or destroys the PostgreSQL server, the Vertex's database or the Vertex's managed identity. The Vertex logs in as that identity: its name is the PostgreSQL role that owns the Vertex's tables, and password authentication is off. See [Destroying a Deployment](#destroying-a-deployment). * The server has private access only, from the virtual network, and no firewall rules. * `database_backup_retention_days` (default `7`, from `7` to `35`) - days of automated backups. You can restore the server to any point in that window; a restore creates a new server. * `database_storage_auto_grow_enabled` (default `true`) - grow the storage (online) before it fills up. A full disk makes the server read-only. #### Kubernetes Upgrades * `kubernetes_version` - when unset, AKS picks the version of a new cluster and an existing cluster keeps its own; AKS applies patch versions automatically. Setting it upgrades the cluster on apply. * `virtual_node_chart_version` (default `1.3410.26082101`) - the [virtual nodes](https://github.com/microsoft/virtualnodesOnAzureContainerInstances) chart. Its 1.34 builds support AKS 1.33 to 1.35. Upgrade it before you upgrade AKS, and only to a version the [compatibility matrix](https://github.com/microsoft/virtualnodesOnAzureContainerInstances/blob/main/Docs/VersionCompatibility.md) lists for your AKS version. #### Observability Both are off by default. * **OpenTelemetry**: set `otel_endpoint` to the OTLP/HTTP endpoint of your collector or observability backend (for example `https://otlp.example.com:4318`), and `otel_headers` to its credentials. Each Vertex pod then runs an OpenTelemetry Collector sidecar that receives the Vertex's traces and metrics and exports them through the NAT gateway. `otel_signals` selects the signals (`traces` and `metrics` default to `true`, `logs` to `false`); enable `logs` to collect the Vertex's logs, because Container Insights doesn't see containers running on ACI. `otel_service_name` (default `sodot-vertex`), `otel_service_namespace` and `otel_environment` set the telemetry's resource attributes. * **Azure Monitor**: set `create_monitoring_workspaces = true` to create a Log Analytics workspace (retention `log_analytics_retention_days`, default `30`) and an Azure Monitor workspace, or pass existing ones with `log_analytics_workspace_id` and `azure_monitor_workspace_id` (both together). This enables Container Insights and managed Prometheus for the cluster, scrapes the virtual nodes' per-pod container metrics, and sends the AKS and PostgreSQL diagnostic logs to the workspace. `key_vault_audit_logs_enabled` also sends the Managed HSM and TLS Key Vault audit logs (key releases and policy changes); leave it off if those resources already send audit logs to the same workspace. `alert_email` enables Azure's recommended AKS alerts. :::warning[Secrets in the confidential computing policy] The secrets the Vertex and the collector receive (the admin token, the Relay API key, `server_share_rsa_private_decryption_key_pem` and `otel_headers`) are part of the confidential computing policy, so anyone who can read the Vertex pods can decode them from the pod's policy annotation. Restrict read access to the `vertex` namespace. ::: #### Upgrading the Module Back up the Terraform state and the database first, and keep the same resource `name`, state and Managed HSM key. Upgrade from module v2.0.0 to v3.0.0 with Vertex 1.18.5. 1. If your configuration sets `admin_access_token`, rename it to `vertex_admin_token`. Remove `unsafe_mode` and `vertex_basic_encryption_key` if you set them: the module no longer has them. 2. Set `vertex_chart_version = "1.18.5"` (the v3.0.0 default). Check your configuration, variable files and command-line overrides for `vertex_image_tag`: remove any existing override to use the chart's pinned 1.18.5 image. An old tag override keeps the old image selected even after changing the chart version. 3. Check the cluster's network mode: ```bash az aks show -g -n --query networkProfile.networkPluginMode ``` If it returns `"overlay"`, set `aks_network_plugin_mode = "overlay"`, or the apply tries to change the cluster's network mode. 4. Run `terraform init -upgrade`, then remove both old kubeconfig actions from the state **before the first plan or apply**: ```bash terraform state rm 'module.aks_cluster.azapi_resource_action.this_admin_kubeconfig[0]' terraform state rm 'module.aks_cluster.azapi_resource_action.this_user_kubeconfig[0]' ``` The v2.0.0 module stored their outputs as non-sensitive, so planning without this step can print cluster-admin credentials into your terminal or CI logs. Removing these state entries does not delete the cluster; apply recreates the actions with sensitive outputs. These addresses assume the downloaded module is your root module; if you use it as a submodule, prepend its module address. 5. Run `terraform plan`. The plan moves the existing resources to their new addresses without replacing them. It also: * removes the PostgreSQL firewall rule that allowed every IP address (the server only has private access); * turns on storage autogrow, and updates the AKS cluster and node pool in place with their current settings; * upgrades the virtual nodes chart to `virtual_node_chart_version`, and adds a second virtual node; 6. Run `terraform apply`. 7. Verify the upgrade before removing the previous release policy: * Confirm the deployed chart and Vertex image are 1.18.5. * Check that `https://URL_ENDPOINT/health` returns an empty `200 OK` response. * Authenticate to the Vertex using an API key created before the upgrade. * Sign and verify a signature using an existing MPC key created before the upgrade, through your application's SDK flow. This checks that existing access and key shares still work. The module reuses the existing HSM key, and adds the new release policy next to the previous one on the same key, so a failed or cancelled rollout doesn't revoke the running Vertex's access to its key. If the HSM key has more than one version, set `secure_key_release_encryption_key_version` to the version the existing database uses: the module doesn't guess. To re-add a release policy branch that was removed by hand, run `terraform apply -replace=terraform_data.key_release_policy`. Remove an old branch from the key's release policy only after the new version is verified and you no longer need to roll back. Never replace the HSM key to upgrade the Vertex: the database can only be read with it. #### Destroying a Deployment To destroy the deployment and keep its data, remove the database, the Vertex's identity and the resource group from the state first: ```bash terraform state rm terraform_data.data_guard module.database module.vertex_identity azurerm_resource_group.this terraform destroy ``` The resource group stays, with the database and the identity in it. To delete everything (test environments only), remove `terraform_data.data_guard` from the state, run `terraform destroy`, then delete the resource group. ### 6. Setting the DNS Record for `URL_ENDPOINT` to Point to the Vertex After running `terraform apply`, the `vertex_load_balancer_ip_address` output holds the public IP address of the Vertex's Ingress: ```bash terraform output vertex_load_balancer_ip_address ``` Create an A record for `URL_ENDPOINT` that points to that IP. Then, at `https://URL_ENDPOINT` you will be able to communicate with your Vertex. Now, you can run: ```bash curl https://URL_ENDPOINT/health -vvv ``` To verify that you get an empty `200 OK` response from the Vertex and that it is indeed up and running.