Version: v23.0

Join a DxEnterprise SQL AG StatefulSet to an Existing Cluster

When running in a StatefulSet, DxEnterprise has two cluster join modes: the default mode creates a brand new DxEnterprise cluster using only the pods, while the second mode allows the pods to join a preexisting DxEnterprise cluster running almost anywhere. Additionally, the pods may also join an existing Vhost and availability group, meaning that you can stretch your availability group across namespaces, Kubernetes clusters, off-site VMs, and more.

The available DxEnterprise cluster-join options covered in this guide are below. Only one type of deployment should be configured across each DxEnterprise cluster.

Join Within a Single Kubernetes Cluster - This option allows joining deployments together within the same Kubernetes cluster, but in different namespaces.
Join with NAT Matchmaking - This option allows joining deployments together between different Kubernetes clusters, on-premises and/or cloud deployments using the DH2i NAT Matchmaking service.
Join Across Kubernetes Clusters using Load Balancers with a Pod Selector - This option allows joining deployments together that are in different Kubernetes clusters using a per-pod load balancer selector.
Join Across Kubernetes Clusters using Load Balancers with a Cluster Selector - This option allows joining deployments together that are in different Kubernetes clusters using a per-cluster load balancer selector.

info

Because of the number of configurations possible with this feature, only a handful of them will be covered in this guide.

Join Within a Single Kubernetes Cluster

In this example, we will create a DxEnterprise SQL AG StatefulSet cluster, and join it to an existing DxEnterprise StatefulSet cluster running in a different namespace in the same Kubernetes cluster.

Additional Prerequisites

A deployed and running DxEnterprise StatefulSet cluster with a unique name. This StatefulSet can be running in any namespace. For more information about creating a new DxEnterprise StatefulSet cluster, see the Deploy SQL Server Availability Groups Quickstart Guide.
DxEnterprise container images must be version 23.0.262.0 or newer

Steps

Create a new namespace.
```
kubectl create namespace test
```
Create secrets for the DxEnterprise passkey and license, and the SQL Server SA account in the new namespace.

Passkeys must match
The DX_PASSKEY environment variable is used to authenticate the cross-cluster join operation, and must match the cluster passkey of the existing cluster.
```
kubectl create secret generic dxe -n test --from-literal=DX_PASSKEY=<pass> --from-literal=DX_LICENSE=<license_key>
```
```
kubectl create secret generic mssql -n test --from-literal=MSSQL_SA_PASSWORD=<pass>
```

Deploy the headless services. This allows your pods to connect to one another using hostnames.

headless.yaml
#Headless services for local connections/resolution
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-0
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-0
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-1
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-1
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022

Use the example StatefulSet YAML below, which has been modified from the Deploy SQL Server Availability Groups in Kubernetes guide. Note that the following values should be changed:

The DX_NEW_CLUSTER environment variable is set to "false" (note the quotes). This indicates that DxEnterprise should not be forming a new cluster, but should instead join an existing one.
The DX_JOIN_TARGET environment variable needs to point to a pod in the target cluster. The <target_name> and <target_namespace> should be the name of the headless service for a pod in the target cluster and the namespace the service/pod is running in, respectively. For example, a target pod service dxemssql-0 and default namespace would be dxemssql-0.default.svc.cluster.local.

headless service name
The name of the headless service for a DxEnterprise StatefulSet pod is identical to the pod name.
The new pods need to have the service sub-domain of the existing set of pods added to their DNS searches list. For example, per the previous step, this might be default.svc.cluster.local. This allows them to resolve their peers in the other namespace when starting up. Likewise, the DNS searches list for the existing pods should be defined to include the namespace of the pods being added.
By default, DxEnterprise will join the new pods to a Vhost named VHOST1, and an AG named AG1. If the existing cluster uses different names for the Vhost or the AG, the environment variables DX_AG_NAME and/or DX_VHOST_NAME will need to be set to match the names of the existing AG and Vhost (respectively) if you want these new pods to join the existing AG.
The name of the StatefulSet should be unique to the Kubernetes cluster since the pod name is only based off the StatefulSet name, not the namespace.

dxemssql.yaml
#DxEnterprise + MSSQL StatefulSet 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dxemssql
spec:
  serviceName: "dxemssql"
  replicas: 2
  selector:
    matchLabels:
      app: dxemssql
  template:
    metadata:
      labels:
        app: dxemssql
    spec:
      securityContext:
        fsGroup: 10001
      dnsConfig:
        searches: [ "default.svc.cluster.local" ]
      containers:
      - name: sql
        image: mcr.microsoft.com/mssql/server:2022-latest
        env:
        - name: ACCEPT_EULA
          value: "Y"
        - name: MSSQL_ENABLE_HADR
          value: "1"
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        volumeMounts:
        - name: mssql
          mountPath: "/var/opt/mssql"
      - name: dxe
        image: docker.io/dh2i/dxe:latest
        env:
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        - name: DX_ACCEPT_EULA
          value: "Y"
        - name: DX_NEW_CLUSTER
          value: "false"
        - name: DX_JOIN_TARGET
          value: "dxemssql-0.default.svc.cluster.local"
        - name: DX_VHOST_NAME
          value: "VHOST1"
        - name: DX_AG_NAME
          value: "AG1"
        envFrom:
        - secretRef:
            name: dxe
        volumeMounts:
        - name: dxe
          mountPath: "/etc/dh2i"
  volumeClaimTemplates:
  - metadata:
      name: dxe
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: mssql
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

Verify the pods are created.
```
kubectl get pods -n test
```
(optional) After giving the cluster a little while to setup, verify the configuration is correct by running dxcli get-ags-detail.
```
kubectl exec -itn test -c dxe dxemssql-0 -- dxcli get-ags-detail <vhost_name> <ag_name>
```

Join with NAT Matchmaking

In this example, we will create a DxEnterprise SQL AG StatefulSet cluster, and join it to an existing VM cluster running outside of Kubernetes, using the NAT join feature of DxEnterprise.

Additional Prerequisites

Any pre-existing DxEnterprise cluster
- If the pre-existing cluster is not in Kubernetes, ensure that the membername.lookup setting is enabled -- see set-globalsetting
DxEnterprise container images used for this StatefulSet deployment must be version 23.0.262.0 or newer
setting up a cluster
The cloud NAT matchmaker provides a mechanism for peers (DxEnterprise cluster nodes) to find each other, and is used to join nodes together and establish connections. This is helpful in situations where it might be very difficult for the nodes to find each other, such as when they are on different networks. More information about setting up a cluster can be found below:
- VMs and bare-metal: See the Linux or Windows quick start guides.
  - If desired, setup an AG in this cluster using the DxCLI or DxAdmin guides.
- Kubernetes: See the DxOperator quick start guide or the Dxemssql StatefulSet guide.
- Docker: Dee the Docker quick start guide.
A valid DxEnterprise license with NAT and tunnels enabled. A fully featured Developer Edition is available free for non-production use. To purchase DxEnterprise software for production workloads, visit the DH2i Store.
Permissive NATs for at least one - but preferably both - clusters. You can test your NAT type by visiting the DH2i NAT test page.

Steps

If not done already, set a cluster passkey on the existing (VM/bare-metal/Kubernetes/Docker) DxEnterprise cluster.
```
dxcli cluster-set-secret-ex <pass>
```
Set a One-Time Passkey on the existing cluster and copy the OTPK.
```
dxcli set-otpk
```
Create secrets for the DxEnterprise passkey, license, and OTPK, and the SQL Server SA account.

Passkeys must match
The DX_PASSKEY and DX_OTPK environment variables below must match the cluster passkey and OTPK of the existing cluster (steps #1 and #2) so the new pods can join the existing cluster.
```
kubectl create secret generic dxe --from-literal=DX_PASSKEY=<pass> --from-literal=DX_LICENSE=<license_key> --from-literal=DX_OTPK=<otpk>
```
```
kubectl create secret generic mssql --from-literal=MSSQL_SA_PASSWORD=<pass>
```

Deploy the headless services. This allows your pods to connect to one another using hostnames.

headless.yaml
#Headless services for local connections/resolution
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-0
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-0
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-1
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-1
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022

Use the example StatefulSet YAML below, which has been modified from the Deploy SQL Server Availability Groups in Kubernetes guide. Note that the following values should be changed:

The DX_NEW_CLUSTER environment variable is set to "false" (note the quotes). This indicates that DxEnterprise should not be forming a new cluster, but should instead join an existing one.
The DX_USE_NAT environment variable has been set to "true". This tells the DxEnterprise containers to use the NAT matchmaker to find the other cluster.
By default, DxEnterprise will join the new pods to a Vhost named VHOST1, and an AG named AG1. If the existing cluster uses different names for the Vhost or the AG, the environment variables DX_AG_NAME and/or DX_VHOST_NAME will need to be set to match the names of the existing AG and Vhost (respectively) if you want these new pods to join the existing AG.
The name of the StatefulSet should be unique to the Kubernetes cluster since the pod name is only based off the StatefulSet name, not the namespace.

dxemssql.yaml
#DxEnterprise + MSSQL StatefulSet 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dxemssql
spec:
  serviceName: "dxemssql"
  replicas: 2
  selector:
    matchLabels:
      app: dxemssql
  template:
    metadata:
      labels:
        app: dxemssql
    spec:
      securityContext:
        fsGroup: 10001
      containers:
      - name: sql
        image: mcr.microsoft.com/mssql/server:2022-latest
        env:
        - name: ACCEPT_EULA
          value: "Y"
        - name: MSSQL_ENABLE_HADR
          value: "1"
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        volumeMounts:
        - name: mssql
          mountPath: "/var/opt/mssql"
      - name: dxe
        image: docker.io/dh2i/dxe:latest
        env:
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        - name: DX_ACCEPT_EULA
          value: "Y"
        - name: DX_NEW_CLUSTER
          value: "false"
        - name: DX_USE_NAT
          value: "true"
        - name: DX_VHOST_NAME
          value: "VHOST1"
        - name: DX_AG_NAME
          value: "AG1"
        envFrom:
        - secretRef:
            name: dxe
        volumeMounts:
        - name: dxe
          mountPath: "/etc/dh2i"
  volumeClaimTemplates:
  - metadata:
      name: dxe
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: mssql
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

Verify the pods are created.
```
kubectl get pods
```
(optional) After giving the cluster a little while to setup, verify that the pods have joined the cluster by running dxcli get-cluster-nodes
```
kubectl exec -itc dxe dxemssql-0 -- dxcli get-cluster-nodes
```

(optional) Verify the configuration is correct by running dxcli get-ags-detail.

kubectl exec -itc dxe dxemssql-0 -- dxcli get-ags-detail <vhost_name> <ag_name>

Join Across Kubernetes Clusters using Load Balancers with a Pod Selector

In this example, we will create DxEnterprise SQL AG StatefulSets in two different Kubernetes clusters, which we will call the A-side and the B-side. We will create load balancers for each pod, and use them to form a single DxEnterprise cluster between the StatefulSets.

Additional Prerequisites

Two Kubernetes clusters
DxEnterprise container images must be version 23.0.262.0 or newer

Steps

On the A-side Kubernetes cluster, create secrets for the DxEnterprise passkey and license, and the SQL Server SA account.

kubectl create secret generic dxe --from-literal=DX_PASSKEY=<pass> --from-literal=DX_LICENSE=<license_key>

kubectl create secret generic mssql --from-literal=MSSQL_SA_PASSWORD=<pass>

Deploy the headless services on the A-side. This allows your pods to connect to one another using hostnames.

headless-a.yaml
#Headless services for local connections/resolution
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-0
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-a-0
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-1
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-a-1
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022

For each pod to be created on the A-side, create a load balancer with a static IP and a selector for the pod. These load balancers let the DxEnterprise cluster nodes (pods) find each other across the network. An example configuration with the necessary ports is given below.

Kubernetes load balancer implementation

Kubernetes load balancers are implemented by outside software stacks and vendors, so the details of assigning a load balancer a static IP will vary. Cloud providers like Azure, AWS, and Google Cloud have documentation on how to assign static IPs to load balancers within their own infrastructure.

warning

Ports mapped to a load balancer within Kubernetes will be accessible to the external network, which in some configurations may be a public network.

lbs-a.yaml
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-0-clb
  # Static IPs are often assigned through annotations or the spec.loadBalancerIP. 
  # The annotation below are given for example only from Azure and MetalLB documentation.
  #annotations:
  #  service.beta.kubernetes.io/azure-load-balancer-resource-group: <node resource group name>
  #  service.beta.kubernetes.io/azure-pip-name: myAKSIP1
  #  metallb.universe.tf/loadBalancerIPs: <ip_address>
spec:
  ports:
  - port: 7980
    targetPort: 7980
    protocol: TCP
    name: join-tcp
  - port: 7981
    targetPort: 7981
    protocol: UDP
    name: group-udp
  # Optional external access to SQL Server and DxAdmin
  - port: 1433
    targetPort: 1433
    protocol: TCP
    name: mssql
  - port: 7979
    targetPort: 7979
    protocol: TCP
    name: dxadmin
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-a-0
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-1-clb
  #annotations:
  #  service.beta.kubernetes.io/azure-load-balancer-resource-group: <node resource group name>
  #  service.beta.kubernetes.io/azure-pip-name: myAKSIP2
  #  metallb.universe.tf/loadBalancerIPs: <ip_address>
spec:
  ports:
  - port: 7980
    targetPort: 7980
    protocol: TCP
    name: join-tcp
  - port: 7981
    targetPort: 7981
    protocol: UDP
    name: group-udp
  # Optional external access to SQL Server and DxAdmin
  - port: 1433
    targetPort: 1433
    protocol: TCP
    name: mssql
  - port: 7979
    targetPort: 7979
    protocol: TCP
    name: dxadmin
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-a-1
  type: LoadBalancer

Create Services on the A-side to map the names of B-side pods to their external IPs. These must be set to the static IPs allocated to the external load balancers in the B-side cluster below

hostmap-a.yaml
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-0
spec:
  clusterIP: None
  ports:
  - name: dxcmonitor-tcp
    protocol: TCP
    port: 7980
  - name: dxcmonitor-udp
    protocol: UDP
    port: 7981
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: dxemssql-b-0-ip
  labels:
    kubernetes.io/service-name: dxemssql-b-0
addressType: IPv4
ports:
- name: dxcmonitor-tcp
  port: 7980
- name: dxcmonitor-udp
  port: 7981
  protocol: UDP
endpoints:
- addresses: [ <dxemssql-b-0_lb_ip> ]
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-1
spec:
  clusterIP: None
  ports:
  - name: dxcmonitor-tcp
    protocol: TCP
    port: 7980
  - name: dxcmonitor-udp
    protocol: UDP
    port: 7981
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: dxemssql-b-1-ip
  labels:
    kubernetes.io/service-name: dxemssql-b-1
addressType: IPv4
ports:
- name: dxcmonitor-tcp
  port: 7980
- name: dxcmonitor-udp
  port: 7981
  protocol: UDP
endpoints:
- addresses: [ <dxemssql-b-1_lb_ip> ]

Deploy the StatefulSet on the A-side using the example YAML below, which has been modified from the Deploy SQL Server Availability Groups in Kubernetes guide. Note that the following values should be changed:

The name of the StatefulSet must be different from that chosen for the B-side. Here, we name it dxemssql-a.
If you wish to customize the name of the Vhost and AG, you may do so here, using the environment variables DX_AG_NAME and DX_VHOST_NAME.
The DX_NEW_CLUSTER and DX_JOIN_TARGET environment variables, which are set in other examples, are not set here. This indicates that DxEnterprise should form a new cluster, as this groups of pods are meant to be.

dxemssql-a.yaml
#DxEnterprise + MSSQL StatefulSet 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dxemssql-a
spec:
  serviceName: "dxemssql-a"
  replicas: 2
  selector:
    matchLabels:
      app: dxemssql-a
  template:
    metadata:
      labels:
        app: dxemssql-a
    spec:
      securityContext:
        fsGroup: 10001
      containers:
      - name: sql
        image: mcr.microsoft.com/mssql/server:2022-latest
        env:
        - name: ACCEPT_EULA
          value: "Y"
        - name: MSSQL_ENABLE_HADR
          value: "1"
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        volumeMounts:
        - name: mssql
          mountPath: "/var/opt/mssql"
      - name: dxe
        image: docker.io/dh2i/dxe:latest
        env:
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        - name: DX_ACCEPT_EULA
          value: "Y"
        - name: DX_VHOST_NAME
          value: "VHOST1"
        - name: DX_AG_NAME
          value: "AG1"
        envFrom:
        - secretRef:
            name: dxe
        volumeMounts:
        - name: dxe
          mountPath: "/etc/dh2i"
  volumeClaimTemplates:
  - metadata:
      name: dxe
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: mssql
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

Switch to the B-side Kubernetes cluster and repeat step #1.

Passkeys must match
The DX_PASSKEY environment variable is used to authenticate the cross-cluster join operation, and must match the cluster passkey of the existing cluster.

Deploy the headless services on the B-side. This allows your pods to connect to one another using hostnames.

headless-b.yaml
#Headless services for local connections/resolution
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-0
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-b-0
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-1
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-b-1
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022

For each pod in the B-side Kubernetes cluster, create a load balancer with a static IP. Example load balancers are given below.

warning

Ports mapped to a load balancer within Kubernetes will be accessible to the external network, which in some configurations may be a public network.

lbs-b.yaml
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-0-clb
  # Static IPs are often assigned through annotations or the spec.loadBalancerIP. 
  # The annotation below are given for example only from Azure and MetalLB documentation.
  #annotations:
  #  service.beta.kubernetes.io/azure-load-balancer-resource-group: <node resource group name>
  #  service.beta.kubernetes.io/azure-pip-name: myAKSIP1
  #  metallb.universe.tf/loadBalancerIPs: <ip_address>
spec:
  ports:
  - port: 7980
    targetPort: 7980
    protocol: TCP
    name: join-tcp
  - port: 7981
    targetPort: 7981
    protocol: UDP
    name: group-udp
  # Optional external access to SQL Server and DxAdmin
  - port: 1433
    targetPort: 1433
    protocol: TCP
    name: mssql
  - port: 7979
    targetPort: 7979
    protocol: TCP
    name: dxadmin
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-b-0
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-1-clb
  #annotations:
  #  service.beta.kubernetes.io/azure-load-balancer-resource-group: <node resource group name>
  #  service.beta.kubernetes.io/azure-pip-name: myAKSIP2
  #  metallb.universe.tf/loadBalancerIPs: <ip_address>
spec:
  ports:
  - port: 7980
    targetPort: 7980
    protocol: TCP
    name: join-tcp
  - port: 7981
    targetPort: 7981
    protocol: UDP
    name: group-udp
  # Optional external access to SQL Server and DxAdmin
  - port: 1433
    targetPort: 1433
    protocol: TCP
    name: mssql
  - port: 7979
    targetPort: 7979
    protocol: TCP
    name: dxadmin
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-b-1
  type: LoadBalancer

Create Services on the B-side to map the names of A-side pods to their external IPs. These must be set to the static IPs allocated to the external load balancers in the A-side cluster above

hostmap-b.yaml
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-0
spec:
  clusterIP: None
  ports:
  - name: dxcmonitor-tcp
    protocol: TCP
    port: 7980
  - name: dxcmonitor-udp
    protocol: UDP
    port: 7981
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: dxemssql-a-0-ip
  labels:
    kubernetes.io/service-name: dxemssql-a-0
addressType: IPv4
ports:
- name: dxcmonitor-tcp
  port: 7980
- name: dxcmonitor-udp
  port: 7981
  protocol: UDP
endpoints:
- addresses: [ <dxemssql-a-0_lb_ip> ]
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-1
spec:
  clusterIP: None
  ports:
  - name: dxcmonitor-tcp
    protocol: TCP
    port: 7980
  - name: dxcmonitor-udp
    protocol: UDP
    port: 7981
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: dxemssql-a-1-ip
  labels:
    kubernetes.io/service-name: dxemssql-a-1
addressType: IPv4
ports:
- name: dxcmonitor-tcp
  port: 7980
- name: dxcmonitor-udp
  port: 7981
  protocol: UDP
endpoints:
- addresses: [ <dxemssql-a-1_lb_ip> ]

Use the example StatefulSet YAML below to deploy the pods in the B-side. This example has been modified from the Deploy SQL Server Availability Groups in Kubernetes guide. Note that the following values should be changed:

The name of the StatefulSet must be different from that chosen for the A-side. Here, we name it dxemssql-b.
The DX_NEW_CLUSTER environment variable is set to "false" (note the quotes). This indicates that DxEnterprise should not be forming a new cluster, but should instead join an existing one.
The DX_JOIN_TARGET environment variable is set to the name of a pod on the A-side, here dxemssql-a-0.
If you chose to customize the name of the Vhost or AG in step 4 above on the A-side, you must set the DX_AG_NAME and DX_VHOST_NAME to match on the B-side.

dxemssql-b.yaml
#DxEnterprise + MSSQL StatefulSet 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dxemssql-b
spec:
  serviceName: "dxemssql-b"
  replicas: 2
  selector:
    matchLabels:
      app: dxemssql-b
  template:
    metadata:
      labels:
        app: dxemssql-b
    spec:
      securityContext:
        fsGroup: 10001
      containers:
      - name: sql
        image: mcr.microsoft.com/mssql/server:2022-latest
        env:
        - name: ACCEPT_EULA
          value: "Y"
        - name: MSSQL_ENABLE_HADR
          value: "1"
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        volumeMounts:
        - name: mssql
          mountPath: "/var/opt/mssql"
      - name: dxe
        image: docker.io/dh2i/dxe:latest
        env:
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        - name: DX_ACCEPT_EULA
          value: "Y"
        - name: DX_NEW_CLUSTER
          value: "false"
        - name: DX_JOIN_TARGET
          value: "dxemssql-a-0"
        - name: DX_VHOST_NAME
          value: "VHOST1"
        - name: DX_AG_NAME
          value: "AG1"
        envFrom:
        - secretRef:
            name: dxe
        volumeMounts:
        - name: dxe
          mountPath: "/etc/dh2i"
  volumeClaimTemplates:
  - metadata:
      name: dxe
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: mssql
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

(optional) After giving the cluster a little while to setup, verify that the pods have joined the cluster by running dxcli get-cluster-nodes
```
kubectl exec -itc dxe dxemssql-a-0 -- dxcli get-cluster-nodes
```

(optional) Verify the configuration is correct by running dxcli get-ags-detail.

kubectl exec -itc dxe dxemssql-a-0 -- dxcli get-ags-detail <vhost_name> <ag_name>

Join Across Kubernetes Clusters using Load Balancers with a Cluster Selector

In this example, we will create DxEnterprise SQL AG StatefulSets in two different Kubernetes clusters, which we will call the A-side and the B-side. We will create one load balancer per StatefulSet, and use them to form a single DxEnterprise cluster between the StatefulSets.

Additional Prerequisites

Two Kubernetes clusters
DxEnterprise container images must be version 23.0.262.0 or newer
Kubernetes clusters must have permissive NAT

Steps

On the A-side Kubernetes cluster, create secrets for the DxEnterprise passkey and license, and the SQL Server SA account.

kubectl create secret generic dxe --from-literal=DX_PASSKEY=<pass> --from-literal=DX_LICENSE=<license_key>

kubectl create secret generic mssql --from-literal=MSSQL_SA_PASSWORD=<pass>

Deploy the headless services on the A-side. This allows your pods to connect to one another using hostnames.

headless-a.yaml
#Headless services for local connections/resolution
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-0
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-a-0
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-1
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-a-1
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022

Create a load balancer with a static IP, an externalTrafficPolicy set to local, and a selector for the entire StatefulSet. The load balancer lets the DxEnterprise cluster nodes (pods) find each other across the network. An example configuration with the necessary ports is given below.

Kubernetes load balancer implementation

lb-a.yaml
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-clb
  # Static IPs are often assigned through annotations or the spec.loadBalancerIP. 
  # The annotation below are given for example only from Azure and MetalLB documentation.
  #annotations:
  #  service.beta.kubernetes.io/azure-load-balancer-resource-group: <node resource group name>
  #  service.beta.kubernetes.io/azure-pip-name: myAKSIP1
  #  metallb.universe.tf/loadBalancerIPs: <ip_address>
spec:
  ports:
  - port: 7980
    targetPort: 7980
    protocol: TCP
    name: join-tcp
  - port: 7981
    targetPort: 7981
    protocol: UDP
    name: group-udp
  selector:
    app: dxemssql-a
  type: LoadBalancer
  externalTrafficPolicy: Local

Create Services on the A-side to map the names of B-side pods to their external IPs. These must be set to the static IP address allocated to the external load balancer in the B-side cluster below

hostmap-a.yaml
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-0
spec:
  clusterIP: None
  ports:
  - name: dxcmonitor-tcp
    protocol: TCP
    port: 7980
  - name: dxcmonitor-udp
    protocol: UDP
    port: 7981
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: dxemssql-b-0-ip
  labels:
    kubernetes.io/service-name: dxemssql-b-0
addressType: IPv4
ports:
- name: dxcmonitor-tcp
  port: 7980
- name: dxcmonitor-udp
  port: 7981
  protocol: UDP
endpoints:
- addresses: [ <dxemssql-b-lb_ip> ]
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-1
spec:
  clusterIP: None
  ports:
  - name: dxcmonitor-tcp
    protocol: TCP
    port: 7980
  - name: dxcmonitor-udp
    protocol: UDP
    port: 7981
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: dxemssql-b-1-ip
  labels:
    kubernetes.io/service-name: dxemssql-b-1
addressType: IPv4
ports:
- name: dxcmonitor-tcp
  port: 7980
- name: dxcmonitor-udp
  port: 7981
  protocol: UDP
endpoints:
- addresses: [ <dxemssql-b-lb_ip> ]

Deploy the StatefulSet on the A-side Kubernetes cluster using the example YAML below, which has been modified from the Deploy SQL Server Availability Groups in Kubernetes guide. Note that the following values should be changed:

The name of the StatefulSet must be different from that chosen for the B-side. Here, we name it dxemssql-a.
If you wish to customize the name of the Vhost and AG, you may do so here, using the environment variables DX_AG_NAME and DX_VHOST_NAME.
The DX_NEW_CLUSTER and DX_JOIN_TARGET environment variables, which are set in other examples, are not set here. This indicates that DxEnterprise should form a new cluster, as this groups of pods are meant to be.

dxemssql-a.yaml
#DxEnterprise + MSSQL StatefulSet 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dxemssql-a
spec:
  serviceName: "dxemssql-a"
  replicas: 2
  selector:
    matchLabels:
      app: dxemssql-a
  template:
    metadata:
      labels:
        app: dxemssql-a
    spec:
      securityContext:
        fsGroup: 10001
      containers:
      - name: sql
        image: mcr.microsoft.com/mssql/server:2022-latest
        env:
        - name: ACCEPT_EULA
          value: "Y"
        - name: MSSQL_ENABLE_HADR
          value: "1"
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        volumeMounts:
        - name: mssql
          mountPath: "/var/opt/mssql"
      - name: dxe
        image: docker.io/dh2i/dxe:latest
        env:
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        - name: DX_ACCEPT_EULA
          value: "Y"
        - name: DX_VHOST_NAME
          value: "VHOST1"
        - name: DX_AG_NAME
          value: "AG1"
        envFrom:
        - secretRef:
            name: dxe
        volumeMounts:
        - name: dxe
          mountPath: "/etc/dh2i"
  volumeClaimTemplates:
  - metadata:
      name: dxe
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: mssql
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

Switch to the B-side Kubernetes cluster and repeat step #1.

Passkeys must match
The DX_PASSKEY environment variable is used to authenticate the cross-cluster join operation, and must match the cluster passkey of the existing cluster.

Deploy the headless services on the B-side. This allows your pods to connect to one another using hostnames.

headless-b.yaml
#Headless services for local connections/resolution
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-0
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-b-0
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-1
spec:
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: dxemssql-b-1
  ports:
  - name: dxl
    protocol: TCP
    port: 7979
  - name: dxc-tcp
    protocol: TCP
    port: 7980
  - name: dxc-udp
    protocol: UDP
    port: 7981
  - name: sql
    protocol: TCP
    port: 1433
  - name: ag-endpoint
    protocol: TCP
    port: 5022

Create a load balancer with a static IP, an externalTrafficPolicy set to local, and a selector for the entire DxEnterprise StatefulSet cluster. Example load balancers are given below.

lb-b.yaml
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-b-clb
  #annotations:
  #  service.beta.kubernetes.io/azure-load-balancer-resource-group: <node resource group name>
  #  service.beta.kubernetes.io/azure-pip-name: myAKSIP3
  #  metallb.universe.tf/loadBalancerIPs: <ip_address>
spec:
  ports:
  - port: 7980
    targetPort: 7980
    protocol: TCP
    name: join-tcp
  - port: 7981
    targetPort: 7981
    protocol: UDP
    name: group-udp
  selector:
    app: dxemssql-b
  type: LoadBalancer
  externalTrafficPolicy: Local

Create Services on the B-side to map the names of A-side pods to their external IPs. These must be set to the static IP address allocated to the external load balancer in the A-side cluster above

hostmap-b.yaml
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-0
spec:
  clusterIP: None
  ports:
  - name: dxcmonitor-tcp
    protocol: TCP
    port: 7980
  - name: dxcmonitor-udp
    protocol: UDP
    port: 7981
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: dxemssql-a-0-ip
  labels:
    kubernetes.io/service-name: dxemssql-a-0
addressType: IPv4
ports:
- name: dxcmonitor-tcp
  port: 7980
- name: dxcmonitor-udp
  port: 7981
  protocol: UDP
endpoints:
- addresses: [ <dxemssql-a-lb_ip> ]
---
apiVersion: v1
kind: Service
metadata:
  name: dxemssql-a-1
spec:
  clusterIP: None
  ports:
  - name: dxcmonitor-tcp
    protocol: TCP
    port: 7980
  - name: dxcmonitor-udp
    protocol: UDP
    port: 7981
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: dxemssql-a-1-ip
  labels:
    kubernetes.io/service-name: dxemssql-a-1
addressType: IPv4
ports:
- name: dxcmonitor-tcp
  port: 7980
- name: dxcmonitor-udp
  port: 7981
  protocol: UDP
endpoints:
- addresses: [ <dxemssql-a-lb_ip> ]

Use the example StatefulSet YAML below to deploy the pods in the B-side Kubernetes cluster. This example has been modified from the Deploy SQL Server Availability Groups in Kubernetes guide. Note that the following values should be changed:

The name of the StatefulSet must be different from that chosen for the A-side. Here, we name it dxemssql-b.
The DX_NEW_CLUSTER environment variable is set to "false" (note the quotes). This indicates that DxEnterprise should not be forming a new cluster, but should instead join an existing one.
The DX_JOIN_TARGET environment variable is set to the name of a pod on the A-side, here dxemssql-a-0.
If you chose to customize the name of the Vhost or AG on the A-side in step 4 above, you must set the DX_AG_NAME and DX_VHOST_NAME on the B-side to match.

dxemssql-b.yaml
#DxEnterprise + MSSQL StatefulSet 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dxemssql-b
spec:
  serviceName: "dxemssql-b"
  replicas: 2
  selector:
    matchLabels:
      app: dxemssql-b
  template:
    metadata:
      labels:
        app: dxemssql-b
    spec:
      securityContext:
        fsGroup: 10001
      containers:
      - name: sql
        image: mcr.microsoft.com/mssql/server:2022-latest
        env:
        - name: ACCEPT_EULA
          value: "Y"
        - name: MSSQL_ENABLE_HADR
          value: "1"
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        volumeMounts:
        - name: mssql
          mountPath: "/var/opt/mssql"
      - name: dxe
        image: docker.io/dh2i/dxe:latest
        env:
        - name: MSSQL_SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: MSSQL_SA_PASSWORD
        - name: DX_ACCEPT_EULA
          value: "Y"
        - name: DX_NEW_CLUSTER
          value: "false"
        - name: DX_JOIN_TARGET
          value: "dxemssql-a-0"
        - name: DX_VHOST_NAME
          value: "VHOST1"
        - name: DX_AG_NAME
          value: "AG1"
        envFrom:
        - secretRef:
            name: dxe
        volumeMounts:
        - name: dxe
          mountPath: "/etc/dh2i"
  volumeClaimTemplates:
  - metadata:
      name: dxe
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: mssql
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

(optional) After giving the cluster a little while to setup, verify that the pods have joined the cluster by running dxcli get-cluster-nodes
```
kubectl exec -itc dxe dxemssql-a-0 -- dxcli get-cluster-nodes
```

(optional) Verify the configuration is correct by running dxcli get-ags-detail.

kubectl exec -itc dxe dxemssql-a-0 -- dxcli get-ags-detail <vhost_name> <ag_name>

Join a DxEnterprise SQL AG StatefulSet to an Existing Cluster

Join Within a Single Kubernetes Cluster​

Additional Prerequisites​

Steps​

Join with NAT Matchmaking​

Additional Prerequisites​

Steps​

Join Across Kubernetes Clusters using Load Balancers with a Pod Selector​

Additional Prerequisites​

Steps​

Join Across Kubernetes Clusters using Load Balancers with a Cluster Selector​

Additional Prerequisites​

Steps​

Join Within a Single Kubernetes Cluster

Additional Prerequisites

Steps

Join with NAT Matchmaking

Additional Prerequisites

Steps

Join Across Kubernetes Clusters using Load Balancers with a Pod Selector

Additional Prerequisites

Steps

Join Across Kubernetes Clusters using Load Balancers with a Cluster Selector

Additional Prerequisites

Steps