Skip to main content
Version: v23.0

Managing Your Cluster During Planned Maintenance Events

Summary​

This article describes how to manage DxEnterprise's failover and monitoring behavior during planned maintenance using three mechanisms:

Each section includes the purpose, recommended use cases, and step-by-step instructions for using both DxCLI and DxAdmin.

Information​

Freezing​

Freezing selectively disables monitoring and automatic failover on a Vhost, or a node. Health monitoring of applications and resources associated with the Vhost will be disabled, and no automatic failover actions will be taken on the Vhost. At the same time, the Vhost is still considered to be active on the node where it was active prior to having been frozen. Unfreezing a Vhost restarts health monitoring and automatic failover, with the assumption that the resources and applications are active and available on that node.

When DxEnterprise is shut down cleanly, it will normally shut down all Vhosts that are active on the node where it is running, allowing those Vhosts to be failed over to other remaining nodes of the cluster. In the case of frozen Vhosts, this shutdown behavior will be skipped - the frozen Vhosts will be left alone.

If a node is rebooted with a frozen Vhost, DxEnterprise on the other nodes of the cluster will not attempt automatic failover of the Vhost. When DxEnterprise restarts on the node after rebooting, it will leave the Vhost in the frozen state. As a side effect of rebooting, the state of the applications and associated Vhost resources such as virtual IP addresses and storage will be reset, and DxEnterprise will make no attempt to re-establish the Vhost resources or restart applications. Unfreezing after reboot will typically result in an immediate automatic failover of the Vhost.

Freezing Vhosts​

Freezing a Vhost can be helpful for application maintenance where:

  • An application running on the Vhost active node must be temporarily shut down or made unavailable
  • Automatic failover actions are not desired and could potentially be disruptive
  • Rebooting is not required as part of the maintenance

To freeze or unfreeze a Vhost:

  1. Freeze the Vhost:

    dxcli freeze-vhost VHOSTNAME

  2. Complete the maintenance task.

  3. Unfreeze the Vhost:

    dxcli unfreeze-vhost VHOSTNAME

Freezing Nodes​

Freezing a node causes all Vhosts that are active on that node to be frozen.

Freezing a node is recommended when upgrading or reinstalling DxEnterprise, and in many cases allows the upgrade to be performed without interrupting applications.

To freeze or unfreeze a node:

  1. Freeze the node:

    dxcli freeze-node NODENAME

  2. Complete the maintenance task.

  3. Unfreeze the node:

    dxcli unfreeze-node NODENAME

Cluster Maintenance Mode​

Cluster Maintenance Mode disables all automatic failover and shutdown behavior cluster-wide. DxEnterprise will continue to monitor application, node, and network health, and will continue to display alerts related to health issues as they are discovered, but no failover actions will be taken. When Cluster Maintenance Mode is enabled, all Vhost start, stop, and failover actions must be performed manually by the administrator.

If DxEnterprise is shut down on a particular node, or the node is rebooted with Cluster Maintenance Mode enabled, all Vhosts and applications that are active on the node will be stopped, and must be restarted manually.

Cluster Maintenance Mode disables all quorum and witness-related shutdown actions.

Cluster Maintenance Mode can be used to prevent DxEnterprise from performing automatic failover or shutdown of applications such as Microsoft SQL Server Availability Groups. However, SQL Server maintains its own quorum requirements for Availability Groups which will be in effect regardless of Cluster Maintenance Mode.

Cluster Maintenance Mode can be broadly helpful in maintenance situations, where:

  • Vhosts and applications should remain active despite apparent unavailability of nodes or quorum
  • Isolated, inactive nodes should not attempt to restart Vhosts or applications
  • Administrators prefer to manually restart Vhosts and applications as needed

To enable or disable Cluster Maintenance Mode:

  1. Enable Cluster Maintenance Mode:

    dxcli set-globalsetting cluster.maintenance_mode

  2. Complete the maintenance task.

  3. Disable Cluster Maintenance Mode:

    dxcli remove-globalsetting cluster.maintenance_mode

Administrative Disablement​

Administratively disabling a Vhost stops its applications and disables failover, providing a controlled shutdown. The Vhost will remain stopped until it is manually restarted.

It can be helpful to administratively disable a Vhost in situations where:

  • The active node of a Vhost must be rebooted
  • Failover to another node is not desired

To administratively disable and re-enable a Vhost:

  1. Disable the Vhost:

    dxcli vhost-stop-node VHOSTNAME "" disabled

  2. Complete the maintenance task.

  3. Re-enable the Vhost:

    dxcli vhost-start-node VHOSTNAME NODENAME

Additional Information​