IT 3300 : Proxmox

Proxmox High Availability

What is high availability?

Usually expressed as a percentage of uptime in a given year. See table 18 here

How to achieve availability

The most elegant solution is to rewrite your software, so that you can run it on several hosts at the same time. The software itself needs to have a way to detect errors and do failover. If you only want to serve read-only web pages, then this is relatively simple. However, this is generally complex and sometimes impossible, because you cannot modify the software yourself.

How to achieve availability (Without modifying the software)

• use reliable server componesnts
• eliminate SPOF
• reduce downtime
  • other cluster nodes
  • automatic error detection (provided by ha-manager)
  • automatic failover (provided by ha-manager)

ha-manager

Proxmox VE ha-manager works like an "automated" administrator. First, you configure what resources (VMs, containers, …) it should manage. Then, ha-manager observes the correct functionality, and handles service failover to another node in case of errors. ha-manager can also handle normal user requests which may start, stop, relocate and migrate a service. ha-manager has typical error detection and failover times of about 2 minutes, so you can get no more than 99.999% availability.

Proxmox HA Requirements

• here
• at least 3 cluster nodes (to get a reliable quorum)
• shared storage for vms
• hardware redundancy

Service states

The cluster resource manager maintains the current service state. You can view it with the ha-manager status command. You can see a list of service states in section 15.4.1

Proxmox HA Resources Tidbits

• max_relocate: how many times we will try to relocate a vm when service fails to start (default 1)
  • This only happens when max_restart value is exceeded on the actual node.
• max_restart: how many times we will try to restart a vm on a node when it failed to start (default 1)
• states:
  • started: CRM tries to start the resource
  • stopped: CRM tries to keep resource in stopped state, but it still tries to relocate on node failures
  • disabled: CRM puts in stopped state, does not relocate
  • ignored: remove from HA

Proxmox HA Groups

• Define groups of cluster nodes
• Then we can ensure a vm is always present on one of those nodes.
• Can also create a group of one if you want to make sure a resource is always on that node.

Proxmox HA Group options

• nofailback: crm tries to run services on the node with the highest priority. If a node with higher priority comes online, the CRM migrates the service to that node. Enabling nofailback prevents that behavior.
• restricted: Resources bound to restricted groups may only run on nodes defined by the group. The resource will be placed in the stopped state if no group node member is online. Resources on unrestricted groups may run on any cluster node if all group members are offline, but they will migrate back as soon as a group member comes online. One can implement a preferred node behavior using an unrestricted group with only one member.

Proxmox HA Fencing

Fencing is a concept of isolating a node or its resources during node failure so that other nodes cannot access the same resources, putting them at risk of data corruption. In Proxmox, fencing prevents multiple nodes from running the same virtual machine or cluster-specific services. Fencing ensures data integrity during a node failure by preventing all nodes from running the same virtual machine or cluster services at the same time.

Fencing

Proxmox HA Howto

• First, put your nodes into an HA group
  • Setting the nofailback flag prevents the recovered services from moving straight back to the fenced node.
  • restricted will only allow services to run on a particular node.
  • The CRM tries to run services on the node with the highest priority. If a node with higher priority comes online, the CRM migrates the service to that node. Enabling nofailback prevents that behavior.
• Add HA resource to the group

Maintenance mode

Only can be done via command line:

• ha-manager crm-command node-maintenance enable NODENAME

• ha-manager crm-command node-maintenance disable NODENAME

What happens when a node shuts down

There are different policies that can be configured:

• migrate (move all vms to different node) (assuming they can be moved)
• Failover: all services get stopped, but that they will also be recovered, if the current node is not online soon. It can be useful when doing maintenance on a cluster scale, where live-migrating VMs may not be possible if too many nodes are powered off at a time, but you still want to ensure HA services get recovered and started again as soon as possible.
• Freeze: keep all services on same node, when it comes back online
• Conditional: ??
• You can also manually move resources over.