Proxmox High Availability and Failover

Overview

In this project, you will investigate how high availability (HA) works in a Proxmox cluster.

You already have a three-node Proxmox cluster with shared storage provided by TrueNAS from the previous project. In this project, you will configure a virtual machine for HA, predict how your cluster will respond to failures, test those predictions, measure recovery time, and analyze the limitations of your HA architecture.

The goal is not simply to configure HA. The goal is to understand what happens when things fail and why.

Learning Objectives

By completing this project, you should be able to:

Configure a virtual machine for high availability in Proxmox.
Explain how Proxmox HA responds to a node failure.
Predict the behavior of a virtual machine during a failure.
Perform a controlled failover test.
Measure the recovery time of a virtual machine.
Analyze the role of quorum in a Proxmox cluster.
Identify potential single points of failure in a highly available infrastructure.
Explain the difference between high availability and fault tolerance.
Evaluate the strengths and limitations of your Proxmox HA configuration.

Part 1 — Configure High Availability

Select one of your existing pokemon virtual machines to use as your HA test VM.

The VM should contain a service that allows you to easily determine whether it is running. For example, you might use a web server.

Configure the VM so that Proxmox will automatically recover it if the physical node hosting it fails.

Requirements

Your VM must:

Be stored on the shared TrueNAS storage.
Be configured as an HA-managed resource.
Be running on one of your three Proxmox nodes.
Have a service that you can test before and after a failure.

Evidence

Provide a screenshot showing that your VM is configured for HA.

Also identify:

The name/ID of your VM
The Proxmox node currently running it
The storage containing the VM’s disk

Part 2 — Predict the Failure

Before performing any failure test, make a prediction.

Assume that the Proxmox node currently running your HA VM suddenly loses power.

Answer the following questions:

What do you expect Proxmox to do?
On which node do you expect the VM to restart?
Do you expect the VM’s IP address to change?
What do you expect to happen to the VM’s disk?
How long do you expect the service to be unavailable?
Why do you expect this behavior?

Do not perform the test until you have recorded your predictions.

The purpose of this section is to compare what you think will happen with what actually happens.

Part 3 — Controlled Node Failure

Now perform a controlled failure test.

Shut down the Proxmox node currently running your HA VM.

Observe what happens to the cluster and the VM.

Record the following information:

Event	Time
Node shutdown initiated
Node becomes unavailable
Proxmox detects the failure
VM begins recovery
VM becomes available

Calculate the approximate amount of downtime experienced by the service.

Answer the following questions

Did the VM automatically restart?
Which Proxmox node restarted the VM?
Did the VM retain its IP address?
Did the VM’s disk remain available?
How long was the service unavailable?
Did the actual behavior match your prediction?
If not, explain the difference.

Provide screenshots showing the cluster before and after the failure.

Part 4 — Unexpected Failure

A planned shutdown is not the same as a hardware failure.

Restore your three-node cluster to normal operation and place your HA VM on a Proxmox node.

Now simulate an unexpected failure by abruptly powering off the node running the VM. (This might be easier by putting your vm on the virtualized proxmox node)

Do not gracefully shut down the Proxmox node.

Observe the behavior of the cluster.

Answer the following questions

How did Proxmox respond to the unexpected failure?
How long did it take for Proxmox to recognize the failure?
How long did it take for the VM to become available again?
Was the recovery time different from the controlled shutdown?
What evidence did Proxmox provide that the node had failed?
What evidence did Proxmox provide that the VM had been recovered?

Document what you observed.

Part 5 — Investigate Quorum

Proxmox uses quorum to help prevent multiple parts of a cluster from making conflicting decisions.

Your cluster contains three nodes.

Research and answer the following questions:

A. What is quorum?

Explain quorum in your own words.

B. Why does Proxmox care about quorum?

Explain why a cluster should not simply allow every surviving node to make independent decisions when communication between nodes is lost.

C. Three-node cluster

If your three-node cluster loses one node, how many nodes remain?

Does the remaining cluster have quorum?

Explain.

D. Two-node failure

If your three-node cluster loses two nodes, what happens to the remaining node’s ability to manage HA resources?

Explain why.

E. Network failure

Consider this situation:

        Node 1
          |
          X
          |
        Node 2 ----- Node 3

Node 1 is still powered on and its VMs are still running, but Node 1 can no longer communicate with Nodes 2 and 3.

Why could this situation be dangerous?

Explain how quorum helps protect against this type of problem.

Part 6 — Is Your Infrastructure Really Highly Available?

Your infrastructure currently looks approximately like this:

             Proxmox Cluster
          /        |        \
       Node 1    Node 2    Node 3
          \        |        /
               TrueNAS
             Shared Storage

You have three Proxmox nodes.

Does that mean your infrastructure is completely highly available?

No configuration changes are required for this section.

Instead, analyze the architecture.

Answer the following questions:

What happens if Node 1 fails?
What happens if Node 2 fails?
What happens if Node 3 fails?
What happens if TrueNAS becomes unavailable?
What happens if the network connection between the Proxmox nodes and TrueNAS fails?
What components of your infrastructure represent potential single points of failure?
Is your current architecture highly available? Explain your answer.

Part 7 — High Availability vs. Fault Tolerance

Explain the difference between:

High Availability

and

Fault Tolerance

Consider your HA VM.

If the physical node fails and Proxmox restarts the VM on another node after two minutes, was the application:

Highly available?
Fault tolerant?
Both?
Neither?

Explain your answer.

Part 8 — Design a Better Architecture

Imagine that you are designing infrastructure for a company that requires its web application to remain available if any one physical Proxmox node fails.

Using what you learned in this project, design an improved infrastructure.

Create a diagram showing:

Proxmox nodes
Storage
Network connections
VMs
HA
Any additional redundancy you believe is necessary

Then explain:

What failures can your architecture tolerate?
What happens when a Proxmox node fails?
What happens when the storage system fails?
What happens when a network component fails?
What single points of failure remain?
What would you change if the company required even higher availability?

Your goal is not necessarily to create a perfect architecture. Your goal is to identify the weaknesses of your architecture and explain how you would improve it.

Final Submission

Submit a single document containing the following:

1. HA Configuration

Provide evidence that your VM is configured for HA.

2. Failure Prediction

Provide your predictions from Part 2.

3. Controlled Failure

Document the results of your controlled failure test.

Include:

Screenshots
Recovery times
Observations
Answers to the questions

4. Unexpected Failure

Document the results of your unexpected failure test.

5. Quorum Analysis

Answer the questions from Part 5.

6. Infrastructure Analysis

Identify the potential single points of failure in your Proxmox/TrueNAS environment.

7. HA vs. Fault Tolerance

Explain the difference between the two concepts.

8. Improved Architecture

Submit your architecture diagram and explanation.