Recently I started building a new home server lab. It consists of Intel NUCs running VMware vSphere. I added the maximum memory to the NUCs. One is a core i7 and the other is a core i5. The both have 64 GB of RAM. I use them for running virtual machines for testing software or developing applications at home.

Currently, my lab is running the following:

  • VMware vCenter Server Appliance 7.0.3
  • VMware ESXi 7.0.3

Depending on the version of vCenter, there are differences in the API calls. The Automation API Reference noted below details the differences.

About a month ago I had a UPS failure that prompted me to setup UPS monitoring. When the power is out, do a graceful shutdown of most things I have plugged into that UPS.

I decided it would be a good opportunity to learn the vCenter APIs. The idea was to have some simple API calls in a Python program that can be called by the UPS when it needs to shutdown everything. I’m using a Raspberry Pi running apcupsd to monitor an older APC UPS with a USB connection.

Reading the VMware site for API documentation was overwhelming. There are several SDKs, including a Python SDK. The information in the site seems to be mixed with a lot of different programming languages with SDKs for SOAP and REST. I don’t want to use the SDK. I feel it adds a lot of complexity for what I need to do. And I want to avoid installing a lot of modules that require a virtual environment in Python. After some searching, I found the information I was looking for in the vSphere Automation API Reference.

One catch is that the APIs are executed through vCenter. This means that when shutting down VMs, the vCenter appliance has to be the last VM shut down. It also means that the API doesn’t shutdown the ESXi host. I use the program to get a list of currently running VMs, shutdown all the VMs except fo vCenter. Finally, it shuts down the vCenter appliance VM. After that I use another program to ssh into ESXi and shut it down.

The program uses the Requests library. It works well and is simple to use.

Certificate checking is disabled to allow the self-signed certificate in vCenter. Do not disable certificate validation for production systems. This program is written for a home computer lab.

I used functions for the API calls. There are several functions for getting the list of VMs and performing power operations. I’m not using all of the functions in this program.

# Function to get the vCenter server session
def get_vc_session(s, vcip, username, password):

# Function to get all the VMs from vCenter inventory
def get_vm_list(s, vcip):

# Function to get all the VMs from vCenter inventory that are powered on
def get_vm_poweredon_list(s, vcip):

# Guest power status
def get_guest_power(s, vmid, vcip):

# Guest shutdown
def guest_shutdown(s, vmid, vcip):

# Power on vm
def vm_poweron(s, vmid, vcip):

# Power off vm (this is not a guest OS shutdown)
def poweroff_vm(s, vmid, vcip):

Creating a session

The first step is to authenticate and create a session to use the API. This function sends the username and password for authentication using Basic Auth. Requests makes this easy. The API returns a header (vmware-api-session-id) that contains the session ID. This is added to the session so it is used for subsequent API calls. The session is returned so that it can be passed to the other API functions. The full program uses logging, but I removed it in this example.

# Function to get the vCenter server session
def get_vc_session(s,vcip,username,password):
    s.verify = False   # disable checking server certificate
    s.auth = (username, password) # Basic authentication
    try:
        r = s.post("https://" + vcip + "/api/session")
    except requests.exceptions.ConnectionError:
        print("Error connecting to vCenter: " + vcip)
        sys.exit(1)
    if r.status_code == 401:
        print("Invalid credentials for vCenter")
        sys.exit(1)
    if r.status_code != 201:
        sys.exit(1)
    s.headers.update({'vmware-api-session-id': r.headers['vmware-api-session-id']})
    return s

Listing virtual machines

This example gets a list of virtual machines (up to 4000 per the API reference).

# Function to get all the VMs from vCenter inventory
def get_vm_list(s, vcip):
    r = s.get("https://" + vcip + "/api/vcenter/vm")
    return r

There is an alternate version that gets the list of powered on VMs. Since the goal is to shutdown all running VMs this is the one the program uses.

# Function to get all the VMs from vCenter inventory that are powered on
def get_vm_poweredon_list(s, vcip):
    vm_query_params = {"power_states":["POWERED_ON"]}
    r = s.get("https://" + vcip + "/api/vcenter/vm", params=vm_query_params)
    return r

All of the API functions return the result from the API call. Enabling debug shows more about the header, request URL, result code and result text.

Shutdown guest

This API call performs a guest OS shutdown operation. When the virtual machine tools are installed, this will gracefully shutdown the virtual machine.

# Guest shutdown
def guest_shutdown(s, vmid, vcip):
    vm_action = {"action": "shutdown"}
    r = s.post("https://" + vcip + "/api/vcenter/vm/" + vmid + "/guest/power", params=vm_action)
    return r

Logging

The program provides status and debugging info with the standard library logging facility. A logger is created and then used to output INFO, DEBUG and ERROR level messages. For now it logs to standard out, but it could also be written to a file. I plan to change it to use syslog.

# Global log
#log_level = logging.DEBUG
log_level = logging.INFO 

logging.basicConfig(level = log_level,
                    format = "%(asctime)s %(levelname)s %(module)s %(message)s")

log = logging.getLogger()

When the log level is set to debug, the program provides more details for each of the API calls. The headers, URL and status code and other items are logged.

log.debug(r.url)
log.debug(r.headers)
log.debug(r.request.headers)
log.debug(str(r.status_code))
log.debug(r.text)

I used a comment to switch between DEBUG and INFO levels. Just uncomment the one you want to use and comment out the other one.

# Global log
#log_level = logging.DEBUG
log_level = logging.INFO

This is an example of the debug output from an API call. It was helpful in determining issues like when the authentication session ID wasn’t in the header. It also helped to see the parameters in the URL for some API calls to make sure they match the documentation.

2022-03-08 23:47:37,295 DEBUG shutdown_vcenter_vms https://X.X.X.X/api/vcenter/vm?power_states=POWERED_ON
2022-03-08 23:47:37,295 DEBUG shutdown_vcenter_vms {'date': 'Wed, 09 Mar 2022 04:47:37 GMT', 'content-type': 'application/json', 'x-envoy-upstream-service-time': '2', 'server': 'envoy', 'transfer-encoding': 'chunked'}
2022-03-08 23:47:37,295 DEBUG shutdown_vcenter_vms {'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'vmware-api-session-id': '01234567890ABCDEFG'}
2022-03-08 23:47:37,295 DEBUG shutdown_vcenter_vms 200
2022-03-08 23:47:37,295 DEBUG shutdown_vcenter_vms [{"memory_size_MiB":12288,"vm":"vm-14","name":"vCenter1","power_state":"POWERED_ON","cpu_count":2},{"memory_size_MiB":2048,"vm":"vm-16","name":"vm-two","power_state":"POWERED_ON","cpu_count":2},{"memory_size_MiB":2048,"vm":"vm-17","name":"vm-three","power_state":"POWERED_ON","cpu_count":2},{"memory_size_MiB":2048,"vm":"vm-19","name":"vm-four","power_state":"POWERED_ON","cpu_count":2}]

Settings

The start of the program has a section with settings. You need to change vcip to use the IP or hostname of your vCenter appliance. I’m using the IP just in case there is a DNS outage when the power is out.

Also change vcacct to the vCenter user that can shutdown the VMs. And vcpw is the password for that user.

The remaining settings control how long the program delays when the VMs are shutdown. When the VM shutdown is called, there is a loop that pauses and then polls for all the power on VMs. The delay is controlled with shutdown_wait_time. The number of times to loop and check VMs is max_shutdown_wait_count. If all of the VMs except for vCenter are shutdown, the loop exits early using a break. For vCenter, it is just a sleep for that time because there isn’t a way to check via API if vCenter is shutdown. It would be possible to ping the vCenter IP and wait for no response. Or to check the ESX host using ssh to see if all the VMs are off.

# Settings
#
vcip = "X.X.X.X" # vCenter server ip address/FQDN
vcacct = "YOUR_USER@vsphere.local"
vcpw = "YOUR_PASSWORD"
max_shutdown_wait_count = 30      # the number of wait loops 30 x 10 = 300 seconds or 5 minutes
shutdown_wait_time = 10           # the number of seconds to sleep between loops
shutdown_vcenter_wait_time = 180  # the number of seconds to sleep after requesting vcenter to shutdown

Testing

When I was developing the program I didn’t want to shutdown all of the VMs. The easy thing to do was to comment out the 2 calls to guest_shutdown(). The first in in the loop that shuts down all the regular VMs. The second is in the last section where vCenter is shut down.

I also used shorter times for the wait loop and timer for vCenter. Comment out the longer times and uncomment the shorter ones. For max_shutdown_wait_count and shutdown_vcenter_wait_time.

max_shutdown_wait_count = 30      # the number of wait loops 30 x 10 = 300 seconds or 5 minutes
#max_shutdown_wait_count = 3      # shorter wait for testing
shutdown_wait_time = 10           # the number of seconds to sleep between loops
#shutdown_vcenter_wait_time = 10  # shorter wait for testing
shutdown_vcenter_wait_time = 180  # the number of seconds to sleep after requesting vcenter to shutdown

The final test was to make sure it shutdown my VMs. So far the system just has a few VMs running.

First remove comments from guest_shutdown if you were testing that way. Change the wait time and count to the larger values. And set the debugging level the way you want. This test used INFO level.

When the program was running I watched the VMs shutdown in the vCenter web UI and in the ESX web UI. vCenter actually shutdown well before the program finished.

user@mb-svr:~ $ python3 shutdown_vcenter_vms.py 
2022-03-24 00:22:29,836 INFO shutdown_vcenter_vms_2 Creating vCenter Rest API session
2022-03-24 00:22:29,963 INFO shutdown_vcenter_vms_2 Number of running VMs not including vCenter: 3
2022-03-24 00:22:29,963 INFO shutdown_vcenter_vms_2 The following VMs will be shutdown: vm-ub-ansible1:vm-16,vm-ub-git1:vm-17,vm-ub-pydev1:vm-19,vCenter1:vm-14
2022-03-24 00:22:29,963 INFO shutdown_vcenter_vms_2 Shutting down VMs
2022-03-24 00:22:29,963 INFO shutdown_vcenter_vms_2 Shutting down: vm-ub-ansible1:vm-16
2022-03-24 00:22:30,282 INFO shutdown_vcenter_vms_2 Shutting down: vm-ub-git1:vm-17
2022-03-24 00:22:30,386 INFO shutdown_vcenter_vms_2 Shutting down: vm-ub-pydev1:vm-19
2022-03-24 00:22:40,493 INFO shutdown_vcenter_vms_2 Timeout remaining: 290
2022-03-24 00:22:40,511 INFO shutdown_vcenter_vms_2 List VMs status_code: 200
2022-03-24 00:22:40,512 INFO shutdown_vcenter_vms_2 Number of running VMs excluding vCenter: 0
2022-03-24 00:22:40,529 INFO shutdown_vcenter_vms_2 List VMs status_code: 200
2022-03-24 00:22:40,530 INFO shutdown_vcenter_vms_2 Number of running VMs remaining: 1
2022-03-24 00:22:40,530 INFO shutdown_vcenter_vms_2 Shutting down: vCenter1:vm-14
2022-03-24 00:22:40,758 INFO shutdown_vcenter_vms_2 Shutdown sent for vCenter. Pausing for 180 seconds.
user@mb-svr:~ $

Everything worked OK. All of the VMs shutdown. Then the vCenter appliance shutdown. I used the other script to shutdown ESX. Then I powered the ESX host back on to make sure it came up OK.

The completed program

The program is available on Github.

homelab-vmware-shutdown