Proxmox-NUT Homelab HOWTO - Step 5 : Install NUT UPS Solution / Configure NUT for Email Alerts and System Shutdown / Test

Step 5: Install NUT UPS Solution / Configure NUT for Email Alerts and System Shutdown / Test

NUT will be installed on Debian (Proxmox) and will be linked to a single APC UPS. When there is a power failure, NUT will send an email. Its status will change from Online (OL) to On Battery (OB).

When either the remain battery percentage charge goes below a certain value (85% in this example) or the remaining run time goes below a certain amount (15 minutes in this example), the UPS status will change from OB to Low Battery (LB).

When the status becomes LB, NUT will send out an email and start shutting down Debian. As part of the Proxmox shutdown process it will shut down all guests running with it and then shut down itself. After that the OS shuts down.

If the LB condition does not trigger within 5 minutes the UPS will shut down Proxmox. This is discussed further below.

NUT manages other UPS status such as whether it is discharging and if the battery needs replacing. These setting will be factored into the configuration files but will not be discussed here.

After the offdelay number of seconds have passed, the UPS powers itself down.

The VMWare Proxmox-NUT VM

Because Proxmox is installed as a guest VM in VMWare and because the session needed to be recorded the UPS setup had to be adjusted to cater for this situation.

 The setup is described in the diagram shown here.

A UPS was connected to the power input of a physical computer. A USB cable was not plugged in.

A second UPS (APC) was connected to the power source but was not plugged into anything. There was a USB cable between the APC and the physical computer.

When VMWare’s Proxmox image was started, the APC was connected to the VM.

 

 


Install and Configure NUT


 The image above, from https://rogerprice.org/NUT/ConfigExamples.A5.pdf, consists of a number of daemons communicating with each other to manage UPS and the devices they are supporting.

NUT is not tied to any brand of UPS. The communication between the NUT and the UPS hardware is achieved through drivers (Driver in the image above). Through the driver the Attachment daemon exposes various ups parameters. The NUT solution will, based on the value of these parameters protect the equipment connected to the ups.

Many of the parameters are RO but some parameters may be RW. RW parameters can be altered by the NUT daemon or a provided NUT utility program.

Even though the above image shows one UPS, NUT is very flexible and can support setups with multiple UPSs working in sync. This topic outside the scope of the Proxmox-NUT topic.

Not all UPSs have the same capabilities or support identical functions. Further different UPS brands may trigger a state differently. For example, with certain UPS, Low Battery status indicates that the UPS is in a critical condition and will imminently power down. Another UPS brand would raise a Low Battery status earlier giving one time to shut down the supported equipment. The NUT website provides a database of supported UPS.

Installing NUT

  1. In Proxmox open a terminal window and type the following:
apt update
apt upgrade
  1. Install nut software.
apt install nut
  1. Identify the Bus and Device of the UPS using the command: lsusb


  2. (Optional) You can extract additional information from the device connected to the Bus / Device using the command lsusb -v -s <bus>:<device>.
  3. Use the utility to get information you need for your nut driver configuration is nut-scanner -U.
nut-scanner -U

Configuring NUT

  1. Create a backup of this file:
    cp /etc/nut/ups.conf /etc/nut/ups.conf.original
  2. Insert the text below.
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
 # Number of attempts to start the driver(s), in case of
# failure, before giving up
maxretry = 3

[pve1]
    driver = "usbhid-ups"
    port = "auto"
    vendorid = "051D"
    productid = "0002"
    product = "Back-UPS XS 1400U FW:926.T1 .I USB FW:T1"
    serial = "3B1451X20989"
    vendor = "American Power Conversion"
    bus = "001"
    
    desc = "Proxmox-NUT UPS"
    # NUT-upsuser: mailing list 'Re: [Nut-upsuser] UPS
    not Shutting Down' Charles Lepple
    # Some drivers reset battery.charge.low to the default value.
    #lowbatt addresses this
    # See chapter 2.7 in https://rogerprice.org/NUT/ConfigExamples.A5.pdf
    override.battery.charge.low = 85
    # lowbatt = 85
    # if the remaining charge is less than 800s
    # trigger a low battery status
    override.battery.runtime.low = 800
    # Wait 2 minutes between the shutdown command and
    # the moment the UPS shuts itself down
    offdelay = 120
    # Wait 5 minutes before attempting to restart the ups
    # load after a power cut
    ondelay = 300
    # wait 5 minutes for ups to power off
    override.ups.delay.shutdown = 300
    # When you specify this, the driver ignores a low
    # battery condition flag that is
    # reported by the UPS ; https://networkupstools.org/docs/man/ups.conf.html
    ignorelb

# from working with heartbeat
# (https://rogerprice.org/NUT/ConfigExamples.A5.pdf)
[heartbeat]
    driver = dummy-ups
    port = heartbeat.conf
    mode = dummy-loop
    desc = "Watch over NUT"
  1. Secure the configuration file by running:
chmod 600 /etc/nut/ups.conf

Each registered device in NUT must have a unique name. In the script above line 04 and 33 define two devices: pve1 and heartbeat.

For physical devices such as pve1, the lines 06 – 13 is copied from the output provided by nut-scanner (see point 5). If nut-scanner returns serial = 0, ignore the line.

Some UPSs take override.battery.charge.low = 85 (line 21) while for other brands the lowbatt = 85 (line 22) must be used. This is because some UPS brands do not support, with some brands ignoring this parameter. You will need to discover what your UPS takes. Try this setup first.

There is another UPS called heartbeat. This is a dummy ups that polls the communication between the computer and the UPS every 10 minutes. It is not evaluated in this HowTo.

  1. Backup and replace the contents of the file /etc/nut/nut.conf as shown below. In this HowTo, both standalone and netserver would have worked. standalone indicates that 1 UPS protecting the local system. netserver enables network connectivity and special care to protect from unauthorised access. If, for example, you add a NUT container on Proxmox to monitor the UPS (https://www.kreaweb.be/diy-home-server-2021-software-proxmox-ups/) this setting is necessary.
    # Appendix A in https://rogerprice.org/NUT/ConfigExamples.A5.pdf
    # MODE=standalone
    MODE=netserver
  2. Secure access:
    chmod 600 /etc/nut/nut.conf
  3. Backup and replace the contents of the /etc/nut/upsd.conf as shown here.
    LISTEN 0.0.0.0 3493
    # for IP V6
    LISTEN :: 3493
  4. Secure access:
    chmod 600 /etc/nut/upsd.conf
  5. Backup and replace /etc/nut/upsd.users as shown hereunder. This file defines the accounts that are authorised to read and adjust the UPS settings. In this example a single password was used everywhere. This should be avoided in production systems.
    [upsadmin]
        # Administrative User
        password = 1234..qwertY
        # Allow changing values of certain variables in the UPS.
        actions = SET
        # Allow setting the Forced Shutdown flag in the UPS.
        actions = FSD
        # Allow all instant commands
        instcmds = ALL
        upsmon primary
  6. Secure access:
    chmod 600 /etc/nut/upsd.users
  7. Create the file /etc/nut/heartbeat.conf. The ups.status on the dummy UPS is switched every 5 minutes (TIMER setting). In another file this will be checked for.
    # heartbeat.conf -- 10 minute heartbeat
    ups.status: OL
    TIMER 300
    ups.status: OB
    TIMER 300
  8. Secure access:
    chmod 600 /etc/nut/heartbeat.conf
  9. Start the driver controller to verify that all settings are correct. With the Proxmox-NUT VM, the bus the APC is connected on can change. The configuration file /etc/nut/ups.conf has to be modified to reflect the bus if this happens.
    upsdrvctl start

The next group of files declare how NUT will handle the various events.

  1. Backup and replace: /etc/nut/upsmon.conf with:
    RUN_AS_USER root
    # defined in /etc/nut/upsd.users 
    MONITOR pve1@localhost 1 upsadmin 1234..qwertY primary
    # zero parameter is because heartbeat does not supply power # https://rogerprice.org/NUT/ConfigExamples.A5.pdf
    MONITOR heartbeat@localhost 0 upsadmin 1234..qwertY primary

    MINSUPPLIES 1
    SHUTDOWNCMD "/sbin/shutdown -h now"
    NOTIFYCMD /usr/sbin/upssched
    POLLFREQ 4
    POLLFREQALERT 2
    HOSTSYNC 15
    DEADTIME 30
    MAXAGE 24
    POWERDOWNFLAG /etc/killpower

    NOTIFYMSG ONLINE "UPS %s on line power"
    NOTIFYMSG ONBATT "UPS %s on battery"
    NOTIFYMSG LOWBATT "UPS %s battary is low"
    NOTIFYMSG FSD "UPS %s: forced shutdown in progress"
    NOTIFYMSG COMMOK "Communications with UPS %s established"
    NOTIFYMSG COMMBAD "Communications with UPS %s lost"
    NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding"
    NOTIFYMSG REPLBATT "UPS %s battery needs to be replaced"
    NOTIFYMSG NOCOMM "UPS %s is unavailable"

    # https://rogerprice.org/NUT/ConfigExamples.A5.pdf
    # To avoid fooding your logs, remove the fags SYSLOG and WALL for the [online] and [onbatt] NOTIFY events
    NOTIFYFLAG ONLINE EXEC
    NOTIFYFLAG ONBATT EXEC
    NOTIFYFLAG LOWBATT SYSLOG+WALL+EXEC
    NOTIFYFLAG FSD SYSLOG+WALL+EXEC
    NOTIFYFLAG COMMOK SYSLOG+WALL+EXEC
    NOTIFYFLAG COMMBAD SYSLOG+WALL+EXEC
    NOTIFYFLAG SHUTDOWN SYSLOG+WALL+EXEC
    NOTIFYFLAG REPLBATT SYSLOG+WALL
    NOTIFYFLAG NOCOMM SYSLOG+WALL+EXEC
    NOTIFYFLAG NOPARENT SYSLOG+WALL

    #Replace Battery Warn Time
    RBWARNTIME 43200
    # No Communication Warn Time
    NOCOMMWARNTIME 640
    # Shutdown immediately after issuing SHUTDOWN notification
    FINALDELAY 0
  2. Secure access:
    chmod 600 /etc/nut/upsmon.conf
  3. Backup and replace: /etc/nut/upssched.conf. “The NOTIFY events detected by upsmon and tagged as EXEC in upsmon.conf become events for upssched when NOTIFYCMD points to upssched. The program upssched provides a richer set of actions than upsmon” [https://rogerprice.org/NUT/ConfigExamples.A5.pdf].
    01
    02
    03
    04
    05
    06
    07
    08
    09
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    		 CMDSCRIPT /etc/nut/upssched-cmd
    PIPEFN /etc/nut/upssched.pipe
    LOCKFN /etc/nut/upssched.lock

    # Restart timer which completes only if the dummy-ups heartbeat
    # has stopped. See timer values in heartbeat.conf
    AT ONBATT heartbeat@localhost CANCEL-TIMER heartbeat-failure-timer
    AT ONBATT heartbeat@localhost START-TIMER heartbeat-failure-timer 660

    # The UPS is on battery.
    # Start a 10 seconds timer, then execute the "On Battery" command.
    AT ONBATT pve1@localhost START-TIMER onbatt 10

    # The UPS is back on line.
    # Cancel any running "On Battery" timer, then execute the "Online" command.
    AT ONLINE pve1@localhost CANCEL-TIMER onbatt online

    # If the Power is out for more than 5 minutes shutdown the UPS
    # (ignore LB condition)
    AT ONBATT pve1@localhost START-TIMER shutafter5min 300
    AT ONLINE * CANCEL-TIMER shutafter5min

    # The UPS battery is low (as determined by the driver).
    # Execute the "Low Battery" command immediately.
    AT LOWBATT pve1@localhost EXECUTE lowbatt

    # The UPS has been commanded into the "Forced Shutdown" mode.
    # Execute the "Forced Shutdown" command immediately.
    AT FSD pve1@localhost EXECUTE fsd

    # Communication with the UPS was just lost.
    # Start a 30 seconds timer, then execute the "Communications Lost" command.
    AT COMMBAD pve1@localhost START-TIMER commbad 30

    # Communication with the UPS has been established.
    # Cancel any running "Communications Lost" timer, then execute the
    # "Communications Restored" command.
    AT COMMOK pve1@localhost CANCEL-TIMER commbad commok

    # The UPS can’t be contacted for monitoring.
    # Start a 5 seconds timer, then execute the "No Communications" command.
    AT NOCOMM pve1@localhost START-TIMER nocomm 15

    AT NOCOMM pve1@localhost EXECUTE commbad

    # The local system is being shut down.
    # Execute the "Notify Shutdown" command immediately.
    AT SHUTDOWN pve1@localhost EXECUTE shutdown

    # The UPS needs to have its battery replaced.
    # Start a 5 minutes timer, then execute the "Replace Battery" command.
    AT REPLBATT pve1@localhost START-TIMER replbatt 300

    In rows 18 – 21, when the UPS first detects that it is OB it sets a 5-minute timer after which it shuts down the UPS. If UPS status changes to OL, the timer is cancelled. This is used to shut down the UPS after a fixed period of time. A LB condition will still cause a shutdown if it happens before the 5 minutes have passed.

  4. Secure access:
    chmod 600 /etc/nut/upssched.conf
  5. Backup and replace: /etc/nut/upssched-cmd. This is a user-defined script defined in /etc/nut/upssched.conf triggered by the EXECUTE You can change the name shown here (you need to change this file and the reference in upssched.conf) but, according to Price, “[while] this is not the most elegant of names but if you use it, people in the NUT community will know immediately what you mean”.
    01
    02
    03
    04
    05
    06
    07
    08
    09
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    		 #!/bin/sh
    case $1 in
      onbatt)
        echo "UPS running on battery" | mail -s "Proxmox-NUT NUT UPS" \
          chribonn.covid@gmail.com
        logger -t "nut_upssched-cmd" "UPS running on battery"
        ;;
      online)
        echo "Power restored on UPS" | mail -s "Proxmox-NUT NUT UPS" \
          chribonn.covid@gmail.com
        logger -t "nut_upssched-cmd" "Power restored on UPS"
        ;;
      lowbatt)
        echo "Low battery on UPS!" | mail -s "Proxmox-NUT NUT UPS" \
          chribonn.covid@gmail.com
        logger -t "nut_upssched-cmd" "Low battery on UPS!"
        ;;
      shutafter5min)
        echo "Shutdown after 5 minutes" | mail -s "Proxmox-NUT NUT UPS" \
          chribonn.covid@gmail.com
        logger -t "nut_upssched-cmd" "Shutdown after 5 minutes"
        upsmon -c fsd
        ;;
      fsd)
        echo "Forced Shutdown from UPS!" | mail -s "Proxmox-NUT NUT UPS" \
          chribonn.covid@gmail.com
        logger -t "nut_upssched-cmd" "Forced Shutdown from UPS!"
        ;;
      commok)
        logger -t "nut_upssched-cmd" "Communications restored with UPS"
        ;;
      commbad)
        logger -t "nut_upssched-cmd" "Warning: Lost communications with UPS"
        ;;
      shutdown)
        echo "System is shutting down now!" | mail -s "Proxmox-NUT NUT UPS" \
          chribonn.covid@gmail.com
        logger -t "nut_upssched-cmd" "System is shutting down now!"
        ;;
      replbatt)
        echo "Replace battery on UPS" | mail -s "Proxmox-NUT NUT UPS" \
          chribonn.covid@gmail.com
        logger -t "nut_upssched-cmd" "Replace battery on UPS"
        ;;
      nocomm)
        logger -t "nut_upssched-cmd" \
          "The UPS can’t be contacted for monitoring!"
        ;;
      heartbeat-failure-timer)
        echo "UPS heartbeat failed." | mail -s "Proxmox-NUT NUT UPS" \
          chribonn.covid@gmail.com
        logger -t "UPS heartbeat failed."
        ;;
      *)
        echo "Unrecognized command: $1" | mail -s "Proxmox-NUT NUT UPS" \
          chribonn.covid@gmail.com
        logger -t "nut_upssched-cmd" "Unrecognized command: $1"
        ;;
    esac

    Lines 18-23 is the block of code that handles a shutdown after 5 minutes not factoring a LB condition.

  6. Grant root use read-write-execute, and group root read access.
    chmod 740 /etc/nut/upssched-cmd
  7. Start NUT solution. You should receive confirmation that both heartbeat and pve1 are connected.
    service nut-server restart
    service nut-client restart
    systemctl restart nut-monitor
    upsdrvctl stop
    upsdrvctl start


Other NUT UPS commands

The following commands allow you to query your UPS and make adjustments. Not all UPSs consider adjusted parameters!!

#query pve1
upsc pve1@localhost
# query heartbeat
upsc heartbeat


The query a single parameter, specify its name upsc pve1@localhost ups.status.

List the parameters that can be adjusted on this UPS:

upsrw -l pve1@localhost

Modify a parameter:

# upsrw -s VAR=VALUE -u <username> -p <password> <ups>
upsrw -s VAR=bsttery.mfr.date -u upsadmin -p 1234..qwertY pve1@localhost
 

Testing the solution

Unplug the UPS (APC in this case) that is connected to the VMWare workstation. You can use upsc to check the variables.

 

NUT’s Versatility and Flexibility

Besides this article and video series, if you want to configure NUT refer to some of the links shared below. Getting a basic understanding of how NUT works and how all the pieces interact with each other before you start slapping code into your solution will pay out almost immediately.


Video


Links

 An Introduction to Network UPS Tools (Editor: Roger Price)
    https://rogerprice.org/NUT/ConfigExamples.A5.pdf
Network UPS Tools
    https://networkupstools.org/
NUT Users mailing list
    http://lists.alioth.debian.org/mailman/listinfo/nut-upsuser
Network UPS Tools (NUT) Ultimate Guide
    https://technotim.live/posts/NUT-server-guide/
DIY Home Server 2021 Software Proxmox UPS
    https://www.kreaweb.be/diy-home-server-2021-software-proxmox-ups/

 

Follow This, That and (Maybe), the Other:

Comments

Post a Comment

Popular posts from this blog

20150628 Giarratana Circular

Statistics Date: Jun 28, 2015 4:34 pm Distance: 15.21 km Duration: 2h:35m:36s Avg Speed: 5.87 km/h Max Speed: 7.40 km/h Min Altitude: 489 m Max Altitude: 594 m Ascent: 249 m Descent: 261 m Route Open route in new window: https://www.google.com/maps/d/edit?mid=z6HKMq7G_wEk.keh2K8cXD6Cc&usp=sharing Photos (Geotagged) Open photos in new Windows: https://plus.google.com/photos/105470926677245165774/albums/6166646868710719889
Read more

HOWTO setup OpenVPN server and client configuration files using EasyRSA

Image
Introduction OpenVPN allows client computers to tunnel into a server over a single UDP or TCP port securely. This HOWTO article is a step-by-step guide that explains how to create the server and client OpenVPN configuration files that makes this possible. In the process this article explains how to create the public key infrastructure (PKI) so that a client can securely communicate with the server. OpenSSL is the foundation for the security functionality of OpenVPN. For this tutorial you will need the following software: OpenVPN. You can download the latest version of OpenVPN from https://openvpn.net/index.php/open-source/downloads.html EasyRSA is the tool people use to create the Public Key Infrastructure (PKI) for OpenVPN. Download the latest release of EasyRSA from https://github.com/OpenVPN/easy-rsa/releases . There is not installation required. Extract the contents of the archive into a folder. OpenVPN is available on various platforms. The generation of the
Read more

Proxmox-NUT Homelab HOWTO : Step 0 : The Proxmox-NUT HomeLab HowTo: Introduction

Image
Step 0: The Proxmox-NUT HomeLab HowTo: Introduction The series includes this article, a 7-part video series, and a VMWare archive. In this HowTo, I cover the following steps: Configure a VM on VMWare Workstation Pro to install Proxmox on it; Install a Windows Server 2022 VM as a VM in Proxmox; Install the NUT (Network UPS Tools) UPS Software. I also configure Proxmox, Windows Server 2022, and NUT to use a Gmail SMTP relay. This will be used by Proxmox and Windows server to send emails when they start and stop. Proxmox and NUT will provide additional status information. Windows Server is configured to start automatically when Proxmox starts and will shut down cleanly without manual intervention when Proxmox is shutdown or rebooted. Every stage is tested out to verify that everything is working as expected. Any limitations are highlighted. A snapshot of the VM state after every Step is taken. You can download the entire archive and run it on your own hardware. If you decide yo
Read more