Send RAID Alerts with my Helios4 via Email

May 3, 2021 — 895 words — 5 min

#email 

---

The goal is to be able to send RAID alerts from my Helios4 to my email address and because my personal IP address provided by my ISP is on a default blocklist and therefore not allowed to send mail, I quickly created a workaround. I already use a VPS, which is set up to send mail, so I figured I just need to connect my Helios4 to my VPS and tell the server to send the email instead.

On the VPS I created an extra user for this, so I can clean this up quickly if it went sideways (but it didn’t) and for some other constraints such as limiting this public key to only executing one script (more on that later).

The new user on the server is quickly created with:

sudo useradd -m heliosmail

Then I created a new keypair (ssh-keygen -t ed25519) on the Helios4 and copied the public key to /home/heliosmail/.ssh/authorized_keys on my server. The user heliosmail (if authenticated with this public key) should only be able to execute one command, so we modify the authorized_keys file slightly. The reason is that the SD card of the Helios4 is not encrypted itself (only the data on the RAID) and if that keypair got compromised, then I don’t want anyone to be able to log into my VPS. Specifically, I inserted this part in front of the public key: command="/home/heliosmail/mail.sh $SSH_ORIGINAL_COMMAND"

command="/home/heliosmail/mail.sh $SSH_ORIGINAL_COMMAND" ssh-ed25519 AAAAC3NzaC1lZ...

Now, when the user heliosmail logs in (actually, authenticates successfully), the mail.sh script is automatically executed. The variable SSH_ORIGINAL_COMMAND contains the original command which was issued by the user. So you could log this if you wanted, but I use this variable to pass an argument (the event why an email is now being sent) to the script.

The mail.sh script has the following content:

#!/bin/bash

EVENT="$1"

# first, strip underscores
CLEAN=${EVENT//_/}
# next, replace spaces with underscores
CLEAN=${CLEAN// /_}
# now, clean out anything that's not alphanumeric or an underscore
CLEAN=${CLEAN//[^a-zA-Z0-9_]/}
# finally get the spaces back because I like them.
CLEAN=${CLEAN//_/ }

(
echo "To: my@email.com"
echo "Subject: Helios RAID alert"
echo "Look at the RAID."
echo "Event: $CLEAN"
echo "."
) | sendmail -t

The first argument which is passed into the script is the event. This is the reason why mdadm triggered the configured script on the Helios4 (see below). And because I pass the argument to the echo command, it should be sanitized in order to avoid command injection.

Then I decided to modify the script which is run by mdadm. This is configured in /etc/mdadm/mdadm.conf on the line which starts with PROGRAM. The line starting with MAILADDR specifies the mail address and I commented it out because the Helios4 itself is not sending any mail. ¹

PROGRAM /usr/sbin/mdadm-fault-led.sh

Actually, this script makes an LED blink on the Helios (or switch it on permanently if a critical error occurred). The only line which I added to this script is the one after the EVENT variable is defined.

#!/bin/bash
#
# Make Red Fault LED (LED2) reports mdadm error events.
#
EVENT=$1

# I added the following line to connect to my VPS
su -c "ssh cloud $EVENT" max >/dev/null
...

I use su in order to change to the user max because the ssh pubkey authentication is only configured for this user and not for root. The -c flag specifies the command which should be run and the $EVENT is the reason why this script was triggered in the first place and I want to see this in the email as described earlier.

The ssh cloud part works because cloud is defined in ~/.ssh/config and $EVENT would be the actual command which should be executed when connected to the server (stored in the SSH_ORIGINAL_COMMAND variable on the server).

Inserting this line of code at the beginning also means that an email for every event is being sent. If this is not desirable, you can move the inserted line into one or more of the following if statements which check for certain events. You can also check the official docs on the Helios4 wiki.

# Active component device of an array has been marked as faulty OR A newly noticed array appears to be degraded.
if [[ $EVENT == "Fail" || $EVENT == "DegradedArray" ]]; then
    echo none > $TRIGGER
    echo 1 > $BRIGHTNESS
fi

# An md array started reconstruction
if [ $EVENT == "RebuildStarted" ]; then
    echo timer > $TRIGGER
    echo 1 > $BRIGHTNESS
fi

# An md array that was rebuilding, isn't any more, either because it finished normally or was aborted.
if [ $EVENT == "RebuildFinished" ]; then
    echo none > $TRIGGER
    echo 0 > $BRIGHTNESS
fi

# Test RED Fault LED
if [ $EVENT == "TestMessage" ]; then
    echo timer > $TRIGGER
    echo 1 > $BRIGHTNESS
    sleep 5
    echo 0 > $BRIGHTNESS
fi

That’s it!

If everything works, you should get an email if you run

sudo mdadm --monitor --scan --test -1

which triggers the event "TestMessage". You should know that if the script /usr/sbin/mdadm-fault-led.sh gets updated, you may need to make the above modifications again.

---

Articles from blogs I follow around the net

Status update, April 2024

Hi! The X.Org Foundation results are in, and I’m now officially part of the Board of Directors. I hope I can be of use to the community on more organizational issues! Speaking of which, I’ve spent quite a bit of time dealing with Code of Conduct matters latel…

via emersion April 16, 2024

Email DNS Records Cheatsheet

A quick summary of the SMTP related DNS records together with brief examples.

via Signs of Triviality April 12, 2024

the best book ever written

I’ve asked my favorite musicians if, when they’re done writing a new song, they feel it’s the best song ever. All of them said yes. I’ve asked my favorite authors if, when they’re done writing a new book, they feel it was the best book ever. All of them said…

via Derek Sivers blog April 12, 2024

Generated by openring