Benchmarking on the Helios4
— 3390 words — 17 min
Table of Contents
If you don’t know about the Helios4, you can read about it here.
First of all, during benchmarking a lot of things can go wrong (I really recommend watching these 5 mintues of a lightning talk about benchmarking). Hence, take the results with a grain of salt. While I’m not doing any fancy calculations here, I am still not an expert. During the tests the Helios was always idle, or, at least, I did not do anything else. For instance, it could have happened that a cronjob was run during a benchmark, but I don’t think that this should have a high impact. Further, I was running the tests (with bonnie) multiple times.
The Helios4 has support for four drives but I use it currently with 2x4TB drives.
At the end of this post, you’ll find a table summarizing the results.
Running OMV with EXT4 on LVM on RAID 1🔗
For the first few tests, I actually installed OpenMediaVault (Helios4 Wiki) as I never used it before and wanted to try it out. I literally followed the instructions on the Official Kobol wiki which meant
- setting up my RAID 1 (mirror)
- installing the LVM plugin and creating the volumes
- creating the EXT4 filesystem
I installed and activated SMB as well and after I was able to connect to it
finally (permission things…), I used dd to push some data over the network
to get a feeling for the speed.
It’s important to note that I did not change any configration or tried to make any optimizations. This was run with the default settings.
SMB Performance🔗
As you can see in the command, I wanted to copy 10 GB originally, but aborted at 3.1 GB, which took 160s at a rate of about 20 MB/s. Using the same command to read the written data again resulted in about the same speed.
w=20MB/s, r=20MB/s
)
)
)
)
Copying a 15 GB video file from my computer onto the Helios4 took almost
exactly 13 minutes, which is pretty much the rate shown by dd: 15.000 MB /
(13*60s) = 19.2 MB/s.
I was a little confused about this speed. The Helios4 has a gigabit ethernet port, my computer and all other involved components as well. Theoretically, this means a throughput of about 125 MB/s. I did not expect to reach this speed in practice, but only 20 MB/s seemed to be a bit low?
SSHFS Performance🔗
Next, I tried SSHFS. You don’t need anything special, the only requirement is the ssh daemon and ssh is running on all my servers, anyway.
Again, I did not change any configurations or made any optimizations. This was run with the default settings when you mount a folder with SSHFS.
The usage of sshfs is sshfs [user@]host:[dir] mountpoint [options]. The
directoy on the Helios4 is mounted to ~/test locally, which you can achieve
with
I used the same dd command but with count=1000 so that 1 GB is transferred
because I did not want to wait for 10 GB.
w=29MB/s, r=31MB/s
)
)
)
)
I transmitted the same 15 GB video file as above as well, which took about 8 minutes: 15.000 MB / (8*60s) = 31 MB/s. That’s definitely better than 19.2 MB/s.
So roughly, SSHFS seems to be about 10 MB/s faster than SMB. Interesting. This actually spiked my interest to actually do some benchmarking on the Helios4 itself. Up until now, I only wanted to get a feel for transmission speeds, which I could expect.
Test on the Helios4 locally🔗
I connected to the Helios4 with ssh and run basically the same dd test as
before, this time with a 10 GB file (count=10000). This shows the writing and
reading speeds to the disk if no network is involved.
w=158MB/s, r=212Mb/s
)
)
)
)
Great! Writing happens with about 150 MB/s and reading at about 210 MB/s. This
shows that the disk I/O is definitely not the limiting factor. Now, let’s do a
test with a proper tool: bonnie++
Test with Bonnie++🔗
With bonnie++ (website) you can test
the performance of your filesystem and hard drives. The manpage tells you
everything you need to know, but let’s look at the used options quickly:
-dsets the directory for the test-c 1the level of concurrency-s 4024the size of the file(s) for IO performance measures in megabytes. 4024 is twice the size the RAM of the Helios4 (2G)-n 1the number of files for the file creation test (measured in multiples of 1024 files)-fspecified without a parameter, this skips the per-char IO tests.-bno write buffering, sofsync()is called after every write.
I really like the result:
w=146MB/s, rw=78MB/s, r=177MB/s
EXT4 on RAID 1🔗
Next, I removed OpenMediaVault and built my own setup. First, I created an EXT4 filesytem directly on the RAID 1 to compare the result to the previous test. The difference is that no LVM is used anymore. Both tests were run with no encryption in use.
The RAID 1 can be accessed with the device /dev/md0.
The following command creates the EXT4 filesystem. By default, a block size of 4096 is used. This is not relevant now, but it will be later when a LUKS container is used for encryption.
Mount and run the test.
Result: w=160MB/s, rw=96MB/s, r=160MB/s
Second run: w=164MB/s, rw=95MB/s, r=167MB/s
Interesting! While reading seems to be a bit slower, the writing and rw value are slightly higher than on the setup with OVM.
EXT4 on LUKS on RAID 1🔗
Instead of creating the EXT4 filesystem directly on the RAID 1 device, we will use a LUKS container to provide encryption and create an EXT4 filesystem on top of it.
It’s important to note that these results cannot be compared to the previous speed measurements because no encryption was used there. As you will see, the additional computing power for the encryption takes its performance toll.
First, let’s briefly look at the two 4 TB drives. Both, /dev/sda and
/dev/sdb report a physical sector size of 4096 bytes and a logical size of
512 bytes. Hence, it should be ensured that the LUKS container and the
filesystem use a block (sector) size of 4096 to use the drive efficiently.
|
|
Unaligned LUKS Sector Size and EXT4 Block Size🔗
The LUKS container was created with default options:
And by default, LUKS uses a sector size of 512 bytes. Information about a LUKS
container can be read with the cryptsetup command as follows.
The LUKS container can be opened with sudo cryptsetup open /dev/md0 cryptroot.
Thereafter, the EXT4 filesystem was created with default options, too:
The default block size is 4096 as can be seen with command dumpe2fs:
|
)
The sector size of the LUKS container and the block size of the filesystem are not identical. This is not recommended and will likely result in a performance loss. But I wanted to know how big the difference between aligned and unaligned sector/blocksize actually is.
A friendly reminder about the speed with an unencrypted EXT4 filesystem on the RAID 1:
w=164MB/s, rw=95MB/s, r=167MB/s
Bonnie++ is run with /usr/sbin/bonnie++ -d /mnt/md0/ -c 1 -s 4024 -n 1 -f -b.
Result: w64MB/s, rw=34MB/s, r=53MB/s
Second run: w=64MB/s, rw=30MB/s, r=53MB/s
I did a few more runs but they all yielded the same results. As you can see, the speed drops significantly.
Aligned LUKS Sector Size and EXT4 Block Size🔗
The following setup uses the correct sector size of 4096 for the LUKS container, which slightly improves the benchmark results.
The commands to create the setup:
Now, the sector size is 4096:
Create the EXT4 filesystem.
|
The block size of EXT4 is again 4096 (as the last time).
Running the test with bonnie++ returns the following result, which is indeed
a few megabytes better than previously. In my opinion, the encryption is worth
it and I’ll happily take the performance trade-off.
w=73MB/s, rw=37MB/s, r=59MB/s
Second go: w=73MB/s, rw=32MB/s, r=59MB/s
Encryption Offloaded to the CESA Unit🔗
Then I remembered that the Helios4 has a CESA unit (Cryptographic Engines and Security Accelerator unit), which can be used to offload encryption. I thought that using the CESA unit will improve the benchmark results further, but the results were quite disappointing.
While creating the LUKS container, the cipher aec-cbc-essiv:sha256 must be
specified to use the unit. This can be achieved with the following command. We
keep the correct sector size, too.
Open the container.
Verify cipher and sector size.
Create filesystem.
Check block size.
|
)
Let’s mount the partition and start the test!
w=48MB/s, rw=29MB/s, r=60MB/s
second run: w=48MB/s, rw=27MB/s, r=59MB/s
Well, the result is not very convincing. With the encryption offloaded to the CESA unit, the results are worse than before.
Table of Results with Bonnie++🔗
All setups use a RAID 1 and the EXT4 filesystem with a block size of 4096.
The exception is the EXT4 filesystem created by OMV. I’m not sure which block size was used as I created it with the OVM web GUI and I didn’t check it in the terminal. However, I assume that the default block size of 4096 was used.
Nonetheless, if another block size than 4096 was used, this might explain the worse performance of the setup of OMV with EXT4 on LVM on RAID 1 versus the plain EXT4 on the RAID 1; but perhaps this is just the indirection layer of LVM?
| Setup | LUKS sector size | encryption | write MB/s | read-write MB/s | read MB/s |
|---|---|---|---|---|---|
| OMV - EXT4 on LVM | no luks | none | 146 | 78 | 177 |
| EXT4 | no luks | none | 160 | 96 | 160 |
| EXT4 on LUKS | 512 | aes-xts-plain64 | 64 | 34 | 53 |
| EXT4 on LUKS | 4096 | aes-xts-plain64 | 73 | 37 | 59 |
| EXT4 on LUKS (CESA) | 4096 | aes-cbc-essiv:sha256 | 48 | 29 | 60 |
Regarding the CESA unit, there exists a HTTPS Benchmark in the official wiki.
Out of Scope🔗
There may be many aspects, which affect the measurement itself as well as how you evaluate the results. For instance, I did not look at the energy usage or CPU usage while the HTTPS benchmark takes the CPU usage into account.
Personally, I settled on the RAID 1 with an EXT4 filesystem on a LUKS container (with the correct sector size, of course). This provides encryption, which is a must-have, and reasonable speed. Furthermore, no other software but ssh is required, which is already running anyway.
Articles from blogs I follow around the net
Status update, April 2024
Hi! The X.Org Foundation results are in, and I’m now officially part of the Board of Directors. I hope I can be of use to the community on more organizational issues! Speaking of which, I’ve spent quite a bit of time dealing with Code of Conduct matters latel…
via emersion April 16, 2024Email DNS Records Cheatsheet
A quick summary of the SMTP related DNS records together with brief examples.
via Signs of Triviality April 12, 2024the best book ever written
I’ve asked my favorite musicians if, when they’re done writing a new song, they feel it’s the best song ever. All of them said yes. I’ve asked my favorite authors if, when they’re done writing a new book, they feel it was the best book ever. All of them said…
via Derek Sivers blog April 12, 2024Generated by openring