Trying to understand bonnie++ output / 2023-08-07

In preparation for a migration at work I wanted to do actual benchmarking of Linux filesystem performance. I think I used bonnie in the last century, so I gave bonnie++ a spin for this.

I have little idea of what 'good' or 'bad' numbers are from bonnie++. I could only compare a "local" filesystem with an NFS filesystem. I use local in quotes because this was in a virtual machine, so it's SSD storage in raid-1, with LVM on top of it, with a logical volume assigned to a KVM-based virtual machine, which uses the virtio disk driver for an ext4 filesystem.

The numbers for the "local" filesystem:

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
gosper          32G  809k  98  440m  38  215m  22 1590k  99  410m  30  4639 135
Latency             25688us     317ms     143ms    9332us   39208us    2089us
Version  1.98       ------Sequential Create------ --------Random Create--------
gosper              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency               488us     684us     762us     236us      87us     262us
1.98,1.98,gosper,1,1691401899,32G,,8192,5,809,98,450230,38,220385,22,1590,99,419827,30,4639,135,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,25688us,317ms,143ms,9332us,39208us,2089us,488us,684us,762us,236us,87us,262us

And for NFS, a Synology NAS with spinning disks in raid-5:

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
gosper          32G 1054k  98 78.7m   7 68.4m  13 1483k  99  109m  10 432.2  12
Latency             11138us     408ms   13261ms   16434us     212ms     274ms
Version  1.98       ------Sequential Create------ --------Random Create--------
gosper              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 16384  10 16384  16 16384  15 16384   9 16384  18 16384  15
Latency             69194us   53194us   98927us   69144us    1240us   94317us
1.98,1.98,gosper,1,1691398200,32G,,8192,5,1054,98,80605,7,70058,13,1483,99,111574,10,432.2,12,16,,,,,705,10,16184,16,2800,15,693,9,3703,18,2226,15,11138us,408ms,13261ms,16434us,212ms,274ms,69194us,53194us,98927us,69144us,1240us,94317us

Now I am somewhat confused. Sequential write to NFS is slightly faster.

Update 2023-08-08

At work I got different but comparable numbers for iscsi attached storage versus vmware storage (and the layers in between). Those numbers helped make decisions about the storage.