Why does E3 SSD dedi seem faster than Ryzen 7950X NVMe 2 vCore VPS at `sha256sum -c`?

Yesterday I uploaded a 4.6 GB Chromebook backup to storage. I uploaded the same file to both an E3 dedicated server and to a 2 vCore Ryzen 7950X VPS.

When I checked the integrity of the uploaded files, I was surprised that running sha256sum -c seemed to take longer on the Ryzen 7950X VPS than on the E3 dedi. For the E3 dedi the wall clock "real" time was 22.655 seconds, and, for the Ryzen, 30.658 seconds.

The E3 dedi has SSD Raid 10, and the Ryzen 7950X 2 vCore VPS has NVMe Raid 10.

The E3 is running Ubuntu 22.04.4 LTS. The Ryzen is running Debian 12.7.

Geekbench and fio scores from Yabs on both machines are shown below along with both machines' time results.

Note that both fio and Geekbench scores are significantly higher on the Ryzen 2 vCore VPS than on the E3 Dedi.

I guess the sha256sum -c execution time might depend on how many threads are being used by sha256sum. From both machines' top results, also shown below, it seems like sha256sum might be single threaded.

I haven't seen any steal on the Ryzen VPS, and I was told that it was on a new node.

In the time results shown below, please note that "real", "user", and "sys" each differ.

I decided to try time sha256sum -c also with another, bigger 17 GB file. For the 17 GB file, the wall clock "real" time was 1m18.146s on the E3 dedi and 2m16.960s on the Ryzen. So, again, the E3 Dedi seemed to beat the 2 vCore Ryzen 7950X.

It really seems like I must be missing something basic here! It doesn't seem sensible that the E3 would be faster than the Ryzen for sha256sum -c. But, what explains the time differences? Why does the E3 seem faster?

Both processors have integrated graphics. The E3 graphics are enabled on the bare metal, and the Ryzen graphics are passed through into and are enabled on the Ryzen VPS. But, are the graphics processors involved with sha256sum?

Assuming these results are not from way out in left field, and assuming the lack of some easy explanation that I am missing, does anybody here have experience checking the time expended on the various operations performed by the sha256sum program and also the detailed E3 and Ryzen architectural specifications? Why does E3 seem faster than Ryzen on sha256sum -c?

E3 Dedi

OS: Ubuntu 22.04.4 LTS

fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/mapper/vg0-root):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 199.08 MB/s  (49.7k) | 180.73 MB/s   (2.8k)
Write      | 199.61 MB/s  (49.9k) | 181.68 MB/s   (2.8k)
Total      | 398.69 MB/s  (99.6k) | 362.42 MB/s   (5.6k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 301.57 MB/s    (589) | 323.23 MB/s    (315)
Write      | 317.60 MB/s    (620) | 344.75 MB/s    (336)
Total      | 619.17 MB/s   (1.2k) | 667.98 MB/s    (651)
Geekbench 6 Benchmark Test:
---------------------------------
Test            | Value                         
                |                               
Single Core     | 1344                          
Multi Core      | 4430                          
Full Test       | https://browser.geekbench.com/v6/cpu/6406219
top - 23:16:48 up 1 day,  5:59,  2 users,  load average: 1.21, 0.93, 0.90
Tasks: 216 total,   2 running, 214 sleeping,   0 stopped,   0 zombie
%Cpu0  :  4.7 us,  0.3 sy,  0.0 ni, 94.3 id,  0.3 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu1  :  1.3 us,  2.3 sy,  0.0 ni, 96.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  : 10.6 us,  1.7 sy,  0.0 ni, 87.1 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu3  : 93.0 us,  1.7 sy,  0.0 ni,  5.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  2.0 us,  2.3 sy,  0.0 ni, 95.4 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu5  :  4.0 us,  1.0 sy,  0.0 ni, 94.4 id,  0.3 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu6  :  1.0 us,  1.0 sy,  0.0 ni, 97.3 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu7  :  6.9 us,  2.3 sy,  0.0 ni, 90.5 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
MiB Mem :  64084.4 total,  27018.1 free,   7788.8 used,  29277.5 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  55571.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND       
  58585 root      20   0    5792   1056    968 R 100.0   0.0   0:24.90 sha256sum     
root@E3-Dedi:~# time sha256sum -c chronos-20240904.tgz.cpt.SHA256 
chronos-20240904.tgz.cpt: OK

real    0m22.655s
user    0m22.090s
sys     0m0.564s
root@E3-Dedi:~#
root@E3-Dedi:~# time sha256sum -c Documents.tgz.cpt.SHA256 
Documents.tgz.cpt: OK

real    1m18.146s
user    1m16.245s
sys     0m1.892s
root@E3-Dedi:~# 

Ryzen 7950X 2 vCore VPS

OS: Debian 12.7

fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/sda1):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 331.18 MB/s  (82.7k) | 1.68 GB/s    (26.3k)
Write      | 332.05 MB/s  (83.0k) | 1.69 GB/s    (26.5k)
Total      | 663.23 MB/s (165.8k) | 3.38 GB/s    (52.8k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 4.98 GB/s     (9.7k) | 4.53 GB/s     (4.4k)
Write      | 5.24 GB/s    (10.2k) | 4.83 GB/s     (4.7k)
Total      | 10.23 GB/s   (19.9k) | 9.36 GB/s     (9.1k)
Geekbench 6 Benchmark Test:
---------------------------------
Test            | Value                         
                |                               
Single Core     | 2534                          
Multi Core      | 4414                          
Full Test       | https://browser.geekbench.com/v6/cpu/7541417
top - 23:15:06 up 4 days,  2:11,  2 users,  load average: 0.44, 0.21, 0.09
Tasks:  91 total,   2 running,  89 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.7 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st 
%Cpu1  : 94.6 us,  4.4 sy,  0.0 ni,  0.0 id,  1.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   3915.5 total,    191.6 free,    329.2 used,   3616.6 buff/cache     
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   3586.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND       
  23848 root      20   0    5484    904    812 R  99.0   0.0   0:25.57 sha256sum   
root@Ryzen-2-vCore-VPS:~# time sha256sum -c chronos-20240904.tgz.cpt.SHA256 
chronos-20240904.tgz.cpt: OK

real    0m30.658s
user    0m9.694s
sys     0m0.668s
root@Ryzen-2-vCore-VPS:~# 
root@Ryzen-2-vCore-VPS:~# time sha256sum -c Documents.tgz.cpt.SHA256
Documents.tgz.cpt: OK

real    2m16.960s
user    0m34.364s
sys     0m1.840s
root@Ryzen-2-vCore-VPS:~# 

Comments

  • @sh97 Interesting guess! I completely missed that possibility! Notice, though, that the top result for Ryzen shows

    @Not_Oles said: MiB Mem : 3915.5 total, 191.6 free, 329.2 used, 3616.6 buff/cache

    Since 191.6 MiB is free during the sha256sum -c execution, I am guessing that the process might not be memory constrained.

    But, still, you make a good point that I completely missed. Thank you so much! :star:

  • The time difference may be caused by I/O caching.
    To isolate them:

    1. Reboot the machine.
    2. Run the command twice, recording their times.

    First run timing reflects uncached I/O plus compute.
    Second run timing reflects cached I/O plus compute.

    We accept Karma donations for the last flan. 🍮 affbrr

  • @sh97 Two additional ideas in response to your kind suggestion about RAM:

    • I could rerun the tests with a smaller file, say 1 GB.

    • I could add a swap file to the VPS and see if the numbers change.

  • Guessing some of the cryptography extensions aren't passing through the virt layer cleanly

  • Just now remembering that the E3 is running ext4 with LVM, whereas the Ryzen is running xfs. I haven't yet studied up on the effect of these filesystem differences. Ideas, please?

  • teamaccteamacc OG teamacc

    Are you sure both sha256sum executables are identical, or at least compiled with identical compile flags?

    Hey teamacc. You're a dick. (c) Jon Biloh, 2020.

  • @teamacc said:
    Are you sure both sha256sum executables are identical, or at least compiled with identical compile flags?

    Another excellent point that I completely missed! The executables could be different or could be compiled with different optimizations. :star:

  • The sha256sum command is single-threaded. Virtualization overhead on the VPS, as well as OS and kernel differences, can also contribute to the performance.

  • The Xeon E3 has 4 times the number of "vcores" and aren't being shared between multiple users in the same server, simple as that.

  • @yoursunny said:
    The time difference may be caused by I/O caching.
    To isolate them:

    1. Reboot the machine.
    2. Run the command twice, recording their times.

    First run timing reflects uncached I/O plus compute.
    Second run timing reflects cached I/O plus compute.

    Reboot plus two successive runs.

    root@Ryzen-2-vCore-VPS:~# uptime
     23:29:49 up 3 min,  1 user,  load average: 0.01, 0.03, 0.01
    root@Ryzen-2-vCore-VPS:~# time sha256sum -c chronos-20240904.tgz.cpt.SHA256
    chronos-20240904.tgz.cpt: OK
    
    real    0m11.104s
    user    0m9.832s
    sys     0m0.557s
    root@Ryzen-2-vCore-VPS:~# time sha256sum -c chronos-20240904.tgz.cpt.SHA256
    chronos-20240904.tgz.cpt: OK
    
    real    0m27.489s
    user    0m9.836s
    sys     0m0.518s
    root@Ryzen-2-vCore-VPS:~# 
    

    Another reboot plus another two successive runs.

    root@Ryzen-2-vCore-VPS:~# date -u
    Thu Sep  5 11:36:00 PM UTC 2024
    root@Ryzen-2-vCore-VPS:~# uptime
     23:36:04 up 0 min,  1 user,  load average: 0.07, 0.03, 0.01
    root@Ryzen-2-vCore-VPS:~# time sha256sum -c chronos-20240904.tgz.cpt.SHA256
    chronos-20240904.tgz.cpt: OK
    
    real    0m34.166s
    user    0m9.814s
    sys     0m0.666s
    root@Ryzen-2-vCore-VPS:~# time sha256sum -c chronos-20240904.tgz.cpt.SHA256
    chronos-20240904.tgz.cpt: OK
    
    real    0m34.701s
    user    0m9.942s
    sys     0m0.501s
    root@Ryzen-2-vCore-VPS:~#
    

    Yet another reboot plus yet another two successive runs.

    root@Ryzen-2-vCore-VPS:~# date -u
    Thu Sep  5 11:41:50 PM UTC 2024
    root@Ryzen-2-vCore-VPS:~# uptime
     23:41:55 up 3 min,  1 user,  load average: 0.01, 0.03, 0.00
    root@Ryzen-2-vCore-VPS:~# time sha256sum -c chronos-20240904.tgz.cpt.SHA256
    chronos-20240904.tgz.cpt: OK
    
    real    0m31.797s
    user    0m9.926s
    sys     0m0.604s
    root@Ryzen-2-vCore-VPS:~# time sha256sum -c chronos-20240904.tgz.cpt.SHA256
    chronos-20240904.tgz.cpt: OK
    
    real    0m35.962s
    user    0m9.780s
    sys     0m0.549s
    root@Ryzen-2-vCore-VPS:~#
    

    One more double try.

    root@Ryzen-2-vCore-VPS:~# date -u
    Thu Sep  5 11:48:17 PM UTC 2024
    root@Ryzen-2-vCore-VPS:~# uptime
     23:48:24 up 1 min,  1 user,  load average: 0.02, 0.01, 0.00
    root@Ryzen-2-vCore-VPS:~# time sha256sum -c chronos-20240904.tgz.cpt.SHA256
    chronos-20240904.tgz.cpt: OK
    
    real    0m32.923s
    user    0m9.913s
    sys     0m0.547s
    root@Ryzen-2-vCore-VPS:~# time sha256sum -c chronos-20240904.tgz.cpt.SHA256
    chronos-20240904.tgz.cpt: OK
    
    real    0m35.545s
    user    0m9.917s
    sys     0m0.456s
    root@Ryzen-2-vCore-VPS:~# 
    
    • Why are the times on the first set of reruns here so different? The wall clock "real" time was 0m11.104s for the first run of the first set and 0m27.489s for the second run of the first set.

    • We might expect the second run of each set (the cached run) to be faster if caching is a significant factor. That's not what happened.

    • Indeed, the first run of the first set of reruns here, the Ryzen VPS at 11.104s was the fastest of all the runs, including the original E3 and Ryzen runs, 22.655s and 30.658s, respectively.

    • Having Ryzen take 11 seconds for something an E3 does in 22 seconds seems reasonable. But, here, Ryzen doesn't consistently run that fast in these tests.

    More ideas, please?

  • The sha256sum -c is a CPU intensive task. So is the Geekbench 6 test. For whatever it is worth, the Geekbench 6 test doesn't seem to show as much variation as the sha256sum -c task.

    chronos@penguin:~/servers/Ryzen-2-vCore-VPS$ grep Single yabs*
    yabs-20240817:Single Core     | 2567                          
    yabs-20240819:Single Core     | 2567                          
    yabs-20240822:Single Core     | 2557                          
    yabs-20240828:Single Core     | 2565                          
    yabs-20240829:Single Core     | 2534                          
    yabs-20240905:Single Core     | 2536                          
    chronos@penguin:~/servers/Ryzen-2-vCore-VPS$ 
    

    I haven't yet compared these yabs scores either with other guys' tests of the 7950X or with previous tests on 7950X bare metal servers I had.

  • AuroraZeroAuroraZero Retired
    edited September 2024

    One is a vps with overhead and what not. The other direct pipe no virtualization.

    Run it on a ryzen with no virtualization and you will see the difference.

    The Yeti has left the building.

  • @AuroraZero said:
    One is a vps with overhead and what not. The other direct pipe no virtualization.

    An enticing answer!

    Run it on a ryzen with no virtualization and you will see the difference.

    Here's a guess: consistently 11 seconds.

    I'm surprised that the variation under virtualization is so high! All the way from 11 seconds to 35 seconds for the same job? And, I really don't think the Node is oversold. I really don't think the Neighbors are problematic, either.

    I've been talking with two different providers and with both about their giving me bare metal Ryzen. So we will see what happens.

    Best wishes!

  • What I find strange is that real time differs so much from user+sys time. In a compute bound single-threaded application you would usually expect real time be roughly the same as user+sys time (user time is the time the application uses the CPU to do the computation, sys time is the time the kernel actively does something, like file system overhead). I would expect real time to differ if either the application doesn't get CPU time when it wants the CPU, or if the system is waiting for I/O, but the top snapshot doesn't seem to show that.

    Maybe you could additionally run vmstat 1 in parallel and show that output. Using /usr/bin/time -v (instead of just time) would also include a bit more information.

  • @cmeerw Wow! Thanks for introducing me to GNU time and vmstat! What follows might be the output you requested. I have to study up a lot to understand the output. If you want something more or something different, please let me know. Thanks again for helping!

    root@Ryzen-2-vCore-VPS:~# who am i
    root     pts/2        2024-09-06 05:32 (xxx.xxx.xxx.xxx)
    root@Ryzen-2-vCore-VPS:~# date
    Fri Sep  6 05:38:32 AM UTC 2024
    root@Ryzen-2-vCore-VPS:~# /usr/bin/time -v sha256sum -c chronos-20240904.tgz.cpt.SHA256 
    chronos-20240904.tgz.cpt: OK
            Command being timed: "sha256sum -c chronos-20240904.tgz.cpt.SHA256"
            User time (seconds): 9.87
            System time (seconds): 0.51
            Percent of CPU this job got: 52%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 0:19.97
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 2008
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 1
            Minor (reclaiming a frame) page faults: 103
            Voluntary context switches: 5322
            Involuntary context switches: 450
            Swaps: 0
            File system inputs: 8005456
            File system outputs: 0
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    root@Ryzen-2-vCore-VPS:~# 
    
    root@Ryzen-2-vCore-VPS:~# who am i
    root     pts/0        2024-09-06 05:19 (xxx.xxx.xxx.xxx)
    root@Ryzen-2-vCore-VPS:~# date
    Fri Sep  6 05:38:05 AM UTC 2024
    root@Ryzen-2-vCore-VPS:~# uptime
     05:38:08 up  5:51,  3 users,  load average: 0.00, 0.00, 0.00
    root@Ryzen-2-vCore-VPS:~# vmstat 1
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     1  0      0 2003564   1876 1879832    0    0   489   273  234  120  1  0 99  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  310  172  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  285  149  0  1 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  244  139  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  247  150  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  287  158  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  283  160  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  305  164  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  305  164  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  305  146  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  316  188  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  341  206  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  305  177  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  318  170  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  361  171  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  316  157  0  0 100  0  0
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     2  0      0 2003564   1876 1879832    0    0     0     0  289  184  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  271  144  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  333  186  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  235  123  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  266  134  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  351  135  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  308  149  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  341  139  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  418  183  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  455  190  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  415  179  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  390  158  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  390  174  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  373  180  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     2  320  194  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  280  170  0  0 100  0  0
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     2  0      0 2003564   1876 1879832    0    0     0     0  299  177  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  270  169  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  257  161  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  275  156  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  303  149  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  327  136  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  333  170  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  337  146  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  339  172  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  329  175  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  250  112  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  284  156  0  0 100  0  0
     2  0      0 2003564   1876 1879832    0    0 126204     0  753  620 14  2 57 27  0
     1  1      0 2003564   1876 1879832    0    0 175872     0  885  749 17  2 50 31  0
     1  1      0 2003564   1876 1879832    0    0 165120     0  851  704 16  2 51 32  0
     1  1      0 2003564   1876 1879832    0    0 205824     0 1102  771 22  1 50 28  0
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     3  0      0 2003564   1876 1879832    0    0 213504     0 1131  744 23  2 49 25  0
     2  0      0 2003564   1876 1879832    0    0 210176     0 1065  742 19  2 52 28  0
     1  1      0 2003564   1876 1879832    0    0 212480     0 1140  789 21  2 50 27  0
     2  0      0 2003564   1876 1879832    0    0 221952     0 1213  718 24  1 50 25  0
     2  0      0 2003564   1876 1879832    0    0 223744     0 1188  718 25  1 49 25  0
     2  0      0 2003564   1876 1879832    0    0 213760     0 1197  930 22  2 50 27  0
     2  0      0 2003564   1876 1879832    0    0 221184     0 1332 1113 23  2 49 26  0
     1  1      0 2003564   1876 1879832    0    0 238848     0 1428 1201 24  3 50 24  0
     1  1      0 2003564   1876 1879832    0    0 229632     0 1366 1185 24  2 49 25  0
     1  1      0 2003564   1876 1879832    0    0 233984     0 1376 1215 24  2 49 25  0
     1  1      0 2003564   1876 1879832    0    0 243200     0 1493 1158 26  2 49 23  0
     2  0      0 2003564   1876 1879832    0    0 241920     0 1525 1198 26  3 49 22  0
     2  0      0 2003564   1876 1879832    0    0 240384     1 1477 1283 24  2 50 23  0
     2  0      0 2003564   1876 1879832    0    0 264448     0 1539 1333 27  2 49 21  0
     2  0      0 2003564   1876 1879832    0    0 120876     0 1025  774 37  3 49 11  0
     2  0      0 2003564   1876 1879832    0    0     0     0  472  174 49  1 50  0  0
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     2  0      0 2003564   1876 1879832    0    0  1032     0  405  371  7  1 92  1  0
     1  0      0 2003564   1876 1879832    0    0     0     0  280  147  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  274  133  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  352  194  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  340  174  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  409  182  0  0 100  0  0
     1  0      0 2003564   1876 1879832    0    0     0     0  396  196  0  0 100  0  0
    ^C
    root@Ryzen-2-vCore-VPS:~# date
    Fri Sep  6 05:39:28 AM UTC 2024
    root@Ryzen-2-vCore-VPS:~# 
    
  • edited September 2024

    The vmstat output shows cpu steal time of around ~25% (st column)

    Oops, sorry, the ~25% should be the cpu wait time (wa column), there's a slight shift in the column alignment of the vmstat output.

  • @Not_Oles said: @cmeerw Wow! Thanks for introducing me to GNU time and vmstat! What follows might be the output you requested. I have to study up a lot to understand the output. If you want something more or something different, please let me know. Thanks again for helping!

    So in this case it's spending half the time waiting for I/O (not sure why the top snapshot didn't show that)

    I am assuming the E3 with 64 GB RAM has all the file contents already cached, so doesn't need to do any I/O.

    You could try re-running the tests on the E3, but with clearing the cache beforehand, e.g.

    echo 1 > /proc/sys/vm/drop_caches
    

    see https://unix.stackexchange.com/questions/87908/how-do-you-empty-the-buffers-and-cache-on-a-linux-system

    I would expect user and system times to remain roughly the same, but real time to go way up (as it will also have to wait for I/O now).

  • @tmntwitw @cmeerw I just want to say that I really appreciate you guys commenting here and on others of my threads, I need to learn more, and comments from you guys and others like you are super helpful to me. So thanks again! Much appreciated! Thanks also to LES, our platform which makes Low End Learning possible! <3

  • @cmeerw said:

    So in this case it's spending half the time waiting for I/O

    The numbers in the I/O wait ("wa") second-to-last column are mostly in the twenties. So, why do you say "half?" Shouldn't we say "spending a quarter of the time waiting for I/O?"

    Also, is there something about sha256sum -c that would make it, as compared with other programs, more likely to succumb to or even create I/O wait time issues?

    I suppose I could run fio with vmstat 1 and see whether I/O wait issues also plague fio?

    (not sure why the top snapshot didn't show that)

    Interesting!

    Thanks again, @cmeerw! :star:

  • @Not_Oles said: The numbers in the I/O wait ("wa") second-to-last column are mostly in the twenties. So, why do you say "half?" Shouldn't we say "spending a quarter of the time waiting for I/O?"

    Those are percentages of total CPU time. You have two cores, but one of them is idle (idle column shows roughly 50 %), and the remaining 50 % are split into roughly 24 % user time, 2 % system time and 24 % wait. So one core is only doing real work half the time and waiting for I/O the other half of the time. This is also consistent with what time tells you above: Percent of CPU this job got: 52% (although that percentage is based on single cores, so for multi-threaded applications it could go higher than 100 %).

    So yes, this can get confusing - and other Unixes might do things differently. If I remember correctly, on HP-UX a load average of 1 meant that all CPU cores were just kept busy, whereas on Linux it means that only 1 CPU core is kept busy.

    Also, is there something about sha256sum -c that would make it, as compared with other programs, more likely to succumb to or even create I/O wait time issues?

    Most Unix utilities are single-threaded and do synchronous I/O, so you will see similar things. Ideally, they could do asynchronous I/O, so they could tell the OS to read some data while they are doing real work - and once they are done with a block of data, the next block is already available to continue doing their processing.

  • @Not_Oles said: Also, is there something about sha256sum -c that would make it, as compared with other programs, more likely to succumb to or even create I/O wait time issues?

    BTW, another thing you could do is use dd to read the file (with a bigger block size) and see if that reduces I/O wait (sha256sum seems to read in chunks of 32 kB only). Unfortunately, you can't do that with the -c options, but you can still just try:

    dd if=chronos-20240904.tgz.cpt bs=1M | /usr/bin/time -v sha256sum -
    
  • @cmeerw said: Those are percentages of total CPU time.

    Okay, got it from your saying that the percentages are of total CPU time and thus need to be doubled because there are two CPUs. Thank you!

  • Not_OlesNot_Oles Provider
    edited September 2024

    @cmeerw said: BTW, another thing you could do is use dd to read the file (with a bigger block size) and see if that reduces I/O wait (sha256sum seems to read in chunks of 32 kB only). Unfortunately, you can't do that with the -c options, but you can still just try:

    dd if=chronos-20240904.tgz.cpt bs=1M | /usr/bin/time -v sha256sum -

    @cmeerw Now it's getting to be almost too much fun! :star:

    root@Ryzen-2-vCore-VPS:~# who am i
    root     pts/1        2024-09-06 20:56 (xxx.xxx.xxx.xxx)
    root@Ryzen-2-vCore-VPS:~# date; \
    > dd if=chronos-20240904.tgz.cpt bs=1M | \
    > /usr/bin/time -v sha256sum -c chronos-20240904.tgz.cpt.SHA256
    Fri Sep  6 09:01:18 PM UTC 2024
    chronos-20240904.tgz.cpt: OK
            Command being timed: "sha256sum -c chronos-20240904.tgz.cpt.SHA256"
            User time (seconds): 9.61
            System time (seconds): 0.42
            Percent of CPU this job got: 93%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 0:10.75
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 2052
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 1
            Minor (reclaiming a frame) page faults: 101
            Voluntary context switches: 399
            Involuntary context switches: 63
            Swaps: 0
            File system inputs: 6446344
            File system outputs: 0
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    root@Ryzen-2-vCore-VPS:~# date
    Fri Sep  6 09:01:33 PM UTC 2024
    root@Ryzen-2-vCore-VPS:~# 
    
    root@Ryzen-2-vCore-VPS:~# who am i
    root     pts/0        2024-09-06 20:33 (xxx.xxx.xxx.xxx)
    root@Ryzen-2-vCore-VPS:~# uptime
     21:00:44 up 21:13,  2 users,  load average: 0.00, 0.00, 0.00
    root@Ryzen-2-vCore-VPS:~# date; vmstat 1
    Fri Sep  6 09:01:15 PM UTC 2024
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     1  0      0 129352   1876 3752600    0    0   161    76  172   84  0  0 100  0  0
     1  0      0 129352   1876 3752600    0    0     0     0  348  154  0  0 100  0  0
     1  0      0 129352   1876 3752600    0    0     0     0  275  132  0  0 100  0  0
     1  0      0 129352   1876 3752600    0    0     0     0  299  117  0  0 100  0  0
     2  0      0 129352   1876 3752600    0    0 189784     0 1300  656 20  2 78  1  0
     2  0      0 129352   1876 3752600    0    0 438016     0 2483 1268 43  3 50  4  0
     2  0      0 129352   1876 3752600    0    0 461312     0 2616 1171 47  3 49  1  0
     2  0      0 129352   1876 3752600    0    0 414464     1 2347  997 42  2 50  7  0
     2  0      0 129352   1876 3752600    0    0 444672     0 2471 1087 46  2 50  3  0
     2  0      0 129352   1876 3752600    0    0 453160     0 2550 1121 46  3 50  2  0
     2  0      0 129352   1876 3752600    0    0 451072     0 2579 1199 46  2 50  3  0
     2  0      0 129352   1876 3752600    0    0 273024     0 1759  714 48  2 50  1  0
     2  0      0 129352   1876 3752600    0    0     0     0  520  156 48  3 50  0  0
     2  0      0 129352   1876 3752600    0    0 99928     0  896  737 33  2 50 16  0
     2  0      0 129352   1876 3752600    0    0     0     0  525  150 48  3 50  0  0
     1  0      0 129352   1876 3752600    0    0  1056     0  461  182 16  1 84  0  0
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     1  0      0 129352   1876 3752600    0    0     0     0  363  175  0  0 100  0  0
     1  0      0 129352   1876 3752600    0    0     0     0  310  135  0  0 100  0  0
     1  0      0 129352   1876 3752600    0    0     0     0  426  174  0  0 100  0  0
     1  0      0 129352   1876 3752600    0    0   164     0  419  184  1  0 100  0  0
     1  0      0 129352   1876 3752600    0    0     0     0  394  178  0  0 100  0  0
     1  0      0 129352   1876 3752600    0    0     0     0  241  121  0  0 100  0  0
     1  0      0 129352   1876 3752600    0    0     0     0  345  154  0  0 100  0  0
    ^C
    root@Ryzen-2-vCore-VPS:~# date
    Fri Sep  6 09:01:39 PM UTC 2024
    root@Ryzen-2-vCore-VPS:~# 
    

    Did another run to test consistency.

    root@Ryzen-2-vCore-VPS:~# date; dd if=chronos-20240904.tgz.cpt bs=1M | /usr/bin/time -v sha256sum -c chronos-20240904.tgz.cpt.SHA256; date
    Fri Sep  6 09:17:26 PM UTC 2024
    chronos-20240904.tgz.cpt: OK
            Command being timed: "sha256sum -c chronos-20240904.tgz.cpt.SHA256"
            User time (seconds): 9.84
            System time (seconds): 0.44
            Percent of CPU this job got: 52%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 0:19.62
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 2152
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 1
            Minor (reclaiming a frame) page faults: 103
            Voluntary context switches: 4994
            Involuntary context switches: 572
            Swaps: 0
            File system inputs: 5861048
            File system outputs: 0
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    Fri Sep  6 09:17:46 PM UTC 2024
    root@Ryzen-2-vCore-VPS:~# 
    

    @cmeerw said: sha256sum seems to read in chunks of 32 kB only

    So, chunk size is the answer, at least on the Ryzen VPS?

    I can see how the E3 could just keep on reading because the E3 has plenty of memory. But the VPS, even without much memory, seems to work fast when we use 1 M block size with dd. So it seems that the chunk size issue is at least partly independent of overall memory size. Maybe the root of the issue not in the memory size, but, instead, is in the increased number of read operations when the chunk size is small? How is the chunk size issue (memory size or number of operations, or maybe both) avoided on the E3 Dedi?

    Thanks yet again!

  • From the OP:

    For the E3 dedi the wall clock "real" time was 22.655 seconds, and, for the Ryzen, 30.658 seconds.

    Now, on the Ryzen 2 vCore VPS, using dd to increase block size for sha256sum.

    • First run

    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:10.75

    This looks enough faster than the E3 Dedi. :)

    • Second run

    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:19.62

    But this one is almost twice as long. And not much faster than the E3 Dedi. Why the inconsistency? :)

    2 0 0 129352 1876 3752600 0 0 99928 0 896 737 33 2 50 16 0

    What's this about the 16 in the wait column? :)

  • I added some swap to the Ryzen KVM VPS.

    root@Ryzen-2-vCore-VPS:~# cd /
    root@Ryzen-2-vCore-VPS:/# dd if=/dev/zero of=/swapfile bs=1024 count=4194304
    4194304+0 records in
    4194304+0 records out
    4294967296 bytes (4.3 GB, 4.0 GiB) copied, 7.20403 s, 596 MB/s
    root@Ryzen-2-vCore-VPS:/# ls -l swapfile 
    -rw-r--r-- 1 root root 4294967296 Sep  7 01:04 swapfile
    root@Ryzen-2-vCore-VPS:/# chown root:root swapfile 
    root@Ryzen-2-vCore-VPS:/# chmod 0600 swapfile 
    root@Ryzen-2-vCore-VPS:/# ls -l swapfile 
    -rw------- 1 root root 4294967296 Sep  7 01:04 swapfile
    root@Ryzen-2-vCore-VPS:/# mkswap swapfile
    Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
    no label, UUID=f4c27178-d1e8-4411-bca8-43aeda70da63
    root@Ryzen-2-vCore-VPS:/# swapon /swapfile 
    root@Ryzen-2-vCore-VPS:/# swapon
    NAME      TYPE SIZE USED PRIO
    /swapfile file   4G   0B   -2
    root@Ryzen-2-vCore-VPS:/# 
    
  • I wanted to see whether another file I/O program would show I/O wait like sha256sum. So, just for some quick fun, I ran a yabs, which calls fio. I ran vmstat while the Yabs was running. Note that the interval for this vmstat was 6 seconds instead of 1 second. This time I ran vmstat in the background instead of in a separate terminal. Here's the Yabs result, and I will post the vmstat results in a moment.

    root@Ryzen-2-vCore-VPS:~# vmstat 6 > vmstat-out 2>&1 &
    [1] 9571
    root@Ryzen-2-vCore-VPS:~# curl -sL yabs.sh | bash
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    #              Yet-Another-Bench-Script              #
    #                     v2024-06-09                    #
    # https://github.com/masonr/yet-another-bench-script #
    # ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
    
    Mon Sep  9 12:03:22 AM UTC 2024
    
    Basic System Information:
    ---------------------------------
    Uptime     : 3 days, 16 hours, 0 minutes
    Processor  : AMD Ryzen 9 7950X 16-Core Processor
    CPU cores  : 2 @ 4499.980 MHz
    AES-NI     : ✔ Enabled
    VM-x/AMD-V : ✔ Enabled
    RAM        : 3.8 GiB
    Swap       : 4.0 GiB
    Disk       : 249.9 GiB
    Distro     : Debian GNU/Linux 12 (bookworm)
    Kernel     : 6.1.0-9-amd64
    VM Type    : KVM
    IPv4/IPv6  : ✔ Online / ✔ Online
    
    IPv6 Network Information:
    ---------------------------------
    ISP        : XXXXXXXXXXXXXXXXXXXXXXXX
    ASN        : XXXXXXXXXXXXXXXXXXXXXXXX
    Host       : XXXXXXXXXXXXXXXXXXXXXXXX
    Location   : XXXXXXXXXXXXXXXXX
    Country    : United States
    
    fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/sda1):
    ---------------------------------
    Block Size | 4k            (IOPS) | 64k           (IOPS)
      ------   | ---            ----  | ----           ---- 
    Read       | 455.40 MB/s (113.8k) | 3.85 GB/s    (60.2k)
    Write      | 456.60 MB/s (114.1k) | 3.87 GB/s    (60.6k)
    Total      | 912.00 MB/s (228.0k) | 7.73 GB/s   (120.9k)
               |                      |                     
    Block Size | 512k          (IOPS) | 1m            (IOPS)
      ------   | ---            ----  | ----           ---- 
    Read       | 5.10 GB/s     (9.9k) | 4.41 GB/s     (4.3k)
    Write      | 5.37 GB/s    (10.5k) | 4.70 GB/s     (4.5k)
    Total      | 10.48 GB/s   (20.4k) | 9.11 GB/s     (8.9k)
    
    iperf3 Network Speed Tests (IPv4):
    ---------------------------------
    Provider        | Location (Link)           | Send Speed      | Recv Speed      | Ping           
    -----           | -----                     | ----            | ----            | ----           
    Clouvider       | London, UK (10G)          | 1.75 Gbits/sec  | 1.79 Gbits/sec  | 103 ms         
    Eranium         | Amsterdam, NL (100G)      | 1.74 Gbits/sec  | 1.64 Gbits/sec  | --             
    Uztelecom       | Tashkent, UZ (10G)        | 1.02 Gbits/sec  | 824 Mbits/sec   | 222 ms         
    Leaseweb        | Singapore, SG (10G)       | 923 Mbits/sec   | 785 Mbits/sec   | 220 ms         
    Clouvider       | Los Angeles, CA, US (10G) | 2.92 Gbits/sec  | 4.17 Gbits/sec  | 37.2 ms        
    Leaseweb        | NYC, NY, US (10G)         | 5.28 Gbits/sec  | 4.43 Gbits/sec  | 35.5 ms        
    Edgoo           | Sao Paulo, BR (1G)        | 1.43 Gbits/sec  | 753 Mbits/sec   | 137 ms         
    
    iperf3 Network Speed Tests (IPv6):
    ---------------------------------
    Provider        | Location (Link)           | Send Speed      | Recv Speed      | Ping           
    -----           | -----                     | ----            | ----            | ----           
    Clouvider       | London, UK (10G)          | 1.70 Gbits/sec  | 1.75 Gbits/sec  | 103 ms         
    Eranium         | Amsterdam, NL (100G)      | 2.00 Gbits/sec  | 1.64 Gbits/sec  | 112 ms         
    Uztelecom       | Tashkent, UZ (10G)        | 987 Mbits/sec   | busy            | 205 ms         
    Leaseweb        | Singapore, SG (10G)       | 926 Mbits/sec   | 774 Mbits/sec   | --             
    Clouvider       | Los Angeles, CA, US (10G) | 2.58 Gbits/sec  | 4.08 Gbits/sec  | 36.2 ms        
    Leaseweb        | NYC, NY, US (10G)         | 5.64 Gbits/sec  | 4.31 Gbits/sec  | 35.4 ms        
    Edgoo           | Sao Paulo, BR (1G)        | 1.00 Gbits/sec  | 1.35 Gbits/sec  | 137 ms         
    
    Geekbench 6 Benchmark Test:
    ---------------------------------
    Test            | Value                         
                    |                               
    Single Core     | 2515                          
    Multi Core      | 4447                          
    Full Test       | https://browser.geekbench.com/v6/cpu/7683692
    
    YABS completed in 11 min 53 sec
    root@Ryzen-2-vCore-VPS:~# fg
    vmstat 6 > vmstat-out 2>&1
    ^C
    root@Ryzen-2-vCore-VPS:~# 
    
  • root@Ryzen-2-vCore-VPS:~# cat -n vmstat-out 
         1  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
         2   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
         3   1  0    572 138928   1876 3746624    0    0   116    74  173   84  0  0 100  0  0
         4   1  0    572 138928   1876 3746624    0    0     0     1  339  189  0  0 100  0  0
         5   1  0    572 138928   1876 3746624    0    0     0     1  285  194  0  0 100  0  0
         6   1  2    572 138928   1876 3746624    0    0    49 240391  844 1028  2  4 51 43  0
         7   2  0    572 138928   1876 3746624    0    0 785761 335577 42194 20746  4 21 52 22  0
         8   1  0    572 138928   1876 3746624    0    0 1150402 1193687 27078 16269  5 17 77  0  0
         9   1  0    572 138928   1876 3746624    0    0     0     0 11713  559  0  2 98  0  0
        10   1  0    572 138928   1876 3746624    0    0     1     0 6734  482  0  3 97  0  0
        11   1  0    572 138928   1876 3746624    0    0     0     0 7719 7060  0  5 95  0  0
        12   1  0    572 138928   1876 3746624    0    0     0     0 5803 5314  0  5 95  0  0
        13   1  0    572 138928   1876 3746624    0    0     0    22 15721  526  0  3 97  0  0
        14   1  0    572 138928   1876 3746624    0    0     0     0 9606  452  0  2 98  0  0
        15   1  0    572 138928   1876 3746624    0    0     0     0 15199 12519  0  2 98  0  0
        16   1  0    572 138928   1876 3746624    0    0     0     0 10142 9770  0  2 98  0  0
        17   1  0    572 138928   1876 3746624    0    0     0     0  286  204  0  0 100  0  0
        18   1  0    572 138928   1876 3746624    0    0     0     0 3120  323  0  4 96  0  0
        19   1  0    572 138928   1876 3746624    0    0     0     9 3474  400  0  2 98  0  0
        20   1  0    572 138928   1876 3746624    0    0     0     0  379  233  0  2 98  0  0
        21   1  0    572 138928   1876 3746624    0    0     0     0 5656 4238  0  2 98  0  0
        22   1  0    572 138928   1876 3746624    0    0     0    29 3223 2815  0  2 98  0  0
        23   1  0    572 138928   1876 3746624    0    0     0     0 3872  320  0  4 96  0  0
        24  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        25   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        26   1  0    572 138928   1876 3746624    0    0     0     3 5337  384  0  3 97  0  0
        27   1  0    572 138928   1876 3746624    0    0     0     1 4484 2900  0  2 97  0  0
        28   1  0    572 138928   1876 3746624    0    0     0     0 12163 8441  0  3 97  0  0
        29   1  0    572 138928   1876 3746624    0    0     0     0 12295 2799  0  3 96  0  0
        30   1  0    572 138928   1876 3746624    0    0     0    90 22461  803  0  4 95  1  0
        31   1  0    572 138928   1876 3746624    0    0     0    11 7195 2004  0  3 97  0  0
        32   1  0    572 138928   1876 3746624    0    0     0     0 19901 18090  0  7 93  0  0
        33   1  0    572 138928   1876 3746624    0    0     0    47 18924 9931  0  7 92  0  0
        34   1  0    572 138928   1876 3746624    0    0     0     0 29803 1175  0  8 92  0  0
        35   1  0    572 138928   1876 3746624    0    0     0     0 24200 9551  0  7 93  0  0
        36   1  0    572 138928   1876 3746624    0    0     0     3 36013 24124  0 15 84  0  0
        37   1  0    572 138928   1876 3746624    0    0     0     0 2456  329  0  3 97  0  0
        38   1  0    572 138928   1876 3746624    0    0     0     0 8374  492  0  3 97  0  0
        39   1  0    572 138928   1876 3746624    0    0     0    57 3260  576  0  3 97  1  0
        40   1  0    572 138928   1876 3746624    0    0     0     0 5966 4256  0  2 98  0  0
        41   1  0    572 138928   1876 3746624    0    0     0     2 6094 3897  0  1 98  0  0
        42   1  0    572 138928   1876 3746624    0    0     0     0 7115  433  0  3 97  0  0
        43   1  0    572 138928   1876 3746624    0    0     0     0 8904  515  0  3 97  0  0
        44   1  0    572 138928   1876 3746624    0    0     0     0 7956 6478  0  3 97  0  0
        45   2  0    572 138928   1876 3746624    0    0     0     1 13184 11447  0  4 96  0  0
        46   1  0    572 138928   1876 3746624    0    0     0     2 7723  519  0  2 97  0  0
        47  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        48   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        49   1  0    572 138928   1876 3746624    0    0     0     0 13976  604  0  4 95  0  0
        50   1  0    572 138928   1876 3746624    0    0     0     0 12171 7910  0  1 99  0  0
        51   2  0    572 138928   1876 3746624    0    0     0     0 17823 13196  0  2 98  0  0
        52   1  0    572 138928   1876 3746624    0    0     0     0 2553 1192  1  3 96  0  0
        53   1  0    572 138928   1876 3746624    0    0     0    54 4026  427  0  1 98  1  0
        54   1  0    572 138928   1876 3746624    0    0     0     0 1149  269  0  1 99  0  0
        55   1  0    572 138928   1876 3746624    0    0     0     0  430  222  0  0 100  0  0
        56   1  0    572 138928   1876 3746624    0    0     0     0  581  241  0  4 96  0  0
        57   1  0    572 138928   1876 3746624    0    0     0     0 6243  396  0  2 98  0  0
        58   1  0    572 138928   1876 3746624    0    0     0     5 2993  306  0  2 98  0  0
        59   2  0    572 138928   1876 3746624    0    0     0     1 9506 6072  0  4 96  0  0
        60   1  0    572 138928   1876 3746624    0    0     0     0 7866 6713  0  2 98  0  0
        61   1  0    572 138928   1876 3746624    0    0     0     0  301  182  0  0 100  0  0
        62   1  0    572 138928   1876 3746624    0    0     0     0 4760  361  0  2 98  0  0
        63   1  0    572 138928   1876 3746624    0    0     0     0 24371  700  0  5 95  0  0
        64   2  0    572 138928   1876 3746624    0    0     0     2 19078 7205  0  5 94  0  0
        65   1  0    572 138928   1876 3746624    0    0     0     1 36257 25194  0  8 92  0  0
        66   1  0    572 138928   1876 3746624    0    0     0     0 23524 9861  0  7 93  0  0
        67   1  0    572 138928   1876 3746624    0    0     0     0 31332 1300  0  9 91  0  0
        68   1  0    572 138928   1876 3746624    0    0     0     0 30865 16805  0 10 90  0  0
        69   1  0    572 138928   1876 3746624    0    0     0     3 24246 20366  0  9 91  0  0
        70  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        71   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        72   1  0    572 138928   1876 3746624    0    0     0     0 4794  398  0  3 97  0  0
        73   1  0    572 138928   1876 3746624    0    0     0    52 9151  412  0  2 98  0  0
        74   2  0    572 138928   1876 3746624    0    0     0     0 5880 3104  0  3 96  0  0
        75   1  0    572 138928   1876 3746624    0    0     0     0 14219 8466  0  4 96  0  0
        76   1  0    572 138928   1876 3746624    0    0    31 29333 6856 10894 15  7 77  1  0
        77   2  0    572 138928   1876 3746624    0    0     0 35825  405  198 14  0 84  2  0
        78   2  0    572 138928   1876 3746624    0    0     0    23  509  188 49  2 50  0  0
        79   1  0    572 138928   1876 3746624    0    0     0     0  414  190 14  0 85  0  0
        80   2  0    572 138928   1876 3746624    0    0     0     0  547  204 43  1 56  0  0
        81   1  0    572 138928   1876 3746624    0    0     0    35  483  216 28  1 70  1  0
        82   1  0    572 138928   1876 3746624    0    0     0 16196  400  208  8  0 91  1  0
        83   1  0    572 138928   1876 3746624    0    0     0     0  461  206 26  1 73  0  0
        84   2  0    572 138928   1876 3746624    0    0     0     0  397  212 11  1 88  0  0
        85   1  0    572 138928   1876 3746624    0    0     0     0  407  208 17  0 83  0  0
        86   1  0    572 138928   1876 3746624    0    0     0     9  425  204 24  0 76  0  0
        87   2  0    572 138928   1876 3746624    0    0     0     0  654  283 25  1 74  0  0
        88   1  0    572 138928   1876 3746624    0    0     0     1  613  307 18  1 81  0  0
        89   2  0    572 138928   1876 3746624    0    0     0     0  597  194 39  1 60  0  0
        90   1  0    572 138928   1876 3746624    0    0     0     0  592  204 34  1 66  0  0
        91   1  0    572 138928   1876 3746624    0    0     0    11  405  199  9  1 90  0  0
        92   2  0    572 138928   1876 3746624    0    0     0     0  529  203 23  1 77  0  0
        93  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        94   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        95   2  0    572 138928   1876 3746624    0    0     0     0  347  182  8  0 91  0  0
        96   1  0    572 138928   1876 3746624    0    0     0     1 2999  189 22  1 77  0  0
        97   2  0    572 138928   1876 3746624    0    0     0     0  533  185 36  1 63  0  0
        98   2  0    572 138928   1876 3746624    0    0     0     3  531  186 50  1 50  0  0
        99   2  0    572 138928   1876 3746624    0    0     0     0  404  184  8  1 92  0  0
       100   1  0    572 138928   1876 3746624    0    0     0     0  437  185 31  1 68  0  0
       101   1  0    572 138928   1876 3746624    0    0     0     0 4093  232 19  4 77  0  0
       102   2  0    572 138928   1876 3746624    0    0     0     7  485  211 12  2 86  0  0
       103   1  0    572 138928   1876 3746624    0    0     0     1  428  213 11  1 88  0  0
       104   1  0    572 138928   1876 3746624    0    0     0     0  474  206 35  1 64  0  0
       105   2  0    572 138928   1876 3746624    0    0  8827     0  508  560 28  1 69  1  0
       106   1  0    572 138928   1876 3746624    0    0     0     0  714  850 64  2 34  0  0
       107   3  0    572 138928   1876 3746624    0    0  5206    51  522  220 32  1 65  2  0
       108   1  0    572 138928   1876 3746624    0    0     0     1  500  221 39  1 60  0  0
       109   1  0    572 138928   1876 3746624    1 4898     1  4902 2737 1163 32  3 65  0  0
       110   3  0    572 138928   1876 3746624   10    0   596     0  927  885 53  2 45  0  0
       111   1  0    572 138928   1876 3746624    0    0     0     0 1944  397 47  2 51  0  0
       112   3  0    572 138928   1876 3746624    1    0  2333     0  973  298 56  2 42  1  0
       113   1  0    572 138928   1876 3746624    0    0     0     0  535  242 32  1 68  0  0
       114   3  0    572 138928   1876 3746624  140    0   247     1  702  304 79  0 20  2  0
       115   3  0    572 138928   1876 3746624    0    0     0     0  664  227 100  0  0  0  0
       116  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
       117   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       118   3  0    572 138928   1876 3746624    2    0     2     0  678  228 100  0  0  0  0
       119   2  0    572 138928   1876 3746624   49    0   246     0  437  216 14  0 85  0  0
       120   1  0    572 138928   1876 3746624   23    0    23     0  861  642 42  1 57  0  0
       121   3  0    572 138928   1876 3746624    0    0  2102     0  590  238 41  0 58  0  0
       122   1  0    572 138928   1876 3746624    0    0     0     1  486  233 33  0 67  0  0
       123   1  0    572 138928   1876 3746624    6  231   311   231 1858  373 38  3 60  0  0
       124   1  0    572 138928   1876 3746624    1    0  1895     0  538  249 32  1 67  1  0
       125   1  0    572 138928   1876 3746624    0    0   683     0  439  226 21  1 78  0  0
       126   2  0    572 138928   1876 3746624    0    0    27     0  478  233 26  0 74  0  0
       127   1  0    572 138928   1876 3746624    0    0     0     0  603  262 70  0 29  0  0
       128   3  0    572 138928   1876 3746624    0    0   961     1 2437  229 38  2 60  0  0
       129   3  0    572 138928   1876 3746624    0    0  2339     0  458  208 11  2 87  0  0
       130   2  0    572 138928   1876 3746624    0    0  8039     0  390  204 11  2 86  1  0
       131   1  0    572 138928   1876 3746624    0    0     0     0  420  191 23  2 75  0  0
       132   1  0    572 138928   1876 3746624    0    0   281    22  504  253 36  1 63  0  0
       133   1  0    572 138928   1876 3746624  215    0   217     0  455  299  0  0 100  0  0
       134   1  0    572 138928   1876 3746624    0    0     0     1  344  174  0  0 100  0  0
       135   1  0    572 138928   1876 3746624   74    0    77     1  327  230  0  0 99  0  0
       136   1  0    572 138928   1876 3746624    0    0     0     7  300  176  0  0 100  0  0
       137   1  0    572 138928   1876 3746624    0    0     0     0  268  169  0  0 100  0  0
       138   1  0    572 138928   1876 3746624    0    0     0    78  301  179  0  0 99  1  0
       139  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
       140   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       141   1  0    572 138928   1876 3746624    0    0     0     0  338  177  0  0 100  0  0
       142   1  0    572 138928   1876 3746624    0    0     0     1  294  180  0  0 100  0  0
       143   1  0    572 138928   1876 3746624    0    0     0    44  280  171  0  0 99  1  0
       144   1  0    572 138928   1876 3746624    0    0     0     0  324  189  0  0 100  0  0
       145   1  0    572 138928   1876 3746624    0    0     0     0  356  196  0  0 100  0  0
       146   1  0    572 138928   1876 3746624    0    0     0     0  349  181  0  0 100  0  0
       147   1  0    572 138928   1876 3746624    0    0     0     0  335  193  0  0 100  0  0
       148   1  0    572 138928   1876 3746624    0    0     0     4  318  184  0  0 100  0  0
       149   1  0    572 138928   1876 3746624    0    0     0     0  295  179  0  0 100  0  0
       150   1  0    572 138928   1876 3746624    0    0     0     0  304  168  0  0 100  0  0
       151   1  0    572 138928   1876 3746624    0    0     0     0  312  158  0  0 100  0  0
       152   1  0    572 138928   1876 3746624    0    0     0     0  361  191  0  0 100  0  0
       153   1  0    572 138928   1876 3746624    0    0     0     0  324  176  0  0 100  0  0
       154   1  0    572 138928   1876 3746624    0    0     3     1  291  180  0  0 100  0  0
       155   1  0    572 138928   1876 3746624    0    0     0     0  374  191  0  0 100  0  0
       156   1  0    572 138928   1876 3746624    0    0     0    74  356  192  0  0 99  1  0
       157   1  0    572 138928   1876 3746624    0    0     0     0  316  183  0  0 100  0  0
       158   1  0    572 138928   1876 3746624    0    0     0     0  332  186  0  0 100  0  0
       159   1  1    572 138928   1876 3746624    0    0     0    24  301  185  0  0 100  0  0
       160   1  0    572 138928   1876 3746624    0    0     0    44  324  178  0  0 99  1  0
       161   1  0    572 138928   1876 3746624    0    0     0     0  376  178  0  0 100  0  0
       162  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
       163   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       164   1  0    572 138928   1876 3746624    0    0     0     0  377  190  0  0 100  0  0
       165   1  0    572 138928   1876 3746624    0    0     0     0  332  178  0  0 100  0  0
       166   1  0    572 138928   1876 3746624    0    0     0     3  309  191  0  0 100  0  0
       167   1  0    572 138928   1876 3746624    0    0     0     0  325  179  0  0 100  0  0
       168   1  0    572 138928   1876 3746624    0    0     0     1  347  186  0  0 100  0  0
       169   1  0    572 138928   1876 3746624    0    0     0     0  332  198  0  0 100  0  0
       170   1  0    572 138928   1876 3746624    0    0     0     0  327  185  0  0 100  0  0
       171   1  0    572 138928   1876 3746624    0    0     0     0  359  195  0  0 100  0  0
       172   1  0    572 138928   1876 3746624    0    0     0     0  336  193  0  0 100  0  0
       173   1  0    572 138928   1876 3746624    0    0     0     0  340  184  0  0 100  0  0
       174   1  0    572 138928   1876 3746624    0    0     0     1  354  196  0  0 100  0  0
       175   1  0    572 138928   1876 3746624    0    0     0     0  415  204  0  0 100  0  0
       176   1  0    572 138928   1876 3746624    0    0     0     0  346  199  0  0 100  0  0
       177   1  0    572 138928   1876 3746624    0    0     0     0  348  193  0  0 100  0  0
       178   1  0    572 138928   1876 3746624    0    0     0     0  323  164  0  0 100  0  0
       179   1  0    572 138928   1876 3746624    0    0     0     0  393  202  0  0 100  0  0
       180   1  0    572 138928   1876 3746624    0    0     0     1  344  192  0  0 100  0  0
       181   1  0    572 138928   1876 3746624    0    0     0     0  308  189  0  0 100  0  0
       182   1  0    572 138928   1876 3746624    0    0     0     0  312  178  0  0 100  0  0
       183   1  0    572 138928   1876 3746624    0    0     0     0  339  188  0  0 100  0  0
       184   1  0    572 138928   1876 3746624    0    0     0     0  301  179  0  0 100  0  0
       185  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
       186   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       187   1  0    572 138928   1876 3746624    0    0     0     0  294  179  0  0 100  0  0
       188   1  0    572 138928   1876 3746624    0    0     0    26  270  189  0  0 99  1  0
       189   1  0    572 138928   1876 3746624    0    0     0     0  292  179  0  0 100  0  0
       190   1  0    572 138928   1876 3746624    0    0     0     0  298  174  0  0 100  0  0
       191   1  0    572 138928   1876 3746624    0    0     0     0  333  206  0  0 100  0  0
       192   1  0    572 138928   1876 3746624    0    0     0     0  275  176  0  0 100  0  0
       193   1  0    572 138928   1876 3746624    0    0     0     8  255  174  0  0 100  0  0
       194   1  0    572 138928   1876 3746624    0    0     0     1  294  186  0  0 100  0  0
       195   1  0    572 138928   1876 3746624    0    0     0     0  289  174  0  0 100  0  0
       196   1  0    572 138928   1876 3746624    0    0     0     0  288  183  0  0 100  0  0
       197   1  0    572 138928   1876 3746624    0    0     0     0  301  189  0  0 100  0  0
       198   1  0    572 138928   1876 3746624    0    0     0     0  260  185  0  0 100  0  0
       199   1  0    572 138928   1876 3746624    0    0     0     0  287  180  0  0 100  0  0
       200   1  0    572 138928   1876 3746624    0    0     0     1  305  174  0  0 100  0  0
       201   1  0    572 138928   1876 3746624    0    0     0     0  282  177  0  0 100  0  0
       202   1  0    572 138928   1876 3746624    0    0     0     0  289  175  0  0 100  0  0
       203   1  0    572 138928   1876 3746624    0    0     0     0  251  168  0  0 100  0  0
       204   1  0    572 138928   1876 3746624    0    0     0     1  282  178  0  0 100  0  0
       205   1  0    572 138928   1876 3746624    0    0     0     0  302  168  0  0 100  0  0
       206   1  0    572 138928   1876 3746624    0    0     0     1  275  176  0  0 100  0  0
       207   1  0    572 138928   1876 3746624    0    0     0     0  268  163  0  0 100  0  0
       208  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
       209   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       210   1  0    572 138928   1876 3746624    0    0     0     0  276  191  0  0 100  0  0
       211   1  0    572 138928   1876 3746624    0    0     0     2  277  184  0  0 100  0  0
       212   1  0    572 138928   1876 3746624    0    0     0     6  314  189  0  0 100  0  0
       213   1  0    572 138928   1876 3746624    0    0     0    49  312  199  0  0 99  1  0
       214   1  0    572 138928   1876 3746624    0    0     0     1  285  174  0  0 100  0  0
       215   1  0    572 138928   1876 3746624    0    0     0     0  241  171  0  0 100  0  0
       216   1  0    572 138928   1876 3746624    0    0     0     0  289  188  0  0 100  0  0
       217   1  0    572 138928   1876 3746624    0    0     0     0  295  176  0  0 100  0  0
       218   1  0    572 138928   1876 3746624    0    0     0     0  271  176  0  0 100  0  0
       219   1  0    572 138928   1876 3746624    0    0     0     0  297  183  0  0 100  0  0
       220   1  0    572 138928   1876 3746624    0    0     0     1  248  175  0  0 100  0  0
       221   1  0    572 138928   1876 3746624    0    0     0     3  342  196  0  0 100  0  0
       222   1  0    572 138928   1876 3746624    0    0     0     0  310  174  0  0 100  0  0
       223   1  0    572 138928   1876 3746624    0    0     0     0  273  183  0  0 100  0  0
       224   1  0    572 138928   1876 3746624    0    0     0     0  279  170  0  0 100  0  0
       225   1  0    572 138928   1876 3746624    0    0     0     0  256  182  0  0 100  0  0
       226   1  0    572 138928   1876 3746624    0    0     0     1  297  180  0  0 100  0  0
       227   1  0    572 138928   1876 3746624    0    0     0     0  341  184  0  0 100  0  0
       228   1  0    572 138928   1876 3746624    0    0     0     0  265  172  0  0 100  0  0
       229   1  0    572 138928   1876 3746624    0    0     0     0  290  178  0  0 100  0  0
       230   1  0    572 138928   1876 3746624    0    0     0     0  249  170  0  0 100  0  0
       231  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
       232   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       233   1  0    572 138928   1876 3746624    0    0     0     0  286  181  0  0 100  0  0
       234   1  0    572 138928   1876 3746624    0    0     0     1  314  176  0  0 100  0  0
       235   1  0    572 138928   1876 3746624    0    0     0     0  339  208  0  0 100  0  0
       236   1  0    572 138928   1876 3746624    0    0     0     0  351  196  0  0 100  0  0
    root@Ryzen-2-vCore-VPS:~#
    
  • Not_OlesNot_Oles Provider
    edited September 2024

    6 1 2 572 138928 1876 3746624 0 0 49 240391 844 1028 2 4 51 43 0
    7 2 0 572 138928 1876 3746624 0 0 785761 335577 42194 20746 4 21 52 22 0

    Note that the time interval is 6 seconds. The command to start vmstat was

    root@Ryzen-2-vCore-VPS:~# vmstat 6 > vmstat-out 2>&1 &
    [1] 9571
    root@Ryzen-2-vCore-VPS:~#
    

    Looks like these two lines 6 and 7 show significant I/O wait, 43% and 22%. So it seems that both fio and sha256sum experience significant I/O wait. Why? :)

  • @Not_Oles said: Looks like these two lines 6 and 7 show significant I/O wait, 43% and 22%. So it seems that both fio and sha256sum experience significant I/O wait. Why?

    The whole point of fio is to do I/O at full speed, so it really is just waiting for the OS to do that I/O.

    The other option would be to just not show that column (like some other Unixes) and just show it as "CPU idle" (as the CPU itself isn't doing anything). But then you would probably wonder why the system is showing that it's mostly idle when there is work to do, but you have an I/O bottleneck.

  • The provider of this very nice VPS asked me to retest with RAM doubled to 4 GB. The short answer seems to be that increasing the RAM doesn't help. Even with 4 GB RAM, there still is a lot of I/O wait.

    "very nice VPS" is not sarcasm. It really is a great VPS! And the provider's customer service is great as well!

    root@Ryzen-4-vCore-VPS:~# date; uptime
    Mon Sep  9 07:20:32 PM UTC 2024
     19:20:32 up 7 min,  2 users,  load average: 0.00, 0.00, 0.00
    root@Ryzen-4-vCore-VPS:~# nproc
    4
    root@Ryzen-4-vCore-VPS:~# vmstat 1 > vmstat-out 2>&1 &
    [1] 997
    root@Ryzen-4-vCore-VPS:~# date; dd if=chronos-20240904.tgz.cpt bs=1M | /usr/bin/time -v sha256sum -c chronos-20240904.tgz.cpt.SHA256; date
    Mon Sep  9 07:21:23 PM UTC 2024
    chronos-20240904.tgz.cpt: OK
            Command being timed: "sha256sum -c chronos-20240904.tgz.cpt.SHA256"
            User time (seconds): 9.80
            System time (seconds): 0.67
            Percent of CPU this job got: 32%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.11
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 1984
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 2
            Minor (reclaiming a frame) page faults: 102
            Voluntary context switches: 9061
            Involuntary context switches: 482
            Swaps: 0
            File system inputs: 9567464
            File system outputs: 0
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    Mon Sep  9 07:21:55 PM UTC 2024
    root@Ryzen-4-vCore-VPS:~# fg
    vmstat 1 > vmstat-out 2>&1
    ^C
    root@Ryzen-4-vCore-VPS:~# cat -n vmstat-out 
         1  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
         2   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
         3   1  0      0 3743848   1876 142920    0    0    78    13  109   81  0  0 100  0  0
         4   1  0      0 3743848   1876 142920    0    0     0     0  293  116  0  0 100  0  0
         5   1  0      0 3743848   1876 142920    0    0     0     0  377  153  0  0 100  0  0
         6   1  0      0 3743848   1876 142920    0    0     0     0  260  123  0  0 100  0  0
         7   1  0      0 3743848   1876 142920    0    0     0     0  288  165  0  0 100  0  0
         8   1  0      0 3743848   1876 142920    0    0     0     0  275  158  0  0 100  0  0
         9   1  0      0 3743848   1876 142920    0    0     0     4  256  169  0  0 100  0  0
        10   1  0      0 3743848   1876 142920    0    0     0     0  203  123  0  0 100  0  0
        11   1  0      0 3743848   1876 142920    0    0     0     0  245  131  0  0 100  0  0
        12   1  0      0 3743848   1876 142920    0    0     0     6  313  163  0  0 100  0  0
        13   1  0      0 3743848   1876 142920    0    0     0     0  311  157  0  0 100  0  0
        14   1  0      0 3743848   1876 142920    0    0     0     0  237  100  0  0 100  0  0
        15   1  0      0 3743848   1876 142920    0    0     0     4  280  152  0  0 100  0  0
        16   1  0      0 3743848   1876 142920    0    0     0     0  274  157  0  0 100  0  0
        17   1  0      0 3743848   1876 142920    0    0     0     0  250  137  0  0 100  0  0
        18   1  0      0 3743848   1876 142920    0    0     0     0  287  139  0  0 100  0  0
        19   1  0      0 3743848   1876 142920    0    0     0     0  283  102  0  0 100  0  0
        20   1  0      0 3743848   1876 142920    0    0     0     0  317  160  0  0 100  0  0
        21   1  0      0 3743848   1876 142920    0    0     0     4  316  159  0  0 100  0  0
        22   1  0      0 3743848   1876 142920    0    0     0     0  247  114  0  0 100  0  0
        23   1  0      0 3743848   1876 142920    0    0     0     0  219  111  0  0 100  0  0
        24  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        25   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        26   1  0      0 3743848   1876 142920    0    0     0     0  263  158  0  0 100  0  0
        27   1  0      0 3743848   1876 142920    0    0     0     0  239  135  0  0 100  0  0
        28   1  0      0 3743848   1876 142920    0    0     0     0  266  120  0  0 100  0  0
        29   1  0      0 3743848   1876 142920    0    0     0     4  268  137  0  0 100  0  0
        30   1  0      0 3743848   1876 142920    0    0     0     0  309  156  0  0 100  0  0
        31   1  0      0 3743848   1876 142920    0    0     0     0  236  130  0  0 100  0  0
        32   1  0      0 3743848   1876 142920    0    0     0     0  276  127  0  0 100  0  0
        33   1  0      0 3743848   1876 142920    0    0     0     0  357  191  0  0 100  0  0
        34   2  0      0 3743848   1876 142920    0    0 71380     0  603  466  4  1 87  8  0
        35   1  1      0 3743848   1876 142920    0    0 148736     4  910  751  8  2 74 17  0
        36   1  1      0 3743848   1876 142920    0    0 145152     0  881  706  6  2 75 17  0
        37   1  1      0 3743848   1876 142920    0    0 154368     0  993  747  7  2 74 16  0
        38   1  1      0 3743848   1876 142920    0    0 146176     0  929  767  8  2 74 17  0
        39   1  1      0 3743848   1876 142920    0    0 147968     0  853  687  7  2 74 16  0
        40   1  1      0 3743848   1876 142920    0    0 154624     0  871  764  7  2 74 16  0
        41   1  1      0 3743848   1876 142920    0    0 146688     4  849  727  8  2 73 17  0
        42   1  1      0 3743848   1876 142920    0    0 139008     0  824  708  6  2 74 17  0
        43   1  1      0 3743848   1876 142920    0    0 149504     0  888  736  7  2 74 16  0
        44   1  1      0 3743848   1876 142920    0    0 149248     0  942  740  9  1 74 17  0
        45   1  1      0 3743848   1876 142920    0    0 145408     9  905  741  8  1 74 17  0
        46   2  0      0 3743848   1876 142920    0    0 141056     0  849  786  7  2 74 17  0
        47  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        48   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        49   1  1      0 3743848   1876 142920    0    0 150528     4  930  817  8  3 73 16  0
        50   1  1      0 3743848   1876 142920    0    0 140544     0  824  730  7  2 74 17  0
        51   2  0      0 3743848   1876 142920    0    0 153344     0  876  793  7  3 74 16  0
        52   1  1      0 3743848   1876 142920    0    0 148992     0  884  773  8  2 73 17  0
        53   2  0      0 3743848   1876 142920    0    0 157184     0  893  720  8  2 74 16  0
        54   1  1      0 3743848   1876 142920    0    0 152064     0  915  808  8  3 72 16  0
        55   1  1      0 3743848   1876 142920    0    0 151296     8  926  771  7  2 74 17  0
        56   2  0      0 3743848   1876 142920    0    0 143104     0  869  773  8  2 74 17  0
        57   2  0      0 3743848   1876 142920    0    0 149760     0  888  788  8  3 72 16  0
        58   2  0      0 3743848   1876 142920    0    0 156416     0  854  774  8  2 74 16  0
        59   2  0      0 3743848   1876 142920    0    0 151296     0  865  775  8  2 74 16  0
        60   1  1      0 3743848   1876 142920    0    0 154112     0  988  887  7  2 74 16  0
        61   1  1      0 3743848   1876 142920    0    0 154112     4  961  943  7  2 74 16  0
        62   1  1      0 3743848   1876 142920    0    0 153088     0  970 1008  8  3 73 16  0
        63   1  1      0 3743848   1876 142920    0    0 152320     0  934 1038  6  2 75 17  0
        64   1  1      0 3743848   1876 142920    0    0 144896     0  909  969  7  3 73 17  0
        65   1  1      0 3743848   1876 142920    0    0 140032     0  888 1062  7  1 74 17  0
        66   2  0      0 3743848   1876 142920    0    0 141056     0  930 1093  6  2 74 17  0
        67   1  1      0 3743848   1876 142920    0    0 152576     4  926 1063  8  3 73 16  0
        68   1  0      0 3743848   1876 142920    0    0 99372     0  820  811  5  1 83 11  0
        69   1  0      0 3743848   1876 142920    0    0     0     0  492  213  0  0 100  0  0
        70  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        71   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        72   1  0      0 3743848   1876 142920    0    0     0     0  285  147  0  0 100  0  0
        73   1  0      0 3743848   1876 142920    0    0     0     0  333  197  0  0 100  0  0
        74   1  0      0 3743848   1876 142920    0    0     0     0  322  198  0  0 100  0  0
        75   1  0      0 3743848   1876 142920    0    0     0     4  338  197  0  0 100  0  0
    root@Ryzen-4-vCore-VPS:~# 
    
  • @Not_Oles said:
    The provider of this very nice VPS asked me to retest with RAM doubled to 4 GB. The short answer seems to be that increasing the RAM doesn't help. Even with 4 GB RAM, there still is a lot of I/O wait.

    "very nice VPS" is not sarcasm. It really is a great VPS! And the provider's customer service is great as well!

    root@Ryzen-4-vCore-VPS:~# date; uptime
    Mon Sep  9 07:20:32 PM UTC 2024
     19:20:32 up 7 min,  2 users,  load average: 0.00, 0.00, 0.00
    root@Ryzen-4-vCore-VPS:~# nproc
    4
    root@Ryzen-4-vCore-VPS:~# vmstat 1 > vmstat-out 2>&1 &
    [1] 997
    root@Ryzen-4-vCore-VPS:~# date; dd if=chronos-20240904.tgz.cpt bs=1M | /usr/bin/time -v sha256sum -c chronos-20240904.tgz.cpt.SHA256; date
    Mon Sep  9 07:21:23 PM UTC 2024
    chronos-20240904.tgz.cpt: OK
            Command being timed: "sha256sum -c chronos-20240904.tgz.cpt.SHA256"
            User time (seconds): 9.80
            System time (seconds): 0.67
            Percent of CPU this job got: 32%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.11
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 1984
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 2
            Minor (reclaiming a frame) page faults: 102
            Voluntary context switches: 9061
            Involuntary context switches: 482
            Swaps: 0
            File system inputs: 9567464
            File system outputs: 0
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    Mon Sep  9 07:21:55 PM UTC 2024
    root@Ryzen-4-vCore-VPS:~# fg
    vmstat 1 > vmstat-out 2>&1
    ^C
    root@Ryzen-4-vCore-VPS:~# cat -n vmstat-out 
         1  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
         2   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
         3   1  0      0 3743848   1876 142920    0    0    78    13  109   81  0  0 100  0  0
         4   1  0      0 3743848   1876 142920    0    0     0     0  293  116  0  0 100  0  0
         5   1  0      0 3743848   1876 142920    0    0     0     0  377  153  0  0 100  0  0
         6   1  0      0 3743848   1876 142920    0    0     0     0  260  123  0  0 100  0  0
         7   1  0      0 3743848   1876 142920    0    0     0     0  288  165  0  0 100  0  0
         8   1  0      0 3743848   1876 142920    0    0     0     0  275  158  0  0 100  0  0
         9   1  0      0 3743848   1876 142920    0    0     0     4  256  169  0  0 100  0  0
        10   1  0      0 3743848   1876 142920    0    0     0     0  203  123  0  0 100  0  0
        11   1  0      0 3743848   1876 142920    0    0     0     0  245  131  0  0 100  0  0
        12   1  0      0 3743848   1876 142920    0    0     0     6  313  163  0  0 100  0  0
        13   1  0      0 3743848   1876 142920    0    0     0     0  311  157  0  0 100  0  0
        14   1  0      0 3743848   1876 142920    0    0     0     0  237  100  0  0 100  0  0
        15   1  0      0 3743848   1876 142920    0    0     0     4  280  152  0  0 100  0  0
        16   1  0      0 3743848   1876 142920    0    0     0     0  274  157  0  0 100  0  0
        17   1  0      0 3743848   1876 142920    0    0     0     0  250  137  0  0 100  0  0
        18   1  0      0 3743848   1876 142920    0    0     0     0  287  139  0  0 100  0  0
        19   1  0      0 3743848   1876 142920    0    0     0     0  283  102  0  0 100  0  0
        20   1  0      0 3743848   1876 142920    0    0     0     0  317  160  0  0 100  0  0
        21   1  0      0 3743848   1876 142920    0    0     0     4  316  159  0  0 100  0  0
        22   1  0      0 3743848   1876 142920    0    0     0     0  247  114  0  0 100  0  0
        23   1  0      0 3743848   1876 142920    0    0     0     0  219  111  0  0 100  0  0
        24  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        25   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        26   1  0      0 3743848   1876 142920    0    0     0     0  263  158  0  0 100  0  0
        27   1  0      0 3743848   1876 142920    0    0     0     0  239  135  0  0 100  0  0
        28   1  0      0 3743848   1876 142920    0    0     0     0  266  120  0  0 100  0  0
        29   1  0      0 3743848   1876 142920    0    0     0     4  268  137  0  0 100  0  0
        30   1  0      0 3743848   1876 142920    0    0     0     0  309  156  0  0 100  0  0
        31   1  0      0 3743848   1876 142920    0    0     0     0  236  130  0  0 100  0  0
        32   1  0      0 3743848   1876 142920    0    0     0     0  276  127  0  0 100  0  0
        33   1  0      0 3743848   1876 142920    0    0     0     0  357  191  0  0 100  0  0
        34   2  0      0 3743848   1876 142920    0    0 71380     0  603  466  4  1 87  8  0
        35   1  1      0 3743848   1876 142920    0    0 148736     4  910  751  8  2 74 17  0
        36   1  1      0 3743848   1876 142920    0    0 145152     0  881  706  6  2 75 17  0
        37   1  1      0 3743848   1876 142920    0    0 154368     0  993  747  7  2 74 16  0
        38   1  1      0 3743848   1876 142920    0    0 146176     0  929  767  8  2 74 17  0
        39   1  1      0 3743848   1876 142920    0    0 147968     0  853  687  7  2 74 16  0
        40   1  1      0 3743848   1876 142920    0    0 154624     0  871  764  7  2 74 16  0
        41   1  1      0 3743848   1876 142920    0    0 146688     4  849  727  8  2 73 17  0
        42   1  1      0 3743848   1876 142920    0    0 139008     0  824  708  6  2 74 17  0
        43   1  1      0 3743848   1876 142920    0    0 149504     0  888  736  7  2 74 16  0
        44   1  1      0 3743848   1876 142920    0    0 149248     0  942  740  9  1 74 17  0
        45   1  1      0 3743848   1876 142920    0    0 145408     9  905  741  8  1 74 17  0
        46   2  0      0 3743848   1876 142920    0    0 141056     0  849  786  7  2 74 17  0
        47  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        48   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        49   1  1      0 3743848   1876 142920    0    0 150528     4  930  817  8  3 73 16  0
        50   1  1      0 3743848   1876 142920    0    0 140544     0  824  730  7  2 74 17  0
        51   2  0      0 3743848   1876 142920    0    0 153344     0  876  793  7  3 74 16  0
        52   1  1      0 3743848   1876 142920    0    0 148992     0  884  773  8  2 73 17  0
        53   2  0      0 3743848   1876 142920    0    0 157184     0  893  720  8  2 74 16  0
        54   1  1      0 3743848   1876 142920    0    0 152064     0  915  808  8  3 72 16  0
        55   1  1      0 3743848   1876 142920    0    0 151296     8  926  771  7  2 74 17  0
        56   2  0      0 3743848   1876 142920    0    0 143104     0  869  773  8  2 74 17  0
        57   2  0      0 3743848   1876 142920    0    0 149760     0  888  788  8  3 72 16  0
        58   2  0      0 3743848   1876 142920    0    0 156416     0  854  774  8  2 74 16  0
        59   2  0      0 3743848   1876 142920    0    0 151296     0  865  775  8  2 74 16  0
        60   1  1      0 3743848   1876 142920    0    0 154112     0  988  887  7  2 74 16  0
        61   1  1      0 3743848   1876 142920    0    0 154112     4  961  943  7  2 74 16  0
        62   1  1      0 3743848   1876 142920    0    0 153088     0  970 1008  8  3 73 16  0
        63   1  1      0 3743848   1876 142920    0    0 152320     0  934 1038  6  2 75 17  0
        64   1  1      0 3743848   1876 142920    0    0 144896     0  909  969  7  3 73 17  0
        65   1  1      0 3743848   1876 142920    0    0 140032     0  888 1062  7  1 74 17  0
        66   2  0      0 3743848   1876 142920    0    0 141056     0  930 1093  6  2 74 17  0
        67   1  1      0 3743848   1876 142920    0    0 152576     4  926 1063  8  3 73 16  0
        68   1  0      0 3743848   1876 142920    0    0 99372     0  820  811  5  1 83 11  0
        69   1  0      0 3743848   1876 142920    0    0     0     0  492  213  0  0 100  0  0
        70  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        71   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        72   1  0      0 3743848   1876 142920    0    0     0     0  285  147  0  0 100  0  0
        73   1  0      0 3743848   1876 142920    0    0     0     0  333  197  0  0 100  0  0
        74   1  0      0 3743848   1876 142920    0    0     0     0  322  198  0  0 100  0  0
        75   1  0      0 3743848   1876 142920    0    0     0     4  338  197  0  0 100  0  0
    root@Ryzen-4-vCore-VPS:~# 
    

    You are wondering why IO wait is higher on a VPS with other people sharing it?

    The Yeti has left the building.

  • @AuroraZero said: You are wondering why IO wait is higher on a VPS with other people sharing it?

    I trust the provider 100%. The provider says it's his least busy Node. Maybe there might be nobody else one it, although the provider didn't say that expressly.

    Compared with their Nodes, I often have seen reduced disk performance on Qemu VPSes. Interestingly, network performance on Qemu VPSes seems pretty good when compared with network performance on the Nodes. However, it's not the same with disk I/O. Even using the virtio Qemu feature instead of Qemu's full hardware emulation still doesn't seem to move Qemu VPS disk I/O up toward parity with the Node. For disk I/O there often seems to be a minimum of approximately 15 or 20% performance penalty.

    Previously I was only looking at overall time for disk I/O tasks. Now that @cmeerw kindly introduced me to vmstat, I am able to see the I/O wait numbers. Thus now I can imagine that the disk I/O performance penalty is due to I/O wait.

    @cmeerw said: it really is just waiting for the OS to do that I/O.

    Of course you are right. What I want to understand, though, is why the VPS OS takes longer to do the same file I/O task even using virtio Qemu than the time taken to do the same task directly on the Node without Qemu. Also, why is there no similar penalty for network I/O? The file I/O penalty shown here seems to exist even on empty Nodes with only one test VM running.

    Thanks (yet again!) to your kindness, @cmeerw, I might recently see a little more deeply. Maybe I now need to find a kind person on the Qemu team who might give me a little tour of the code and a little hint about why Qemu seems, even with virtio, to take longer for file I/O.

    Do we have someone here who knows the Qemu code?

  • What I did this morning was create a 2 core, 4 GB VPS on my E3 machine. Then I ran a test on the E3 VPS. Here are the results. Almost no I/O wait!

    Why is there so much I/O wait on the Ryzen VPS results shown above and practically none on the E3 VPS?


    bashvm@E3-VPS:~$ vmstat 1 > vmstat-out 2>&1 & [1] 566 bashvm@E3-VPS:~$ date; dd if=chronos-20240904.tgz.cpt bs=1M | /usr/bin/time -v sha256sum -c 6 Wed Sep 11 17:31:01 UTC 2024 chronos-20240904.tgz.cpt: OK Command being timed: "sha256sum -c chronos-20240904.tgz.cpt.SHA256" User time (seconds): 21.54 System time (seconds): 2.50 Percent of CPU this job got: 99% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:24.05 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 1780 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 1 Minor (reclaiming a frame) page faults: 106 Voluntary context switches: 6 Involuntary context switches: 132 Swaps: 0 File system inputs: 9562160 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 bashvm@E3-VPS:~$ cat vmstat-out procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 71896 12556 1828908 0 0 9238 2323 211 98 5 1 93 0 0 1 0 0 71896 12556 1828908 0 0 0 0 33 39 0 0 100 0 1 1 0 0 71896 12556 1828908 0 0 0 4 24 32 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 24 30 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 21 36 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 23 30 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 24 32 54 0 0 100 1 0 1 0 0 71896 12556 1828908 0 0 0 0 15 18 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 24 36 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 19 26 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 14 19 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 12 31 48 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 34 47 0 0 100 0 0 2 0 0 71896 12556 1828908 0 0 57564 0 367 103 13 4 84 0 0 2 0 0 71896 12556 1828908 0 0 197376 0 1081 159 41 10 49 0 0 2 0 0 71896 12556 1828908 0 0 198400 0 1089 156 44 8 49 0 0 2 0 0 71896 12556 1828908 0 0 197120 0 1087 154 45 7 49 0 0 2 0 0 71896 12556 1828908 0 0 198912 12 1100 167 44 7 48 1 0 2 0 0 71896 12556 1828908 0 0 198400 0 1090 159 44 8 49 0 0 2 0 0 71896 12556 1828908 0 0 197632 0 1082 153 43 8 49 0 0 2 0 0 71896 12556 1828908 0 0 198144 0 1090 148 47 5 49 0 0 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 71896 12556 1828908 0 0 200704 0 1100 169 44 7 49 0 0 2 0 0 71896 12556 1828908 0 0 199936 0 1099 167 45 7 49 0 0 2 0 0 71896 12556 1828908 0 0 193792 12 1070 165 44 7 49 0 0 2 0 0 71896 12556 1828908 0 0 201984 0 1103 169 46 5 49 0 0 2 0 0 71896 12556 1828908 0 0 204032 0 1113 150 45 6 49 0 0 2 0 0 71896 12556 1828908 0 0 198144 0 1095 170 47 4 49 0 0 2 0 0 71896 12556 1828908 0 0 198400 0 1091 160 45 6 49 0 0 2 0 0 71896 12556 1828908 0 0 200960 0 1100 160 47 5 48 0 0 2 0 0 71896 12556 1828908 0 0 199936 12 1104 171 47 4 48 1 0 2 0 0 71896 12556 1828908 0 0 198912 0 1104 184 45 6 48 0 0 2 0 0 71896 12556 1828908 0 0 197888 4 1086 148 46 5 49 0 0 2 0 0 71896 12556 1828908 0 0 201476 0 1113 171 45 7 48 0 0 2 0 0 71896 12556 1828908 0 0 195328 0 1079 153 46 6 49 0 0 2 0 0 71896 12556 1828908 0 0 200960 0 1104 156 45 6 49 0 0 2 0 0 71896 12556 1828908 0 0 200192 20 1103 178 45 6 48 0 0 2 0 0 71896 12556 1828908 0 0 198656 12 1098 157 44 8 49 0 0 1 0 0 71896 12556 1828908 0 0 146256 0 2276 161 33 8 59 0 0 1 0 0 71896 12556 1828908 0 0 0 0 15 23 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 20 34 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 26 39 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 12 21 36 0 0 100 1 0 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 71896 12556 1828908 0 0 0 0 30 33 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 23 26 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 15 23 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 30 47 0 1 100 0 0 1 0 0 71896 12556 1828908 0 0 0 12 30 47 0 0 100 0 0 1 0 0 71896 12556 1828908 0 0 0 0 35 45 0 1 100 0 0 1 0 0 71896 12556 1828908 0 0 76 0 61 70 1 1 99 0 0 1 0 0 71896 12556 1828908 0 0 0 0 18 25 0 0 100 0 0 bashvm@E3-VPS:~$
  • @Not_Oles said: Why is there so much I/O wait on the Ryzen VPS results shown above and practically none on the E3 VPS?

    Actually looking at the Ryzen results again, why is it only doing around 150 MB/s (even with a 1M block size)?

    Could you maybe just do a dd if=chronos-20240904.tgz.cpt of=/dev/null bs=1M? Surely, that should be closer to some GB/s (similar to the fio results you had earlier)

    One other thing that might be an advantage for the E3 VPS could be that it still benefits from the host system's cache (depending on how the storage is set up)

  • The kind provider and I finally got around to testing, as @zakkuuno suggested on OGF, with increased RAM on the Ryzen VPS.

    Bingo! No I/O wait!

    Why does the Ryzen VPS need more RAM than the E3 VPS needs?

    root@Ryzen-2-vCore-VPS:~# # Retest with RAM increase
    root@Ryzen-2-vCore-VPS:~# free -h
                   total        used        free      shared  buff/cache   available
    Mem:           7.8Gi       360Mi       2.9Gi       524Ki       4.7Gi       7.4Gi
    Swap:             0B          0B          0B
    root@Ryzen-2-vCore-VPS:~# ls -l chronos-20240904.tgz.cpt
    -rw------- 1 root root 4900030457 Sep  4 21:29 chronos-20240904.tgz.cpt
    root@Ryzen-2-vCore-VPS:~# vmstat 1 > vmstat-out 2>&1 &
    [1] 1042
    root@Ryzen-2-vCore-VPS:~# date; dd if=chronos-20240904.tgz.cpt bs=1M | /usr/bin/time -v sha256sum -c chronos-20240904.tgz.cpt.SHA256; date
    Wed Sep 11 07:41:17 PM UTC 2024
    chronos-20240904.tgz.cpt: OK
            Command being timed: "sha256sum -c chronos-20240904.tgz.cpt.SHA256"
            User time (seconds): 9.74
            System time (seconds): 0.23
            Percent of CPU this job got: 99%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.99
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 1960
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 0
            Minor (reclaiming a frame) page faults: 101
            Voluntary context switches: 2
            Involuntary context switches: 99
            Swaps: 0
            File system inputs: 8
            File system outputs: 0
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    Wed Sep 11 07:41:27 PM UTC 2024
    root@Ryzen-2-vCore-VPS:~# fg
    vmstat 1 > vmstat-out 2>&1
    ^C
    root@Ryzen-2-vCore-VPS:~# cat -n vmstat-out 
         1  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
         2   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
         3   1  0      0 3020332   1876 4974188    0    0   646     4  181  150  0  0 100  0  0
         4   1  0      0 3020332   1876 4974188    0    0     0     0  271  153  0  0 100  0  0
         5   1  0      0 3020332   1876 4974188    0    0     0     0  263  138  0  0 100  0  0
         6   1  0      0 3020332   1876 4974188    0    0     0     0  339  163  0  0 100  0  0
         7   2  0      0 3020332   1876 4974188    0    0     4     0  350  179 11  1 89  0  0
         8   2  0      0 3020332   1876 4974188    0    0     0     0  599  159 48  2 50  0  0
         9   2  0      0 3020332   1876 4974188    0    0     0     4  524  137 49  1 50  0  0
        10   2  0      0 3020332   1876 4974188    0    0     0     0  581  144 49  1 50  0  0
        11   2  0      0 3020332   1876 4974188    0    0     0     0  591  146 48  2 50  0  0
        12   2  0      0 3020332   1876 4974188    0    0     0     0  611  143 49  2 49  0  0
        13   2  0      0 3020332   1876 4974188    0    0     0     0  635  168 48  2 50  0  0
        14   2  0      0 3020332   1876 4974188    0    0     0     0  666  149 49  1 50  0  0
        15   2  0      0 3020332   1876 4974188    0    0     0     4  575  140 49  2 50  0  0
        16   2  0      0 3020332   1876 4974188    0    0     0     0  591  140 49  1 50  0  0
        17   1  0      0 3020332   1876 4974188    0    0     0     0  466 1527 38  1 62  0  0
        18   1  0      0 3020332   1876 4974188    0    0     0     0  306  147  0  0 100  0  0
        19   1  0      0 3020332   1876 4974188    0    0     0     0  263  137  0  0 100  0  0
        20   1  0      0 3020332   1876 4974188    0    0     0     0  257  170  0  0 100  0  0
        21   1  0      0 3020332   1876 4974188    0    0     0     4  323  134  0  0 100  0  0
        22   1  0      0 3020332   1876 4974188    0    0     0     0  267  118  0  0 100  0  0
        23   1  0      0 3020332   1876 4974188    0    0     0     0  276  102  0  0 100  0  0
        24  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
        25   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
        26   1  0      0 3020332   1876 4974188    0    0     0     0  342  153  0  0 100  0  0
    root@Ryzen-2-vCore-VPS:~# 
    
  • bashvm@E3-VPS:~$ date; dd if=chronos-20240904.tgz.cpt bs=1M | /usr/bin/time -v sha256sum -c 6

    This line is a weird cut and paste error. It should have been

    bashvm@E3-VPS:~$ date; dd if=chronos-20240904.tgz.cpt bs=1M | /usr/bin/time -v sha256sum -c chronos-20240904.tgz.cpt.SHA256; date

  • @Not_Oles said: Bingo! No I/O wait!

    Right, there is no I/O wait, because there is no I/O now ("bi" column in vmstat are all "0" - because it's cached now).

  • @Not_Oles said: Why does the Ryzen VPS need more RAM than the E3 VPS needs?

    Possible answer: The Ryzen needs more RAM because it is using NVMe instead of the SSD that is in the E3? So it's not the single threaded sha256sum that makes the difference here? Instead it's the difference in the disk setup?

  • The time discrepancies you're seeing when running sha256sum -c between your E3 dedicated server and Ryzen 7950X VPS are puzzling, especially given that the Ryzen 7950X is a far more powerful processor and is equipped with NVMe storage. However, there are multiple factors that could explain why the E3 dedicated server is outperforming the Ryzen VPS for this specific task, despite the hardware advantages of the VPS.

    Let's break this down:

    1. Single-threaded Nature of sha256sum
      The sha256sum utility, as you're observing, is likely single-threaded, meaning it can only use one CPU core for computing the checksum at a time. This explains why even a more powerful CPU like the Ryzen 7950X may not show its full potential in this case. It's not leveraging multiple cores, so the additional power of the Ryzen CPU is not fully utilized.
      In your case, the E3 has fewer but dedicated cores with no virtualization overhead, and that may be enough to perform similarly or even better for single-threaded workloads like sha256sum.

    2. VPS Virtualization Overhead
      Even though the Ryzen VPS has powerful hardware, virtualization adds a layer of overhead. This includes managing the CPU, memory, and I/O operations. The performance of vCores in a VPS may not match the physical cores of a dedicated server, even if the VPS hardware is theoretically faster.
      The dedicated server on the E3 is running without any virtualization overhead, so it can access the hardware directly without hypervisor intervention. This direct access can make a noticeable difference in I/O-heavy operations like checksum verification.

    3. I/O Performance (SSD RAID 10 vs NVMe RAID 10)
      You mentioned that the E3 dedicated server has SSD RAID 10, while the Ryzen VPS has NVMe RAID 10. Although NVMe is generally faster than SSD, performance in a VPS can suffer due to I/O throttling, contention with other VPSs, or underlying shared infrastructure that limits the maximum throughput of the storage system.
      RAID 10 on SSDs in your dedicated E3 might be very well-optimized and not subject to any sharing. Meanwhile, the NVMe RAID 10 in the VPS could be shared across multiple tenants. Even if it's fast, contention for resources might slow down the actual I/O performance you're seeing on the VPS.
      Impact on sha256sum -c: Since this command reads through the entire file, the I/O performance will heavily affect the overall execution time. If the VPS has any bottlenecks or restrictions on its NVMe performance, this could lead to slower I/O compared to the dedicated SSD RAID 10 setup.

    4. Different Operating Systems
      The E3 is running Ubuntu 22.04.4 LTS, while the Ryzen VPS is running Debian 12.7. While both are Linux distributions, there might be differences in file system tuning, kernel versions, or I/O scheduling mechanisms that could impact how efficiently each system handles large file reads and checksumming operations.
      The Ubuntu installation on the E3 could have a more optimized I/O scheduler for this specific workload, or it might handle file caching differently than Debian on the VPS.

    5. I/O Caching and File System Differences
      The file system in use (ext4, XFS, etc.) could impact performance as well. Different file systems have varying levels of efficiency for reading large files. In addition, Linux systems often cache recently accessed files in memory, so depending on when and how the sha256sum -c was run, the E3 might have benefited from better caching of the file in memory compared to the Ryzen VPS.
      RAID configurations on the VPS may also introduce additional latency due to the abstraction layer needed to manage the RAID arrays. Even though NVMe is fast, the RAID controller (especially in virtualized environments) can sometimes add latency.

    6. Disk I/O Saturation and Contention
      On a VPS, you're often sharing resources like the disk subsystem with other virtual machines. Even if your VPS has NVMe storage, if other users on the same physical machine are performing heavy disk operations at the same time, your VPS might experience lower-than-expected I/O performance.
      This would explain why real-world performance on the VPS is lower than expected, even though benchmarks like Geekbench and fio show the Ryzen VPS is generally faster in isolated tests.

    7. real, user, and sys Time Differences
      The "real" time is the actual elapsed time from start to finish of the command.
      The "user" time refers to the amount of time the CPU spent executing user-space (application-level) code.
      The "sys" time is the amount of time the CPU spent executing system-level (kernel) operations, such as I/O operations.
      If you see a significant increase in sys time on the VPS, it could indicate that more time is being spent on I/O operations, which would make sense if the I/O system on the VPS is slower due to contention, throttling, or other factors.

    8. Graphics Processors (GPU)
      The graphics processors (iGPUs) on either system are not involved in the sha256sum operation. SHA-256 hashing is a CPU-bound operation, and the typical sha256sum utility does not leverage GPU acceleration unless specifically configured to do so (which is not common with basic file checksum verification tasks).

    9. CPU Frequency Differences
      Although the Ryzen 7950X is much newer and faster overall, the E3's CPU might have a higher base clock speed for a single core, or it might sustain its boost frequencies for longer periods under load, particularly since it's dedicated hardware. The Ryzen VPS, on the other hand, might throttle CPU frequency slightly in a virtualized environment to balance resource usage across tenants, impacting single-threaded tasks like sha256sum.

    Conclusion:
    The E3 dedicated server is outperforming the Ryzen 7950X VPS in your sha256sum -c tests due to a combination of factors:

    Single-threaded nature of sha256sum – the additional cores and raw power of the Ryzen CPU are not fully utilized.

    Virtualization overhead – the VPS introduces some overhead that affects I/O performance and CPU efficiency.

    I/O performance bottlenecks – while the VPS uses NVMe storage, it may be throttled or subject to resource contention, whereas the E3 has direct and optimized access to its SSD RAID 10 array.

    File system and kernel differences – slight differences in how the operating systems handle I/O could also contribute to the disparity.
    Your results are not unusual given these factors, and the key issue seems to be the VPS’s I/O performance and the single-threaded nature of sha256sum -c.

  • imokimok OG Not Administrator

    @hostmaze shut up AI!

Sign In or Register to comment.