== sanity test 398b: DIO and buffer IO race ============== 18:47:59 (1710110879) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 0.985357 s, 51.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13405: Sun Mar 10 18:48:15 2024 read: IOPS=114, BW=458KiB/s (469kB/s)(5992KiB/13088msec) clat (usec): min=44, max=140228, avg=4810.21, stdev=4403.37 lat (usec): min=44, max=140230, avg=4811.52, stdev=4404.66 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 766], 10.00th=[ 2245], 20.00th=[ 2737], | 30.00th=[ 3195], 40.00th=[ 3621], 50.00th=[ 4178], 60.00th=[ 4883], | 70.00th=[ 5604], 80.00th=[ 6652], 90.00th=[ 7898], 95.00th=[ 9503], | 99.00th=[ 14484], 99.50th=[ 16319], 99.90th=[ 21365], 99.95th=[139461], | 99.99th=[139461] bw ( KiB/s): min= 360, max= 600, per=25.41%, avg=456.69, stdev=64.38, samples=26 iops : min= 90, max= 150, avg=114.08, stdev=16.15, samples=26 write: IOPS=120, BW=481KiB/s (493kB/s)(6296KiB/13088msec) clat (usec): min=355, max=19919, avg=3710.64, stdev=3242.35 lat (usec): min=355, max=19920, avg=3714.04, stdev=3244.79 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 498], 10.00th=[ 537], 20.00th=[ 676], | 30.00th=[ 906], 40.00th=[ 2278], 50.00th=[ 3097], 60.00th=[ 3818], | 70.00th=[ 4752], 80.00th=[ 6259], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[14091], 99.50th=[15139], 99.90th=[18482], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 384, max= 624, per=25.47%, avg=479.42, stdev=60.71, samples=26 iops : min= 96, max= 156, avg=119.77, stdev=15.26, samples=26 lat (usec) : 50=0.03%, 100=2.05%, 250=0.33%, 500=2.70%, 750=9.47% lat (usec) : 1000=4.43% lat (msec) : 2=3.94%, 4=31.67%, 10=40.59%, 20=4.72%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.59%, ctx=6402, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13406: Sun Mar 10 18:48:15 2024 read: IOPS=113, BW=454KiB/s (465kB/s)(5940KiB/13085msec) clat (usec): min=41, max=158836, avg=4859.59, stdev=4916.07 lat (usec): min=42, max=158837, avg=4860.82, stdev=4916.23 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 105], 10.00th=[ 2180], 20.00th=[ 2671], | 30.00th=[ 3097], 40.00th=[ 3556], 50.00th=[ 4228], 60.00th=[ 4948], | 70.00th=[ 5735], 80.00th=[ 6652], 90.00th=[ 8160], 95.00th=[ 9765], | 99.00th=[ 15139], 99.50th=[ 16909], 99.90th=[ 23462], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 368, max= 640, per=25.22%, avg=453.12, stdev=57.20, samples=26 iops : min= 92, max= 160, avg=113.19, stdev=14.30, samples=26 write: IOPS=121, BW=485KiB/s (497kB/s)(6348KiB/13085msec) clat (usec): min=383, max=21753, avg=3673.57, stdev=3290.30 lat (usec): min=384, max=21754, avg=3674.62, stdev=3290.32 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 506], 10.00th=[ 562], 20.00th=[ 693], | 30.00th=[ 889], 40.00th=[ 1975], 50.00th=[ 3032], 60.00th=[ 3720], | 70.00th=[ 4686], 80.00th=[ 6194], 90.00th=[ 8356], 95.00th=[10159], | 99.00th=[14353], 99.50th=[15664], 99.90th=[19792], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 352, max= 704, per=25.65%, avg=482.65, stdev=84.25, samples=26 iops : min= 88, max= 176, avg=120.58, stdev=21.12, samples=26 lat (usec) : 50=0.07%, 100=2.21%, 250=0.49%, 500=2.34%, 750=10.09% lat (usec) : 1000=4.69% lat (msec) : 2=4.49%, 4=31.22%, 10=39.58%, 20=4.69%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=23.09%, ctx=6418, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13407: Sun Mar 10 18:48:15 2024 read: IOPS=115, BW=461KiB/s (472kB/s)(6028KiB/13066msec) clat (usec): min=42, max=27815, avg=4713.45, stdev=2770.14 lat (usec): min=42, max=27816, avg=4715.31, stdev=2769.93 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 97], 10.00th=[ 2180], 20.00th=[ 2671], | 30.00th=[ 3097], 40.00th=[ 3556], 50.00th=[ 4178], 60.00th=[ 5014], | 70.00th=[ 5800], 80.00th=[ 6652], 90.00th=[ 8029], 95.00th=[ 9372], | 99.00th=[13829], 99.50th=[15533], 99.90th=[21365], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 360, max= 632, per=25.62%, avg=460.42, stdev=68.73, samples=26 iops : min= 90, max= 158, avg=115.00, stdev=17.21, samples=26 write: IOPS=119, BW=479KiB/s (491kB/s)(6260KiB/13066msec) clat (usec): min=429, max=156190, avg=3784.96, stdev=5035.77 lat (usec): min=430, max=156191, avg=3787.43, stdev=5035.99 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 510], 10.00th=[ 562], 20.00th=[ 676], | 30.00th=[ 914], 40.00th=[ 2147], 50.00th=[ 3032], 60.00th=[ 3752], | 70.00th=[ 4817], 80.00th=[ 6390], 90.00th=[ 8094], 95.00th=[ 10028], | 99.00th=[ 13304], 99.50th=[ 16057], 99.90th=[ 23200], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 368, max= 592, per=25.36%, avg=477.35, stdev=56.82, samples=26 iops : min= 92, max= 148, avg=119.23, stdev=14.24, samples=26 lat (usec) : 50=0.07%, 100=2.57%, 250=0.29%, 500=2.21%, 750=10.32% lat (usec) : 1000=3.78% lat (msec) : 2=4.23%, 4=31.87%, 10=40.10%, 20=4.36%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.11%, ctx=6351, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13408: Sun Mar 10 18:48:15 2024 read: IOPS=113, BW=453KiB/s (464kB/s)(6052KiB/13358msec) clat (usec): min=41, max=27699, avg=4837.14, stdev=2920.50 lat (usec): min=41, max=27700, avg=4838.47, stdev=2920.60 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 1762], 10.00th=[ 2278], 20.00th=[ 2704], | 30.00th=[ 3163], 40.00th=[ 3621], 50.00th=[ 4293], 60.00th=[ 5014], | 70.00th=[ 5800], 80.00th=[ 6718], 90.00th=[ 8094], 95.00th=[ 9372], | 99.00th=[15795], 99.50th=[17433], 99.90th=[26084], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 304, max= 560, per=24.87%, avg=446.92, stdev=69.16, samples=26 iops : min= 76, max= 140, avg=111.65, stdev=17.32, samples=26 write: IOPS=116, BW=467KiB/s (478kB/s)(6236KiB/13358msec) clat (usec): min=397, max=137877, avg=3852.46, stdev=4839.29 lat (usec): min=398, max=137878, avg=3853.53, stdev=4839.31 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 502], 10.00th=[ 545], 20.00th=[ 652], | 30.00th=[ 832], 40.00th=[ 2212], 50.00th=[ 3064], 60.00th=[ 3851], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8356], 95.00th=[ 10421], | 99.00th=[ 14877], 99.50th=[ 17695], 99.90th=[ 28967], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 336, max= 688, per=24.54%, avg=461.92, stdev=73.96, samples=26 iops : min= 84, max= 172, avg=115.42, stdev=18.48, samples=26 lat (usec) : 50=0.07%, 100=1.92%, 250=0.33%, 500=2.41%, 750=10.87% lat (usec) : 1000=3.97% lat (msec) : 2=3.22%, 4=31.18%, 10=40.98%, 20=4.79%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.04%, ctx=6315, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1798KiB/s (1841kB/s), 453KiB/s-461KiB/s (464kB/s-472kB/s), io=23.4MiB (24.6MB), run=13066-13358msec WRITE: bw=1882KiB/s (1927kB/s), 467KiB/s-485KiB/s (478kB/s-497kB/s), io=24.6MiB (25.7MB), run=13066-13358msec rand-rw: (groupid=0, jobs=1): err= 0: pid=13400: Sun Mar 10 18:48:18 2024 read: IOPS=90, BW=363KiB/s (372kB/s)(5992KiB/16504msec) clat (usec): min=1553, max=161650, avg=4450.09, stdev=5372.14 lat (usec): min=1554, max=161651, avg=4451.10, stdev=5372.17 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 2057], | 30.00th=[ 2212], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3359], | 70.00th=[ 4293], 80.00th=[ 6194], 90.00th=[ 9241], 95.00th=[ 12125], | 99.00th=[ 17171], 99.50th=[ 19530], 99.90th=[ 29230], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 200, max= 776, per=24.97%, avg=362.88, stdev=180.53, samples=33 iops : min= 50, max= 194, avg=90.52, stdev=45.22, samples=33 write: IOPS=95, BW=381KiB/s (391kB/s)(6296KiB/16504msec) clat (usec): min=2134, max=32295, avg=6216.55, stdev=3917.36 lat (usec): min=2134, max=32296, avg=6217.64, stdev=3917.34 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2802], 10.00th=[ 2999], 20.00th=[ 3228], | 30.00th=[ 3458], 40.00th=[ 3851], 50.00th=[ 4555], 60.00th=[ 5735], | 70.00th=[ 7242], 80.00th=[ 9110], 90.00th=[11863], 95.00th=[13829], | 99.00th=[19792], 99.50th=[22938], 99.90th=[26870], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 216, max= 816, per=25.05%, avg=381.03, stdev=185.28, samples=33 iops : min= 54, max= 204, avg=95.06, stdev=46.40, samples=33 lat (msec) : 2=8.20%, 4=46.87%, 10=32.42%, 20=11.88%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.57%, ctx=3639, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13402: Sun Mar 10 18:48:18 2024 read: IOPS=90, BW=362KiB/s (371kB/s)(5940KiB/16394msec) clat (usec): min=1516, max=135518, avg=4301.86, stdev=4640.41 lat (usec): min=1517, max=135519, avg=4302.68, stdev=4640.41 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1926], 20.00th=[ 2057], | 30.00th=[ 2245], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3326], | 70.00th=[ 4146], 80.00th=[ 5932], 90.00th=[ 9372], 95.00th=[ 11469], | 99.00th=[ 15401], 99.50th=[ 16909], 99.90th=[ 20579], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 176, max= 832, per=24.65%, avg=358.09, stdev=185.43, samples=32 iops : min= 44, max= 208, avg=89.41, stdev=46.42, samples=32 write: IOPS=96, BW=387KiB/s (397kB/s)(6348KiB/16394msec) clat (usec): min=2221, max=30044, avg=6272.06, stdev=3910.25 lat (usec): min=2223, max=30045, avg=6273.14, stdev=3910.28 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2737], 10.00th=[ 2999], 20.00th=[ 3228], | 30.00th=[ 3490], 40.00th=[ 3916], 50.00th=[ 4686], 60.00th=[ 5932], | 70.00th=[ 7570], 80.00th=[ 9372], 90.00th=[11469], 95.00th=[13698], | 99.00th=[19006], 99.50th=[25297], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 232, max= 776, per=24.79%, avg=377.09, stdev=169.99, samples=32 iops : min= 58, max= 194, avg=94.16, stdev=42.55, samples=32 lat (msec) : 2=7.58%, 4=47.10%, 10=32.39%, 20=12.40%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=10.57%, ctx=3619, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13403: Sun Mar 10 18:48:18 2024 read: IOPS=91, BW=365KiB/s (374kB/s)(6028KiB/16524msec) clat (usec): min=1604, max=29562, avg=4454.07, stdev=3575.17 lat (usec): min=1605, max=29563, avg=4454.96, stdev=3575.17 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 2024], | 30.00th=[ 2180], 40.00th=[ 2474], 50.00th=[ 2868], 60.00th=[ 3523], | 70.00th=[ 4555], 80.00th=[ 6521], 90.00th=[10028], 95.00th=[12125], | 99.00th=[16581], 99.50th=[19268], 99.90th=[26870], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 200, max= 816, per=25.08%, avg=364.48, stdev=185.45, samples=33 iops : min= 50, max= 204, avg=91.03, stdev=46.40, samples=33 write: IOPS=94, BW=379KiB/s (388kB/s)(6260KiB/16524msec) clat (msec): min=2, max=135, avg= 6.23, stdev= 5.10 lat (msec): min=2, max=135, avg= 6.24, stdev= 5.10 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 22], 99.50th=[ 24], 99.90th=[ 34], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 183, max= 808, per=24.86%, avg=378.06, stdev=191.16, samples=33 iops : min= 45, max= 202, avg=94.42, stdev=47.82, samples=33 lat (msec) : 2=8.66%, 4=45.25%, 10=33.40%, 20=11.88%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=10.60%, ctx=3576, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13404: Sun Mar 10 18:48:18 2024 read: IOPS=91, BW=366KiB/s (375kB/s)(6052KiB/16517msec) clat (usec): min=1463, max=26763, avg=4356.77, stdev=3495.40 lat (usec): min=1464, max=26763, avg=4357.64, stdev=3495.36 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1909], 20.00th=[ 2040], | 30.00th=[ 2212], 40.00th=[ 2540], 50.00th=[ 2868], 60.00th=[ 3392], | 70.00th=[ 4359], 80.00th=[ 6063], 90.00th=[ 9765], 95.00th=[11600], | 99.00th=[17171], 99.50th=[19530], 99.90th=[25822], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 191, max= 800, per=25.20%, avg=366.12, stdev=183.86, samples=33 iops : min= 47, max= 200, avg=91.39, stdev=46.03, samples=33 write: IOPS=94, BW=378KiB/s (387kB/s)(6236KiB/16517msec) clat (msec): min=2, max=136, avg= 6.33, stdev= 5.21 lat (msec): min=2, max=136, avg= 6.33, stdev= 5.21 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 26], 99.90th=[ 34], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 208, max= 784, per=24.77%, avg=376.79, stdev=186.76, samples=33 iops : min= 52, max= 196, avg=94.06, stdev=46.77, samples=33 lat (msec) : 2=8.46%, 4=45.87%, 10=32.62%, 20=12.37%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=10.44%, ctx=3682, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1453KiB/s (1488kB/s), 362KiB/s-366KiB/s (371kB/s-375kB/s), io=23.4MiB (24.6MB), run=16394-16524msec WRITE: bw=1521KiB/s (1558kB/s), 378KiB/s-387KiB/s (387kB/s-397kB/s), io=24.6MiB (25.7MB), run=16394-16524msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13435: Sun Mar 10 18:48:25 2024 read: IOPS=55, BW=887KiB/s (909kB/s)(5920KiB/6672msec) clat (usec): min=65, max=122349, avg=11432.89, stdev=6856.16 lat (usec): min=66, max=122350, avg=11433.75, stdev=6856.16 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 7177], 10.00th=[ 7767], 20.00th=[ 8848], | 30.00th=[ 9634], 40.00th=[ 10290], 50.00th=[ 10945], 60.00th=[ 11600], | 70.00th=[ 12256], 80.00th=[ 13173], 90.00th=[ 14615], 95.00th=[ 16909], | 99.00th=[ 27132], 99.50th=[ 27919], 99.90th=[122160], 99.95th=[122160], | 99.99th=[122160] bw ( KiB/s): min= 640, max= 1024, per=24.21%, avg=868.77, stdev=116.70, samples=13 iops : min= 40, max= 64, avg=54.23, stdev= 7.28, samples=13 write: IOPS=59, BW=954KiB/s (977kB/s)(6368KiB/6672msec) clat (usec): min=669, max=47589, avg=6110.49, stdev=7044.52 lat (usec): min=670, max=47590, avg=6111.87, stdev=7044.56 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 807], 10.00th=[ 955], 20.00th=[ 1696], | 30.00th=[ 2606], 40.00th=[ 2900], 50.00th=[ 3490], 60.00th=[ 4555], | 70.00th=[ 6063], 80.00th=[ 8848], 90.00th=[13960], 95.00th=[17957], | 99.00th=[40109], 99.50th=[42206], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 352, max= 1344, per=25.16%, avg=949.92, stdev=229.71, samples=13 iops : min= 22, max= 84, avg=59.31, stdev=14.34, samples=13 lat (usec) : 100=0.78%, 250=0.39%, 750=1.43%, 1000=4.69% lat (msec) : 2=4.56%, 4=18.36%, 10=29.56%, 20=36.59%, 50=3.52% lat (msec) : 250=0.13% cpu : usr=0.06%, sys=19.64%, ctx=2760, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13436: Sun Mar 10 18:48:25 2024 read: IOPS=54, BW=871KiB/s (892kB/s)(5744KiB/6597msec) clat (usec): min=65, max=40990, avg=11111.16, stdev=3409.03 lat (usec): min=66, max=40991, avg=11112.00, stdev=3409.01 clat percentiles (usec): | 1.00th=[ 108], 5.00th=[ 7570], 10.00th=[ 8160], 20.00th=[ 9110], | 30.00th=[ 9634], 40.00th=[10159], 50.00th=[10814], 60.00th=[11338], | 70.00th=[11994], 80.00th=[13042], 90.00th=[14353], 95.00th=[16319], | 99.00th=[20317], 99.50th=[23200], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 480, max= 1088, per=24.28%, avg=871.23, stdev=176.93, samples=13 iops : min= 30, max= 68, avg=54.38, stdev=11.12, samples=13 write: IOPS=61, BW=992KiB/s (1016kB/s)(6544KiB/6597msec) clat (usec): min=665, max=153169, avg=6352.80, stdev=9885.76 lat (usec): min=666, max=153170, avg=6354.35, stdev=9886.01 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 750], 10.00th=[ 898], 20.00th=[ 1287], | 30.00th=[ 2507], 40.00th=[ 2900], 50.00th=[ 3589], 60.00th=[ 4817], | 70.00th=[ 6259], 80.00th=[ 9241], 90.00th=[ 13960], 95.00th=[ 16909], | 99.00th=[ 36963], 99.50th=[ 42730], 99.90th=[154141], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 704, max= 1184, per=26.33%, avg=994.23, stdev=135.37, samples=13 iops : min= 44, max= 74, avg=62.08, stdev= 8.45, samples=13 lat (usec) : 100=0.39%, 250=0.39%, 750=2.34%, 1000=4.30% lat (msec) : 2=5.73%, 4=16.54%, 10=30.73%, 20=37.37%, 50=2.08% lat (msec) : 250=0.13% cpu : usr=0.20%, sys=17.94%, ctx=2686, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13437: Sun Mar 10 18:48:25 2024 read: IOPS=59, BW=957KiB/s (980kB/s)(6256KiB/6535msec) clat (usec): min=67, max=31479, avg=11094.43, stdev=3978.72 lat (usec): min=68, max=31480, avg=11095.22, stdev=3978.72 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 7242], 10.00th=[ 7963], 20.00th=[ 8717], | 30.00th=[ 9503], 40.00th=[10028], 50.00th=[10683], 60.00th=[11207], | 70.00th=[12125], 80.00th=[13042], 90.00th=[14746], 95.00th=[17957], | 99.00th=[27657], 99.50th=[29754], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 766, max= 1152, per=26.74%, avg=959.85, stdev=114.92, samples=13 iops : min= 47, max= 72, avg=59.92, stdev= 7.31, samples=13 write: IOPS=57, BW=923KiB/s (945kB/s)(6032KiB/6535msec) clat (usec): min=599, max=164039, avg=5802.68, stdev=10043.11 lat (usec): min=600, max=164040, avg=5804.12, stdev=10043.09 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 758], 10.00th=[ 914], 20.00th=[ 1221], | 30.00th=[ 2376], 40.00th=[ 2704], 50.00th=[ 3261], 60.00th=[ 4424], | 70.00th=[ 5735], 80.00th=[ 7504], 90.00th=[ 13698], 95.00th=[ 17433], | 99.00th=[ 33162], 99.50th=[ 41157], 99.90th=[164627], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 352, max= 1280, per=24.12%, avg=910.62, stdev=244.56, samples=13 iops : min= 22, max= 80, avg=56.85, stdev=15.26, samples=13 lat (usec) : 100=0.91%, 250=0.65%, 500=0.13%, 750=2.08%, 1000=5.21% lat (msec) : 2=4.30%, 4=16.93%, 10=31.38%, 20=34.90%, 50=3.39% lat (msec) : 250=0.13% cpu : usr=0.14%, sys=18.53%, ctx=2808, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13438: Sun Mar 10 18:48:25 2024 read: IOPS=57, BW=915KiB/s (937kB/s)(6032KiB/6592msec) clat (usec): min=63, max=39310, avg=11295.82, stdev=3968.69 lat (usec): min=64, max=39310, avg=11297.00, stdev=3968.80 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 7373], 10.00th=[ 8225], 20.00th=[ 8979], | 30.00th=[ 9503], 40.00th=[10159], 50.00th=[10683], 60.00th=[11469], | 70.00th=[12125], 80.00th=[13173], 90.00th=[15008], 95.00th=[17433], | 99.00th=[26084], 99.50th=[35914], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 608, max= 1216, per=25.24%, avg=905.69, stdev=142.23, samples=13 iops : min= 38, max= 76, avg=56.54, stdev= 8.93, samples=13 write: IOPS=59, BW=949KiB/s (972kB/s)(6256KiB/6592msec) clat (usec): min=655, max=164320, avg=5941.60, stdev=10335.77 lat (usec): min=656, max=164321, avg=5943.07, stdev=10335.83 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 775], 10.00th=[ 898], 20.00th=[ 1156], | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3326], 60.00th=[ 4146], | 70.00th=[ 5538], 80.00th=[ 8225], 90.00th=[ 13304], 95.00th=[ 16450], | 99.00th=[ 41157], 99.50th=[ 51119], 99.90th=[164627], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 416, max= 1280, per=24.90%, avg=940.15, stdev=225.02, samples=13 iops : min= 26, max= 80, avg=58.69, stdev=14.09, samples=13 lat (usec) : 100=0.91%, 750=1.43%, 1000=5.08% lat (msec) : 2=7.03%, 4=16.15%, 10=29.17%, 20=37.50%, 50=2.47% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.11%, sys=18.07%, ctx=2733, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3590KiB/s (3676kB/s), 871KiB/s-957KiB/s (892kB/s-980kB/s), io=23.4MiB (24.5MB), run=6535-6672msec WRITE: bw=3777KiB/s (3868kB/s), 923KiB/s-992KiB/s (945kB/s-1016kB/s), io=24.6MiB (25.8MB), run=6535-6672msec rand-rw: (groupid=0, jobs=1): err= 0: pid=13440: Sun Mar 10 18:48:25 2024 read: IOPS=55, BW=889KiB/s (910kB/s)(5920KiB/6659msec) clat (usec): min=1886, max=127752, avg=7157.73, stdev=8833.92 lat (usec): min=1886, max=127753, avg=7158.56, stdev=8833.92 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 17], | 99.00th=[ 40], 99.50th=[ 55], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 640, max= 1053, per=24.01%, avg=863.31, stdev=130.36, samples=13 iops : min= 40, max= 65, avg=53.69, stdev= 8.10, samples=13 write: IOPS=59, BW=956KiB/s (979kB/s)(6368KiB/6659msec) clat (usec): min=2677, max=59534, avg=10053.96, stdev=6632.55 lat (usec): min=2678, max=59536, avg=10055.63, stdev=6632.54 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3621], 10.00th=[ 4047], 20.00th=[ 5080], | 30.00th=[ 6390], 40.00th=[ 7373], 50.00th=[ 8356], 60.00th=[ 9634], | 70.00th=[11338], 80.00th=[14091], 90.00th=[17957], 95.00th=[20841], | 99.00th=[39060], 99.50th=[51119], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 352, max= 1184, per=24.89%, avg=941.92, stdev=209.11, samples=13 iops : min= 22, max= 74, avg=58.62, stdev=12.92, samples=13 lat (msec) : 2=0.26%, 4=26.82%, 10=42.97%, 20=25.00%, 50=4.30% lat (msec) : 100=0.52%, 250=0.13% cpu : usr=0.15%, sys=9.42%, ctx=1232, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13441: Sun Mar 10 18:48:25 2024 read: IOPS=54, BW=873KiB/s (894kB/s)(5744KiB/6581msec) clat (usec): min=1885, max=32739, avg=6706.73, stdev=5139.08 lat (usec): min=1886, max=32740, avg=6707.64, stdev=5139.44 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2343], 10.00th=[ 2474], 20.00th=[ 2737], | 30.00th=[ 3130], 40.00th=[ 3589], 50.00th=[ 4555], 60.00th=[ 6194], | 70.00th=[ 8225], 80.00th=[10421], 90.00th=[13698], 95.00th=[17695], | 99.00th=[23987], 99.50th=[27919], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 416, max= 1152, per=24.08%, avg=865.85, stdev=198.25, samples=13 iops : min= 26, max= 72, avg=53.92, stdev=12.26, samples=13 write: IOPS=62, BW=994KiB/s (1018kB/s)(6544KiB/6581msec) clat (msec): min=2, max=182, avg=10.17, stdev=10.87 lat (msec): min=2, max=182, avg=10.17, stdev=10.87 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 17], 95.00th=[ 21], | 99.00th=[ 45], 99.50th=[ 51], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 672, max= 1216, per=26.00%, avg=984.00, stdev=161.65, samples=13 iops : min= 42, max= 76, avg=61.31, stdev=10.14, samples=13 lat (msec) : 2=0.26%, 4=27.21%, 10=41.41%, 20=26.69%, 50=3.91% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.18%, sys=9.47%, ctx=1167, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13442: Sun Mar 10 18:48:25 2024 read: IOPS=59, BW=955KiB/s (978kB/s)(6256KiB/6553msec) clat (usec): min=1860, max=128829, avg=7355.21, stdev=8430.86 lat (usec): min=1861, max=128829, avg=7356.10, stdev=8430.85 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 17], | 99.00th=[ 41], 99.50th=[ 48], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 768, max= 1184, per=26.69%, avg=959.62, stdev=138.14, samples=13 iops : min= 48, max= 74, avg=59.85, stdev= 8.61, samples=13 write: IOPS=57, BW=920KiB/s (943kB/s)(6032KiB/6553msec) clat (usec): min=2947, max=47428, avg=9722.83, stdev=6183.71 lat (usec): min=2949, max=47429, avg=9724.55, stdev=6183.55 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3589], 10.00th=[ 4047], 20.00th=[ 5014], | 30.00th=[ 6390], 40.00th=[ 7177], 50.00th=[ 8586], 60.00th=[ 9372], | 70.00th=[10552], 80.00th=[13042], 90.00th=[16712], 95.00th=[20841], | 99.00th=[36963], 99.50th=[46924], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 320, max= 1344, per=24.13%, avg=912.92, stdev=259.64, samples=13 iops : min= 20, max= 84, avg=56.92, stdev=16.29, samples=13 lat (msec) : 2=0.39%, 4=24.74%, 10=45.83%, 20=24.61%, 50=4.30% lat (msec) : 250=0.13% cpu : usr=0.24%, sys=9.51%, ctx=1229, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13443: Sun Mar 10 18:48:25 2024 read: IOPS=57, BW=914KiB/s (936kB/s)(6032KiB/6596msec) clat (usec): min=1853, max=126024, avg=7057.27, stdev=8614.71 lat (usec): min=1853, max=126025, avg=7058.07, stdev=8614.71 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 18], | 99.00th=[ 40], 99.50th=[ 42], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 608, max= 1216, per=25.25%, avg=908.00, stdev=147.06, samples=13 iops : min= 38, max= 76, avg=56.62, stdev= 9.24, samples=13 write: IOPS=59, BW=948KiB/s (971kB/s)(6256KiB/6596msec) clat (usec): min=2741, max=48320, avg=10031.66, stdev=6486.14 lat (usec): min=2742, max=48322, avg=10033.25, stdev=6486.07 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3720], 10.00th=[ 4146], 20.00th=[ 4752], | 30.00th=[ 5997], 40.00th=[ 7308], 50.00th=[ 8848], 60.00th=[10290], | 70.00th=[11600], 80.00th=[13566], 90.00th=[17171], 95.00th=[21365], | 99.00th=[44827], 99.50th=[47973], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 384, max= 1149, per=24.84%, avg=939.85, stdev=209.59, samples=13 iops : min= 24, max= 71, avg=58.62, stdev=12.98, samples=13 lat (msec) : 2=0.39%, 4=26.82%, 10=41.15%, 20=26.56%, 50=4.95% lat (msec) : 250=0.13% cpu : usr=0.21%, sys=9.51%, ctx=1215, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3597KiB/s (3683kB/s), 873KiB/s-955KiB/s (894kB/s-978kB/s), io=23.4MiB (24.5MB), run=6553-6659msec WRITE: bw=3784KiB/s (3875kB/s), 920KiB/s-994KiB/s (943kB/s-1018kB/s), io=24.6MiB (25.8MB), run=6553-6659msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13453: Sun Mar 10 18:48:29 2024 read: IOPS=2, BW=3052KiB/s (3125kB/s)(7168KiB/2349msec) clat (usec): min=439, max=545392, avg=308252.13, stdev=240023.55 lat (usec): min=439, max=545393, avg=308253.65, stdev=240023.73 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 441], 10.00th=[ 441], 20.00th=[ 33424], | 30.00th=[141558], 40.00th=[141558], 50.00th=[450888], 60.00th=[459277], | 70.00th=[459277], 80.00th=[526386], 90.00th=[549454], 95.00th=[549454], | 99.00th=[549454], 99.50th=[549454], 99.90th=[549454], 99.95th=[549454], | 99.99th=[549454] bw ( KiB/s): min= 2048, max= 2048, per=27.21%, avg=2048.00, stdev= 0.00, samples=4 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=4 write: IOPS=2, BW=2180KiB/s (2232kB/s)(5120KiB/2349msec) clat (usec): min=23815, max=90123, avg=37780.35, stdev=29267.23 lat (usec): min=23882, max=90189, avg=37838.52, stdev=29271.32 clat percentiles (usec): | 1.00th=[23725], 5.00th=[23725], 10.00th=[23725], 20.00th=[23725], | 30.00th=[24511], 40.00th=[24511], 50.00th=[24773], 60.00th=[24773], | 70.00th=[25560], 80.00th=[25560], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 2048, max= 2048, per=19.44%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=8.33%, 250=8.33%, 500=16.67%, 750=16.67% cpu : usr=0.00%, sys=20.77%, ctx=1246, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13454: Sun Mar 10 18:48:29 2024 read: IOPS=0, BW=753KiB/s (771kB/s)(2048KiB/2721msec) clat (msec): min=32, max=469, avg=251.02, stdev=309.32 lat (msec): min=32, max=469, avg=251.02, stdev=309.32 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 468], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 2048, max= 2048, per=27.21%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3763KiB/s (3854kB/s)(10.0MiB/2721msec) clat (msec): min=17, max=1040, avg=221.72, stdev=339.99 lat (msec): min=18, max=1040, avg=221.79, stdev=339.99 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 24], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 38], 60.00th=[ 58], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 531], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2048, max=10240, per=51.83%, avg=5461.33, stdev=4263.25, samples=3 iops : min= 2, max= 10, avg= 5.33, stdev= 4.16, samples=3 lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 500=16.67%, 750=8.33% cpu : usr=0.04%, sys=10.88%, ctx=339, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13455: Sun Mar 10 18:48:29 2024 read: IOPS=2, BW=2331KiB/s (2387kB/s)(6144KiB/2636msec) clat (usec): min=552, max=478438, avg=142184.76, stdev=203755.48 lat (usec): min=554, max=478440, avg=142186.08, stdev=203755.40 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 7504], | 30.00th=[ 7504], 40.00th=[ 13042], 50.00th=[ 13042], 60.00th=[ 40109], | 70.00th=[312476], 80.00th=[312476], 90.00th=[480248], 95.00th=[480248], | 99.00th=[480248], 99.50th=[480248], 99.90th=[480248], 99.95th=[480248], | 99.99th=[480248] bw ( KiB/s): min= 4096, max= 6144, per=68.03%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2331KiB/s (2387kB/s)(6144KiB/2636msec) clat (msec): min=21, max=1002, avg=296.79, stdev=383.93 lat (msec): min=21, max=1002, avg=296.83, stdev=383.92 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 192], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 2048, max= 4096, per=29.15%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 750=8.33% lat (msec) : 10=8.33%, 20=8.33%, 50=25.00%, 100=8.33%, 250=8.33% lat (msec) : 500=25.00% cpu : usr=0.00%, sys=9.72%, ctx=638, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13456: Sun Mar 10 18:48:29 2024 read: IOPS=1, BW=1942KiB/s (1989kB/s)(5120KiB/2636msec) clat (usec): min=534, max=505507, avg=274957.02, stdev=237643.02 lat (usec): min=535, max=505508, avg=274958.20, stdev=237642.92 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[ 537], | 30.00th=[ 43254], 40.00th=[ 43254], 50.00th=[358613], 60.00th=[358613], | 70.00th=[467665], 80.00th=[467665], 90.00th=[505414], 95.00th=[505414], | 99.00th=[505414], 99.50th=[505414], 99.90th=[505414], 99.95th=[505414], | 99.99th=[505414] bw ( KiB/s): min= 2048, max= 4096, per=36.28%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=2, BW=2719KiB/s (2785kB/s)(7168KiB/2636msec) clat (msec): min=20, max=997, avg=179.93, stdev=360.87 lat (msec): min=20, max=997, avg=179.99, stdev=360.88 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 23], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 44], 60.00th=[ 59], | 70.00th=[ 59], 80.00th=[ 80], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 2048, max= 6144, per=45.35%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=41.67%, 100=16.67%, 500=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=13.06%, ctx=809, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7527KiB/s (7707kB/s), 753KiB/s-3052KiB/s (771kB/s-3125kB/s), io=20.0MiB (20.0MB), run=2349-2721msec WRITE: bw=10.3MiB/s (10.8MB/s), 2180KiB/s-3763KiB/s (2232kB/s-3854kB/s), io=28.0MiB (29.4MB), run=2349-2721msec rand-rw: (groupid=0, jobs=1): err= 0: pid=13458: Sun Mar 10 18:48:29 2024 read: IOPS=2, BW=3019KiB/s (3092kB/s)(7168KiB/2374msec) clat (msec): min=18, max=616, avg=301.83, stdev=264.18 lat (msec): min=18, max=616, avg=301.84, stdev=264.18 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 31], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 380], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 527], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 2043, max= 4096, per=35.94%, avg=2729.00, stdev=1183.86, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 write: IOPS=2, BW=2157KiB/s (2208kB/s)(5120KiB/2374msec) clat (usec): min=34785, max=73848, avg=51260.77, stdev=14488.93 lat (usec): min=34868, max=73929, avg=51343.26, stdev=14486.77 clat percentiles (usec): | 1.00th=[34866], 5.00th=[34866], 10.00th=[34866], 20.00th=[34866], | 30.00th=[44303], 40.00th=[44303], 50.00th=[50070], 60.00th=[50070], | 70.00th=[53740], 80.00th=[53740], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 2048, max= 2048, per=19.26%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 500=8.33%, 750=25.00% cpu : usr=0.00%, sys=5.44%, ctx=67, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13459: Sun Mar 10 18:48:29 2024 read: IOPS=0, BW=759KiB/s (778kB/s)(2048KiB/2697msec) clat (msec): min=65, max=365, avg=215.59, stdev=212.05 lat (msec): min=65, max=365, avg=215.59, stdev=212.05 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 2048, max= 2048, per=26.97%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3797KiB/s (3888kB/s)(10.0MiB/2697msec) clat (msec): min=24, max=978, avg=225.98, stdev=314.21 lat (msec): min=24, max=978, avg=226.05, stdev=314.20 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 41], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 68], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 518], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 2048, max=10219, per=51.31%, avg=5454.33, stdev=4251.48, samples=3 iops : min= 2, max= 9, avg= 5.00, stdev= 3.61, samples=3 lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 500=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.07%, sys=6.49%, ctx=79, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13460: Sun Mar 10 18:48:29 2024 read: IOPS=2, BW=2299KiB/s (2354kB/s)(6144KiB/2673msec) clat (msec): min=21, max=509, avg=181.77, stdev=210.73 lat (msec): min=21, max=509, avg=181.77, stdev=210.73 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 124], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 4096, max= 6144, per=67.43%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2299KiB/s (2354kB/s)(6144KiB/2673msec) clat (msec): min=40, max=971, avg=262.09, stdev=370.02 lat (msec): min=40, max=971, avg=262.18, stdev=370.01 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 77], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 2048, max= 4096, per=28.90%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 500=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.11%, sys=4.75%, ctx=155, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13461: Sun Mar 10 18:48:29 2024 read: IOPS=1, BW=1905KiB/s (1950kB/s)(5120KiB/2688msec) clat (msec): min=20, max=512, avg=272.99, stdev=234.60 lat (msec): min=20, max=512, avg=272.99, stdev=234.60 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 418], 80.00th=[ 418], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 2048, max= 4096, per=35.96%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=2, BW=2667KiB/s (2731kB/s)(7168KiB/2688msec) clat (msec): min=26, max=1001, avg=187.96, stdev=359.33 lat (msec): min=26, max=1001, avg=188.03, stdev=359.33 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 32], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 60], 60.00th=[ 81], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 2048, max= 6144, per=44.95%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (msec) : 50=41.67%, 100=25.00%, 500=16.67%, 750=8.33% cpu : usr=0.04%, sys=5.36%, ctx=173, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7594KiB/s (7776kB/s), 759KiB/s-3019KiB/s (778kB/s-3092kB/s), io=20.0MiB (20.0MB), run=2374-2697msec WRITE: bw=10.4MiB/s (10.9MB/s), 2157KiB/s-3797KiB/s (2208kB/s-3888kB/s), io=28.0MiB (29.4MB), run=2374-2697msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13469: Sun Mar 10 18:48:33 2024 read: IOPS=1, BW=4313KiB/s (4417kB/s)(12.0MiB/2849msec) clat (msec): min=184, max=1952, avg=944.08, stdev=910.12 lat (msec): min=184, max=1952, avg=944.09, stdev=910.12 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 693], | 70.00th=[ 1955], 80.00th=[ 1955], 90.00th=[ 1955], 95.00th=[ 1955], | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7301, max= 8192, per=100.00%, avg=7746.50, stdev=630.03, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=37.46%, ctx=1322, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13470: Sun Mar 10 18:48:33 2024 read: IOPS=0, BW=1595KiB/s (1633kB/s)(4096KiB/2568msec) clat (nsec): min=224261k, max=224261k, avg=224261333.00, stdev= 0.00 lat (nsec): min=224263k, max=224263k, avg=224262966.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 8143, max= 8143, per=100.00%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3190KiB/s (3267kB/s)(8192KiB/2568msec) clat (msec): min=289, max=2052, avg=1170.98, stdev=1247.21 lat (msec): min=289, max=2053, avg=1171.24, stdev=1247.22 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 2056], | 70.00th=[ 2056], 80.00th=[ 2056], 90.00th=[ 2056], 95.00th=[ 2056], | 99.00th=[ 2056], 99.50th=[ 2056], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 8192, max= 8192, per=70.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=12.19%, ctx=232, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13471: Sun Mar 10 18:48:33 2024 write: IOPS=1, BW=4354KiB/s (4459kB/s)(12.0MiB/2822msec) clat (msec): min=212, max=1894, avg=933.90, stdev=866.05 lat (msec): min=212, max=1894, avg=934.18, stdev=866.00 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 693], | 70.00th=[ 1888], 80.00th=[ 1888], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7670, max= 8192, per=68.31%, avg=7931.00, stdev=369.11, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.04%, sys=12.55%, ctx=681, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13472: Sun Mar 10 18:48:33 2024 write: IOPS=1, BW=5324KiB/s (5452kB/s)(12.0MiB/2308msec) clat (msec): min=83, max=1967, avg=767.16, stdev=1043.20 lat (msec): min=83, max=1968, avg=767.42, stdev=1043.23 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 1972], 80.00th=[ 1972], 90.00th=[ 1972], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 8192, max= 8192, per=70.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=10.32%, ctx=39, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5751KiB/s (5889kB/s), 1595KiB/s-4313KiB/s (1633kB/s-4417kB/s), io=16.0MiB (16.8MB), run=2568-2849msec WRITE: bw=11.3MiB/s (11.9MB/s), 3190KiB/s-5324KiB/s (3267kB/s-5452kB/s), io=32.0MiB (33.6MB), run=2308-2822msec rand-rw: (groupid=0, jobs=1): err= 0: pid=13473: Sun Mar 10 18:48:33 2024 read: IOPS=1, BW=4194KiB/s (4295kB/s)(12.0MiB/2930msec) clat (msec): min=433, max=1973, avg=973.71, stdev=866.87 lat (msec): min=433, max=1973, avg=973.71, stdev=866.87 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 1972], 80.00th=[ 1972], 90.00th=[ 1972], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=4.40%, ctx=42, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13474: Sun Mar 10 18:48:33 2024 read: IOPS=0, BW=1335KiB/s (1367kB/s)(4096KiB/3068msec) clat (nsec): min=377303k, max=377303k, avg=377302747.00, stdev= 0.00 lat (nsec): min=377306k, max=377306k, avg=377305549.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 376], 10.00th=[ 376], 20.00th=[ 376], | 30.00th=[ 376], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2670KiB/s (2734kB/s)(8192KiB/3068msec) clat (msec): min=280, max=2404, avg=1342.67, stdev=1501.43 lat (msec): min=281, max=2404, avg=1342.96, stdev=1501.47 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 2400], | 70.00th=[ 2400], 80.00th=[ 2400], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 8175, max= 8175, per=76.67%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=4.34%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13475: Sun Mar 10 18:48:33 2024 write: IOPS=0, BW=4063KiB/s (4161kB/s)(12.0MiB/3024msec) clat (msec): min=68, max=2390, avg=1007.16, stdev=1222.75 lat (msec): min=69, max=2390, avg=1007.43, stdev=1222.74 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 2400], 80.00th=[ 2400], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 8192, max= 8192, per=76.83%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.03%, sys=5.33%, ctx=67, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13476: Sun Mar 10 18:48:33 2024 write: IOPS=0, BW=3999KiB/s (4095kB/s)(12.0MiB/3073msec) clat (msec): min=77, max=2372, avg=1018.18, stdev=1202.31 lat (msec): min=77, max=2372, avg=1018.51, stdev=1202.27 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 609], | 70.00th=[ 2366], 80.00th=[ 2366], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 8192, max= 8192, per=76.83%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=5.34%, ctx=110, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5340KiB/s (5468kB/s), 1335KiB/s-4194KiB/s (1367kB/s-4295kB/s), io=16.0MiB (16.8MB), run=2930-3068msec WRITE: bw=10.4MiB/s (10.9MB/s), 2670KiB/s-4063KiB/s (2734kB/s-4161kB/s), io=32.0MiB (33.6MB), run=3024-3073msec