== sanity test 398b: DIO and buffer IO race ============== 09:13:45 (1726838025) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.82169 s, 27.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6209: Fri Sep 20 09:14:33 2024 read: IOPS=33, BW=133KiB/s (137kB/s)(5992KiB/44888msec) clat (usec): min=62, max=248471, avg=16983.54, stdev=13107.92 lat (usec): min=63, max=248473, avg=16985.07, stdev=13107.90 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 108], 10.00th=[ 4621], 20.00th=[ 8225], | 30.00th=[ 10421], 40.00th=[ 12518], 50.00th=[ 14746], 60.00th=[ 17433], | 70.00th=[ 20579], 80.00th=[ 25035], 90.00th=[ 31065], 95.00th=[ 37487], | 99.00th=[ 55313], 99.50th=[ 69731], 99.90th=[107480], 99.95th=[248513], | 99.99th=[248513] bw ( KiB/s): min= 64, max= 272, per=24.71%, avg=131.98, stdev=33.48, samples=89 iops : min= 16, max= 68, avg=32.62, stdev= 8.41, samples=89 write: IOPS=35, BW=140KiB/s (144kB/s)(6296KiB/44888msec) clat (usec): min=482, max=85035, avg=12274.80, stdev=13795.05 lat (usec): min=484, max=85037, avg=12280.66, stdev=13803.20 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 766], 10.00th=[ 807], 20.00th=[ 947], | 30.00th=[ 1827], 40.00th=[ 3752], 50.00th=[ 8094], 60.00th=[11863], | 70.00th=[15401], 80.00th=[21365], 90.00th=[31327], 95.00th=[39584], | 99.00th=[63177], 99.50th=[69731], 99.90th=[82314], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 29, max= 240, per=24.71%, avg=138.36, stdev=40.80, samples=89 iops : min= 7, max= 60, avg=34.20, stdev=10.25, samples=89 lat (usec) : 100=1.79%, 250=2.25%, 500=0.10%, 750=1.86%, 1000=9.47% lat (msec) : 2=4.69%, 4=5.08%, 10=16.47%, 20=31.58%, 50=24.74% lat (msec) : 100=1.92%, 250=0.07% cpu : usr=0.18%, sys=10.79%, ctx=5008, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6210: Fri Sep 20 09:14:33 2024 read: IOPS=33, BW=133KiB/s (136kB/s)(5940KiB/44660msec) clat (usec): min=62, max=243839, avg=16997.42, stdev=13454.12 lat (usec): min=63, max=243841, avg=16999.36, stdev=13454.09 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 112], 10.00th=[ 5080], 20.00th=[ 8029], | 30.00th=[ 10421], 40.00th=[ 12649], 50.00th=[ 14746], 60.00th=[ 17433], | 70.00th=[ 20055], 80.00th=[ 24249], 90.00th=[ 31589], 95.00th=[ 37487], | 99.00th=[ 62129], 99.50th=[ 72877], 99.90th=[100140], 99.95th=[244319], | 99.99th=[244319] bw ( KiB/s): min= 16, max= 239, per=24.74%, avg=132.09, stdev=39.03, samples=89 iops : min= 4, max= 59, avg=32.75, stdev= 9.79, samples=89 write: IOPS=35, BW=142KiB/s (146kB/s)(6348KiB/44660msec) clat (usec): min=553, max=94846, avg=12131.75, stdev=13934.32 lat (usec): min=554, max=94848, avg=12136.70, stdev=13935.13 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 930], | 30.00th=[ 1582], 40.00th=[ 3326], 50.00th=[ 8225], 60.00th=[11338], | 70.00th=[15139], 80.00th=[21627], 90.00th=[30802], 95.00th=[38536], | 99.00th=[58983], 99.50th=[81265], 99.90th=[88605], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 48, max= 288, per=25.12%, avg=140.66, stdev=49.82, samples=89 iops : min= 12, max= 72, avg=34.90, stdev=12.52, samples=89 lat (usec) : 100=1.92%, 250=1.89%, 500=0.13%, 750=2.18%, 1000=9.44% lat (msec) : 2=5.70%, 4=5.01%, 10=16.11%, 20=31.54%, 50=23.99% lat (msec) : 100=2.02%, 250=0.07% cpu : usr=0.26%, sys=10.64%, ctx=5017, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6211: Fri Sep 20 09:14:33 2024 read: IOPS=33, BW=135KiB/s (138kB/s)(6028KiB/44611msec) clat (usec): min=72, max=76870, avg=16643.91, stdev=11167.59 lat (usec): min=72, max=76871, avg=16645.47, stdev=11167.44 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 110], 10.00th=[ 5276], 20.00th=[ 8225], | 30.00th=[10290], 40.00th=[12387], 50.00th=[14746], 60.00th=[17433], | 70.00th=[20841], 80.00th=[24773], 90.00th=[30278], 95.00th=[36439], | 99.00th=[52167], 99.50th=[59507], 99.90th=[76022], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 56, max= 256, per=25.18%, avg=134.47, stdev=38.73, samples=89 iops : min= 14, max= 64, avg=33.30, stdev= 9.75, samples=89 write: IOPS=35, BW=140KiB/s (144kB/s)(6260KiB/44611msec) clat (usec): min=594, max=225200, avg=12351.91, stdev=15055.31 lat (usec): min=595, max=225203, avg=12354.74, stdev=15054.68 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 938], | 30.00th=[ 1696], 40.00th=[ 3556], 50.00th=[ 7767], 60.00th=[ 11469], | 70.00th=[ 15664], 80.00th=[ 21103], 90.00th=[ 31065], 95.00th=[ 39584], | 99.00th=[ 68682], 99.50th=[ 73925], 99.90th=[103285], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 39, max= 240, per=24.92%, avg=139.55, stdev=42.92, samples=89 iops : min= 9, max= 60, avg=34.57, stdev=10.82, samples=89 lat (usec) : 100=1.69%, 250=2.38%, 500=0.07%, 750=2.05%, 1000=9.34% lat (msec) : 2=5.40%, 4=4.82%, 10=17.12%, 20=30.11%, 50=24.97% lat (msec) : 100=1.99%, 250=0.07% cpu : usr=0.11%, sys=11.04%, ctx=5078, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6212: Fri Sep 20 09:14:33 2024 read: IOPS=34, BW=137KiB/s (141kB/s)(6052KiB/44016msec) clat (usec): min=65, max=106678, avg=16629.79, stdev=11428.30 lat (usec): min=66, max=106680, avg=16632.49, stdev=11427.28 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 111], 10.00th=[ 5211], 20.00th=[ 8455], | 30.00th=[ 10290], 40.00th=[ 12387], 50.00th=[ 14615], 60.00th=[ 16909], | 70.00th=[ 20055], 80.00th=[ 23987], 90.00th=[ 30278], 95.00th=[ 38536], | 99.00th=[ 52691], 99.50th=[ 62129], 99.90th=[ 80217], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 52, max= 248, per=25.65%, avg=136.97, stdev=40.39, samples=88 iops : min= 13, max= 62, avg=33.94, stdev=10.16, samples=88 write: IOPS=35, BW=142KiB/s (145kB/s)(6236KiB/44016msec) clat (usec): min=474, max=218456, avg=11952.58, stdev=14281.59 lat (usec): min=475, max=218459, avg=11954.40, stdev=14281.46 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 930], | 30.00th=[ 1663], 40.00th=[ 3523], 50.00th=[ 7635], 60.00th=[ 11469], | 70.00th=[ 15533], 80.00th=[ 21103], 90.00th=[ 29492], 95.00th=[ 38536], | 99.00th=[ 56361], 99.50th=[ 73925], 99.90th=[110625], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 39, max= 263, per=25.19%, avg=141.09, stdev=39.74, samples=88 iops : min= 9, max= 65, avg=34.98, stdev=10.00, samples=88 lat (usec) : 100=1.92%, 250=1.86%, 500=0.07%, 750=2.41%, 1000=9.15% lat (msec) : 2=5.11%, 4=4.85%, 10=17.35%, 20=31.54%, 50=23.96% lat (msec) : 100=1.69%, 250=0.10% cpu : usr=0.17%, sys=10.96%, ctx=5146, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=535KiB/s (548kB/s), 133KiB/s-137KiB/s (136kB/s-141kB/s), io=23.4MiB (24.6MB), run=44016-44888msec WRITE: bw=560KiB/s (574kB/s), 140KiB/s-142KiB/s (144kB/s-146kB/s), io=24.6MiB (25.7MB), run=44016-44888msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6204: Fri Sep 20 09:14:57 2024 read: IOPS=21, BW=87.8KiB/s (89.9kB/s)(5992KiB/68279msec) clat (msec): min=3, max=287, avg=17.22, stdev=16.78 lat (msec): min=3, max=287, avg=17.22, stdev=16.78 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 36], 95.00th=[ 51], | 99.00th=[ 74], 99.50th=[ 93], 99.90th=[ 153], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 23, max= 240, per=24.85%, avg=86.98, stdev=46.37, samples=136 iops : min= 5, max= 60, avg=21.43, stdev=11.68, samples=136 write: IOPS=23, BW=92.2KiB/s (94.4kB/s)(6296KiB/68279msec) clat (msec): min=5, max=134, avg=26.90, stdev=16.71 lat (msec): min=5, max=134, avg=26.90, stdev=16.71 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 26], | 70.00th=[ 31], 80.00th=[ 39], 90.00th=[ 50], 95.00th=[ 59], | 99.00th=[ 85], 99.50th=[ 102], 99.90th=[ 134], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 16, max= 200, per=24.90%, avg=91.38, stdev=44.41, samples=136 iops : min= 4, max= 50, avg=22.53, stdev=11.20, samples=136 lat (msec) : 4=0.52%, 10=20.64%, 20=38.87%, 50=32.68%, 100=6.84% lat (msec) : 250=0.42%, 500=0.03% cpu : usr=0.10%, sys=4.93%, ctx=3612, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6205: Fri Sep 20 09:14:57 2024 read: IOPS=21, BW=86.0KiB/s (89.0kB/s)(5940KiB/68312msec) clat (msec): min=2, max=270, avg=17.29, stdev=16.66 lat (msec): min=2, max=270, avg=17.30, stdev=16.66 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 34], 95.00th=[ 50], | 99.00th=[ 85], 99.50th=[ 99], 99.90th=[ 144], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 16, max= 256, per=24.68%, avg=86.38, stdev=47.54, samples=136 iops : min= 4, max= 64, avg=21.32, stdev=11.95, samples=136 write: IOPS=23, BW=92.9KiB/s (95.2kB/s)(6348KiB/68312msec) clat (msec): min=4, max=116, avg=26.76, stdev=16.74 lat (msec): min=4, max=116, avg=26.77, stdev=16.74 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 30], 80.00th=[ 37], 90.00th=[ 50], 95.00th=[ 62], | 99.00th=[ 90], 99.50th=[ 94], 99.90th=[ 111], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 32, max= 224, per=25.11%, avg=92.15, stdev=44.49, samples=136 iops : min= 8, max= 56, avg=22.76, stdev=11.23, samples=136 lat (msec) : 4=0.52%, 10=20.57%, 20=38.35%, 50=33.17%, 100=7.00% lat (msec) : 250=0.36%, 500=0.03% cpu : usr=0.12%, sys=4.96%, ctx=3661, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6206: Fri Sep 20 09:14:57 2024 read: IOPS=22, BW=88.0KiB/s (90.1kB/s)(6028KiB/68497msec) clat (msec): min=3, max=148, avg=17.41, stdev=15.24 lat (msec): min=3, max=148, avg=17.41, stdev=15.24 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 37], 95.00th=[ 49], | 99.00th=[ 73], 99.50th=[ 87], 99.90th=[ 122], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 16, max= 248, per=24.89%, avg=87.12, stdev=44.84, samples=136 iops : min= 4, max= 62, avg=21.45, stdev=11.29, samples=136 write: IOPS=22, BW=91.4KiB/s (93.6kB/s)(6260KiB/68497msec) clat (msec): min=5, max=280, avg=26.91, stdev=18.14 lat (msec): min=5, max=280, avg=26.91, stdev=18.14 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 30], 80.00th=[ 37], 90.00th=[ 50], 95.00th=[ 63], | 99.00th=[ 89], 99.50th=[ 105], 99.90th=[ 134], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 23, max= 215, per=24.58%, avg=90.19, stdev=44.83, samples=136 iops : min= 5, max= 53, avg=22.21, stdev=11.30, samples=136 lat (msec) : 4=0.65%, 10=20.67%, 20=38.12%, 50=33.50%, 100=6.54% lat (msec) : 250=0.49%, 500=0.03% cpu : usr=0.06%, sys=5.04%, ctx=3693, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6208: Fri Sep 20 09:14:57 2024 read: IOPS=22, BW=88.7KiB/s (90.8kB/s)(6052KiB/68221msec) clat (msec): min=3, max=114, avg=17.11, stdev=14.73 lat (msec): min=3, max=114, avg=17.11, stdev=14.73 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 34], 95.00th=[ 50], | 99.00th=[ 78], 99.50th=[ 89], 99.90th=[ 112], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 8, max= 223, per=25.20%, avg=88.20, stdev=47.15, samples=136 iops : min= 2, max= 55, avg=21.76, stdev=11.84, samples=136 write: IOPS=22, BW=91.4KiB/s (93.6kB/s)(6236KiB/68221msec) clat (msec): min=6, max=281, avg=27.06, stdev=17.69 lat (msec): min=6, max=281, avg=27.06, stdev=17.69 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 26], | 70.00th=[ 31], 80.00th=[ 37], 90.00th=[ 50], 95.00th=[ 62], | 99.00th=[ 89], 99.50th=[ 97], 99.90th=[ 129], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 23, max= 224, per=24.72%, avg=90.73, stdev=44.26, samples=136 iops : min= 5, max= 56, avg=22.40, stdev=11.13, samples=136 lat (msec) : 4=0.33%, 10=20.25%, 20=38.31%, 50=33.89%, 100=6.90% lat (msec) : 250=0.29%, 500=0.03% cpu : usr=0.20%, sys=4.87%, ctx=3625, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=351KiB/s (359kB/s), 86.0KiB/s-88.7KiB/s (89.0kB/s-90.8kB/s), io=23.4MiB (24.6MB), run=68221-68497msec WRITE: bw=367KiB/s (376kB/s), 91.4KiB/s-92.9KiB/s (93.6kB/s-95.2kB/s), io=24.6MiB (25.7MB), run=68221-68497msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6289: Fri Sep 20 09:15:20 2024 read: IOPS=17, BW=284KiB/s (291kB/s)(5920KiB/20858msec) clat (usec): min=91, max=146767, avg=41968.92, stdev=16347.95 lat (usec): min=93, max=146769, avg=41970.51, stdev=16347.88 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 22414], 10.00th=[ 26608], 20.00th=[ 31065], | 30.00th=[ 34341], 40.00th=[ 37487], 50.00th=[ 40109], 60.00th=[ 43779], | 70.00th=[ 47449], 80.00th=[ 52691], 90.00th=[ 60031], 95.00th=[ 67634], | 99.00th=[ 91751], 99.50th=[115868], 99.90th=[145753], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 159, max= 384, per=24.95%, avg=279.44, stdev=54.94, samples=41 iops : min= 9, max= 24, avg=16.98, stdev= 3.52, samples=41 write: IOPS=19, BW=305KiB/s (313kB/s)(6368KiB/20858msec) clat (usec): min=987, max=87764, avg=13244.77, stdev=14288.92 lat (usec): min=991, max=87767, avg=13250.36, stdev=14286.82 clat percentiles (usec): | 1.00th=[ 1090], 5.00th=[ 1188], 10.00th=[ 1270], 20.00th=[ 1942], | 30.00th=[ 4146], 40.00th=[ 6849], 50.00th=[ 9634], 60.00th=[11207], | 70.00th=[14877], 80.00th=[19792], 90.00th=[32113], 95.00th=[45351], | 99.00th=[74974], 99.50th=[82314], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 96, max= 640, per=25.84%, avg=304.39, stdev=123.57, samples=41 iops : min= 6, max= 40, avg=18.54, stdev= 7.83, samples=41 lat (usec) : 100=0.13%, 250=1.30%, 1000=0.13% lat (msec) : 2=10.68%, 4=4.43%, 10=11.98%, 20=15.10%, 50=42.32% lat (msec) : 100=13.67%, 250=0.26% cpu : usr=0.09%, sys=10.01%, ctx=2538, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6290: Fri Sep 20 09:15:20 2024 read: IOPS=17, BW=275KiB/s (281kB/s)(5744KiB/20903msec) clat (usec): min=78, max=91448, avg=41508.41, stdev=15742.67 lat (usec): min=79, max=91450, avg=41509.92, stdev=15742.71 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[15401], 10.00th=[25560], 20.00th=[30540], | 30.00th=[33817], 40.00th=[38536], 50.00th=[42206], 60.00th=[44303], | 70.00th=[47449], 80.00th=[52167], 90.00th=[59507], 95.00th=[69731], | 99.00th=[82314], 99.50th=[89654], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 95, max= 480, per=24.47%, avg=274.10, stdev=67.87, samples=41 iops : min= 5, max= 30, avg=16.68, stdev= 4.30, samples=41 write: IOPS=19, BW=313KiB/s (321kB/s)(6544KiB/20903msec) clat (usec): min=1089, max=91415, avg=14391.32, stdev=14623.48 lat (usec): min=1091, max=91418, avg=14394.10, stdev=14623.50 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 1221], 10.00th=[ 1500], 20.00th=[ 2933], | 30.00th=[ 6259], 40.00th=[ 8356], 50.00th=[10421], 60.00th=[12518], | 70.00th=[16057], 80.00th=[21365], 90.00th=[30802], 95.00th=[45876], | 99.00th=[70779], 99.50th=[72877], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 64, max= 672, per=26.24%, avg=309.15, stdev=115.25, samples=41 iops : min= 4, max= 42, avg=18.88, stdev= 7.20, samples=41 lat (usec) : 100=0.65%, 250=0.52%, 500=0.39%, 750=0.13%, 1000=0.26% lat (msec) : 2=7.81%, 4=4.69%, 10=13.80%, 20=15.10%, 50=42.84% lat (msec) : 100=13.80% cpu : usr=0.11%, sys=10.01%, ctx=2530, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6291: Fri Sep 20 09:15:20 2024 read: IOPS=18, BW=294KiB/s (301kB/s)(6256KiB/21310msec) clat (usec): min=88, max=98187, avg=41340.08, stdev=15582.41 lat (usec): min=89, max=98189, avg=41343.46, stdev=15580.79 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[20579], 10.00th=[24773], 20.00th=[31327], | 30.00th=[34866], 40.00th=[37487], 50.00th=[40633], 60.00th=[43254], | 70.00th=[47449], 80.00th=[51643], 90.00th=[61080], 95.00th=[67634], | 99.00th=[82314], 99.50th=[86508], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 191, max= 416, per=26.12%, avg=292.55, stdev=64.85, samples=42 iops : min= 11, max= 26, avg=17.79, stdev= 4.10, samples=42 write: IOPS=17, BW=283KiB/s (290kB/s)(6032KiB/21310msec) clat (usec): min=746, max=91947, avg=13432.03, stdev=14776.65 lat (usec): min=748, max=91950, avg=13434.59, stdev=14776.74 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1188], 10.00th=[ 1287], 20.00th=[ 2376], | 30.00th=[ 4752], 40.00th=[ 7177], 50.00th=[ 8848], 60.00th=[10945], | 70.00th=[14222], 80.00th=[19792], 90.00th=[33162], 95.00th=[46924], | 99.00th=[70779], 99.50th=[74974], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 95, max= 512, per=23.55%, avg=277.38, stdev=101.03, samples=42 iops : min= 5, max= 32, avg=16.83, stdev= 6.40, samples=42 lat (usec) : 100=0.65%, 250=1.17%, 500=0.13%, 750=0.13%, 1000=0.52% lat (msec) : 2=7.94%, 4=5.47%, 10=13.41%, 20=12.50%, 50=44.01% lat (msec) : 100=14.06% cpu : usr=0.15%, sys=9.93%, ctx=2611, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6292: Fri Sep 20 09:15:20 2024 read: IOPS=17, BW=282KiB/s (289kB/s)(6032KiB/21384msec) clat (usec): min=78, max=87407, avg=41361.86, stdev=15076.60 lat (usec): min=79, max=87409, avg=41363.44, stdev=15076.65 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[19530], 10.00th=[25297], 20.00th=[30802], | 30.00th=[34866], 40.00th=[38536], 50.00th=[40633], 60.00th=[44303], | 70.00th=[48497], 80.00th=[52691], 90.00th=[59507], 95.00th=[66847], | 99.00th=[82314], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 96, max= 384, per=24.84%, avg=278.21, stdev=61.59, samples=42 iops : min= 6, max= 24, avg=16.90, stdev= 3.80, samples=42 write: IOPS=18, BW=293KiB/s (300kB/s)(6256KiB/21384msec) clat (usec): min=804, max=92045, avg=14334.01, stdev=15447.29 lat (usec): min=806, max=92047, avg=14336.80, stdev=15447.46 clat percentiles (usec): | 1.00th=[ 1106], 5.00th=[ 1254], 10.00th=[ 1483], 20.00th=[ 2737], | 30.00th=[ 5538], 40.00th=[ 7701], 50.00th=[ 9372], 60.00th=[11469], | 70.00th=[14877], 80.00th=[21365], 90.00th=[36963], 95.00th=[49021], | 99.00th=[65799], 99.50th=[84411], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 63, max= 704, per=24.66%, avg=290.45, stdev=107.71, samples=42 iops : min= 3, max= 44, avg=17.67, stdev= 6.88, samples=42 lat (usec) : 100=0.52%, 250=1.17%, 1000=0.26% lat (msec) : 2=7.68%, 4=4.56%, 10=14.71%, 20=13.80%, 50=42.97% lat (msec) : 100=14.32% cpu : usr=0.16%, sys=9.82%, ctx=2583, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1120KiB/s (1147kB/s), 275KiB/s-294KiB/s (281kB/s-301kB/s), io=23.4MiB (24.5MB), run=20858-21384msec WRITE: bw=1178KiB/s (1207kB/s), 283KiB/s-313KiB/s (290kB/s-321kB/s), io=24.6MiB (25.8MB), run=20858-21384msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6285: Fri Sep 20 09:15:20 2024 read: IOPS=17, BW=278KiB/s (284kB/s)(5920KiB/21317msec) clat (usec): min=3579, max=85871, avg=21229.94, stdev=16035.91 lat (usec): min=3580, max=85873, avg=21233.33, stdev=16036.60 clat percentiles (usec): | 1.00th=[ 4752], 5.00th=[ 5866], 10.00th=[ 6718], 20.00th=[ 8979], | 30.00th=[10552], 40.00th=[11994], 50.00th=[14484], 60.00th=[18482], | 70.00th=[25035], 80.00th=[36439], 90.00th=[45351], 95.00th=[54264], | 99.00th=[69731], 99.50th=[82314], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 159, max= 480, per=24.36%, avg=271.36, stdev=75.87, samples=42 iops : min= 9, max= 30, avg=16.52, stdev= 4.69, samples=42 write: IOPS=18, BW=299KiB/s (306kB/s)(6368KiB/21317msec) clat (msec): min=8, max=140, avg=33.34, stdev=20.45 lat (msec): min=8, max=140, avg=33.34, stdev=20.45 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 28], 60.00th=[ 33], | 70.00th=[ 39], 80.00th=[ 48], 90.00th=[ 62], 95.00th=[ 78], | 99.00th=[ 109], 99.50th=[ 128], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 32, max= 494, per=25.14%, avg=294.64, stdev=100.21, samples=42 iops : min= 2, max= 30, avg=17.98, stdev= 6.24, samples=42 lat (msec) : 4=0.26%, 10=13.80%, 20=30.99%, 50=42.06%, 100=12.24% lat (msec) : 250=0.65% cpu : usr=0.05%, sys=4.60%, ctx=1019, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6286: Fri Sep 20 09:15:20 2024 read: IOPS=17, BW=272KiB/s (279kB/s)(5744KiB/21117msec) clat (usec): min=4383, max=94318, avg=20721.00, stdev=15613.40 lat (usec): min=4384, max=94319, avg=20723.41, stdev=15613.04 clat percentiles (usec): | 1.00th=[ 4948], 5.00th=[ 5997], 10.00th=[ 7504], 20.00th=[ 8586], | 30.00th=[10290], 40.00th=[11600], 50.00th=[14353], 60.00th=[18220], | 70.00th=[24773], 80.00th=[32375], 90.00th=[42730], 95.00th=[51119], | 99.00th=[76022], 99.50th=[79168], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 64, max= 448, per=24.35%, avg=271.31, stdev=83.91, samples=42 iops : min= 4, max= 28, avg=16.52, stdev= 5.26, samples=42 write: IOPS=19, BW=310KiB/s (317kB/s)(6544KiB/21117msec) clat (msec): min=9, max=114, avg=33.26, stdev=19.29 lat (msec): min=9, max=114, avg=33.27, stdev=19.29 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 13], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 22], 40.00th=[ 24], 50.00th=[ 28], 60.00th=[ 32], | 70.00th=[ 38], 80.00th=[ 48], 90.00th=[ 61], 95.00th=[ 75], | 99.00th=[ 95], 99.50th=[ 101], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 191, max= 670, per=26.26%, avg=307.74, stdev=98.91, samples=42 iops : min= 11, max= 41, avg=18.81, stdev= 6.18, samples=42 lat (msec) : 10=13.02%, 20=30.47%, 50=45.05%, 100=11.07%, 250=0.39% cpu : usr=0.18%, sys=4.38%, ctx=960, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6287: Fri Sep 20 09:15:20 2024 read: IOPS=18, BW=291KiB/s (298kB/s)(6256KiB/21498msec) clat (usec): min=3584, max=80718, avg=19782.51, stdev=14859.44 lat (usec): min=3585, max=80720, avg=19784.17, stdev=14859.37 clat percentiles (usec): | 1.00th=[ 4555], 5.00th=[ 6128], 10.00th=[ 7308], 20.00th=[ 8717], | 30.00th=[ 9896], 40.00th=[11600], 50.00th=[13960], 60.00th=[16581], | 70.00th=[22152], 80.00th=[30016], 90.00th=[43254], 95.00th=[51643], | 99.00th=[65799], 99.50th=[73925], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 128, max= 480, per=26.02%, avg=289.86, stdev=93.34, samples=42 iops : min= 8, max= 30, avg=17.81, stdev= 5.88, samples=42 write: IOPS=17, BW=281KiB/s (287kB/s)(6032KiB/21498msec) clat (msec): min=8, max=161, avg=35.98, stdev=21.57 lat (msec): min=8, max=161, avg=35.99, stdev=21.57 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 20], | 30.00th=[ 23], 40.00th=[ 27], 50.00th=[ 32], 60.00th=[ 36], | 70.00th=[ 42], 80.00th=[ 50], 90.00th=[ 63], 95.00th=[ 80], | 99.00th=[ 117], 99.50th=[ 134], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 128, max= 448, per=23.44%, avg=274.74, stdev=85.55, samples=42 iops : min= 8, max= 28, avg=16.86, stdev= 5.37, samples=42 lat (msec) : 4=0.13%, 10=16.15%, 20=29.17%, 50=41.93%, 100=11.72% lat (msec) : 250=0.91% cpu : usr=0.11%, sys=4.61%, ctx=1014, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6288: Fri Sep 20 09:15:20 2024 read: IOPS=18, BW=293KiB/s (300kB/s)(6032KiB/20619msec) clat (usec): min=4491, max=95735, avg=20335.56, stdev=15399.71 lat (usec): min=4492, max=95737, avg=20337.60, stdev=15399.96 clat percentiles (usec): | 1.00th=[ 5342], 5.00th=[ 6915], 10.00th=[ 7701], 20.00th=[ 9110], | 30.00th=[10814], 40.00th=[12125], 50.00th=[14484], 60.00th=[17957], | 70.00th=[22676], 80.00th=[31065], 90.00th=[40633], 95.00th=[52691], | 99.00th=[85459], 99.50th=[94897], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 64, max= 576, per=26.09%, avg=290.66, stdev=115.71, samples=41 iops : min= 4, max= 36, avg=17.78, stdev= 7.17, samples=41 write: IOPS=18, BW=303KiB/s (311kB/s)(6256KiB/20619msec) clat (msec): min=7, max=127, avg=33.00, stdev=20.88 lat (msec): min=7, max=127, avg=33.00, stdev=20.88 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 32], | 70.00th=[ 37], 80.00th=[ 45], 90.00th=[ 63], 95.00th=[ 74], | 99.00th=[ 112], 99.50th=[ 123], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 95, max= 512, per=25.99%, avg=304.66, stdev=78.70, samples=41 iops : min= 5, max= 32, avg=18.66, stdev= 4.97, samples=41 lat (msec) : 10=13.41%, 20=32.94%, 50=42.71%, 100=10.03%, 250=0.91% cpu : usr=0.07%, sys=4.41%, ctx=940, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1114KiB/s (1141kB/s), 272KiB/s-293KiB/s (279kB/s-300kB/s), io=23.4MiB (24.5MB), run=20619-21498msec WRITE: bw=1172KiB/s (1200kB/s), 281KiB/s-310KiB/s (287kB/s-317kB/s), io=24.6MiB (25.8MB), run=20619-21498msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6317: Fri Sep 20 09:15:24 2024 read: IOPS=4, BW=4995KiB/s (5115kB/s)(7168KiB/1435msec) clat (msec): min=45, max=141, avg=100.09, stdev=36.25 lat (msec): min=45, max=141, avg=100.09, stdev=36.25 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 70], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 107], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 142], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 2048, max= 8192, per=68.28%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=3, BW=3568KiB/s (3654kB/s)(5120KiB/1435msec) clat (msec): min=58, max=326, avg=138.12, stdev=108.53 lat (msec): min=58, max=326, avg=138.20, stdev=108.54 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 88], 40.00th=[ 88], 50.00th=[ 88], 60.00th=[ 88], | 70.00th=[ 131], 80.00th=[ 131], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 2048, max= 4096, per=29.26%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=8.33%, 100=41.67%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=9.83%, ctx=223, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6319: Fri Sep 20 09:15:24 2024 read: IOPS=0, BW=750KiB/s (768kB/s)(2048KiB/2731msec) clat (msec): min=82, max=154, avg=118.32, stdev=51.34 lat (msec): min=82, max=154, avg=118.32, stdev=51.34 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 83], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 1387, max= 2048, per=22.90%, avg=1717.50, stdev=467.40, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3750KiB/s (3840kB/s)(10.0MiB/2731msec) clat (msec): min=27, max=1404, avg=246.77, stdev=422.25 lat (msec): min=27, max=1404, avg=246.92, stdev=422.22 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 42], 60.00th=[ 120], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 326], 95.00th=[ 1401], | 99.00th=[ 1401], 99.50th=[ 1401], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 1387, max= 6144, per=30.42%, avg=3193.00, stdev=2576.92, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 lat (msec) : 50=41.67%, 100=8.33%, 250=25.00%, 500=16.67% cpu : usr=0.00%, sys=5.60%, ctx=72, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6320: Fri Sep 20 09:15:24 2024 read: IOPS=4, BW=4376KiB/s (4481kB/s)(6144KiB/1404msec) clat (msec): min=62, max=254, avg=123.51, stdev=69.06 lat (msec): min=62, max=254, avg=123.51, stdev=69.06 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 100], 50.00th=[ 100], 60.00th=[ 111], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 2039, max= 6144, per=54.56%, avg=4091.50, stdev=2902.67, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=4, BW=4376KiB/s (4481kB/s)(6144KiB/1404msec) clat (msec): min=47, max=211, avg=101.87, stdev=62.63 lat (msec): min=47, max=211, avg=101.95, stdev=62.61 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 93], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 2048, max= 6119, per=38.90%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=41.67%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=7.27%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6322: Fri Sep 20 09:15:24 2024 read: IOPS=3, BW=3642KiB/s (3729kB/s)(5120KiB/1406msec) clat (msec): min=59, max=166, avg=123.06, stdev=39.15 lat (msec): min=59, max=166, avg=123.06, stdev=39.15 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 128], 60.00th=[ 128], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 167], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5098KiB/s (5221kB/s)(7168KiB/1406msec) clat (msec): min=51, max=217, avg=109.49, stdev=63.49 lat (msec): min=51, max=217, avg=109.57, stdev=63.48 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 82], 60.00th=[ 101], | 70.00th=[ 101], 80.00th=[ 178], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 8192, max= 8192, per=78.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 100=50.00%, 250=50.00% cpu : usr=0.00%, sys=10.04%, ctx=67, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7499KiB/s (7679kB/s), 750KiB/s-4995KiB/s (768kB/s-5115kB/s), io=20.0MiB (20.0MB), run=1404-2731msec WRITE: bw=10.3MiB/s (10.8MB/s), 3568KiB/s-5098KiB/s (3654kB/s-5221kB/s), io=28.0MiB (29.4MB), run=1404-2731msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6316: Fri Sep 20 09:15:28 2024 read: IOPS=1, BW=1047KiB/s (1072kB/s)(7168KiB/6849msec) clat (usec): min=534, max=1415.5k, avg=915675.34, stdev=616089.43 lat (usec): min=535, max=1415.5k, avg=915676.92, stdev=616089.63 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], | 20.00th=[ 143655], 30.00th=[ 784335], 40.00th=[ 784335], | 50.00th=[1333789], 60.00th=[1367344], 70.00th=[1367344], | 80.00th=[1367344], 90.00th=[1417675], 95.00th=[1417675], | 99.00th=[1417675], 99.50th=[1417675], 99.90th=[1417675], | 99.95th=[1417675], 99.99th=[1417675] bw ( KiB/s): min= 1400, max= 4087, per=64.80%, avg=1937.40, stdev=1201.66, samples=5 iops : min= 1, max= 3, avg= 1.40, stdev= 0.89, samples=5 write: IOPS=0, BW=748KiB/s (765kB/s)(5120KiB/6849msec) clat (usec): min=37820, max=67606, avg=51475.77, stdev=13956.66 lat (usec): min=38013, max=67732, avg=51612.01, stdev=13960.62 clat percentiles (usec): | 1.00th=[38011], 5.00th=[38011], 10.00th=[38011], 20.00th=[38011], | 30.00th=[41681], 40.00th=[41681], 50.00th=[44827], 60.00th=[44827], | 70.00th=[65274], 80.00th=[65274], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 1400, max= 4202, per=55.76%, avg=2334.00, stdev=1617.74, samples=3 iops : min= 1, max= 4, avg= 2.00, stdev= 1.73, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.12%, sys=15.67%, ctx=1286, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6318: Fri Sep 20 09:15:28 2024 read: IOPS=1, BW=1476KiB/s (1511kB/s)(2048KiB/1388msec) clat (usec): min=463, max=82177, avg=41320.29, stdev=57780.68 lat (usec): min=463, max=82179, avg=41321.86, stdev=57781.78 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], 20.00th=[ 465], | 30.00th=[ 465], 40.00th=[ 465], 50.00th=[ 465], 60.00th=[82314], | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 2027, max= 2048, per=68.14%, avg=2037.50, stdev=14.85, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=7, BW=7378KiB/s (7555kB/s)(10.0MiB/1388msec) clat (msec): min=25, max=287, avg=126.13, stdev=86.12 lat (msec): min=25, max=287, avg=126.35, stdev=86.04 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 62], | 30.00th=[ 71], 40.00th=[ 82], 50.00th=[ 83], 60.00th=[ 93], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 239], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 4096, max= 6083, per=100.00%, avg=5089.50, stdev=1405.02, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=8.33%, 100=50.00%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=22.42%, ctx=88, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6321: Fri Sep 20 09:15:28 2024 read: IOPS=5, BW=5678KiB/s (5815kB/s)(6144KiB/1082msec) clat (usec): min=368, max=198603, avg=70726.90, stdev=83185.27 lat (usec): min=368, max=198604, avg=70728.30, stdev=83185.41 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 562], | 30.00th=[ 562], 40.00th=[ 889], 50.00th=[ 889], 60.00th=[105382], | 70.00th=[119014], 80.00th=[119014], 90.00th=[198181], 95.00th=[198181], | 99.00th=[198181], 99.50th=[198181], 99.90th=[198181], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min=10219, max=10219, per=100.00%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5678KiB/s (5815kB/s)(6144KiB/1082msec) clat (msec): min=36, max=252, avg=93.18, stdev=82.69 lat (msec): min=36, max=252, avg=93.23, stdev=82.70 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 73], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 6131, max= 6131, per=100.00%, avg=6131.00, stdev= 0.00, samples=2 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=2 lat (usec) : 500=8.33%, 750=8.33%, 1000=8.33% lat (msec) : 50=16.67%, 100=16.67%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=28.12%, ctx=231, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6323: Fri Sep 20 09:15:28 2024 read: IOPS=4, BW=4736KiB/s (4850kB/s)(5120KiB/1081msec) clat (usec): min=502, max=174516, avg=67894.11, stdev=92336.16 lat (usec): min=504, max=174518, avg=67895.52, stdev=92335.90 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 553], 60.00th=[ 553], | 70.00th=[162530], 80.00th=[162530], 90.00th=[175113], 95.00th=[175113], | 99.00th=[175113], 99.50th=[175113], 99.90th=[175113], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 2048, max= 6083, per=100.00%, avg=4065.50, stdev=2853.18, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=6, BW=6631KiB/s (6790kB/s)(7168KiB/1081msec) clat (msec): min=42, max=126, avg=81.81, stdev=30.92 lat (msec): min=42, max=126, avg=81.88, stdev=30.92 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 54], | 30.00th=[ 69], 40.00th=[ 69], 50.00th=[ 77], 60.00th=[ 92], | 70.00th=[ 92], 80.00th=[ 115], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 4055, max= 8192, per=100.00%, avg=6123.50, stdev=2925.30, samples=2 iops : min= 3, max= 8, avg= 5.50, stdev= 3.54, samples=2 lat (usec) : 750=25.00% lat (msec) : 50=8.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=23.61%, ctx=150, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2990KiB/s (3062kB/s), 1047KiB/s-5678KiB/s (1072kB/s-5815kB/s), io=20.0MiB (20.0MB), run=1081-6849msec WRITE: bw=4186KiB/s (4287kB/s), 748KiB/s-7378KiB/s (765kB/s-7555kB/s), io=28.0MiB (29.4MB), run=1081-6849msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6334: Fri Sep 20 09:15:31 2024 read: IOPS=2, BW=10.2MiB/s (10.7MB/s)(12.0MiB/1174msec) clat (msec): min=35, max=547, avg=371.47, stdev=290.83 lat (msec): min=35, max=547, avg=371.47, stdev=290.83 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 531], 50.00th=[ 531], 60.00th=[ 531], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min=15456, max=15456, per=100.00%, avg=15456.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 750=66.67% cpu : usr=0.09%, sys=15.77%, ctx=134, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6335: Fri Sep 20 09:15:31 2024 read: IOPS=0, BW=2809KiB/s (2877kB/s)(4096KiB/1458msec) clat (nsec): min=507429k, max=507429k, avg=507429321.00, stdev= 0.00 lat (nsec): min=507433k, max=507433k, avg=507432998.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 6206, max= 6206, per=55.23%, avg=6206.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5619KiB/s (5754kB/s)(8192KiB/1458msec) clat (msec): min=180, max=709, avg=445.30, stdev=374.24 lat (msec): min=181, max=710, avg=445.94, stdev=374.15 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 709], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.00%, sys=20.32%, ctx=144, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6336: Fri Sep 20 09:15:31 2024 write: IOPS=3, BW=15.4MiB/s (16.1MB/s)(12.0MiB/781msec) clat (msec): min=206, max=280, avg=246.73, stdev=37.15 lat (msec): min=207, max=280, avg=247.27, stdev=37.12 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 207], 10.00th=[ 207], 20.00th=[ 207], | 30.00th=[ 207], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 253], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min=16351, max=16351, per=72.76%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=37.69%, ctx=89, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6338: Fri Sep 20 09:15:31 2024 write: IOPS=2, BW=11.5MiB/s (12.1MB/s)(12.0MiB/1041msec) clat (msec): min=244, max=496, avg=329.63, stdev=144.89 lat (msec): min=245, max=497, avg=330.19, stdev=144.87 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 245], 10.00th=[ 245], 20.00th=[ 245], | 30.00th=[ 245], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 7861, max= 8192, per=35.71%, avg=8026.50, stdev=234.05, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.67%, sys=28.56%, ctx=90, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.0MiB/s (11.5MB/s), 2809KiB/s-10.2MiB/s (2877kB/s-10.7MB/s), io=16.0MiB (16.8MB), run=1174-1458msec WRITE: bw=21.9MiB/s (23.0MB/s), 5619KiB/s-15.4MiB/s (5754kB/s-16.1MB/s), io=32.0MiB (33.6MB), run=781-1458msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6337: Fri Sep 20 09:15:31 2024 read: IOPS=1, BW=7520KiB/s (7701kB/s)(12.0MiB/1634msec) clat (msec): min=466, max=570, avg=508.83, stdev=54.70 lat (msec): min=466, max=570, avg=508.83, stdev=54.70 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 468], 10.00th=[ 468], 20.00th=[ 468], | 30.00th=[ 468], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 489], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 8159, max= 8192, per=87.53%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.12%, sys=4.72%, ctx=57, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6339: Fri Sep 20 09:15:31 2024 read: IOPS=0, BW=2335KiB/s (2391kB/s)(4096KiB/1754msec) clat (nsec): min=510142k, max=510142k, avg=510141644.00, stdev= 0.00 lat (nsec): min=510146k, max=510146k, avg=510145830.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 8192, max= 8192, per=87.71%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4670KiB/s (4783kB/s)(8192KiB/1754msec) clat (msec): min=241, max=899, avg=570.48, stdev=464.62 lat (msec): min=242, max=899, avg=570.94, stdev=464.56 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 243], 10.00th=[ 243], 20.00th=[ 243], | 30.00th=[ 243], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 902], | 70.00th=[ 902], 80.00th=[ 902], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.46%, sys=10.84%, ctx=84, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6340: Fri Sep 20 09:15:31 2024 write: IOPS=1, BW=7132KiB/s (7303kB/s)(12.0MiB/1723msec) clat (msec): min=425, max=607, avg=517.32, stdev=91.43 lat (msec): min=425, max=608, avg=517.89, stdev=91.44 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 518], 50.00th=[ 518], 60.00th=[ 518], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 8192, max= 8192, per=43.85%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.17%, sys=6.33%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6341: Fri Sep 20 09:15:31 2024 write: IOPS=1, BW=7714KiB/s (7899kB/s)(12.0MiB/1593msec) clat (msec): min=297, max=744, avg=509.03, stdev=224.45 lat (msec): min=297, max=744, avg=509.57, stdev=224.32 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 7984, max= 8159, per=43.21%, avg=8071.50, stdev=123.74, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.57%, sys=11.31%, ctx=117, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9341KiB/s (9565kB/s), 2335KiB/s-7520KiB/s (2391kB/s-7701kB/s), io=16.0MiB (16.8MB), run=1634-1754msec WRITE: bw=18.2MiB/s (19.1MB/s), 4670KiB/s-7714KiB/s (4783kB/s-7899kB/s), io=32.0MiB (33.6MB), run=1593-1754msec