== sanity test 398b: DIO and buffer IO race ============== 07:59:03 (1715342343) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.19369 s, 42.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17387: Fri May 10 07:59:17 2024 read: IOPS=126, BW=507KiB/s (519kB/s)(5992KiB/11815msec) clat (usec): min=53, max=188790, avg=4424.05, stdev=5380.83 lat (usec): min=53, max=188791, avg=4424.87, stdev=5380.71 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 97], 10.00th=[ 2008], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3326], 50.00th=[ 3916], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 5997], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[ 12911], 99.50th=[ 14353], 99.90th=[ 29492], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 216, max= 664, per=24.55%, avg=498.87, stdev=95.34, samples=23 iops : min= 54, max= 166, avg=124.57, stdev=23.93, samples=23 write: IOPS=133, BW=533KiB/s (546kB/s)(6296KiB/11815msec) clat (usec): min=399, max=18224, avg=3263.09, stdev=2925.39 lat (usec): min=400, max=18225, avg=3265.17, stdev=2925.00 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 766], 40.00th=[ 1958], 50.00th=[ 2606], 60.00th=[ 3195], | 70.00th=[ 4146], 80.00th=[ 5538], 90.00th=[ 7308], 95.00th=[ 9241], | 99.00th=[12125], 99.50th=[14091], 99.90th=[17957], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 256, max= 648, per=24.55%, avg=522.09, stdev=82.71, samples=23 iops : min= 64, max= 162, avg=130.39, stdev=20.69, samples=23 lat (usec) : 100=2.47%, 250=0.20%, 500=1.04%, 750=14.23%, 1000=2.28% lat (msec) : 2=5.27%, 4=34.77%, 10=36.52%, 20=3.16%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.93%, ctx=5754, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17388: Fri May 10 07:59:17 2024 read: IOPS=125, BW=504KiB/s (516kB/s)(5940KiB/11792msec) clat (usec): min=48, max=17607, avg=4325.11, stdev=2437.23 lat (usec): min=48, max=17608, avg=4326.60, stdev=2437.03 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 1778], 10.00th=[ 2073], 20.00th=[ 2474], | 30.00th=[ 2868], 40.00th=[ 3326], 50.00th=[ 3851], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5866], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[13173], 99.50th=[13829], 99.90th=[17171], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 224, max= 632, per=24.59%, avg=499.70, stdev=81.24, samples=23 iops : min= 56, max= 158, avg=124.87, stdev=20.30, samples=23 write: IOPS=134, BW=538KiB/s (551kB/s)(6348KiB/11792msec) clat (usec): min=407, max=210497, avg=3354.87, stdev=5903.04 lat (usec): min=408, max=210499, avg=3356.05, stdev=5902.91 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 816], 40.00th=[ 1958], 50.00th=[ 2704], 60.00th=[ 3326], | 70.00th=[ 4293], 80.00th=[ 5342], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[ 11600], 99.50th=[ 12649], 99.90th=[ 25297], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 288, max= 688, per=24.83%, avg=528.17, stdev=89.37, samples=23 iops : min= 72, max= 172, avg=132.00, stdev=22.29, samples=23 lat (usec) : 50=0.03%, 100=1.89%, 250=0.23%, 500=0.72%, 750=13.51% lat (usec) : 1000=3.16% lat (msec) : 2=5.50%, 4=35.35%, 10=36.46%, 20=3.09%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.73%, ctx=5953, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17389: Fri May 10 07:59:17 2024 read: IOPS=128, BW=514KiB/s (526kB/s)(6028KiB/11728msec) clat (usec): min=50, max=214979, avg=4356.68, stdev=5917.53 lat (usec): min=50, max=214979, avg=4358.14, stdev=5917.27 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 318], 10.00th=[ 2040], 20.00th=[ 2442], | 30.00th=[ 2868], 40.00th=[ 3294], 50.00th=[ 3818], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 6915], 95.00th=[ 8356], | 99.00th=[ 13173], 99.50th=[ 14091], 99.90th=[ 19006], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 232, max= 688, per=25.03%, avg=508.65, stdev=87.35, samples=23 iops : min= 58, max= 172, avg=127.00, stdev=21.81, samples=23 write: IOPS=133, BW=534KiB/s (547kB/s)(6260KiB/11728msec) clat (usec): min=425, max=19170, avg=3264.75, stdev=2931.45 lat (usec): min=426, max=19171, avg=3267.01, stdev=2931.75 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 766], 40.00th=[ 1713], 50.00th=[ 2606], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5473], 90.00th=[ 7570], 95.00th=[ 8979], | 99.00th=[12256], 99.50th=[12649], 99.90th=[18482], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 312, max= 688, per=24.75%, avg=526.43, stdev=85.62, samples=23 iops : min= 78, max= 172, avg=131.43, stdev=21.41, samples=23 lat (usec) : 100=2.12%, 250=0.33%, 500=0.81%, 750=14.19%, 1000=2.64% lat (msec) : 2=5.21%, 4=35.55%, 10=36.30%, 20=2.83%, 250=0.03% cpu : usr=0.14%, sys=22.77%, ctx=5943, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17390: Fri May 10 07:59:17 2024 read: IOPS=128, BW=514KiB/s (526kB/s)(6052KiB/11777msec) clat (usec): min=46, max=21664, avg=4220.42, stdev=2398.87 lat (usec): min=47, max=21665, avg=4222.39, stdev=2401.15 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 90], 10.00th=[ 2008], 20.00th=[ 2507], | 30.00th=[ 2835], 40.00th=[ 3261], 50.00th=[ 3818], 60.00th=[ 4424], | 70.00th=[ 5014], 80.00th=[ 5800], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[13173], 99.50th=[14615], 99.90th=[17433], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 200, max= 664, per=24.98%, avg=507.65, stdev=85.14, samples=23 iops : min= 50, max= 166, avg=126.87, stdev=21.28, samples=23 write: IOPS=132, BW=530KiB/s (542kB/s)(6236KiB/11777msec) clat (usec): min=378, max=227500, avg=3428.91, stdev=6420.19 lat (usec): min=379, max=227501, avg=3429.60, stdev=6420.20 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 594], | 30.00th=[ 725], 40.00th=[ 1893], 50.00th=[ 2606], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5669], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[ 12649], 99.50th=[ 15401], 99.90th=[ 25822], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 216, max= 816, per=24.59%, avg=523.00, stdev=105.30, samples=23 iops : min= 54, max= 204, avg=130.70, stdev=26.35, samples=23 lat (usec) : 50=0.07%, 100=2.51%, 250=0.13%, 500=1.01%, 750=14.75% lat (usec) : 1000=1.86% lat (msec) : 2=5.27%, 4=34.57%, 10=37.01%, 20=2.67%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.72%, ctx=5878, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2032KiB/s (2081kB/s), 504KiB/s-514KiB/s (516kB/s-526kB/s), io=23.4MiB (24.6MB), run=11728-11815msec WRITE: bw=2128KiB/s (2179kB/s), 530KiB/s-538KiB/s (542kB/s-551kB/s), io=24.6MiB (25.7MB), run=11728-11815msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17382: Fri May 10 07:59:21 2024 read: IOPS=93, BW=373KiB/s (382kB/s)(5992KiB/16050msec) clat (usec): min=1563, max=203540, avg=4330.29, stdev=5909.92 lat (usec): min=1564, max=203542, avg=4330.98, stdev=5909.93 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2474], 40.00th=[ 2704], 50.00th=[ 3064], 60.00th=[ 3556], | 70.00th=[ 4178], 80.00th=[ 5669], 90.00th=[ 8225], 95.00th=[ 10683], | 99.00th=[ 15270], 99.50th=[ 17171], 99.90th=[ 20841], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 112, max= 712, per=25.04%, avg=372.13, stdev=135.59, samples=32 iops : min= 28, max= 178, avg=92.97, stdev=33.91, samples=32 write: IOPS=98, BW=392KiB/s (402kB/s)(6296KiB/16050msec) clat (usec): min=2380, max=24820, avg=6024.90, stdev=3252.45 lat (usec): min=2381, max=24821, avg=6025.90, stdev=3252.37 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2966], 10.00th=[ 3163], 20.00th=[ 3458], | 30.00th=[ 3818], 40.00th=[ 4228], 50.00th=[ 4817], 60.00th=[ 5866], | 70.00th=[ 6915], 80.00th=[ 8160], 90.00th=[10552], 95.00th=[12387], | 99.00th=[17695], 99.50th=[19006], 99.90th=[22152], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 128, max= 744, per=25.15%, avg=391.38, stdev=144.52, samples=32 iops : min= 32, max= 186, avg=97.78, stdev=36.13, samples=32 lat (msec) : 2=3.65%, 4=46.91%, 10=40.53%, 20=8.69%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.85%, ctx=3688, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17383: Fri May 10 07:59:21 2024 read: IOPS=91, BW=368KiB/s (377kB/s)(5940KiB/16150msec) clat (usec): min=1570, max=215464, avg=4405.66, stdev=6205.52 lat (usec): min=1571, max=215465, avg=4406.44, stdev=6205.50 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2278], | 30.00th=[ 2474], 40.00th=[ 2704], 50.00th=[ 3064], 60.00th=[ 3621], | 70.00th=[ 4359], 80.00th=[ 5735], 90.00th=[ 8586], 95.00th=[ 10814], | 99.00th=[ 13829], 99.50th=[ 16450], 99.90th=[ 22414], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 80, max= 704, per=24.70%, avg=366.97, stdev=149.41, samples=32 iops : min= 20, max= 176, avg=91.72, stdev=37.37, samples=32 write: IOPS=98, BW=393KiB/s (402kB/s)(6348KiB/16150msec) clat (usec): min=2316, max=27237, avg=6004.48, stdev=3322.26 lat (usec): min=2317, max=27238, avg=6005.40, stdev=3322.23 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2933], 10.00th=[ 3163], 20.00th=[ 3490], | 30.00th=[ 3818], 40.00th=[ 4178], 50.00th=[ 4817], 60.00th=[ 5669], | 70.00th=[ 6783], 80.00th=[ 8094], 90.00th=[10552], 95.00th=[12518], | 99.00th=[17957], 99.50th=[19792], 99.90th=[24249], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 152, max= 712, per=25.06%, avg=389.97, stdev=136.76, samples=32 iops : min= 38, max= 178, avg=97.47, stdev=34.20, samples=32 lat (msec) : 2=3.45%, 4=45.96%, 10=41.44%, 20=8.82%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.83%, ctx=3705, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17384: Fri May 10 07:59:21 2024 read: IOPS=93, BW=375KiB/s (384kB/s)(6028KiB/16092msec) clat (usec): min=1657, max=22643, avg=4239.16, stdev=2931.30 lat (usec): min=1657, max=22644, avg=4239.81, stdev=2931.29 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1958], 10.00th=[ 2073], 20.00th=[ 2278], | 30.00th=[ 2474], 40.00th=[ 2704], 50.00th=[ 3032], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5604], 90.00th=[ 8848], 95.00th=[10683], | 99.00th=[14615], 99.50th=[16188], 99.90th=[21103], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 120, max= 736, per=25.13%, avg=373.41, stdev=148.70, samples=32 iops : min= 30, max= 184, avg=93.28, stdev=37.18, samples=32 write: IOPS=97, BW=389KiB/s (398kB/s)(6260KiB/16092msec) clat (msec): min=2, max=215, avg= 6.15, stdev= 6.21 lat (msec): min=2, max=215, avg= 6.15, stdev= 6.21 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 20], 99.90th=[ 26], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 160, max= 720, per=24.88%, avg=387.16, stdev=138.64, samples=32 iops : min= 40, max= 180, avg=96.72, stdev=34.71, samples=32 lat (msec) : 2=3.29%, 4=47.33%, 10=40.33%, 20=8.79%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.80%, ctx=3696, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17385: Fri May 10 07:59:21 2024 read: IOPS=94, BW=377KiB/s (386kB/s)(6052KiB/16045msec) clat (usec): min=1593, max=20447, avg=4202.73, stdev=2869.49 lat (usec): min=1593, max=20448, avg=4203.61, stdev=2869.39 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1909], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3556], | 70.00th=[ 4293], 80.00th=[ 5866], 90.00th=[ 8586], 95.00th=[10683], | 99.00th=[13435], 99.50th=[15795], 99.90th=[19268], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 112, max= 712, per=25.35%, avg=376.66, stdev=144.84, samples=32 iops : min= 28, max= 178, avg=94.09, stdev=36.22, samples=32 write: IOPS=97, BW=389KiB/s (398kB/s)(6236KiB/16045msec) clat (msec): min=2, max=225, avg= 6.16, stdev= 6.46 lat (msec): min=2, max=225, avg= 6.16, stdev= 6.46 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 22], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 128, max= 720, per=24.88%, avg=387.16, stdev=144.07, samples=32 iops : min= 32, max= 180, avg=96.72, stdev=36.06, samples=32 lat (msec) : 2=4.10%, 4=46.29%, 10=40.33%, 20=8.98%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.88%, ctx=3638, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1487KiB/s (1522kB/s), 368KiB/s-377KiB/s (377kB/s-386kB/s), io=23.4MiB (24.6MB), run=16045-16150msec WRITE: bw=1557KiB/s (1594kB/s), 389KiB/s-393KiB/s (398kB/s-402kB/s), io=24.6MiB (25.7MB), run=16045-16150msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17414: Fri May 10 07:59:28 2024 read: IOPS=76, BW=1222KiB/s (1252kB/s)(5920KiB/4843msec) clat (usec): min=1781, max=18744, avg=5203.15, stdev=3487.40 lat (usec): min=1781, max=18745, avg=5203.88, stdev=3487.30 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2474], | 30.00th=[ 2671], 40.00th=[ 3064], 50.00th=[ 3523], 60.00th=[ 4621], | 70.00th=[ 6194], 80.00th=[ 8455], 90.00th=[10683], 95.00th=[12125], | 99.00th=[17433], 99.50th=[17695], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 928, max= 1376, per=25.72%, avg=1187.22, stdev=149.23, samples=9 iops : min= 58, max= 86, avg=74.11, stdev= 9.23, samples=9 write: IOPS=82, BW=1315KiB/s (1346kB/s)(6368KiB/4843msec) clat (usec): min=2639, max=27721, avg=7311.49, stdev=4490.51 lat (usec): min=2641, max=27722, avg=7312.64, stdev=4490.51 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3195], 10.00th=[ 3425], 20.00th=[ 3752], | 30.00th=[ 4146], 40.00th=[ 4883], 50.00th=[ 5866], 60.00th=[ 6980], | 70.00th=[ 8586], 80.00th=[10028], 90.00th=[14222], 95.00th=[17171], | 99.00th=[24249], 99.50th=[26346], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1088, max= 1472, per=26.71%, avg=1297.44, stdev=140.30, samples=9 iops : min= 68, max= 92, avg=81.00, stdev= 8.72, samples=9 lat (msec) : 2=1.43%, 4=38.80%, 10=43.23%, 20=15.62%, 50=0.91% cpu : usr=0.17%, sys=9.89%, ctx=906, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17415: Fri May 10 07:59:28 2024 read: IOPS=69, BW=1107KiB/s (1134kB/s)(5744KiB/5188msec) clat (usec): min=1807, max=27571, avg=5710.50, stdev=4262.92 lat (usec): min=1807, max=27572, avg=5711.07, stdev=4262.92 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2507], | 30.00th=[ 2802], 40.00th=[ 3163], 50.00th=[ 3916], 60.00th=[ 5211], | 70.00th=[ 6915], 80.00th=[ 8979], 90.00th=[10421], 95.00th=[14484], | 99.00th=[21103], 99.50th=[25297], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 736, max= 1277, per=23.91%, avg=1103.70, stdev=170.49, samples=10 iops : min= 46, max= 79, avg=68.90, stdev=10.57, samples=10 write: IOPS=78, BW=1261KiB/s (1292kB/s)(6544KiB/5188msec) clat (usec): min=2633, max=32267, avg=7653.63, stdev=4972.50 lat (usec): min=2634, max=32269, avg=7654.77, stdev=4972.49 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3130], 10.00th=[ 3392], 20.00th=[ 3687], | 30.00th=[ 4228], 40.00th=[ 5080], 50.00th=[ 5932], 60.00th=[ 7177], | 70.00th=[ 8717], 80.00th=[10683], 90.00th=[14746], 95.00th=[17433], | 99.00th=[25560], 99.50th=[28443], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 1056, max= 1472, per=25.43%, avg=1234.90, stdev=117.28, samples=10 iops : min= 66, max= 92, avg=77.10, stdev= 7.40, samples=10 lat (msec) : 2=1.30%, 4=36.20%, 10=44.14%, 20=16.15%, 50=2.21% cpu : usr=0.12%, sys=9.50%, ctx=926, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17416: Fri May 10 07:59:28 2024 read: IOPS=79, BW=1268KiB/s (1299kB/s)(6256KiB/4933msec) clat (usec): min=1868, max=22118, avg=5176.20, stdev=3873.36 lat (usec): min=1869, max=22118, avg=5176.77, stdev=3873.37 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2073], 10.00th=[ 2212], 20.00th=[ 2442], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3490], 60.00th=[ 4146], | 70.00th=[ 5407], 80.00th=[ 8029], 90.00th=[10683], 95.00th=[13173], | 99.00th=[19530], 99.50th=[21365], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 990, max= 1696, per=27.34%, avg=1262.00, stdev=267.56, samples=9 iops : min= 61, max= 106, avg=78.78, stdev=16.84, samples=9 write: IOPS=76, BW=1223KiB/s (1252kB/s)(6032KiB/4933msec) clat (usec): min=2579, max=41091, avg=7700.88, stdev=5244.31 lat (usec): min=2580, max=41092, avg=7702.05, stdev=5244.33 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3195], 10.00th=[ 3392], 20.00th=[ 3785], | 30.00th=[ 4113], 40.00th=[ 4621], 50.00th=[ 5669], 60.00th=[ 6980], | 70.00th=[ 8455], 80.00th=[11469], 90.00th=[15926], 95.00th=[18220], | 99.00th=[23725], 99.50th=[28967], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 928, max= 1344, per=24.74%, avg=1201.44, stdev=128.07, samples=9 iops : min= 58, max= 84, avg=75.00, stdev= 8.00, samples=9 lat (msec) : 2=1.30%, 4=41.15%, 10=40.10%, 20=16.02%, 50=1.43% cpu : usr=0.18%, sys=9.73%, ctx=927, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17417: Fri May 10 07:59:28 2024 read: IOPS=74, BW=1198KiB/s (1227kB/s)(6032KiB/5033msec) clat (usec): min=1871, max=30562, avg=5534.57, stdev=4261.58 lat (usec): min=1872, max=30562, avg=5535.14, stdev=4261.59 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2089], 10.00th=[ 2278], 20.00th=[ 2474], | 30.00th=[ 2802], 40.00th=[ 3163], 50.00th=[ 3589], 60.00th=[ 4490], | 70.00th=[ 6194], 80.00th=[ 8586], 90.00th=[11076], 95.00th=[14746], | 99.00th=[20055], 99.50th=[23987], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 958, max= 1440, per=25.78%, avg=1190.20, stdev=169.50, samples=10 iops : min= 59, max= 90, avg=74.30, stdev=10.73, samples=10 write: IOPS=77, BW=1243KiB/s (1273kB/s)(6256KiB/5033msec) clat (usec): min=2548, max=26892, avg=7518.86, stdev=4908.38 lat (usec): min=2549, max=26893, avg=7520.04, stdev=4908.39 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3163], 10.00th=[ 3392], 20.00th=[ 3720], | 30.00th=[ 4228], 40.00th=[ 4817], 50.00th=[ 5538], 60.00th=[ 6849], | 70.00th=[ 8586], 80.00th=[10552], 90.00th=[15008], 95.00th=[18482], | 99.00th=[25297], 99.50th=[26346], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 1088, max= 1469, per=25.62%, avg=1244.50, stdev=112.67, samples=10 iops : min= 68, max= 91, avg=77.70, stdev= 6.86, samples=10 lat (msec) : 2=1.17%, 4=39.45%, 10=40.62%, 20=16.54%, 50=2.21% cpu : usr=0.06%, sys=9.64%, ctx=924, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4617KiB/s (4728kB/s), 1107KiB/s-1268KiB/s (1134kB/s-1299kB/s), io=23.4MiB (24.5MB), run=4843-5188msec WRITE: bw=4857KiB/s (4974kB/s), 1223KiB/s-1315KiB/s (1252kB/s-1346kB/s), io=24.6MiB (25.8MB), run=4843-5188msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17419: Fri May 10 07:59:28 2024 read: IOPS=69, BW=1111KiB/s (1138kB/s)(5920KiB/5327msec) clat (usec): min=56, max=24688, avg=10602.13, stdev=4134.92 lat (usec): min=56, max=24689, avg=10602.86, stdev=4135.02 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 83], 10.00th=[ 7177], 20.00th=[ 8586], | 30.00th=[ 9372], 40.00th=[10028], 50.00th=[10814], 60.00th=[11338], | 70.00th=[12125], 80.00th=[13042], 90.00th=[14746], 95.00th=[16712], | 99.00th=[22414], 99.50th=[24511], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 864, max= 1216, per=23.37%, avg=1036.50, stdev=120.47, samples=10 iops : min= 54, max= 76, avg=64.70, stdev= 7.42, samples=10 write: IOPS=74, BW=1195KiB/s (1224kB/s)(6368KiB/5327msec) clat (usec): min=620, max=23981, avg=3506.97, stdev=3544.19 lat (usec): min=620, max=23982, avg=3508.03, stdev=3544.18 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 889], | 30.00th=[ 1090], 40.00th=[ 2147], 50.00th=[ 2442], 60.00th=[ 2999], | 70.00th=[ 3621], 80.00th=[ 4883], 90.00th=[ 7570], 95.00th=[11338], | 99.00th=[17957], 99.50th=[19268], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 736, max= 1504, per=24.07%, avg=1122.90, stdev=232.90, samples=10 iops : min= 46, max= 94, avg=70.10, stdev=14.55, samples=10 lat (usec) : 100=2.73%, 250=0.39%, 750=2.21%, 1000=11.07% lat (msec) : 2=5.99%, 4=19.40%, 10=25.52%, 20=31.64%, 50=1.04% cpu : usr=0.15%, sys=20.67%, ctx=2603, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17420: Fri May 10 07:59:28 2024 read: IOPS=66, BW=1071KiB/s (1097kB/s)(5744KiB/5361msec) clat (usec): min=57, max=23055, avg=10888.52, stdev=4073.84 lat (usec): min=57, max=23056, avg=10889.12, stdev=4073.87 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 79], 10.00th=[ 7242], 20.00th=[ 8717], | 30.00th=[ 9503], 40.00th=[10159], 50.00th=[10945], 60.00th=[11731], | 70.00th=[12649], 80.00th=[13698], 90.00th=[15139], 95.00th=[16581], | 99.00th=[21103], 99.50th=[22152], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 864, max= 1152, per=23.15%, avg=1026.90, stdev=102.41, samples=10 iops : min= 54, max= 72, avg=64.10, stdev= 6.30, samples=10 write: IOPS=76, BW=1221KiB/s (1250kB/s)(6544KiB/5361msec) clat (usec): min=616, max=20185, avg=3528.90, stdev=3258.07 lat (usec): min=617, max=20186, avg=3529.92, stdev=3258.14 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 914], | 30.00th=[ 1467], 40.00th=[ 2311], 50.00th=[ 2540], 60.00th=[ 2966], | 70.00th=[ 3884], 80.00th=[ 4948], 90.00th=[ 7439], 95.00th=[ 9765], | 99.00th=[16581], 99.50th=[17171], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 896, max= 1696, per=24.89%, avg=1161.30, stdev=243.41, samples=10 iops : min= 56, max= 106, avg=72.50, stdev=15.26, samples=10 lat (usec) : 100=2.60%, 250=0.26%, 750=1.30%, 1000=11.72% lat (msec) : 2=4.69%, 4=20.31%, 10=27.34%, 20=30.60%, 50=1.17% cpu : usr=0.09%, sys=21.27%, ctx=2564, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17421: Fri May 10 07:59:28 2024 read: IOPS=72, BW=1161KiB/s (1189kB/s)(6256KiB/5387msec) clat (usec): min=51, max=22661, avg=10329.07, stdev=4529.80 lat (usec): min=51, max=22662, avg=10329.63, stdev=4529.82 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 68], 10.00th=[ 196], 20.00th=[ 8160], | 30.00th=[ 9241], 40.00th=[ 9896], 50.00th=[10552], 60.00th=[11469], | 70.00th=[12125], 80.00th=[13435], 90.00th=[15401], 95.00th=[17171], | 99.00th=[21103], 99.50th=[21890], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 800, max= 1376, per=24.16%, avg=1071.70, stdev=180.97, samples=10 iops : min= 50, max= 86, avg=66.90, stdev=11.26, samples=10 write: IOPS=69, BW=1120KiB/s (1147kB/s)(6032KiB/5387msec) clat (usec): min=627, max=28655, avg=3551.34, stdev=3606.92 lat (usec): min=628, max=28656, avg=3552.66, stdev=3606.83 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 766], 10.00th=[ 799], 20.00th=[ 898], | 30.00th=[ 1172], 40.00th=[ 2212], 50.00th=[ 2606], 60.00th=[ 3064], | 70.00th=[ 3720], 80.00th=[ 5080], 90.00th=[ 7111], 95.00th=[12125], | 99.00th=[18220], 99.50th=[19530], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 734, max= 1344, per=22.70%, avg=1059.00, stdev=196.73, samples=10 iops : min= 45, max= 84, avg=66.10, stdev=12.46, samples=10 lat (usec) : 100=4.82%, 250=0.39%, 750=1.82%, 1000=10.03% lat (msec) : 2=5.86%, 4=18.10%, 10=26.30%, 20=31.77%, 50=0.91% cpu : usr=0.09%, sys=20.70%, ctx=2583, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17422: Fri May 10 07:59:28 2024 read: IOPS=69, BW=1117KiB/s (1144kB/s)(6032KiB/5400msec) clat (usec): min=58, max=30651, avg=10682.55, stdev=4368.65 lat (usec): min=59, max=30651, avg=10683.11, stdev=4368.66 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 77], 10.00th=[ 7373], 20.00th=[ 8586], | 30.00th=[ 9503], 40.00th=[10159], 50.00th=[10683], 60.00th=[11338], | 70.00th=[12125], 80.00th=[13173], 90.00th=[15270], 95.00th=[16909], | 99.00th=[23987], 99.50th=[28967], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 896, max= 1245, per=23.59%, avg=1046.10, stdev=117.30, samples=10 iops : min= 56, max= 77, avg=65.30, stdev= 7.18, samples=10 write: IOPS=72, BW=1159KiB/s (1186kB/s)(6256KiB/5400msec) clat (usec): min=616, max=27803, avg=3489.31, stdev=3894.15 lat (usec): min=616, max=27805, avg=3490.23, stdev=3894.16 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 775], 10.00th=[ 791], 20.00th=[ 857], | 30.00th=[ 1045], 40.00th=[ 1926], 50.00th=[ 2474], 60.00th=[ 2868], | 70.00th=[ 3523], 80.00th=[ 4686], 90.00th=[ 7504], 95.00th=[11731], | 99.00th=[20841], 99.50th=[25560], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 832, max= 1472, per=23.38%, avg=1091.00, stdev=206.16, samples=10 iops : min= 52, max= 92, avg=68.10, stdev=12.97, samples=10 lat (usec) : 100=3.65%, 250=0.13%, 750=1.43%, 1000=13.02% lat (msec) : 2=6.12%, 4=17.45%, 10=23.31%, 20=33.33%, 50=1.56% cpu : usr=0.13%, sys=20.67%, ctx=2528, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4436KiB/s (4542kB/s), 1071KiB/s-1161KiB/s (1097kB/s-1189kB/s), io=23.4MiB (24.5MB), run=5327-5400msec WRITE: bw=4667KiB/s (4779kB/s), 1120KiB/s-1221KiB/s (1147kB/s-1250kB/s), io=24.6MiB (25.8MB), run=5327-5400msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17433: Fri May 10 07:59:32 2024 read: IOPS=2, BW=2181KiB/s (2233kB/s)(7168KiB/3287msec) clat (usec): min=502, max=890358, avg=396181.76, stdev=362327.83 lat (usec): min=503, max=890358, avg=396182.70, stdev=362327.75 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 15533], | 30.00th=[ 86508], 40.00th=[ 86508], 50.00th=[455082], 60.00th=[666895], | 70.00th=[666895], 80.00th=[666895], 90.00th=[893387], 95.00th=[893387], | 99.00th=[893387], 99.50th=[893387], 99.90th=[893387], 99.95th=[893387], | 99.99th=[893387] bw ( KiB/s): min= 1965, max= 4096, per=48.03%, avg=2992.00, stdev=1185.33, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 write: IOPS=1, BW=1558KiB/s (1595kB/s)(5120KiB/3287msec) clat (msec): min=46, max=288, avg=102.13, stdev=104.58 lat (msec): min=46, max=288, avg=102.17, stdev=104.59 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 63], 60.00th=[ 63], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 1969, max= 5907, per=37.63%, avg=3281.67, stdev=2273.61, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 500=16.67%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=38.34%, ctx=749, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17435: Fri May 10 07:59:32 2024 read: IOPS=2, BW=2612KiB/s (2675kB/s)(2048KiB/784msec) clat (msec): min=37, max=101, avg=69.71, stdev=45.08 lat (msec): min=37, max=101, avg=69.71, stdev=45.08 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 2048, max= 2048, per=32.87%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.8MiB/s (13.4MB/s)(10.0MiB/784msec) clat (msec): min=30, max=235, avg=62.16, stdev=61.21 lat (msec): min=30, max=235, avg=62.20, stdev=61.22 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 40], 50.00th=[ 46], 60.00th=[ 47], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 53], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 8192, max= 8192, per=93.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=30.27%, ctx=149, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17437: Fri May 10 07:59:32 2024 read: IOPS=9, BW=9.88MiB/s (10.4MB/s)(6144KiB/607msec) clat (usec): min=316, max=162909, avg=58582.68, stdev=71340.65 lat (usec): min=317, max=162910, avg=58583.33, stdev=71340.66 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 318], 10.00th=[ 318], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 392], 50.00th=[ 392], 60.00th=[ 62653], | 70.00th=[124257], 80.00th=[124257], 90.00th=[162530], 95.00th=[162530], | 99.00th=[162530], 99.50th=[162530], 99.90th=[162530], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9.88MiB/s (10.4MB/s)(6144KiB/607msec) clat (usec): min=24628, max=67629, avg=38089.59, stdev=16638.45 lat (usec): min=24638, max=67736, avg=38134.72, stdev=16677.86 clat percentiles (usec): | 1.00th=[24511], 5.00th=[24511], 10.00th=[24511], 20.00th=[26870], | 30.00th=[26870], 40.00th=[30016], 50.00th=[30016], 60.00th=[31589], | 70.00th=[47973], 80.00th=[47973], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 50=41.67%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=42.74%, ctx=182, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17439: Fri May 10 07:59:32 2024 read: IOPS=8, BW=9094KiB/s (9312kB/s)(5120KiB/563msec) clat (usec): min=342, max=64201, avg=24762.87, stdev=32493.24 lat (usec): min=343, max=64201, avg=24763.56, stdev=32493.13 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 343], 10.00th=[ 343], 20.00th=[ 343], | 30.00th=[ 424], 40.00th=[ 424], 50.00th=[ 2638], 60.00th=[ 2638], | 70.00th=[56361], 80.00th=[56361], 90.00th=[64226], 95.00th=[64226], | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.4MiB/s (13.0MB/s)(7168KiB/563msec) clat (msec): min=23, max=103, avg=61.00, stdev=29.81 lat (msec): min=23, max=103, avg=61.05, stdev=29.82 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 38], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 50], 60.00th=[ 68], | 70.00th=[ 68], 80.00th=[ 96], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min=12288, max=12288, per=100.00%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 4=8.33%, 50=33.33%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=35.77%, ctx=137, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6231KiB/s (6380kB/s), 2181KiB/s-9.88MiB/s (2233kB/s-10.4MB/s), io=20.0MiB (20.0MB), run=563-3287msec WRITE: bw=8723KiB/s (8932kB/s), 1558KiB/s-12.8MiB/s (1595kB/s-13.4MB/s), io=28.0MiB (29.4MB), run=563-3287msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17434: Fri May 10 07:59:32 2024 read: IOPS=2, BW=2122KiB/s (2173kB/s)(7168KiB/3378msec) clat (msec): min=21, max=799, avg=215.13, stdev=289.35 lat (msec): min=21, max=799, avg=215.13, stdev=289.35 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 25], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 118], | 70.00th=[ 118], 80.00th=[ 405], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 1961, max= 4096, per=49.25%, avg=2985.25, stdev=1184.81, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 write: IOPS=1, BW=1516KiB/s (1552kB/s)(5120KiB/3378msec) clat (msec): min=74, max=944, avg=374.04, stdev=356.90 lat (msec): min=74, max=944, avg=374.10, stdev=356.90 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 75], | 30.00th=[ 91], 40.00th=[ 91], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 1961, max= 3923, per=30.81%, avg=2615.00, stdev=1132.76, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=8.33%, 500=25.00%, 1000=16.67% cpu : usr=0.06%, sys=5.18%, ctx=46, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17436: Fri May 10 07:59:32 2024 read: IOPS=2, BW=2260KiB/s (2315kB/s)(2048KiB/906msec) clat (msec): min=85, max=106, avg=95.91, stdev=14.98 lat (msec): min=85, max=106, avg=95.91, stdev=14.98 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 86], 50.00th=[ 86], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 2048, max= 2048, per=33.78%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.0MiB/s (11.6MB/s)(10.0MiB/906msec) clat (msec): min=17, max=237, avg=71.06, stdev=63.01 lat (msec): min=17, max=237, avg=71.11, stdev=63.00 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 28], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 48], 60.00th=[ 57], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 101], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 6144, max= 6144, per=72.39%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=25.00% cpu : usr=0.22%, sys=14.92%, ctx=123, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17438: Fri May 10 07:59:32 2024 read: IOPS=7, BW=7959KiB/s (8150kB/s)(6144KiB/772msec) clat (msec): min=15, max=180, avg=54.70, stdev=62.73 lat (msec): min=15, max=180, avg=54.70, stdev=62.73 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[ 31], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 4096, max= 4096, per=67.57%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7959KiB/s (8150kB/s)(6144KiB/772msec) clat (msec): min=28, max=136, avg=73.12, stdev=39.68 lat (msec): min=28, max=136, avg=73.17, stdev=39.68 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 73], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 8192, max= 8192, per=96.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=8.33%, 250=25.00% cpu : usr=0.00%, sys=11.80%, ctx=71, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17440: Fri May 10 07:59:32 2024 read: IOPS=6, BW=6448KiB/s (6603kB/s)(5120KiB/794msec) clat (usec): min=15684, max=76342, avg=38614.31, stdev=26498.26 lat (usec): min=15685, max=76343, avg=38614.94, stdev=26498.25 clat percentiles (usec): | 1.00th=[15664], 5.00th=[15664], 10.00th=[15664], 20.00th=[15664], | 30.00th=[19006], 40.00th=[19006], 50.00th=[25560], 60.00th=[25560], | 70.00th=[56361], 80.00th=[56361], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 4096, max= 4096, per=67.57%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9028KiB/s (9244kB/s)(7168KiB/794msec) clat (msec): min=20, max=217, avg=85.17, stdev=69.94 lat (msec): min=20, max=217, avg=85.21, stdev=69.94 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 29], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 61], 60.00th=[ 81], | 70.00th=[ 81], 80.00th=[ 136], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 8192, max= 8192, per=96.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=41.67%, 250=16.67% cpu : usr=0.13%, sys=15.76%, ctx=126, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6063KiB/s (6208kB/s), 2122KiB/s-7959KiB/s (2173kB/s-8150kB/s), io=20.0MiB (20.0MB), run=772-3378msec WRITE: bw=8488KiB/s (8692kB/s), 1516KiB/s-11.0MiB/s (1552kB/s-11.6MB/s), io=28.0MiB (29.4MB), run=772-3378msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17454: Fri May 10 07:59:38 2024 read: IOPS=0, BW=2994KiB/s (3066kB/s)(12.0MiB/4104msec) clat (msec): min=47, max=2756, avg=1364.08, stdev=1355.89 lat (msec): min=47, max=2756, avg=1364.08, stdev=1355.89 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 1284], 50.00th=[ 1284], 60.00th=[ 1284], | 70.00th=[ 2769], 80.00th=[ 2769], 90.00th=[ 2769], 95.00th=[ 2769], | 99.00th=[ 2769], 99.50th=[ 2769], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 50=33.33% cpu : usr=0.02%, sys=30.86%, ctx=1067, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17455: Fri May 10 07:59:38 2024 read: IOPS=0, BW=874KiB/s (895kB/s)(4096KiB/4688msec) clat (nsec): min=83044k, max=83044k, avg=83044059.00, stdev= 0.00 lat (nsec): min=83046k, max=83046k, avg=83045511.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[83362], 5.00th=[83362], 10.00th=[83362], 20.00th=[83362], | 30.00th=[83362], 40.00th=[83362], 50.00th=[83362], 60.00th=[83362], | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1747KiB/s (1789kB/s)(8192KiB/4688msec) clat (msec): min=557, max=4044, avg=2301.01, stdev=2465.35 lat (msec): min=558, max=4044, avg=2301.19, stdev=2465.22 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 558], 10.00th=[ 558], 20.00th=[ 558], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 4044], | 70.00th=[ 4044], 80.00th=[ 4044], 90.00th=[ 4044], 95.00th=[ 4044], | 99.00th=[ 4044], 99.50th=[ 4044], 99.90th=[ 4044], 99.95th=[ 4044], | 99.99th=[ 4044] bw ( KiB/s): min= 7816, max= 7816, per=100.00%, avg=7816.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.02%, sys=5.78%, ctx=776, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17456: Fri May 10 07:59:38 2024 write: IOPS=0, BW=2734KiB/s (2800kB/s)(12.0MiB/4494msec) clat (msec): min=117, max=3966, avg=1494.96, stdev=2145.37 lat (msec): min=118, max=3967, avg=1495.15, stdev=2145.39 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 3977], 80.00th=[ 3977], 90.00th=[ 3977], 95.00th=[ 3977], | 99.00th=[ 3977], 99.50th=[ 3977], 99.90th=[ 3977], 99.95th=[ 3977], | 99.99th=[ 3977] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.90%, ctx=16, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17458: Fri May 10 07:59:38 2024 write: IOPS=0, BW=2919KiB/s (2989kB/s)(12.0MiB/4210msec) clat (msec): min=128, max=3926, avg=1402.85, stdev=2185.55 lat (msec): min=128, max=3926, avg=1403.14, stdev=2185.55 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 3943], 80.00th=[ 3943], 90.00th=[ 3943], 95.00th=[ 3943], | 99.00th=[ 3943], 99.50th=[ 3943], 99.90th=[ 3943], 99.95th=[ 3943], | 99.99th=[ 3943] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=8.60%, ctx=71, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3495KiB/s (3579kB/s), 874KiB/s-2994KiB/s (895kB/s-3066kB/s), io=16.0MiB (16.8MB), run=4104-4688msec WRITE: bw=6990KiB/s (7158kB/s), 1747KiB/s-2919KiB/s (1789kB/s-2989kB/s), io=32.0MiB (33.6MB), run=4210-4688msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17457: Fri May 10 07:59:38 2024 read: IOPS=0, BW=2695KiB/s (2759kB/s)(12.0MiB/4560msec) clat (msec): min=237, max=3039, avg=1517.74, stdev=1416.35 lat (msec): min=237, max=3039, avg=1517.74, stdev=1416.35 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 239], 10.00th=[ 239], 20.00th=[ 239], | 30.00th=[ 239], 40.00th=[ 1284], 50.00th=[ 1284], 60.00th=[ 1284], | 70.00th=[ 3037], 80.00th=[ 3037], 90.00th=[ 3037], 95.00th=[ 3037], | 99.00th=[ 3037], 99.50th=[ 3037], 99.90th=[ 3037], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=2.70%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17459: Fri May 10 07:59:38 2024 read: IOPS=0, BW=864KiB/s (885kB/s)(4096KiB/4741msec) clat (nsec): min=3990.8M, max=3990.8M, avg=3990817118.00, stdev= 0.00 lat (nsec): min=3990.8M, max=3990.8M, avg=3990818047.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 3977], 5.00th=[ 3977], 10.00th=[ 3977], 20.00th=[ 3977], | 30.00th=[ 3977], 40.00th=[ 3977], 50.00th=[ 3977], 60.00th=[ 3977], | 70.00th=[ 3977], 80.00th=[ 3977], 90.00th=[ 3977], 95.00th=[ 3977], | 99.00th=[ 3977], 99.50th=[ 3977], 99.90th=[ 3977], 99.95th=[ 3977], | 99.99th=[ 3977] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1728KiB/s (1769kB/s)(8192KiB/4741msec) clat (msec): min=289, max=452, avg=371.23, stdev=115.52 lat (msec): min=289, max=453, avg=371.47, stdev=115.58 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.02%, sys=2.26%, ctx=39, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17460: Fri May 10 07:59:38 2024 write: IOPS=0, BW=2637KiB/s (2700kB/s)(12.0MiB/4660msec) clat (msec): min=108, max=3963, avg=1550.23, stdev=2103.08 lat (msec): min=109, max=3963, avg=1550.39, stdev=2103.00 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 3977], 80.00th=[ 3977], 90.00th=[ 3977], 95.00th=[ 3977], | 99.00th=[ 3977], 99.50th=[ 3977], 99.90th=[ 3977], 99.95th=[ 3977], | 99.99th=[ 3977] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=2.23%, ctx=43, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17461: Fri May 10 07:59:38 2024 write: IOPS=0, BW=2667KiB/s (2731kB/s)(12.0MiB/4607msec) clat (msec): min=134, max=3173, avg=1532.50, stdev=1534.18 lat (msec): min=135, max=3174, avg=1532.86, stdev=1534.23 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 1284], 50.00th=[ 1284], 60.00th=[ 1284], | 70.00th=[ 3171], 80.00th=[ 3171], 90.00th=[ 3171], 95.00th=[ 3171], | 99.00th=[ 3171], 99.50th=[ 3171], 99.90th=[ 3171], 99.95th=[ 3171], | 99.99th=[ 3171] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=3.26%, ctx=86, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3456KiB/s (3539kB/s), 864KiB/s-2695KiB/s (885kB/s-2759kB/s), io=16.0MiB (16.8MB), run=4560-4741msec WRITE: bw=6912KiB/s (7078kB/s), 1728KiB/s-2667KiB/s (1769kB/s-2731kB/s), io=32.0MiB (33.6MB), run=4607-4741msec