== sanity test 398b: DIO and buffer IO race ============== 10:47:05 (1749480425) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18155 s, 42.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23692: Mon Jun 9 10:47:20 2025 read: IOPS=121, BW=485KiB/s (497kB/s)(5992KiB/12353msec) clat (usec): min=53, max=150287, avg=4540.67, stdev=4614.83 lat (usec): min=54, max=150289, avg=4541.29, stdev=4614.86 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 83], 10.00th=[ 2057], 20.00th=[ 2540], | 30.00th=[ 2933], 40.00th=[ 3458], 50.00th=[ 4015], 60.00th=[ 4621], | 70.00th=[ 5407], 80.00th=[ 6259], 90.00th=[ 7570], 95.00th=[ 9241], | 99.00th=[ 13698], 99.50th=[ 14484], 99.90th=[ 23987], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 271, max= 606, per=24.66%, avg=479.08, stdev=70.35, samples=24 iops : min= 67, max= 151, avg=119.67, stdev=17.68, samples=24 write: IOPS=127, BW=510KiB/s (522kB/s)(6296KiB/12353msec) clat (usec): min=414, max=24264, avg=3486.99, stdev=3215.76 lat (usec): min=414, max=24265, avg=3487.74, stdev=3215.80 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 635], | 30.00th=[ 824], 40.00th=[ 1565], 50.00th=[ 2835], 60.00th=[ 3425], | 70.00th=[ 4424], 80.00th=[ 5997], 90.00th=[ 7963], 95.00th=[ 9765], | 99.00th=[13698], 99.50th=[15401], 99.90th=[19530], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 327, max= 696, per=24.61%, avg=500.79, stdev=76.52, samples=24 iops : min= 81, max= 174, avg=125.08, stdev=19.24, samples=24 lat (usec) : 100=2.86%, 250=0.59%, 500=0.23%, 750=13.57%, 1000=4.52% lat (msec) : 2=3.91%, 4=31.87%, 10=38.25%, 20=4.04%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.12%, ctx=5398, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23693: Mon Jun 9 10:47:20 2025 read: IOPS=122, BW=489KiB/s (500kB/s)(5940KiB/12156msec) clat (usec): min=53, max=18005, avg=4325.23, stdev=2458.92 lat (usec): min=53, max=18006, avg=4327.08, stdev=2461.29 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 95], 10.00th=[ 2057], 20.00th=[ 2540], | 30.00th=[ 2868], 40.00th=[ 3261], 50.00th=[ 3884], 60.00th=[ 4555], | 70.00th=[ 5276], 80.00th=[ 6259], 90.00th=[ 7570], 95.00th=[ 8848], | 99.00th=[11731], 99.50th=[13698], 99.90th=[16909], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 351, max= 632, per=24.97%, avg=485.12, stdev=67.32, samples=24 iops : min= 87, max= 158, avg=121.17, stdev=16.90, samples=24 write: IOPS=130, BW=522KiB/s (535kB/s)(6348KiB/12156msec) clat (usec): min=444, max=164833, avg=3582.98, stdev=5203.29 lat (usec): min=444, max=164833, avg=3584.00, stdev=5203.20 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 660], | 30.00th=[ 857], 40.00th=[ 1663], 50.00th=[ 2835], 60.00th=[ 3523], | 70.00th=[ 4359], 80.00th=[ 5866], 90.00th=[ 7898], 95.00th=[ 9765], | 99.00th=[ 13960], 99.50th=[ 16188], 99.90th=[ 37487], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 384, max= 672, per=25.26%, avg=514.08, stdev=78.37, samples=24 iops : min= 96, max= 168, avg=128.42, stdev=19.64, samples=24 lat (usec) : 100=2.57%, 250=0.75%, 500=0.39%, 750=13.02%, 1000=4.39% lat (msec) : 2=4.69%, 4=33.46%, 10=37.27%, 20=3.32%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.65%, ctx=5324, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23694: Mon Jun 9 10:47:20 2025 read: IOPS=122, BW=489KiB/s (500kB/s)(6028KiB/12338msec) clat (usec): min=56, max=16332, avg=4407.78, stdev=2496.56 lat (usec): min=56, max=16332, avg=4409.04, stdev=2496.33 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 2057], 20.00th=[ 2540], | 30.00th=[ 2933], 40.00th=[ 3425], 50.00th=[ 4047], 60.00th=[ 4752], | 70.00th=[ 5407], 80.00th=[ 6194], 90.00th=[ 7635], 95.00th=[ 8848], | 99.00th=[12125], 99.50th=[13960], 99.90th=[15270], 99.95th=[16319], | 99.99th=[16319] bw ( KiB/s): min= 344, max= 616, per=24.88%, avg=483.46, stdev=65.11, samples=24 iops : min= 86, max= 154, avg=120.75, stdev=16.29, samples=24 write: IOPS=126, BW=507KiB/s (520kB/s)(6260KiB/12338msec) clat (usec): min=418, max=134759, avg=3594.97, stdev=4589.94 lat (usec): min=419, max=134760, avg=3595.68, stdev=4589.96 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 635], | 30.00th=[ 832], 40.00th=[ 1467], 50.00th=[ 2835], 60.00th=[ 3621], | 70.00th=[ 4621], 80.00th=[ 6063], 90.00th=[ 8160], 95.00th=[ 9634], | 99.00th=[ 13173], 99.50th=[ 14353], 99.90th=[ 23987], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 352, max= 624, per=24.54%, avg=499.42, stdev=65.91, samples=24 iops : min= 88, max= 156, avg=124.75, stdev=16.46, samples=24 lat (usec) : 100=2.54%, 250=0.94%, 500=0.55%, 750=13.25%, 1000=4.17% lat (msec) : 2=4.56%, 4=30.92%, 10=39.55%, 20=3.45%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.19%, ctx=5426, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23695: Mon Jun 9 10:47:20 2025 read: IOPS=122, BW=490KiB/s (502kB/s)(6052KiB/12345msec) clat (usec): min=53, max=28618, avg=4411.92, stdev=2682.85 lat (usec): min=53, max=28619, avg=4412.54, stdev=2682.90 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 95], 10.00th=[ 2008], 20.00th=[ 2474], | 30.00th=[ 2868], 40.00th=[ 3359], 50.00th=[ 3982], 60.00th=[ 4686], | 70.00th=[ 5407], 80.00th=[ 6259], 90.00th=[ 7767], 95.00th=[ 9110], | 99.00th=[12911], 99.50th=[15139], 99.90th=[22152], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 232, max= 584, per=24.88%, avg=483.50, stdev=72.01, samples=24 iops : min= 58, max= 146, avg=120.79, stdev=18.00, samples=24 write: IOPS=126, BW=505KiB/s (517kB/s)(6236KiB/12345msec) clat (usec): min=404, max=134407, avg=3590.94, stdev=4601.72 lat (usec): min=404, max=134409, avg=3591.79, stdev=4601.68 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 627], | 30.00th=[ 783], 40.00th=[ 1745], 50.00th=[ 2900], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 8029], 95.00th=[ 9896], | 99.00th=[ 13435], 99.50th=[ 15926], 99.90th=[ 27657], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 296, max= 672, per=24.56%, avg=499.83, stdev=77.65, samples=24 iops : min= 74, max= 168, avg=124.88, stdev=19.45, samples=24 lat (usec) : 100=2.73%, 250=0.85%, 500=0.68%, 750=13.74%, 1000=3.65% lat (msec) : 2=3.97%, 4=32.29%, 10=38.28%, 20=3.68%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.16%, ctx=5462, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1944KiB/s (1990kB/s), 485KiB/s-490KiB/s (497kB/s-502kB/s), io=23.4MiB (24.6MB), run=12156-12353msec WRITE: bw=2035KiB/s (2084kB/s), 505KiB/s-522KiB/s (517kB/s-535kB/s), io=24.6MiB (25.7MB), run=12156-12353msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23688: Mon Jun 9 10:47:24 2025 read: IOPS=92, BW=370KiB/s (379kB/s)(5992KiB/16209msec) clat (usec): min=1512, max=166723, avg=4326.95, stdev=5381.22 lat (usec): min=1513, max=166724, avg=4327.61, stdev=5381.22 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1926], 20.00th=[ 2073], | 30.00th=[ 2212], 40.00th=[ 2442], 50.00th=[ 2769], 60.00th=[ 3326], | 70.00th=[ 4293], 80.00th=[ 5735], 90.00th=[ 8979], 95.00th=[ 11600], | 99.00th=[ 17433], 99.50th=[ 19268], 99.90th=[ 30540], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 176, max= 800, per=24.86%, avg=367.41, stdev=187.33, samples=32 iops : min= 44, max= 200, avg=91.81, stdev=46.83, samples=32 write: IOPS=97, BW=388KiB/s (398kB/s)(6296KiB/16209msec) clat (usec): min=2238, max=29837, avg=6144.12, stdev=3919.38 lat (usec): min=2238, max=29838, avg=6144.86, stdev=3919.37 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2835], 10.00th=[ 3032], 20.00th=[ 3294], | 30.00th=[ 3523], 40.00th=[ 3818], 50.00th=[ 4359], 60.00th=[ 5407], | 70.00th=[ 7111], 80.00th=[ 8979], 90.00th=[11469], 95.00th=[13698], | 99.00th=[20055], 99.50th=[24249], 99.90th=[29492], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 184, max= 808, per=24.85%, avg=384.62, stdev=190.99, samples=32 iops : min= 46, max= 202, avg=96.13, stdev=47.72, samples=32 lat (msec) : 2=7.29%, 4=47.88%, 10=33.04%, 20=11.07%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=10.78%, ctx=3592, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23689: Mon Jun 9 10:47:24 2025 read: IOPS=91, BW=367KiB/s (375kB/s)(5940KiB/16201msec) clat (usec): min=1543, max=153585, avg=4273.21, stdev=5149.39 lat (usec): min=1543, max=153587, avg=4273.88, stdev=5149.42 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1844], 10.00th=[ 1926], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3195], | 70.00th=[ 4047], 80.00th=[ 5669], 90.00th=[ 8848], 95.00th=[ 11731], | 99.00th=[ 16319], 99.50th=[ 20055], 99.90th=[ 40633], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 160, max= 752, per=24.75%, avg=365.87, stdev=191.70, samples=32 iops : min= 40, max= 188, avg=91.44, stdev=47.90, samples=32 write: IOPS=97, BW=392KiB/s (401kB/s)(6348KiB/16201msec) clat (usec): min=2256, max=30304, avg=6175.31, stdev=3930.39 lat (usec): min=2257, max=30304, avg=6176.08, stdev=3930.38 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2868], 10.00th=[ 3064], 20.00th=[ 3294], | 30.00th=[ 3556], 40.00th=[ 3884], 50.00th=[ 4359], 60.00th=[ 5342], | 70.00th=[ 7242], 80.00th=[ 8979], 90.00th=[11600], 95.00th=[13698], | 99.00th=[20055], 99.50th=[22938], 99.90th=[30016], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 216, max= 760, per=25.02%, avg=387.38, stdev=182.73, samples=32 iops : min= 54, max= 190, avg=96.81, stdev=45.66, samples=32 lat (msec) : 2=7.45%, 4=48.37%, 10=32.52%, 20=10.87%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=10.87%, ctx=3522, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23690: Mon Jun 9 10:47:24 2025 read: IOPS=93, BW=374KiB/s (383kB/s)(6028KiB/16136msec) clat (usec): min=1562, max=26686, avg=4273.44, stdev=3310.95 lat (usec): min=1563, max=26687, avg=4274.12, stdev=3310.97 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1860], 10.00th=[ 1958], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3326], | 70.00th=[ 4228], 80.00th=[ 5997], 90.00th=[ 9110], 95.00th=[11469], | 99.00th=[15533], 99.50th=[16909], 99.90th=[26084], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 176, max= 792, per=25.19%, avg=372.38, stdev=180.99, samples=32 iops : min= 44, max= 198, avg=93.06, stdev=45.23, samples=32 write: IOPS=96, BW=388KiB/s (397kB/s)(6260KiB/16136msec) clat (msec): min=2, max=157, avg= 6.16, stdev= 5.36 lat (msec): min=2, max=157, avg= 6.16, stdev= 5.36 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 23], 99.90th=[ 35], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 208, max= 800, per=24.93%, avg=385.88, stdev=191.25, samples=32 iops : min= 52, max= 200, avg=96.44, stdev=47.79, samples=32 lat (msec) : 2=6.54%, 4=48.76%, 10=33.07%, 20=11.07%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=10.92%, ctx=3559, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23691: Mon Jun 9 10:47:24 2025 read: IOPS=93, BW=373KiB/s (382kB/s)(6052KiB/16240msec) clat (usec): min=1648, max=41372, avg=4284.22, stdev=3566.81 lat (usec): min=1649, max=41373, avg=4284.88, stdev=3566.82 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2073], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 4178], 80.00th=[ 5932], 90.00th=[ 9241], 95.00th=[12125], | 99.00th=[18220], 99.50th=[20841], 99.90th=[25035], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 176, max= 816, per=25.01%, avg=369.62, stdev=187.70, samples=32 iops : min= 44, max= 204, avg=92.38, stdev=46.90, samples=32 write: IOPS=95, BW=384KiB/s (393kB/s)(6236KiB/16240msec) clat (msec): min=2, max=156, avg= 6.22, stdev= 5.42 lat (msec): min=2, max=156, avg= 6.22, stdev= 5.42 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 31], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 192, max= 752, per=24.56%, avg=380.12, stdev=190.55, samples=32 iops : min= 48, max= 188, avg=95.00, stdev=47.62, samples=32 lat (msec) : 2=6.87%, 4=48.47%, 10=32.65%, 20=11.20%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=10.73%, ctx=3533, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1479KiB/s (1514kB/s), 367KiB/s-374KiB/s (375kB/s-383kB/s), io=23.4MiB (24.6MB), run=16136-16240msec WRITE: bw=1548KiB/s (1585kB/s), 384KiB/s-392KiB/s (393kB/s-401kB/s), io=24.6MiB (25.7MB), run=16136-16240msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23723: Mon Jun 9 10:47:28 2025 read: IOPS=108, BW=1735KiB/s (1777kB/s)(5920KiB/3412msec) clat (usec): min=56, max=20118, avg=4681.13, stdev=3107.42 lat (usec): min=56, max=20119, avg=4683.73, stdev=3109.23 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 73], 10.00th=[ 89], 20.00th=[ 2606], | 30.00th=[ 3228], 40.00th=[ 3752], 50.00th=[ 4359], 60.00th=[ 4948], | 70.00th=[ 6063], 80.00th=[ 7046], 90.00th=[ 8160], 95.00th=[ 9896], | 99.00th=[15401], 99.50th=[16450], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1216, max= 2048, per=23.96%, avg=1658.67, stdev=280.01, samples=6 iops : min= 76, max= 128, avg=103.67, stdev=17.50, samples=6 write: IOPS=116, BW=1866KiB/s (1911kB/s)(6368KiB/3412msec) clat (usec): min=702, max=20596, avg=4173.67, stdev=3522.19 lat (usec): min=703, max=20597, avg=4174.56, stdev=3522.23 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 816], 10.00th=[ 840], 20.00th=[ 947], | 30.00th=[ 1254], 40.00th=[ 2409], 50.00th=[ 3261], 60.00th=[ 4178], | 70.00th=[ 5538], 80.00th=[ 7177], 90.00th=[ 8717], 95.00th=[10683], | 99.00th=[16712], 99.50th=[19530], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 1440, max= 2368, per=24.97%, avg=1818.67, stdev=425.21, samples=6 iops : min= 90, max= 148, avg=113.67, stdev=26.58, samples=6 lat (usec) : 100=5.34%, 250=1.69%, 500=0.13%, 750=1.17%, 1000=10.68% lat (msec) : 2=7.81%, 4=25.00%, 10=42.84%, 20=5.08%, 50=0.26% cpu : usr=0.09%, sys=24.83%, ctx=1469, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23725: Mon Jun 9 10:47:28 2025 read: IOPS=103, BW=1660KiB/s (1700kB/s)(5744KiB/3460msec) clat (usec): min=58, max=21236, avg=4877.33, stdev=3025.45 lat (usec): min=58, max=21237, avg=4878.13, stdev=3025.39 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 85], 10.00th=[ 139], 20.00th=[ 2868], | 30.00th=[ 3326], 40.00th=[ 3818], 50.00th=[ 4293], 60.00th=[ 5145], | 70.00th=[ 6128], 80.00th=[ 6915], 90.00th=[ 8455], 95.00th=[10421], | 99.00th=[14091], 99.50th=[17171], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1152, max= 1792, per=22.96%, avg=1589.33, stdev=240.89, samples=6 iops : min= 72, max= 112, avg=99.33, stdev=15.06, samples=6 write: IOPS=118, BW=1891KiB/s (1937kB/s)(6544KiB/3460msec) clat (usec): min=630, max=19036, avg=3976.48, stdev=3561.15 lat (usec): min=631, max=19037, avg=3977.36, stdev=3561.17 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 791], 10.00th=[ 840], 20.00th=[ 938], | 30.00th=[ 1287], 40.00th=[ 2040], 50.00th=[ 2999], 60.00th=[ 3654], | 70.00th=[ 4817], 80.00th=[ 6718], 90.00th=[ 8979], 95.00th=[11469], | 99.00th=[15270], 99.50th=[17433], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1408, max= 1984, per=24.31%, avg=1770.67, stdev=223.24, samples=6 iops : min= 88, max= 124, avg=110.67, stdev=13.95, samples=6 lat (usec) : 100=3.12%, 250=2.08%, 750=0.91%, 1000=10.94% lat (msec) : 2=9.11%, 4=27.73%, 10=39.58%, 20=6.38%, 50=0.13% cpu : usr=0.17%, sys=24.69%, ctx=1486, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23727: Mon Jun 9 10:47:28 2025 read: IOPS=113, BW=1813KiB/s (1856kB/s)(6256KiB/3451msec) clat (usec): min=60, max=22377, avg=4593.50, stdev=2983.62 lat (usec): min=60, max=22377, avg=4594.09, stdev=2983.63 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 76], 10.00th=[ 102], 20.00th=[ 2704], | 30.00th=[ 3195], 40.00th=[ 3654], 50.00th=[ 4293], 60.00th=[ 5014], | 70.00th=[ 5932], 80.00th=[ 6849], 90.00th=[ 8094], 95.00th=[ 9372], | 99.00th=[13698], 99.50th=[16319], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1152, max= 2016, per=24.50%, avg=1696.00, stdev=320.00, samples=6 iops : min= 72, max= 126, avg=106.00, stdev=20.00, samples=6 write: IOPS=109, BW=1748KiB/s (1790kB/s)(6032KiB/3451msec) clat (usec): min=664, max=18775, avg=4186.12, stdev=3586.36 lat (usec): min=665, max=18776, avg=4186.99, stdev=3586.37 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 807], 10.00th=[ 848], 20.00th=[ 1012], | 30.00th=[ 1401], 40.00th=[ 2278], 50.00th=[ 3163], 60.00th=[ 3851], | 70.00th=[ 5407], 80.00th=[ 7111], 90.00th=[ 9372], 95.00th=[11207], | 99.00th=[16188], 99.50th=[16909], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1376, max= 2176, per=23.29%, avg=1696.00, stdev=330.70, samples=6 iops : min= 86, max= 136, avg=106.00, stdev=20.67, samples=6 lat (usec) : 100=5.08%, 250=1.95%, 500=0.39%, 750=1.30%, 1000=8.20% lat (msec) : 2=9.37%, 4=26.17%, 10=42.06%, 20=5.34%, 50=0.13% cpu : usr=0.17%, sys=24.26%, ctx=1461, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23728: Mon Jun 9 10:47:28 2025 read: IOPS=111, BW=1783KiB/s (1825kB/s)(6032KiB/3384msec) clat (usec): min=60, max=17531, avg=4883.09, stdev=3290.37 lat (usec): min=61, max=17531, avg=4883.69, stdev=3290.39 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 81], 10.00th=[ 88], 20.00th=[ 2704], | 30.00th=[ 3228], 40.00th=[ 3884], 50.00th=[ 4686], 60.00th=[ 5473], | 70.00th=[ 6128], 80.00th=[ 7111], 90.00th=[ 8848], 95.00th=[10552], | 99.00th=[15664], 99.50th=[17171], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1216, max= 1952, per=24.66%, avg=1706.67, stdev=269.00, samples=6 iops : min= 76, max= 122, avg=106.67, stdev=16.81, samples=6 write: IOPS=115, BW=1849KiB/s (1893kB/s)(6256KiB/3384msec) clat (usec): min=639, max=21017, avg=3824.16, stdev=3552.68 lat (usec): min=639, max=21018, avg=3825.59, stdev=3552.66 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 865], | 30.00th=[ 1004], 40.00th=[ 1369], 50.00th=[ 2769], 60.00th=[ 3654], | 70.00th=[ 4817], 80.00th=[ 6783], 90.00th=[ 9372], 95.00th=[10945], | 99.00th=[14091], 99.50th=[18220], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1280, max= 2176, per=24.46%, avg=1781.33, stdev=310.03, samples=6 iops : min= 80, max= 136, avg=111.33, stdev=19.38, samples=6 lat (usec) : 100=5.47%, 250=1.82%, 500=0.26%, 750=2.21%, 1000=13.02% lat (msec) : 2=8.85%, 4=20.31%, 10=41.41%, 20=6.51%, 50=0.13% cpu : usr=0.09%, sys=24.30%, ctx=1425, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6923KiB/s (7089kB/s), 1660KiB/s-1813KiB/s (1700kB/s-1856kB/s), io=23.4MiB (24.5MB), run=3384-3460msec WRITE: bw=7283KiB/s (7458kB/s), 1748KiB/s-1891KiB/s (1790kB/s-1937kB/s), io=24.6MiB (25.8MB), run=3384-3460msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23721: Mon Jun 9 10:47:29 2025 read: IOPS=80, BW=1281KiB/s (1312kB/s)(5920KiB/4621msec) clat (usec): min=1812, max=21152, avg=4611.84, stdev=3417.90 lat (usec): min=1812, max=21153, avg=4612.47, stdev=3417.91 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2147], 10.00th=[ 2278], 20.00th=[ 2442], | 30.00th=[ 2704], 40.00th=[ 2933], 50.00th=[ 3195], 60.00th=[ 3556], | 70.00th=[ 4228], 80.00th=[ 5735], 90.00th=[ 9765], 95.00th=[12780], | 99.00th=[17171], 99.50th=[17695], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 768, max= 2144, per=24.25%, avg=1240.89, stdev=522.09, samples=9 iops : min= 48, max= 134, avg=77.56, stdev=32.63, samples=9 write: IOPS=86, BW=1378KiB/s (1411kB/s)(6368KiB/4621msec) clat (usec): min=3024, max=33046, avg=7281.03, stdev=4286.00 lat (usec): min=3025, max=33047, avg=7282.12, stdev=4285.98 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3326], 10.00th=[ 3589], 20.00th=[ 4080], | 30.00th=[ 4490], 40.00th=[ 4948], 50.00th=[ 5538], 60.00th=[ 6652], | 70.00th=[ 8094], 80.00th=[10421], 90.00th=[13566], 95.00th=[16188], | 99.00th=[20841], 99.50th=[27657], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 768, max= 2528, per=25.16%, avg=1354.67, stdev=579.54, samples=9 iops : min= 48, max= 158, avg=84.67, stdev=36.22, samples=9 lat (msec) : 2=0.65%, 4=40.62%, 10=42.97%, 20=14.97%, 50=0.78% cpu : usr=0.19%, sys=11.13%, ctx=900, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23722: Mon Jun 9 10:47:29 2025 read: IOPS=76, BW=1227KiB/s (1257kB/s)(5744KiB/4680msec) clat (usec): min=1826, max=24627, avg=4730.43, stdev=3780.43 lat (usec): min=1826, max=24627, avg=4731.11, stdev=3780.41 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2180], 10.00th=[ 2343], 20.00th=[ 2507], | 30.00th=[ 2671], 40.00th=[ 2900], 50.00th=[ 3130], 60.00th=[ 3621], | 70.00th=[ 4293], 80.00th=[ 5932], 90.00th=[10159], 95.00th=[14091], | 99.00th=[18482], 99.50th=[22676], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 736, max= 2112, per=23.28%, avg=1191.11, stdev=534.21, samples=9 iops : min= 46, max= 132, avg=74.44, stdev=33.39, samples=9 write: IOPS=87, BW=1398KiB/s (1432kB/s)(6544KiB/4680msec) clat (usec): min=2722, max=27371, avg=7117.77, stdev=4221.79 lat (usec): min=2723, max=27372, avg=7119.03, stdev=4221.82 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3326], 10.00th=[ 3556], 20.00th=[ 3949], | 30.00th=[ 4228], 40.00th=[ 4686], 50.00th=[ 5276], 60.00th=[ 6390], | 70.00th=[ 8225], 80.00th=[10159], 90.00th=[13566], 95.00th=[15926], | 99.00th=[19792], 99.50th=[22676], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 832, max= 2624, per=25.16%, avg=1354.67, stdev=604.62, samples=9 iops : min= 52, max= 164, avg=84.67, stdev=37.79, samples=9 lat (msec) : 2=0.65%, 4=41.54%, 10=41.93%, 20=15.10%, 50=0.78% cpu : usr=0.19%, sys=11.18%, ctx=901, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23724: Mon Jun 9 10:47:29 2025 read: IOPS=85, BW=1364KiB/s (1397kB/s)(6256KiB/4585msec) clat (usec): min=1887, max=24601, avg=4730.77, stdev=3625.22 lat (usec): min=1888, max=24602, avg=4731.42, stdev=3625.20 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2089], 10.00th=[ 2311], 20.00th=[ 2442], | 30.00th=[ 2704], 40.00th=[ 2966], 50.00th=[ 3294], 60.00th=[ 3752], | 70.00th=[ 4490], 80.00th=[ 6063], 90.00th=[10028], 95.00th=[13173], | 99.00th=[18744], 99.50th=[21890], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 768, max= 2816, per=26.40%, avg=1351.11, stdev=647.51, samples=9 iops : min= 48, max= 176, avg=84.44, stdev=40.47, samples=9 write: IOPS=82, BW=1316KiB/s (1347kB/s)(6032KiB/4585msec) clat (usec): min=2702, max=25350, avg=7100.69, stdev=4167.09 lat (usec): min=2703, max=25352, avg=7101.86, stdev=4167.10 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3261], 10.00th=[ 3589], 20.00th=[ 4047], | 30.00th=[ 4424], 40.00th=[ 4817], 50.00th=[ 5407], 60.00th=[ 6587], | 70.00th=[ 7701], 80.00th=[ 9896], 90.00th=[13042], 95.00th=[15926], | 99.00th=[22938], 99.50th=[25297], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 960, max= 2176, per=23.71%, avg=1276.44, stdev=497.40, samples=9 iops : min= 60, max= 136, avg=79.78, stdev=31.09, samples=9 lat (msec) : 2=0.91%, 4=41.28%, 10=43.10%, 20=13.67%, 50=1.04% cpu : usr=0.17%, sys=11.37%, ctx=900, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23726: Mon Jun 9 10:47:29 2025 read: IOPS=82, BW=1313KiB/s (1344kB/s)(6032KiB/4595msec) clat (usec): min=1838, max=24327, avg=4870.97, stdev=3825.64 lat (usec): min=1839, max=24328, avg=4871.65, stdev=3825.61 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2540], | 30.00th=[ 2704], 40.00th=[ 2933], 50.00th=[ 3261], 60.00th=[ 3752], | 70.00th=[ 4621], 80.00th=[ 6259], 90.00th=[11207], 95.00th=[13566], | 99.00th=[20841], 99.50th=[22414], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 896, max= 2560, per=25.01%, avg=1280.00, stdev=545.88, samples=9 iops : min= 56, max= 160, avg=80.00, stdev=34.12, samples=9 write: IOPS=85, BW=1361KiB/s (1394kB/s)(6256KiB/4595msec) clat (usec): min=2745, max=28197, avg=6951.37, stdev=4043.32 lat (usec): min=2745, max=28198, avg=6952.59, stdev=4043.57 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3228], 10.00th=[ 3556], 20.00th=[ 4047], | 30.00th=[ 4424], 40.00th=[ 4883], 50.00th=[ 5342], 60.00th=[ 6259], | 70.00th=[ 7767], 80.00th=[ 9634], 90.00th=[12256], 95.00th=[15270], | 99.00th=[22152], 99.50th=[25297], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 896, max= 2336, per=24.83%, avg=1336.89, stdev=561.32, samples=9 iops : min= 56, max= 146, avg=83.56, stdev=35.08, samples=9 lat (msec) : 2=0.65%, 4=40.10%, 10=44.27%, 20=13.80%, 50=1.17% cpu : usr=0.20%, sys=11.28%, ctx=914, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5118KiB/s (5241kB/s), 1227KiB/s-1364KiB/s (1257kB/s-1397kB/s), io=23.4MiB (24.5MB), run=4585-4680msec WRITE: bw=5385KiB/s (5514kB/s), 1316KiB/s-1398KiB/s (1347kB/s-1432kB/s), io=24.6MiB (25.8MB), run=4585-4680msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23742: Mon Jun 9 10:47:31 2025 read: IOPS=15, BW=15.0MiB/s (16.8MB/s)(7168KiB/438msec) clat (usec): min=422, max=59681, avg=28944.32, stdev=22639.55 lat (usec): min=424, max=59682, avg=28945.54, stdev=22639.65 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 603], | 30.00th=[25035], 40.00th=[25035], 50.00th=[32375], 60.00th=[34341], | 70.00th=[34341], 80.00th=[50594], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] write: IOPS=11, BW=11.4MiB/s (11.0MB/s)(5120KiB/438msec) clat (usec): min=26817, max=33122, avg=29914.77, stdev=2630.42 lat (usec): min=26849, max=33188, avg=29956.69, stdev=2631.52 clat percentiles (usec): | 1.00th=[26870], 5.00th=[26870], 10.00th=[26870], 20.00th=[26870], | 30.00th=[27919], 40.00th=[27919], 50.00th=[30016], 60.00th=[30016], | 70.00th=[31851], 80.00th=[31851], 90.00th=[33162], 95.00th=[33162], | 99.00th=[33162], 99.50th=[33162], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=66.67%, 100=16.67% cpu : usr=0.00%, sys=43.71%, ctx=1236, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23743: Mon Jun 9 10:47:31 2025 read: IOPS=3, BW=3580KiB/s (3666kB/s)(2048KiB/572msec) clat (usec): min=420, max=30207, avg=15313.98, stdev=21062.60 lat (usec): min=421, max=30209, avg=15315.32, stdev=21063.08 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 420], 10.00th=[ 420], 20.00th=[ 420], | 30.00th=[ 420], 40.00th=[ 420], 50.00th=[ 420], 60.00th=[30278], | 70.00th=[30278], 80.00th=[30278], 90.00th=[30278], 95.00th=[30278], | 99.00th=[30278], 99.50th=[30278], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 4096, max= 4096, per=13.86%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=17, BW=17.5MiB/s (18.3MB/s)(10.0MiB/572msec) clat (usec): min=17757, max=94371, avg=52756.85, stdev=22557.78 lat (usec): min=17792, max=94456, avg=52820.12, stdev=22574.85 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[27395], | 30.00th=[40633], 40.00th=[41681], 50.00th=[50594], 60.00th=[51643], | 70.00th=[68682], 80.00th=[68682], 90.00th=[70779], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min=16384, max=16384, per=39.60%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=33.33%, 100=50.00% cpu : usr=0.35%, sys=43.61%, ctx=128, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23744: Mon Jun 9 10:47:31 2025 read: IOPS=8, BW=8904KiB/s (9118kB/s)(6144KiB/690msec) clat (usec): min=410, max=61585, avg=28435.01, stdev=24217.88 lat (usec): min=410, max=61586, avg=28438.57, stdev=24219.87 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[ 429], | 30.00th=[ 429], 40.00th=[28705], 50.00th=[28705], 60.00th=[38011], | 70.00th=[41681], 80.00th=[41681], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 4087, max= 4087, per=13.83%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8904KiB/s (9118kB/s)(6144KiB/690msec) clat (msec): min=33, max=165, avg=72.87, stdev=50.15 lat (msec): min=33, max=165, avg=72.92, stdev=50.16 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 51], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 6131, max= 6131, per=14.82%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=28.30%, ctx=283, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23745: Mon Jun 9 10:47:31 2025 read: IOPS=7, BW=7388KiB/s (7565kB/s)(5120KiB/693msec) clat (usec): min=450, max=58690, avg=19623.19, stdev=27198.74 lat (usec): min=451, max=58691, avg=19624.55, stdev=27198.37 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 449], 10.00th=[ 449], 20.00th=[ 449], | 30.00th=[ 465], 40.00th=[ 465], 50.00th=[ 578], 60.00th=[ 578], | 70.00th=[38011], 80.00th=[38011], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 2048, max= 2048, per=6.93%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.1MiB/s (10.6MB/s)(7168KiB/693msec) clat (msec): min=21, max=151, avg=72.70, stdev=46.18 lat (msec): min=21, max=151, avg=72.76, stdev=46.18 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 36], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 61], 60.00th=[ 74], | 70.00th=[ 74], 80.00th=[ 116], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 8192, max= 8192, per=19.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=33.33%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=25.00%, ctx=309, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.9MiB/s (30.3MB/s), 3580KiB/s-15.0MiB/s (3666kB/s-16.8MB/s), io=20.0MiB (20.0MB), run=438-693msec WRITE: bw=40.4MiB/s (42.4MB/s), 8904KiB/s-17.5MiB/s (9118kB/s-18.3MB/s), io=28.0MiB (29.4MB), run=438-693msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23738: Mon Jun 9 10:47:31 2025 read: IOPS=9, BW=9545KiB/s (9774kB/s)(7168KiB/751msec) clat (msec): min=14, max=140, avg=67.26, stdev=48.91 lat (msec): min=14, max=140, avg=67.26, stdev=48.91 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 30], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 46], 60.00th=[ 85], | 70.00th=[ 85], 80.00th=[ 123], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 6144, max= 6144, per=24.39%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6818KiB/s (6981kB/s)(5120KiB/751msec) clat (usec): min=16616, max=88624, avg=42526.28, stdev=31820.24 lat (usec): min=16687, max=88748, avg=42618.02, stdev=31821.21 clat percentiles (usec): | 1.00th=[16581], 5.00th=[16581], 10.00th=[16581], 20.00th=[16581], | 30.00th=[19792], 40.00th=[19792], 50.00th=[24511], 60.00th=[24511], | 70.00th=[63177], 80.00th=[63177], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 2048, max= 2048, per=5.81%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 250=16.67% cpu : usr=0.40%, sys=13.01%, ctx=78, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23739: Mon Jun 9 10:47:31 2025 read: IOPS=2, BW=2573KiB/s (2635kB/s)(2048KiB/796msec) clat (msec): min=17, max=179, avg=98.53, stdev=114.82 lat (msec): min=17, max=179, avg=98.53, stdev=114.82 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 18], 60.00th=[ 180], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 4096, max= 4096, per=16.26%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.6MiB/s (13.2MB/s)(10.0MiB/796msec) clat (msec): min=15, max=149, avg=58.45, stdev=48.33 lat (msec): min=16, max=149, avg=58.52, stdev=48.33 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 24], 40.00th=[ 26], 50.00th=[ 32], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 136], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min=10240, max=10240, per=29.04%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=25.00%, 100=25.00%, 250=25.00% cpu : usr=0.13%, sys=15.60%, ctx=63, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23740: Mon Jun 9 10:47:31 2025 read: IOPS=8, BW=9129KiB/s (9348kB/s)(6144KiB/673msec) clat (usec): min=14255, max=89585, avg=54899.11, stdev=29028.03 lat (usec): min=14256, max=89586, avg=54900.32, stdev=29028.20 clat percentiles (usec): | 1.00th=[14222], 5.00th=[14222], 10.00th=[14222], 20.00th=[35390], | 30.00th=[35390], 40.00th=[46400], 50.00th=[46400], 60.00th=[58459], | 70.00th=[84411], 80.00th=[84411], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 8192, max= 8192, per=32.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9129KiB/s (9348kB/s)(6144KiB/673msec) clat (msec): min=29, max=108, avg=50.02, stdev=29.83 lat (msec): min=29, max=109, avg=50.07, stdev=29.83 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 42], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 109], 95.00th=[ 109], | 99.00th=[ 109], 99.50th=[ 109], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 8192, max= 8192, per=23.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=14.29%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23741: Mon Jun 9 10:47:31 2025 read: IOPS=6, BW=6298KiB/s (6449kB/s)(5120KiB/813msec) clat (usec): min=13329, max=66057, avg=42369.91, stdev=24501.74 lat (usec): min=13331, max=66059, avg=42371.33, stdev=24501.42 clat percentiles (usec): | 1.00th=[13304], 5.00th=[13304], 10.00th=[13304], 20.00th=[13304], | 30.00th=[18744], 40.00th=[18744], 50.00th=[53216], 60.00th=[53216], | 70.00th=[60556], 80.00th=[60556], 90.00th=[65799], 95.00th=[65799], | 99.00th=[65799], 99.50th=[65799], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] write: IOPS=8, BW=8817KiB/s (9028kB/s)(7168KiB/813msec) clat (msec): min=18, max=150, avg=76.99, stdev=42.32 lat (msec): min=18, max=150, avg=77.06, stdev=42.32 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 53], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 71], 60.00th=[ 86], | 70.00th=[ 86], 80.00th=[ 105], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 8192, max= 8192, per=23.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 100=58.33%, 250=16.67% cpu : usr=0.00%, sys=13.30%, ctx=64, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.6MiB/s (25.8MB/s), 2573KiB/s-9545KiB/s (2635kB/s-9774kB/s), io=20.0MiB (20.0MB), run=673-813msec WRITE: bw=34.4MiB/s (36.1MB/s), 6818KiB/s-12.6MiB/s (6981kB/s-13.2MB/s), io=28.0MiB (29.4MB), run=673-813msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23755: Mon Jun 9 10:47:33 2025 read: IOPS=4, BW=19.7MiB/s (20.6MB/s)(12.0MiB/610msec) clat (msec): min=20, max=537, avg=199.19, stdev=292.78 lat (msec): min=20, max=537, avg=199.19, stdev=292.78 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 41], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 50=66.67%, 750=33.33% cpu : usr=0.00%, sys=7.55%, ctx=905, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23756: Mon Jun 9 10:47:33 2025 read: IOPS=1, BW=4108KiB/s (4207kB/s)(4096KiB/997msec) clat (nsec): min=591685k, max=591685k, avg=591685296.00, stdev= 0.00 lat (nsec): min=591687k, max=591687k, avg=591686785.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 592], 10.00th=[ 592], 20.00th=[ 592], | 30.00th=[ 592], 40.00th=[ 592], 50.00th=[ 592], 60.00th=[ 592], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] write: IOPS=2, BW=8217KiB/s (8414kB/s)(8192KiB/997msec) clat (msec): min=88, max=307, avg=197.74, stdev=155.18 lat (msec): min=88, max=307, avg=198.00, stdev=155.19 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 88], 50.00th=[ 88], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=30.22%, ctx=989, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23757: Mon Jun 9 10:47:33 2025 write: IOPS=2, BW=10.4MiB/s (10.9MB/s)(12.0MiB/1158msec) clat (msec): min=98, max=680, avg=378.54, stdev=291.75 lat (msec): min=98, max=680, avg=378.81, stdev=291.78 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 99], | 30.00th=[ 99], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 8192, max= 8192, per=28.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=23.77%, ctx=18, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23758: Mon Jun 9 10:47:33 2025 write: IOPS=2, BW=11.8MiB/s (12.4MB/s)(12.0MiB/1014msec) clat (msec): min=123, max=503, avg=332.74, stdev=193.03 lat (msec): min=123, max=504, avg=333.00, stdev=193.03 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 8192, max= 8192, per=28.95%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=31.10%, ctx=40, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.0MiB/s (16.8MB/s), 4108KiB/s-19.7MiB/s (4207kB/s-20.6MB/s), io=16.0MiB (16.8MB), run=610-997msec WRITE: bw=27.6MiB/s (28.0MB/s), 8217KiB/s-11.8MiB/s (8414kB/s-12.4MB/s), io=32.0MiB (33.6MB), run=997-1158msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23751: Mon Jun 9 10:47:34 2025 read: IOPS=3, BW=13.0MiB/s (13.7MB/s)(12.0MiB/921msec) clat (msec): min=55, max=525, avg=295.02, stdev=235.08 lat (msec): min=55, max=525, avg=295.02, stdev=235.08 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=11.96%, ctx=76, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23752: Mon Jun 9 10:47:34 2025 read: IOPS=0, BW=3419KiB/s (3501kB/s)(4096KiB/1198msec) clat (nsec): min=788653k, max=788653k, avg=788652845.00, stdev= 0.00 lat (nsec): min=788654k, max=788654k, avg=788653825.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 793], 5.00th=[ 793], 10.00th=[ 793], 20.00th=[ 793], | 30.00th=[ 793], 40.00th=[ 793], 50.00th=[ 793], 60.00th=[ 793], | 70.00th=[ 793], 80.00th=[ 793], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] write: IOPS=1, BW=6838KiB/s (7002kB/s)(8192KiB/1198msec) clat (msec): min=161, max=238, avg=199.84, stdev=54.32 lat (msec): min=161, max=238, avg=200.09, stdev=54.35 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 239], | 70.00th=[ 239], 80.00th=[ 239], 90.00th=[ 239], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=5.51%, ctx=43, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23753: Mon Jun 9 10:47:34 2025 write: IOPS=2, BW=9352KiB/s (9576kB/s)(12.0MiB/1314msec) clat (msec): min=123, max=868, avg=430.22, stdev=389.67 lat (msec): min=123, max=869, avg=430.50, stdev=389.67 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 8143, max= 8143, per=32.65%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=11.96%, ctx=208, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23754: Mon Jun 9 10:47:34 2025 write: IOPS=3, BW=13.2MiB/s (13.9MB/s)(12.0MiB/908msec) clat (msec): min=59, max=468, avg=292.43, stdev=210.44 lat (msec): min=59, max=468, avg=292.74, stdev=210.45 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 8175, max= 8175, per=32.78%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.11%, sys=7.39%, ctx=52, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.4MiB/s (14.0MB/s), 3419KiB/s-13.0MiB/s (3501kB/s-13.7MB/s), io=16.0MiB (16.8MB), run=921-1198msec WRITE: bw=24.4MiB/s (25.5MB/s), 6838KiB/s-13.2MiB/s (7002kB/s-13.9MB/s), io=32.0MiB (33.6MB), run=908-1314msec