== sanity test 398b: DIO and buffer IO race ============== 07:22:44 (1744543364) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 0.972695 s, 51.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7501: Sun Apr 13 07:22:58 2025 read: IOPS=127, BW=509KiB/s (521kB/s)(5992KiB/11773msec) clat (usec): min=54, max=174198, avg=3332.03, stdev=5355.11 lat (usec): min=54, max=174199, avg=3332.60, stdev=5355.11 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 68], 10.00th=[ 71], 20.00th=[ 93], | 30.00th=[ 1909], 40.00th=[ 2245], 50.00th=[ 2606], 60.00th=[ 3032], | 70.00th=[ 3884], 80.00th=[ 5342], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[ 13698], 99.50th=[ 16450], 99.90th=[ 34866], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 248, max= 760, per=24.52%, avg=499.96, stdev=116.80, samples=23 iops : min= 62, max= 190, avg=124.91, stdev=29.15, samples=23 write: IOPS=133, BW=535KiB/s (548kB/s)(6296KiB/11773msec) clat (usec): min=407, max=53479, avg=4228.41, stdev=4382.25 lat (usec): min=408, max=53479, avg=4229.40, stdev=4382.44 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 799], 40.00th=[ 2024], 50.00th=[ 3294], 60.00th=[ 4178], | 70.00th=[ 5342], 80.00th=[ 6652], 90.00th=[ 9634], 95.00th=[12256], | 99.00th=[19792], 99.50th=[23200], 99.90th=[29492], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 296, max= 824, per=24.49%, avg=522.87, stdev=107.69, samples=23 iops : min= 74, max= 206, avg=130.65, stdev=26.87, samples=23 lat (usec) : 100=10.12%, 250=1.66%, 500=1.56%, 750=13.48%, 1000=2.15% lat (msec) : 2=7.49%, 4=27.80%, 10=29.69%, 20=5.47%, 50=0.52% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=21.93%, ctx=5504, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7502: Sun Apr 13 07:22:58 2025 read: IOPS=127, BW=510KiB/s (522kB/s)(5940KiB/11654msec) clat (usec): min=51, max=42072, avg=3173.84, stdev=3060.35 lat (usec): min=52, max=42073, avg=3174.60, stdev=3060.41 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 68], 10.00th=[ 71], 20.00th=[ 85], | 30.00th=[ 1942], 40.00th=[ 2278], 50.00th=[ 2671], 60.00th=[ 3130], | 70.00th=[ 3916], 80.00th=[ 5080], 90.00th=[ 6849], 95.00th=[ 8455], | 99.00th=[12125], 99.50th=[14091], 99.90th=[28705], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 256, max= 768, per=24.89%, avg=507.61, stdev=132.03, samples=23 iops : min= 64, max= 192, avg=126.83, stdev=32.97, samples=23 write: IOPS=136, BW=545KiB/s (558kB/s)(6348KiB/11654msec) clat (usec): min=417, max=173538, avg=4295.75, stdev=5943.44 lat (usec): min=417, max=173539, avg=4296.70, stdev=5943.38 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 799], 40.00th=[ 1631], 50.00th=[ 3130], 60.00th=[ 4113], | 70.00th=[ 5342], 80.00th=[ 7177], 90.00th=[ 10159], 95.00th=[ 12649], | 99.00th=[ 17695], 99.50th=[ 19530], 99.90th=[ 32637], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 312, max= 880, per=25.19%, avg=537.83, stdev=135.56, samples=23 iops : min= 78, max= 220, avg=134.39, stdev=33.86, samples=23 lat (usec) : 100=10.61%, 250=1.50%, 500=1.53%, 750=13.41%, 1000=2.67% lat (msec) : 2=7.03%, 4=27.80%, 10=29.04%, 20=6.05%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=21.96%, ctx=5560, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7503: Sun Apr 13 07:22:58 2025 read: IOPS=129, BW=517KiB/s (530kB/s)(6028KiB/11652msec) clat (usec): min=53, max=173579, avg=3356.88, stdev=5491.03 lat (usec): min=54, max=173580, avg=3357.52, stdev=5491.06 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 67], 10.00th=[ 70], 20.00th=[ 83], | 30.00th=[ 1844], 40.00th=[ 2212], 50.00th=[ 2573], 60.00th=[ 3097], | 70.00th=[ 3851], 80.00th=[ 5276], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[ 12649], 99.50th=[ 19530], 99.90th=[ 45351], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 288, max= 776, per=25.22%, avg=514.26, stdev=108.11, samples=23 iops : min= 72, max= 194, avg=128.48, stdev=27.05, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6260KiB/11652msec) clat (usec): min=437, max=30941, avg=4131.04, stdev=3973.10 lat (usec): min=437, max=30941, avg=4134.17, stdev=3974.63 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 824], 40.00th=[ 1975], 50.00th=[ 3195], 60.00th=[ 4228], | 70.00th=[ 5342], 80.00th=[ 6915], 90.00th=[ 9634], 95.00th=[12256], | 99.00th=[16712], 99.50th=[19268], 99.90th=[24511], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 352, max= 728, per=25.00%, avg=533.70, stdev=98.18, samples=23 iops : min= 88, max= 182, avg=133.35, stdev=24.54, samples=23 lat (usec) : 100=11.17%, 250=1.40%, 500=1.14%, 750=13.54%, 1000=2.44% lat (msec) : 2=7.19%, 4=27.51%, 10=29.36%, 20=5.96%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=21.71%, ctx=5665, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7505: Sun Apr 13 07:22:58 2025 read: IOPS=131, BW=525KiB/s (537kB/s)(6052KiB/11533msec) clat (usec): min=52, max=167891, avg=3269.10, stdev=5148.25 lat (usec): min=52, max=167892, avg=3269.68, stdev=5148.24 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 68], 10.00th=[ 71], 20.00th=[ 92], | 30.00th=[ 1958], 40.00th=[ 2245], 50.00th=[ 2573], 60.00th=[ 3064], | 70.00th=[ 3818], 80.00th=[ 4948], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[ 11731], 99.50th=[ 15401], 99.90th=[ 34341], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 288, max= 832, per=25.72%, avg=524.35, stdev=120.21, samples=23 iops : min= 72, max= 208, avg=131.04, stdev=30.06, samples=23 write: IOPS=135, BW=541KiB/s (554kB/s)(6236KiB/11533msec) clat (usec): min=428, max=32798, avg=4144.70, stdev=4142.70 lat (usec): min=429, max=32799, avg=4145.50, stdev=4142.76 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 510], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 840], 40.00th=[ 1778], 50.00th=[ 3195], 60.00th=[ 4146], | 70.00th=[ 5211], 80.00th=[ 6915], 90.00th=[ 9765], 95.00th=[12387], | 99.00th=[17695], 99.50th=[21103], 99.90th=[28967], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 368, max= 800, per=25.29%, avg=540.00, stdev=107.17, samples=23 iops : min= 92, max= 200, avg=134.96, stdev=26.78, samples=23 lat (usec) : 100=10.42%, 250=1.30%, 500=1.79%, 750=12.57%, 1000=2.64% lat (msec) : 2=7.81%, 4=28.84%, 10=28.78%, 20=5.40%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.06%, ctx=5570, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2040KiB/s (2089kB/s), 509KiB/s-525KiB/s (521kB/s-537kB/s), io=23.4MiB (24.6MB), run=11533-11773msec WRITE: bw=2135KiB/s (2187kB/s), 535KiB/s-545KiB/s (548kB/s-558kB/s), io=24.6MiB (25.7MB), run=11533-11773msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7506: Sun Apr 13 07:23:19 2025 read: IOPS=45, BW=183KiB/s (188kB/s)(5992KiB/32687msec) clat (usec): min=1523, max=177423, avg=3327.33, stdev=5413.53 lat (usec): min=1524, max=177425, avg=3327.96, stdev=5413.55 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1795], 10.00th=[ 1844], 20.00th=[ 1909], | 30.00th=[ 1958], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2212], | 70.00th=[ 2442], 80.00th=[ 3261], 90.00th=[ 7046], 95.00th=[ 9372], | 99.00th=[ 14222], 99.50th=[ 16712], 99.90th=[ 45876], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 40, max= 304, per=25.06%, avg=182.91, stdev=72.03, samples=65 iops : min= 10, max= 76, avg=45.65, stdev=18.01, samples=65 write: IOPS=48, BW=193KiB/s (197kB/s)(6296KiB/32687msec) clat (usec): min=8465, max=57210, avg=17534.36, stdev=7317.98 lat (usec): min=8466, max=57210, avg=17535.10, stdev=7317.97 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[10290], 10.00th=[10814], 20.00th=[11600], | 30.00th=[12780], 40.00th=[14484], 50.00th=[16057], 60.00th=[17171], | 70.00th=[18482], 80.00th=[20317], 90.00th=[28967], 95.00th=[33817], | 99.00th=[43254], 99.50th=[45351], 99.90th=[55313], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 56, max= 272, per=25.17%, avg=192.26, stdev=63.48, samples=65 iops : min= 14, max= 68, avg=47.98, stdev=15.89, samples=65 lat (msec) : 2=18.13%, 4=22.69%, 10=7.49%, 20=40.92%, 50=10.64% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=5.23%, ctx=3467, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7507: Sun Apr 13 07:23:19 2025 read: IOPS=45, BW=181KiB/s (185kB/s)(5940KiB/32889msec) clat (usec): min=1602, max=178603, avg=3358.94, stdev=5453.75 lat (usec): min=1602, max=178604, avg=3359.63, stdev=5453.83 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1893], | 30.00th=[ 1958], 40.00th=[ 2008], 50.00th=[ 2114], 60.00th=[ 2212], | 70.00th=[ 2474], 80.00th=[ 3359], 90.00th=[ 6980], 95.00th=[ 9634], | 99.00th=[ 15401], 99.50th=[ 20055], 99.90th=[ 36439], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 16, max= 336, per=24.75%, avg=180.69, stdev=79.67, samples=65 iops : min= 4, max= 84, avg=45.09, stdev=19.97, samples=65 write: IOPS=48, BW=193KiB/s (198kB/s)(6348KiB/32889msec) clat (usec): min=8688, max=72235, avg=17514.37, stdev=7304.73 lat (usec): min=8689, max=72235, avg=17515.49, stdev=7304.88 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10290], 10.00th=[10945], 20.00th=[11731], | 30.00th=[13042], 40.00th=[14615], 50.00th=[16057], 60.00th=[17433], | 70.00th=[18482], 80.00th=[20317], 90.00th=[27395], 95.00th=[32375], | 99.00th=[41681], 99.50th=[53216], 99.90th=[67634], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 64, max= 264, per=25.15%, avg=192.14, stdev=61.57, samples=65 iops : min= 16, max= 66, avg=47.95, stdev=15.40, samples=65 lat (msec) : 2=18.33%, 4=21.74%, 10=7.78%, 20=41.02%, 50=10.84% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.07%, sys=5.27%, ctx=3425, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7508: Sun Apr 13 07:23:19 2025 read: IOPS=45, BW=184KiB/s (188kB/s)(6028KiB/32798msec) clat (usec): min=1558, max=43107, avg=3379.63, stdev=3265.15 lat (usec): min=1558, max=43108, avg=3380.28, stdev=3265.14 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1844], 20.00th=[ 1909], | 30.00th=[ 1975], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2573], 80.00th=[ 3785], 90.00th=[ 7767], 95.00th=[ 9765], | 99.00th=[15139], 99.50th=[20317], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 40, max= 352, per=25.14%, avg=183.55, stdev=71.16, samples=65 iops : min= 10, max= 88, avg=45.83, stdev=17.78, samples=65 write: IOPS=47, BW=191KiB/s (195kB/s)(6260KiB/32798msec) clat (msec): min=8, max=215, avg=17.64, stdev= 8.38 lat (msec): min=8, max=215, avg=17.64, stdev= 8.38 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 41], 99.50th=[ 44], 99.90th=[ 49], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 64, max= 264, per=24.88%, avg=190.08, stdev=61.77, samples=65 iops : min= 16, max= 66, avg=47.46, stdev=15.45, samples=65 lat (msec) : 2=17.35%, 4=22.17%, 10=8.85%, 20=40.43%, 50=11.17% lat (msec) : 250=0.03% cpu : usr=0.09%, sys=5.17%, ctx=3474, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7509: Sun Apr 13 07:23:19 2025 read: IOPS=46, BW=187KiB/s (191kB/s)(6052KiB/32447msec) clat (usec): min=1583, max=28980, avg=3191.49, stdev=2724.25 lat (usec): min=1583, max=28980, avg=3192.11, stdev=2724.26 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1795], 10.00th=[ 1844], 20.00th=[ 1909], | 30.00th=[ 1975], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2507], 80.00th=[ 3228], 90.00th=[ 6718], 95.00th=[ 9372], | 99.00th=[14091], 99.50th=[16712], 99.90th=[20579], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 56, max= 360, per=25.42%, avg=185.55, stdev=75.42, samples=64 iops : min= 14, max= 90, avg=46.33, stdev=18.88, samples=64 write: IOPS=48, BW=192KiB/s (197kB/s)(6236KiB/32447msec) clat (msec): min=8, max=194, avg=17.64, stdev= 8.50 lat (msec): min=8, max=194, avg=17.64, stdev= 8.50 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 29], 95.00th=[ 34], | 99.00th=[ 42], 99.50th=[ 46], 99.90th=[ 64], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 56, max= 256, per=25.07%, avg=191.55, stdev=62.70, samples=64 iops : min= 14, max= 64, avg=47.83, stdev=15.70, samples=64 lat (msec) : 2=17.22%, 4=23.86%, 10=7.65%, 20=40.40%, 50=10.64% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.07%, sys=5.31%, ctx=3436, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=730KiB/s (748kB/s), 181KiB/s-187KiB/s (185kB/s-191kB/s), io=23.4MiB (24.6MB), run=32447-32889msec WRITE: bw=764KiB/s (783kB/s), 191KiB/s-193KiB/s (195kB/s-198kB/s), io=24.6MiB (25.7MB), run=32447-32889msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7545: Sun Apr 13 07:23:23 2025 read: IOPS=145, BW=2331KiB/s (2387kB/s)(5920KiB/2540msec) clat (usec): min=60, max=26313, avg=3024.08, stdev=2726.89 lat (usec): min=61, max=26314, avg=3024.71, stdev=2726.86 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 77], 20.00th=[ 92], | 30.00th=[ 1958], 40.00th=[ 2442], 50.00th=[ 2835], 60.00th=[ 3326], | 70.00th=[ 3949], 80.00th=[ 4752], 90.00th=[ 5866], 95.00th=[ 7308], | 99.00th=[11076], 99.50th=[12911], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 1309, max= 2816, per=25.48%, avg=2322.60, stdev=583.75, samples=5 iops : min= 81, max= 176, avg=145.00, stdev=36.84, samples=5 write: IOPS=156, BW=2507KiB/s (2567kB/s)(6368KiB/2540msec) clat (usec): min=664, max=44009, avg=3357.13, stdev=4324.07 lat (usec): min=664, max=44011, avg=3359.06, stdev=4323.66 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 775], 20.00th=[ 857], | 30.00th=[ 963], 40.00th=[ 1205], 50.00th=[ 1680], 60.00th=[ 2835], | 70.00th=[ 3589], 80.00th=[ 5538], 90.00th=[ 7177], 95.00th=[ 9503], | 99.00th=[22676], 99.50th=[37487], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 1532, max= 3200, per=26.15%, avg=2508.00, stdev=621.08, samples=5 iops : min= 95, max= 200, avg=156.60, stdev=39.11, samples=5 lat (usec) : 100=10.42%, 250=2.60%, 500=0.13%, 750=3.26%, 1000=13.67% lat (msec) : 2=11.98%, 4=29.30%, 10=25.91%, 20=1.82%, 50=0.91% cpu : usr=0.12%, sys=27.69%, ctx=1508, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7547: Sun Apr 13 07:23:23 2025 read: IOPS=147, BW=2361KiB/s (2418kB/s)(5744KiB/2433msec) clat (usec): min=59, max=20221, avg=3196.17, stdev=2644.35 lat (usec): min=59, max=20221, avg=3196.77, stdev=2644.39 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 75], 10.00th=[ 84], 20.00th=[ 127], | 30.00th=[ 2212], 40.00th=[ 2573], 50.00th=[ 2868], 60.00th=[ 3458], | 70.00th=[ 4146], 80.00th=[ 4817], 90.00th=[ 5932], 95.00th=[ 7439], | 99.00th=[14877], 99.50th=[15270], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 1980, max= 2784, per=25.96%, avg=2367.00, stdev=335.12, samples=4 iops : min= 123, max= 174, avg=147.75, stdev=21.23, samples=4 write: IOPS=168, BW=2690KiB/s (2754kB/s)(6544KiB/2433msec) clat (usec): min=673, max=30381, avg=3093.06, stdev=3328.96 lat (usec): min=674, max=30382, avg=3094.07, stdev=3329.15 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 873], | 30.00th=[ 988], 40.00th=[ 1221], 50.00th=[ 1860], 60.00th=[ 2868], | 70.00th=[ 3523], 80.00th=[ 4686], 90.00th=[ 6849], 95.00th=[ 8848], | 99.00th=[17957], 99.50th=[21890], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 2235, max= 3104, per=27.34%, avg=2622.75, stdev=408.87, samples=4 iops : min= 139, max= 194, avg=163.75, stdev=25.77, samples=4 lat (usec) : 100=7.94%, 250=2.47%, 500=0.13%, 750=1.30%, 1000=15.23% lat (msec) : 2=11.85%, 4=32.55%, 10=26.30%, 20=1.69%, 50=0.52% cpu : usr=0.16%, sys=29.89%, ctx=1495, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7549: Sun Apr 13 07:23:23 2025 read: IOPS=152, BW=2446KiB/s (2504kB/s)(6256KiB/2558msec) clat (usec): min=57, max=21972, avg=3177.45, stdev=3138.21 lat (usec): min=57, max=21973, avg=3177.98, stdev=3138.24 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 68], 10.00th=[ 73], 20.00th=[ 86], | 30.00th=[ 338], 40.00th=[ 2442], 50.00th=[ 2835], 60.00th=[ 3261], | 70.00th=[ 4015], 80.00th=[ 5211], 90.00th=[ 6652], 95.00th=[ 7504], | 99.00th=[17433], 99.50th=[21103], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 928, max= 3168, per=27.03%, avg=2464.00, stdev=900.28, samples=5 iops : min= 58, max= 198, avg=154.00, stdev=56.27, samples=5 write: IOPS=147, BW=2358KiB/s (2415kB/s)(6032KiB/2558msec) clat (usec): min=679, max=37868, avg=3195.53, stdev=3922.61 lat (usec): min=680, max=37870, avg=3198.13, stdev=3921.77 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 873], | 30.00th=[ 947], 40.00th=[ 1074], 50.00th=[ 1483], 60.00th=[ 2704], | 70.00th=[ 3490], 80.00th=[ 5014], 90.00th=[ 7308], 95.00th=[ 9110], | 99.00th=[20841], 99.50th=[29754], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 1248, max= 2784, per=24.42%, avg=2342.40, stdev=634.54, samples=5 iops : min= 78, max= 174, avg=146.40, stdev=39.66, samples=5 lat (usec) : 100=13.02%, 250=2.21%, 500=0.13%, 750=2.73%, 1000=14.84% lat (msec) : 2=9.77%, 4=28.65%, 10=25.52%, 20=2.34%, 50=0.78% cpu : usr=0.16%, sys=26.87%, ctx=1504, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7551: Sun Apr 13 07:23:23 2025 read: IOPS=143, BW=2296KiB/s (2351kB/s)(6032KiB/2627msec) clat (usec): min=61, max=32344, avg=2891.97, stdev=3216.23 lat (usec): min=62, max=32344, avg=2892.56, stdev=3216.19 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 72], 10.00th=[ 76], 20.00th=[ 87], | 30.00th=[ 137], 40.00th=[ 2245], 50.00th=[ 2737], 60.00th=[ 3195], | 70.00th=[ 3916], 80.00th=[ 4752], 90.00th=[ 5735], 95.00th=[ 6980], | 99.00th=[13042], 99.50th=[31589], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 640, max= 2816, per=24.36%, avg=2220.80, stdev=891.88, samples=5 iops : min= 40, max= 176, avg=138.80, stdev=55.74, samples=5 write: IOPS=148, BW=2381KiB/s (2439kB/s)(6256KiB/2627msec) clat (usec): min=651, max=42688, avg=3652.10, stdev=5085.87 lat (usec): min=652, max=42689, avg=3652.91, stdev=5085.96 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 783], 20.00th=[ 865], | 30.00th=[ 971], 40.00th=[ 1237], 50.00th=[ 1876], 60.00th=[ 3097], | 70.00th=[ 3752], 80.00th=[ 5145], 90.00th=[ 7111], 95.00th=[10552], | 99.00th=[30278], 99.50th=[38536], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 768, max= 3584, per=24.49%, avg=2348.80, stdev=1040.27, samples=5 iops : min= 48, max= 224, avg=146.80, stdev=65.02, samples=5 lat (usec) : 100=13.02%, 250=2.86%, 500=0.26%, 750=2.73%, 1000=13.93% lat (msec) : 2=10.29%, 4=29.04%, 10=24.35%, 20=1.95%, 50=1.56% cpu : usr=0.27%, sys=25.10%, ctx=1495, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9118KiB/s (9336kB/s), 2296KiB/s-2446KiB/s (2351kB/s-2504kB/s), io=23.4MiB (24.5MB), run=2433-2627msec WRITE: bw=9593KiB/s (9823kB/s), 2358KiB/s-2690KiB/s (2415kB/s-2754kB/s), io=24.6MiB (25.8MB), run=2433-2627msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7546: Sun Apr 13 07:23:29 2025 read: IOPS=45, BW=725KiB/s (742kB/s)(5920KiB/8170msec) clat (usec): min=1784, max=21653, avg=3431.73, stdev=2861.06 lat (usec): min=1784, max=21654, avg=3432.40, stdev=2861.17 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1991], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2474], | 70.00th=[ 2737], 80.00th=[ 3720], 90.00th=[ 5735], 95.00th=[10159], | 99.00th=[17695], 99.50th=[20317], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 447, max= 1248, per=25.66%, avg=713.81, stdev=248.06, samples=16 iops : min= 27, max= 78, avg=44.50, stdev=15.54, samples=16 write: IOPS=48, BW=779KiB/s (798kB/s)(6368KiB/8170msec) clat (usec): min=9972, max=42489, avg=17300.29, stdev=5706.70 lat (usec): min=9973, max=42490, avg=17301.36, stdev=5706.79 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11207], 10.00th=[11731], 20.00th=[12387], | 30.00th=[13435], 40.00th=[14615], 50.00th=[16712], 60.00th=[17957], | 70.00th=[19006], 80.00th=[19792], 90.00th=[23725], 95.00th=[29230], | 99.00th=[39584], 99.50th=[42206], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 448, max= 992, per=26.43%, avg=773.75, stdev=193.53, samples=16 iops : min= 28, max= 62, avg=48.25, stdev=12.11, samples=16 lat (msec) : 2=2.86%, 4=36.20%, 10=6.64%, 20=43.75%, 50=10.55% cpu : usr=0.05%, sys=6.40%, ctx=875, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7548: Sun Apr 13 07:23:29 2025 read: IOPS=41, BW=667KiB/s (683kB/s)(5744KiB/8608msec) clat (usec): min=1824, max=18551, avg=3305.41, stdev=2473.37 lat (usec): min=1824, max=18551, avg=3306.09, stdev=2473.38 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1975], 10.00th=[ 2008], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2245], 50.00th=[ 2343], 60.00th=[ 2442], | 70.00th=[ 2802], 80.00th=[ 3621], 90.00th=[ 5800], 95.00th=[ 9372], | 99.00th=[14353], 99.50th=[15139], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 288, max= 1024, per=24.04%, avg=668.76, stdev=204.81, samples=17 iops : min= 18, max= 64, avg=41.65, stdev=12.75, samples=17 write: IOPS=47, BW=760KiB/s (778kB/s)(6544KiB/8608msec) clat (usec): min=10508, max=61041, avg=18040.63, stdev=5852.04 lat (usec): min=10509, max=61042, avg=18041.84, stdev=5852.22 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11338], 10.00th=[11731], 20.00th=[12911], | 30.00th=[14615], 40.00th=[16188], 50.00th=[17695], 60.00th=[18482], | 70.00th=[19268], 80.00th=[20841], 90.00th=[25035], 95.00th=[29230], | 99.00th=[35914], 99.50th=[39584], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 448, max= 928, per=25.95%, avg=759.53, stdev=173.28, samples=17 iops : min= 28, max= 58, avg=47.29, stdev=10.86, samples=17 lat (msec) : 2=4.04%, 4=34.11%, 10=6.64%, 20=41.93%, 50=13.15% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=6.12%, ctx=874, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7550: Sun Apr 13 07:23:29 2025 read: IOPS=49, BW=785KiB/s (804kB/s)(6256KiB/7966msec) clat (usec): min=1703, max=20902, avg=3324.41, stdev=2608.74 lat (usec): min=1704, max=20902, avg=3325.14, stdev=2608.73 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2474], | 70.00th=[ 2802], 80.00th=[ 3294], 90.00th=[ 5997], 95.00th=[ 8979], | 99.00th=[15926], 99.50th=[16909], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 352, max= 1440, per=28.75%, avg=799.73, stdev=343.75, samples=15 iops : min= 22, max= 90, avg=49.87, stdev=21.56, samples=15 write: IOPS=47, BW=757KiB/s (775kB/s)(6032KiB/7966msec) clat (usec): min=10161, max=42427, avg=17548.26, stdev=5231.24 lat (usec): min=10162, max=42428, avg=17549.43, stdev=5231.36 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11469], 10.00th=[11863], 20.00th=[13042], | 30.00th=[14615], 40.00th=[15795], 50.00th=[16909], 60.00th=[17957], | 70.00th=[18744], 80.00th=[20317], 90.00th=[23725], 95.00th=[28181], | 99.00th=[36963], 99.50th=[41157], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 480, max= 928, per=25.57%, avg=748.53, stdev=153.81, samples=15 iops : min= 30, max= 58, avg=46.67, stdev= 9.60, samples=15 lat (msec) : 2=2.47%, 4=40.10%, 10=6.12%, 20=40.62%, 50=10.68% cpu : usr=0.14%, sys=6.24%, ctx=872, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7552: Sun Apr 13 07:23:29 2025 read: IOPS=46, BW=744KiB/s (762kB/s)(6032KiB/8106msec) clat (usec): min=1762, max=25919, avg=3305.52, stdev=2815.98 lat (usec): min=1763, max=25920, avg=3306.15, stdev=2815.97 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1975], 10.00th=[ 2024], 20.00th=[ 2114], | 30.00th=[ 2147], 40.00th=[ 2212], 50.00th=[ 2311], 60.00th=[ 2409], | 70.00th=[ 2540], 80.00th=[ 3326], 90.00th=[ 5800], 95.00th=[ 9241], | 99.00th=[17957], 99.50th=[20579], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 320, max= 1184, per=26.52%, avg=737.81, stdev=238.16, samples=16 iops : min= 20, max= 74, avg=46.00, stdev=14.94, samples=16 write: IOPS=48, BW=772KiB/s (790kB/s)(6256KiB/8106msec) clat (usec): min=10451, max=42395, avg=17270.83, stdev=5497.04 lat (usec): min=10452, max=42396, avg=17271.97, stdev=5497.10 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11207], 10.00th=[11731], 20.00th=[12387], | 30.00th=[13566], 40.00th=[15139], 50.00th=[16581], 60.00th=[17957], | 70.00th=[19006], 80.00th=[20317], 90.00th=[22676], 95.00th=[28967], | 99.00th=[39060], 99.50th=[41681], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 415, max= 928, per=26.23%, avg=767.81, stdev=184.81, samples=16 iops : min= 25, max= 58, avg=47.88, stdev=11.64, samples=16 lat (msec) : 2=3.65%, 4=37.11%, 10=6.64%, 20=41.15%, 50=11.46% cpu : usr=0.10%, sys=6.12%, ctx=880, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2783KiB/s (2849kB/s), 667KiB/s-785KiB/s (683kB/s-804kB/s), io=23.4MiB (24.5MB), run=7966-8608msec WRITE: bw=2928KiB/s (2998kB/s), 757KiB/s-779KiB/s (775kB/s-798kB/s), io=24.6MiB (25.8MB), run=7966-8608msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7563: Sun Apr 13 07:23:30 2025 read: IOPS=15, BW=15.1MiB/s (15.9MB/s)(7168KiB/463msec) clat (usec): min=469, max=54485, avg=30173.61, stdev=21220.66 lat (usec): min=469, max=54486, avg=30174.55, stdev=21220.82 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 469], 10.00th=[ 469], 20.00th=[ 1029], | 30.00th=[32900], 40.00th=[32900], 50.00th=[36963], 60.00th=[40633], | 70.00th=[40633], 80.00th=[44827], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] write: IOPS=10, BW=10.8MiB/s (11.3MB/s)(5120KiB/463msec) clat (usec): min=22950, max=58067, avg=35529.98, stdev=13924.98 lat (usec): min=23011, max=58090, avg=35566.11, stdev=13913.51 clat percentiles (usec): | 1.00th=[22938], 5.00th=[22938], 10.00th=[22938], 20.00th=[22938], | 30.00th=[28443], 40.00th=[28443], 50.00th=[28705], 60.00th=[28705], | 70.00th=[39584], 80.00th=[39584], 90.00th=[57934], 95.00th=[57934], | 99.00th=[57934], 99.50th=[57934], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] lat (usec) : 500=8.33% lat (msec) : 2=8.33%, 50=66.67%, 100=16.67% cpu : usr=0.00%, sys=39.18%, ctx=636, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7564: Sun Apr 13 07:23:30 2025 read: IOPS=3, BW=3303KiB/s (3383kB/s)(2048KiB/620msec) clat (usec): min=30392, max=40441, avg=35416.68, stdev=7105.51 lat (usec): min=30392, max=40445, avg=35419.46, stdev=7108.55 clat percentiles (usec): | 1.00th=[30278], 5.00th=[30278], 10.00th=[30278], 20.00th=[30278], | 30.00th=[30278], 40.00th=[30278], 50.00th=[30278], 60.00th=[40633], | 70.00th=[40633], 80.00th=[40633], 90.00th=[40633], 95.00th=[40633], | 99.00th=[40633], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 3351, max= 3351, per=10.14%, avg=3351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.1MiB/s (16.9MB/s)(10.0MiB/620msec) clat (usec): min=18494, max=73674, avg=50277.46, stdev=19208.17 lat (usec): min=18542, max=73716, avg=50325.42, stdev=19198.80 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[25822], | 30.00th=[39584], 40.00th=[39584], 50.00th=[51119], 60.00th=[52691], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min=15083, max=15083, per=32.62%, avg=15083.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=50.00% cpu : usr=0.00%, sys=35.54%, ctx=214, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7565: Sun Apr 13 07:23:30 2025 read: IOPS=11, BW=11.5MiB/s (12.1MB/s)(6144KiB/521msec) clat (usec): min=470, max=56908, avg=23273.68, stdev=25631.20 lat (usec): min=471, max=56908, avg=23274.73, stdev=25631.08 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 469], 10.00th=[ 469], 20.00th=[ 570], | 30.00th=[ 570], 40.00th=[ 644], 50.00th=[ 644], 60.00th=[38011], | 70.00th=[42730], 80.00th=[42730], 90.00th=[56886], 95.00th=[56886], | 99.00th=[56886], 99.50th=[56886], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] write: IOPS=11, BW=11.5MiB/s (12.1MB/s)(6144KiB/521msec) clat (usec): min=23443, max=98090, avg=51203.06, stdev=27611.52 lat (usec): min=23464, max=98137, avg=51241.20, stdev=27623.09 clat percentiles (usec): | 1.00th=[23462], 5.00th=[23462], 10.00th=[23462], 20.00th=[29492], | 30.00th=[29492], 40.00th=[40633], 50.00th=[40633], 60.00th=[48497], | 70.00th=[66847], 80.00th=[66847], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=50.00%, 100=25.00% cpu : usr=0.00%, sys=29.42%, ctx=236, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7566: Sun Apr 13 07:23:30 2025 read: IOPS=11, BW=11.6MiB/s (12.1MB/s)(5120KiB/432msec) clat (usec): min=309, max=43018, avg=23768.35, stdev=15753.31 lat (usec): min=309, max=43019, avg=23769.09, stdev=15753.27 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 310], 10.00th=[ 310], 20.00th=[ 310], | 30.00th=[19268], 40.00th=[19268], 50.00th=[25560], 60.00th=[25560], | 70.00th=[30802], 80.00th=[30802], 90.00th=[43254], 95.00th=[43254], | 99.00th=[43254], 99.50th=[43254], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] write: IOPS=16, BW=16.2MiB/s (16.0MB/s)(7168KiB/432msec) clat (msec): min=20, max=100, avg=43.46, stdev=26.16 lat (msec): min=21, max=100, avg=43.50, stdev=26.16 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 28], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 38], 60.00th=[ 38], | 70.00th=[ 38], 80.00th=[ 46], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=75.00%, 250=8.33% cpu : usr=0.00%, sys=36.43%, ctx=83, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=32.3MiB/s (33.8MB/s), 3303KiB/s-15.1MiB/s (3383kB/s-15.9MB/s), io=20.0MiB (20.0MB), run=432-620msec WRITE: bw=45.2MiB/s (47.4MB/s), 10.8MiB/s-16.2MiB/s (11.3MB/s-16.0MB/s), io=28.0MiB (29.4MB), run=432-620msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7568: Sun Apr 13 07:23:30 2025 read: IOPS=10, BW=10.5MiB/s (11.1MB/s)(7168KiB/664msec) clat (msec): min=16, max=120, avg=55.46, stdev=38.70 lat (msec): min=16, max=120, avg=55.47, stdev=38.70 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 28], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 37], 60.00th=[ 60], | 70.00th=[ 60], 80.00th=[ 95], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min=10219, max=10219, per=37.17%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7711KiB/s (7896kB/s)(5120KiB/664msec) clat (usec): min=27254, max=86897, avg=44302.28, stdev=24600.70 lat (usec): min=27319, max=86932, avg=44340.92, stdev=24595.00 clat percentiles (usec): | 1.00th=[27132], 5.00th=[27132], 10.00th=[27132], 20.00th=[27132], | 30.00th=[29492], 40.00th=[29492], 50.00th=[34866], 60.00th=[34866], | 70.00th=[43254], 80.00th=[43254], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 2043, max= 2043, per=5.31%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=12.97%, ctx=49, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7569: Sun Apr 13 07:23:30 2025 read: IOPS=2, BW=2749KiB/s (2815kB/s)(2048KiB/745msec) clat (usec): min=13440, max=16751, avg=15095.97, stdev=2341.44 lat (usec): min=13441, max=16753, avg=15097.12, stdev=2341.87 clat percentiles (usec): | 1.00th=[13435], 5.00th=[13435], 10.00th=[13435], 20.00th=[13435], | 30.00th=[13435], 40.00th=[13435], 50.00th=[13435], 60.00th=[16712], | 70.00th=[16712], 80.00th=[16712], 90.00th=[16712], 95.00th=[16712], | 99.00th=[16712], 99.50th=[16712], 99.90th=[16712], 99.95th=[16712], | 99.99th=[16712] bw ( KiB/s): min= 2048, max= 2048, per=7.45%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.4MiB/s (14.1MB/s)(10.0MiB/745msec) clat (msec): min=28, max=138, avg=67.28, stdev=38.58 lat (msec): min=28, max=138, avg=67.34, stdev=38.58 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 36], 50.00th=[ 52], 60.00th=[ 62], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 113], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min=10240, max=10240, per=26.61%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=11.29%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7570: Sun Apr 13 07:23:30 2025 read: IOPS=8, BW=9049KiB/s (9266kB/s)(6144KiB/679msec) clat (msec): min=29, max=127, avg=58.77, stdev=35.41 lat (msec): min=29, max=127, avg=58.77, stdev=35.41 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 50], | 70.00th=[ 61], 80.00th=[ 61], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 8192, max= 8192, per=29.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9049KiB/s (9266kB/s)(6144KiB/679msec) clat (usec): min=28653, max=71871, avg=49475.14, stdev=17231.04 lat (usec): min=28683, max=71940, avg=49539.37, stdev=17228.82 clat percentiles (usec): | 1.00th=[28705], 5.00th=[28705], 10.00th=[28705], 20.00th=[36963], | 30.00th=[36963], 40.00th=[38536], 50.00th=[38536], 60.00th=[56886], | 70.00th=[64226], 80.00th=[64226], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 8192, max= 8192, per=21.29%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=12.09%, ctx=38, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7571: Sun Apr 13 07:23:30 2025 read: IOPS=7, BW=7388KiB/s (7565kB/s)(5120KiB/693msec) clat (usec): min=24724, max=89437, avg=43630.36, stdev=26152.57 lat (usec): min=24725, max=89438, avg=43631.20, stdev=26152.54 clat percentiles (usec): | 1.00th=[24773], 5.00th=[24773], 10.00th=[24773], 20.00th=[24773], | 30.00th=[29754], 40.00th=[29754], 50.00th=[36439], 60.00th=[36439], | 70.00th=[38011], 80.00th=[38011], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 4096, max= 4096, per=14.90%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.1MiB/s (10.6MB/s)(7168KiB/693msec) clat (msec): min=31, max=118, avg=60.44, stdev=30.78 lat (msec): min=31, max=118, avg=60.50, stdev=30.78 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 35], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 55], 60.00th=[ 62], | 70.00th=[ 62], 80.00th=[ 81], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 8192, max= 8192, per=21.29%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 250=8.33% cpu : usr=0.14%, sys=14.02%, ctx=61, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.8MiB/s (28.1MB/s), 2749KiB/s-10.5MiB/s (2815kB/s-11.1MB/s), io=20.0MiB (20.0MB), run=664-745msec WRITE: bw=37.6MiB/s (39.4MB/s), 7711KiB/s-13.4MiB/s (7896kB/s-14.1MB/s), io=28.0MiB (29.4MB), run=664-745msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7578: Sun Apr 13 07:23:32 2025 read: IOPS=6, BW=26.9MiB/s (28.2MB/s)(12.0MiB/446msec) clat (msec): min=112, max=175, avg=145.39, stdev=31.55 lat (msec): min=112, max=175, avg=145.39, stdev=31.55 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=12.36%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7579: Sun Apr 13 07:23:32 2025 read: IOPS=1, BW=6273KiB/s (6423kB/s)(4096KiB/653msec) clat (nsec): min=409747k, max=409747k, avg=409746610.00, stdev= 0.00 lat (nsec): min=409748k, max=409748k, avg=409748036.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 409], 10.00th=[ 409], 20.00th=[ 409], | 30.00th=[ 409], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 409], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] write: IOPS=3, BW=12.3MiB/s (12.8MB/s)(8192KiB/653msec) clat (msec): min=68, max=159, avg=114.42, stdev=64.40 lat (msec): min=69, max=160, avg=114.65, stdev=64.47 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=11.04%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7581: Sun Apr 13 07:23:32 2025 write: IOPS=7, BW=30.2MiB/s (31.7MB/s)(12.0MiB/397msec) clat (msec): min=47, max=262, avg=127.38, stdev=117.45 lat (msec): min=47, max=264, avg=128.47, stdev=118.74 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 72], 50.00th=[ 72], 60.00th=[ 72], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.25%, sys=21.72%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7582: Sun Apr 13 07:23:32 2025 write: IOPS=5, BW=21.1MiB/s (22.1MB/s)(12.0MiB/569msec) clat (msec): min=54, max=440, avg=186.23, stdev=220.25 lat (msec): min=54, max=440, avg=186.49, stdev=220.34 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 64], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min=16351, max=16351, per=32.58%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.18%, sys=9.33%, ctx=42, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.5MiB/s (25.7MB/s), 6273KiB/s-26.9MiB/s (6423kB/s-28.2MB/s), io=16.0MiB (16.8MB), run=446-653msec WRITE: bw=49.0MiB/s (51.4MB/s), 12.3MiB/s-30.2MiB/s (12.8MB/s-31.7MB/s), io=32.0MiB (33.6MB), run=397-653msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7580: Sun Apr 13 07:23:32 2025 read: IOPS=10, BW=41.8MiB/s (43.8MB/s)(12.0MiB/287msec) clat (msec): min=53, max=110, avg=90.89, stdev=32.33 lat (msec): min=53, max=110, avg=90.89, stdev=32.33 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 109], 50.00th=[ 109], 60.00th=[ 109], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=28.28%, ctx=1043, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7583: Sun Apr 13 07:23:32 2025 read: IOPS=2, BW=9942KiB/s (10.2MB/s)(4096KiB/412msec) clat (nsec): min=123250k, max=123250k, avg=123250158.00, stdev= 0.00 lat (nsec): min=123251k, max=123251k, avg=123251125.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] write: IOPS=4, BW=19.4MiB/s (20.4MB/s)(8192KiB/412msec) clat (msec): min=102, max=175, avg=139.31, stdev=51.49 lat (msec): min=103, max=176, avg=140.07, stdev=52.15 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] lat (msec) : 250=100.00% cpu : usr=0.24%, sys=47.20%, ctx=901, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7584: Sun Apr 13 07:23:32 2025 write: IOPS=4, BW=17.9MiB/s (18.8MB/s)(12.0MiB/671msec) clat (msec): min=169, max=311, avg=218.80, stdev=80.73 lat (msec): min=169, max=312, avg=219.10, stdev=80.79 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min=16384, max=16384, per=34.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=36.27%, ctx=51, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7585: Sun Apr 13 07:23:32 2025 write: IOPS=4, BW=17.4MiB/s (18.2MB/s)(12.0MiB/690msec) clat (msec): min=125, max=406, avg=225.52, stdev=157.02 lat (msec): min=126, max=406, avg=225.81, stdev=156.97 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min=16384, max=16384, per=34.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=33.38%, ctx=92, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=38.8MiB/s (40.7MB/s), 9942KiB/s-41.8MiB/s (10.2MB/s-43.8MB/s), io=16.0MiB (16.8MB), run=287-412msec WRITE: bw=46.4MiB/s (48.6MB/s), 17.4MiB/s-19.4MiB/s (18.2MB/s-20.4MB/s), io=32.0MiB (33.6MB), run=412-690msec