== sanity test 398b: DIO and buffer IO race ============== 01:08:25 (1747804105) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18064 s, 42.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2575: Wed May 21 01:08:39 2025 read: IOPS=130, BW=521KiB/s (533kB/s)(5992KiB/11507msec) clat (usec): min=64, max=155605, avg=4281.86, stdev=4508.93 lat (usec): min=64, max=155607, avg=4283.41, stdev=4508.92 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1811], 10.00th=[ 2073], 20.00th=[ 2507], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3785], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 5735], 90.00th=[ 6783], 95.00th=[ 8160], | 99.00th=[ 12518], 99.50th=[ 13435], 99.90th=[ 23462], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 272, max= 640, per=25.35%, avg=520.35, stdev=78.55, samples=23 iops : min= 68, max= 160, avg=130.09, stdev=19.64, samples=23 write: IOPS=136, BW=547KiB/s (560kB/s)(6296KiB/11507msec) clat (usec): min=448, max=19925, avg=3181.40, stdev=2845.08 lat (usec): min=449, max=19926, avg=3182.10, stdev=2845.08 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 627], | 30.00th=[ 832], 40.00th=[ 1975], 50.00th=[ 2606], 60.00th=[ 3195], | 70.00th=[ 3949], 80.00th=[ 5276], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[12649], 99.50th=[16581], 99.90th=[19530], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 328, max= 744, per=25.41%, avg=546.09, stdev=91.06, samples=23 iops : min= 82, max= 186, avg=136.52, stdev=22.77, samples=23 lat (usec) : 100=1.76%, 250=0.29%, 500=0.33%, 750=13.51%, 1000=3.32% lat (msec) : 2=5.37%, 4=38.02%, 10=35.25%, 20=2.08%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.82%, ctx=5633, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2577: Wed May 21 01:08:39 2025 read: IOPS=128, BW=514KiB/s (526kB/s)(5940KiB/11557msec) clat (usec): min=54, max=23902, avg=4223.34, stdev=2425.19 lat (usec): min=54, max=23902, avg=4224.92, stdev=2424.48 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 128], 10.00th=[ 2073], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3228], 50.00th=[ 3720], 60.00th=[ 4359], | 70.00th=[ 5080], 80.00th=[ 5866], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[12518], 99.50th=[14222], 99.90th=[16909], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 304, max= 640, per=24.93%, avg=511.83, stdev=70.59, samples=23 iops : min= 76, max= 160, avg=127.87, stdev=17.69, samples=23 write: IOPS=137, BW=549KiB/s (562kB/s)(6348KiB/11557msec) clat (usec): min=435, max=164914, avg=3283.22, stdev=4904.74 lat (usec): min=435, max=164916, avg=3283.95, stdev=4904.77 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 627], | 30.00th=[ 799], 40.00th=[ 1909], 50.00th=[ 2606], 60.00th=[ 3261], | 70.00th=[ 4080], 80.00th=[ 5342], 90.00th=[ 7111], 95.00th=[ 8717], | 99.00th=[ 11994], 99.50th=[ 13435], 99.90th=[ 17695], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 360, max= 712, per=25.38%, avg=545.52, stdev=87.65, samples=23 iops : min= 90, max= 178, avg=136.30, stdev=21.94, samples=23 lat (usec) : 100=2.05%, 250=0.49%, 500=0.36%, 750=14.55%, 1000=3.03% lat (msec) : 2=4.65%, 4=37.21%, 10=35.16%, 20=2.44%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.81%, ctx=5521, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2578: Wed May 21 01:08:39 2025 read: IOPS=128, BW=515KiB/s (528kB/s)(6028KiB/11696msec) clat (usec): min=56, max=19621, avg=4179.79, stdev=2323.73 lat (usec): min=57, max=19621, avg=4183.14, stdev=2322.02 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 100], 10.00th=[ 2040], 20.00th=[ 2442], | 30.00th=[ 2868], 40.00th=[ 3261], 50.00th=[ 3785], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 6980], 95.00th=[ 8094], | 99.00th=[11863], 99.50th=[12911], 99.90th=[16909], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 160, max= 656, per=24.85%, avg=510.26, stdev=102.68, samples=23 iops : min= 40, max= 164, avg=127.57, stdev=25.67, samples=23 write: IOPS=133, BW=535KiB/s (548kB/s)(6260KiB/11696msec) clat (usec): min=460, max=164783, avg=3392.78, stdev=5071.22 lat (usec): min=460, max=164785, avg=3393.47, stdev=5071.25 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 627], | 30.00th=[ 799], 40.00th=[ 1844], 50.00th=[ 2606], 60.00th=[ 3228], | 70.00th=[ 4113], 80.00th=[ 5538], 90.00th=[ 7504], 95.00th=[ 9110], | 99.00th=[ 13304], 99.50th=[ 14353], 99.90th=[ 25560], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 208, max= 656, per=24.59%, avg=528.35, stdev=96.83, samples=23 iops : min= 52, max= 164, avg=132.09, stdev=24.21, samples=23 lat (usec) : 100=2.47%, 250=0.42%, 500=0.23%, 750=14.32%, 1000=3.26% lat (msec) : 2=4.95%, 4=35.42%, 10=35.97%, 20=2.86%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.57%, ctx=5508, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2579: Wed May 21 01:08:39 2025 read: IOPS=130, BW=522KiB/s (535kB/s)(6052KiB/11594msec) clat (usec): min=54, max=20933, avg=4295.23, stdev=2473.95 lat (usec): min=54, max=20933, avg=4295.84, stdev=2473.93 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 128], 10.00th=[ 2057], 20.00th=[ 2540], | 30.00th=[ 2900], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4490], | 70.00th=[ 5080], 80.00th=[ 5866], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[12780], 99.50th=[15401], 99.90th=[18744], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 248, max= 688, per=25.31%, avg=519.61, stdev=92.95, samples=23 iops : min= 62, max= 172, avg=129.87, stdev=23.25, samples=23 write: IOPS=134, BW=538KiB/s (551kB/s)(6236KiB/11594msec) clat (usec): min=433, max=152342, avg=3208.94, stdev=4717.06 lat (usec): min=434, max=152344, avg=3209.62, stdev=4717.07 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 611], | 30.00th=[ 717], 40.00th=[ 1369], 50.00th=[ 2606], 60.00th=[ 3130], | 70.00th=[ 4047], 80.00th=[ 5276], 90.00th=[ 7177], 95.00th=[ 8586], | 99.00th=[ 11469], 99.50th=[ 13173], 99.90th=[ 23987], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 328, max= 800, per=24.97%, avg=536.65, stdev=107.90, samples=23 iops : min= 82, max= 200, avg=134.13, stdev=27.00, samples=23 lat (usec) : 100=2.25%, 250=0.39%, 500=0.42%, 750=15.49%, 1000=2.57% lat (msec) : 2=5.53%, 4=34.51%, 10=36.20%, 20=2.51%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.27%, ctx=5506, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2053KiB/s (2102kB/s), 514KiB/s-522KiB/s (526kB/s-535kB/s), io=23.4MiB (24.6MB), run=11507-11696msec WRITE: bw=2149KiB/s (2201kB/s), 535KiB/s-549KiB/s (548kB/s-562kB/s), io=24.6MiB (25.7MB), run=11507-11696msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2572: Wed May 21 01:08:43 2025 read: IOPS=96, BW=386KiB/s (395kB/s)(5992KiB/15529msec) clat (usec): min=1440, max=169045, avg=4234.84, stdev=5225.50 lat (usec): min=1440, max=169046, avg=4235.60, stdev=5225.46 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1893], 20.00th=[ 2073], | 30.00th=[ 2278], 40.00th=[ 2507], 50.00th=[ 2933], 60.00th=[ 3490], | 70.00th=[ 4228], 80.00th=[ 5473], 90.00th=[ 8586], 95.00th=[ 10945], | 99.00th=[ 15533], 99.50th=[ 17695], 99.90th=[ 21890], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 183, max= 808, per=24.96%, avg=385.94, stdev=165.33, samples=31 iops : min= 45, max= 202, avg=96.39, stdev=41.40, samples=31 write: IOPS=101, BW=405KiB/s (415kB/s)(6296KiB/15529msec) clat (usec): min=2167, max=27564, avg=5789.94, stdev=3394.74 lat (usec): min=2168, max=27565, avg=5790.64, stdev=3394.73 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2769], 10.00th=[ 2933], 20.00th=[ 3228], | 30.00th=[ 3490], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5407], | 70.00th=[ 6652], 80.00th=[ 8160], 90.00th=[10159], 95.00th=[12256], | 99.00th=[18220], 99.50th=[20317], 99.90th=[26084], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 223, max= 768, per=25.03%, avg=405.03, stdev=164.62, samples=31 iops : min= 55, max= 192, avg=101.16, stdev=41.23, samples=31 lat (msec) : 2=7.81%, 4=46.61%, 10=36.91%, 20=8.27%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=10.98%, ctx=3614, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2573: Wed May 21 01:08:43 2025 read: IOPS=95, BW=383KiB/s (392kB/s)(5940KiB/15497msec) clat (usec): min=1426, max=168339, avg=4168.52, stdev=5249.98 lat (usec): min=1427, max=168342, avg=4169.14, stdev=5250.00 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 2040], | 30.00th=[ 2245], 40.00th=[ 2474], 50.00th=[ 2835], 60.00th=[ 3359], | 70.00th=[ 4015], 80.00th=[ 5538], 90.00th=[ 8455], 95.00th=[ 10552], | 99.00th=[ 15664], 99.50th=[ 18220], 99.90th=[ 27657], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 151, max= 768, per=24.69%, avg=381.73, stdev=176.33, samples=30 iops : min= 37, max= 192, avg=95.33, stdev=44.15, samples=30 write: IOPS=102, BW=410KiB/s (419kB/s)(6348KiB/15497msec) clat (usec): min=2215, max=27351, avg=5818.99, stdev=3349.10 lat (usec): min=2216, max=27352, avg=5819.72, stdev=3349.10 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2835], 10.00th=[ 3064], 20.00th=[ 3294], | 30.00th=[ 3589], 40.00th=[ 3916], 50.00th=[ 4490], 60.00th=[ 5407], | 70.00th=[ 6587], 80.00th=[ 8160], 90.00th=[10552], 95.00th=[12518], | 99.00th=[17695], 99.50th=[20055], 99.90th=[26346], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 215, max= 776, per=24.86%, avg=402.27, stdev=160.22, samples=30 iops : min= 53, max= 194, avg=100.47, stdev=40.14, samples=30 lat (msec) : 2=8.30%, 4=47.14%, 10=35.16%, 20=8.98%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=11.06%, ctx=3579, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2574: Wed May 21 01:08:43 2025 read: IOPS=98, BW=393KiB/s (402kB/s)(6028KiB/15341msec) clat (usec): min=1490, max=21957, avg=4101.96, stdev=2967.95 lat (usec): min=1490, max=21957, avg=4102.57, stdev=2967.94 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1811], 10.00th=[ 1909], 20.00th=[ 2073], | 30.00th=[ 2278], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 3458], | 70.00th=[ 4178], 80.00th=[ 5407], 90.00th=[ 8586], 95.00th=[10552], | 99.00th=[14877], 99.50th=[16057], 99.90th=[19792], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 159, max= 792, per=25.24%, avg=390.27, stdev=165.94, samples=30 iops : min= 39, max= 198, avg=97.47, stdev=41.56, samples=30 write: IOPS=102, BW=408KiB/s (418kB/s)(6260KiB/15341msec) clat (msec): min=2, max=172, avg= 5.81, stdev= 5.45 lat (msec): min=2, max=172, avg= 5.81, stdev= 5.45 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 19], 99.50th=[ 24], 99.90th=[ 34], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 183, max= 752, per=24.96%, avg=403.87, stdev=164.92, samples=30 iops : min= 45, max= 188, avg=100.87, stdev=41.30, samples=30 lat (msec) : 2=7.62%, 4=48.50%, 10=35.12%, 20=8.37%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.25%, ctx=3577, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2576: Wed May 21 01:08:43 2025 read: IOPS=98, BW=393KiB/s (402kB/s)(6052KiB/15412msec) clat (usec): min=1521, max=21212, avg=4044.70, stdev=2855.89 lat (usec): min=1522, max=21212, avg=4045.32, stdev=2855.87 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1926], 20.00th=[ 2073], | 30.00th=[ 2278], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3425], | 70.00th=[ 4146], 80.00th=[ 5342], 90.00th=[ 8356], 95.00th=[10552], | 99.00th=[13698], 99.50th=[16319], 99.90th=[20579], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 143, max= 816, per=25.09%, avg=387.90, stdev=163.82, samples=30 iops : min= 35, max= 204, avg=96.90, stdev=41.02, samples=30 write: IOPS=101, BW=405KiB/s (414kB/s)(6236KiB/15412msec) clat (msec): min=2, max=171, avg= 5.91, stdev= 5.40 lat (msec): min=2, max=171, avg= 5.91, stdev= 5.40 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 23], 99.90th=[ 34], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 167, max= 736, per=24.75%, avg=400.43, stdev=160.33, samples=30 iops : min= 41, max= 184, avg=100.03, stdev=40.14, samples=30 lat (msec) : 2=7.62%, 4=46.74%, 10=36.95%, 20=8.30%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=10.96%, ctx=3578, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1546KiB/s (1583kB/s), 383KiB/s-393KiB/s (392kB/s-402kB/s), io=23.4MiB (24.6MB), run=15341-15529msec WRITE: bw=1619KiB/s (1658kB/s), 405KiB/s-410KiB/s (414kB/s-419kB/s), io=24.6MiB (25.7MB), run=15341-15529msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2609: Wed May 21 01:08:47 2025 read: IOPS=112, BW=1802KiB/s (1845kB/s)(5920KiB/3286msec) clat (usec): min=58, max=20577, avg=4464.80, stdev=2814.84 lat (usec): min=59, max=20578, avg=4465.37, stdev=2814.89 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 76], 10.00th=[ 277], 20.00th=[ 2704], | 30.00th=[ 3097], 40.00th=[ 3490], 50.00th=[ 4080], 60.00th=[ 4555], | 70.00th=[ 5342], 80.00th=[ 6259], 90.00th=[ 7635], 95.00th=[ 9241], | 99.00th=[13698], 99.50th=[19006], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 1056, max= 2048, per=23.80%, avg=1728.00, stdev=359.77, samples=6 iops : min= 66, max= 128, avg=108.00, stdev=22.49, samples=6 write: IOPS=121, BW=1938KiB/s (1984kB/s)(6368KiB/3286msec) clat (usec): min=620, max=22714, avg=3917.47, stdev=3382.21 lat (usec): min=621, max=22715, avg=3918.42, stdev=3382.28 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 865], | 30.00th=[ 1188], 40.00th=[ 2343], 50.00th=[ 2966], 60.00th=[ 3916], | 70.00th=[ 4686], 80.00th=[ 6390], 90.00th=[ 8455], 95.00th=[11076], | 99.00th=[14222], 99.50th=[16581], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1248, max= 2208, per=24.71%, avg=1888.00, stdev=357.48, samples=6 iops : min= 78, max= 138, avg=118.00, stdev=22.34, samples=6 lat (usec) : 100=4.17%, 250=0.52%, 500=0.13%, 750=2.08%, 1000=11.98% lat (msec) : 2=4.95%, 4=31.38%, 10=39.71%, 20=4.82%, 50=0.26% cpu : usr=0.18%, sys=24.05%, ctx=1595, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2610: Wed May 21 01:08:47 2025 read: IOPS=108, BW=1742KiB/s (1783kB/s)(5744KiB/3298msec) clat (usec): min=63, max=18728, avg=4513.76, stdev=2640.90 lat (usec): min=64, max=18728, avg=4514.32, stdev=2640.91 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 86], 10.00th=[ 2024], 20.00th=[ 2638], | 30.00th=[ 3130], 40.00th=[ 3654], 50.00th=[ 4293], 60.00th=[ 4948], | 70.00th=[ 5538], 80.00th=[ 6194], 90.00th=[ 7701], 95.00th=[ 8848], | 99.00th=[13173], 99.50th=[17171], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1216, max= 2080, per=23.43%, avg=1701.33, stdev=303.86, samples=6 iops : min= 76, max= 130, avg=106.33, stdev=18.99, samples=6 write: IOPS=124, BW=1984KiB/s (2032kB/s)(6544KiB/3298msec) clat (usec): min=623, max=18670, avg=3910.41, stdev=3242.65 lat (usec): min=623, max=18671, avg=3911.29, stdev=3242.63 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 889], | 30.00th=[ 1172], 40.00th=[ 2311], 50.00th=[ 3163], 60.00th=[ 4047], | 70.00th=[ 5145], 80.00th=[ 6587], 90.00th=[ 8094], 95.00th=[10159], | 99.00th=[13829], 99.50th=[15139], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1440, max= 2400, per=25.13%, avg=1920.00, stdev=338.66, samples=6 iops : min= 90, max= 150, avg=120.00, stdev=21.17, samples=6 lat (usec) : 100=3.39%, 250=0.52%, 500=0.13%, 750=2.47%, 1000=12.11% lat (msec) : 2=5.73%, 4=28.39%, 10=43.36%, 20=3.91% cpu : usr=0.18%, sys=24.08%, ctx=1542, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2611: Wed May 21 01:08:47 2025 read: IOPS=118, BW=1897KiB/s (1942kB/s)(6256KiB/3298msec) clat (usec): min=67, max=17765, avg=4695.37, stdev=2688.11 lat (usec): min=68, max=17766, avg=4695.93, stdev=2688.12 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 89], 10.00th=[ 2180], 20.00th=[ 2835], | 30.00th=[ 3228], 40.00th=[ 3752], 50.00th=[ 4293], 60.00th=[ 4948], | 70.00th=[ 5538], 80.00th=[ 6456], 90.00th=[ 7767], 95.00th=[10028], | 99.00th=[15270], 99.50th=[16581], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1152, max= 2235, per=25.83%, avg=1875.67, stdev=414.50, samples=6 iops : min= 72, max= 139, avg=117.00, stdev=25.69, samples=6 write: IOPS=114, BW=1829KiB/s (1873kB/s)(6032KiB/3298msec) clat (usec): min=654, max=16403, avg=3721.35, stdev=3090.09 lat (usec): min=654, max=16403, avg=3722.22, stdev=3090.08 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 906], | 30.00th=[ 1172], 40.00th=[ 2114], 50.00th=[ 2802], 60.00th=[ 3589], | 70.00th=[ 4686], 80.00th=[ 6456], 90.00th=[ 8356], 95.00th=[ 9896], | 99.00th=[12911], 99.50th=[14222], 99.90th=[16450], 99.95th=[16450], | 99.99th=[16450] bw ( KiB/s): min= 1344, max= 2016, per=23.51%, avg=1796.00, stdev=254.17, samples=6 iops : min= 84, max= 126, avg=112.00, stdev=15.94, samples=6 lat (usec) : 100=2.86%, 250=0.39%, 750=2.08%, 1000=10.16% lat (msec) : 2=7.29%, 4=30.99%, 10=41.54%, 20=4.69% cpu : usr=0.09%, sys=24.29%, ctx=1580, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2612: Wed May 21 01:08:47 2025 read: IOPS=119, BW=1907KiB/s (1953kB/s)(6032KiB/3163msec) clat (usec): min=67, max=16195, avg=4503.47, stdev=2579.85 lat (usec): min=68, max=16196, avg=4504.04, stdev=2579.87 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 81], 10.00th=[ 101], 20.00th=[ 2868], | 30.00th=[ 3228], 40.00th=[ 3752], 50.00th=[ 4424], 60.00th=[ 4883], | 70.00th=[ 5407], 80.00th=[ 6325], 90.00th=[ 7635], 95.00th=[ 8717], | 99.00th=[13435], 99.50th=[14353], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 1600, max= 2304, per=25.56%, avg=1856.00, stdev=291.18, samples=6 iops : min= 100, max= 144, avg=116.00, stdev=18.20, samples=6 write: IOPS=123, BW=1978KiB/s (2025kB/s)(6256KiB/3163msec) clat (usec): min=637, max=16772, avg=3705.22, stdev=3246.07 lat (usec): min=637, max=16773, avg=3708.76, stdev=3247.91 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 775], 10.00th=[ 807], 20.00th=[ 889], | 30.00th=[ 1020], 40.00th=[ 1745], 50.00th=[ 2704], 60.00th=[ 3458], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 8455], 95.00th=[ 9765], | 99.00th=[14484], 99.50th=[16581], 99.90th=[16712], 99.95th=[16712], | 99.99th=[16712] bw ( KiB/s): min= 1760, max= 2144, per=25.62%, avg=1957.33, stdev=153.36, samples=6 iops : min= 110, max= 134, avg=122.33, stdev= 9.58, samples=6 lat (usec) : 100=4.82%, 250=0.52%, 750=2.21%, 1000=12.76% lat (msec) : 2=6.38%, 4=27.08%, 10=42.71%, 20=3.52% cpu : usr=0.03%, sys=24.67%, ctx=1488, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7263KiB/s (7437kB/s), 1742KiB/s-1907KiB/s (1783kB/s-1953kB/s), io=23.4MiB (24.5MB), run=3163-3298msec WRITE: bw=7641KiB/s (7824kB/s), 1829KiB/s-1984KiB/s (1873kB/s-2032kB/s), io=24.6MiB (25.8MB), run=3163-3298msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2613: Wed May 21 01:08:48 2025 read: IOPS=88, BW=1409KiB/s (1443kB/s)(5920KiB/4202msec) clat (usec): min=1761, max=19548, avg=4531.93, stdev=3323.10 lat (usec): min=1761, max=19548, avg=4532.50, stdev=3323.11 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1991], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3097], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 6652], 90.00th=[ 9634], 95.00th=[12256], | 99.00th=[15664], 99.50th=[17433], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 928, max= 2368, per=24.09%, avg=1343.38, stdev=488.40, samples=8 iops : min= 58, max= 148, avg=83.75, stdev=30.67, samples=8 write: IOPS=94, BW=1515KiB/s (1552kB/s)(6368KiB/4202msec) clat (usec): min=2495, max=27255, avg=6290.55, stdev=3768.01 lat (usec): min=2496, max=27256, avg=6291.68, stdev=3768.02 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 2999], 10.00th=[ 3130], 20.00th=[ 3523], | 30.00th=[ 3851], 40.00th=[ 4146], 50.00th=[ 4752], 60.00th=[ 5735], | 70.00th=[ 7111], 80.00th=[ 8586], 90.00th=[11469], 95.00th=[13960], | 99.00th=[20317], 99.50th=[23987], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 960, max= 2528, per=24.94%, avg=1463.25, stdev=547.36, samples=8 iops : min= 60, max= 158, avg=91.25, stdev=34.30, samples=8 lat (msec) : 2=2.73%, 4=47.66%, 10=37.76%, 20=11.33%, 50=0.52% cpu : usr=0.17%, sys=11.78%, ctx=887, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2614: Wed May 21 01:08:48 2025 read: IOPS=83, BW=1342KiB/s (1374kB/s)(5744KiB/4281msec) clat (usec): min=1793, max=21576, avg=4428.67, stdev=3321.81 lat (usec): min=1793, max=21576, avg=4429.28, stdev=3321.79 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1991], 10.00th=[ 2089], 20.00th=[ 2278], | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4228], 80.00th=[ 5735], 90.00th=[ 9503], 95.00th=[12256], | 99.00th=[16581], 99.50th=[17695], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 608, max= 2304, per=22.96%, avg=1280.00, stdev=536.55, samples=8 iops : min= 38, max= 144, avg=80.00, stdev=33.53, samples=8 write: IOPS=95, BW=1529KiB/s (1565kB/s)(6544KiB/4281msec) clat (usec): min=2441, max=41584, avg=6334.98, stdev=4156.55 lat (usec): min=2442, max=41585, avg=6336.29, stdev=4156.44 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2868], 10.00th=[ 3130], 20.00th=[ 3523], | 30.00th=[ 3851], 40.00th=[ 4228], 50.00th=[ 4752], 60.00th=[ 5538], | 70.00th=[ 6718], 80.00th=[ 8717], 90.00th=[12125], 95.00th=[14091], | 99.00th=[19006], 99.50th=[20841], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 1024, max= 2688, per=24.61%, avg=1444.00, stdev=557.53, samples=8 iops : min= 64, max= 168, avg=90.25, stdev=34.85, samples=8 lat (msec) : 2=2.47%, 4=46.74%, 10=37.76%, 20=12.50%, 50=0.52% cpu : usr=0.09%, sys=11.82%, ctx=911, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2615: Wed May 21 01:08:48 2025 read: IOPS=91, BW=1468KiB/s (1503kB/s)(6256KiB/4261msec) clat (usec): min=1747, max=19326, avg=4550.56, stdev=3453.43 lat (usec): min=1747, max=19326, avg=4551.14, stdev=3453.42 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2507], 40.00th=[ 2769], 50.00th=[ 3130], 60.00th=[ 3556], | 70.00th=[ 4228], 80.00th=[ 6063], 90.00th=[ 9634], 95.00th=[12518], | 99.00th=[17171], 99.50th=[18482], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 736, max= 2624, per=25.02%, avg=1395.38, stdev=601.52, samples=8 iops : min= 46, max= 164, avg=87.00, stdev=37.75, samples=8 write: IOPS=88, BW=1416KiB/s (1450kB/s)(6032KiB/4261msec) clat (usec): min=2589, max=25586, avg=6345.11, stdev=3777.77 lat (usec): min=2591, max=25587, avg=6346.32, stdev=3778.07 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 2999], 10.00th=[ 3195], 20.00th=[ 3523], | 30.00th=[ 3851], 40.00th=[ 4146], 50.00th=[ 4621], 60.00th=[ 5604], | 70.00th=[ 7177], 80.00th=[ 9241], 90.00th=[12125], 95.00th=[13435], | 99.00th=[19006], 99.50th=[22414], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 864, max= 2304, per=22.96%, avg=1347.25, stdev=446.11, samples=8 iops : min= 54, max= 144, avg=84.00, stdev=27.95, samples=8 lat (msec) : 2=2.47%, 4=48.96%, 10=35.42%, 20=12.89%, 50=0.26% cpu : usr=0.26%, sys=11.69%, ctx=886, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2616: Wed May 21 01:08:48 2025 read: IOPS=87, BW=1404KiB/s (1438kB/s)(6032KiB/4295msec) clat (usec): min=1844, max=21529, avg=4654.32, stdev=3470.99 lat (usec): min=1845, max=21529, avg=4655.01, stdev=3470.92 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2474], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 3720], | 70.00th=[ 4883], 80.00th=[ 6521], 90.00th=[10028], 95.00th=[12125], | 99.00th=[17695], 99.50th=[20579], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 896, max= 2336, per=23.73%, avg=1323.25, stdev=463.14, samples=8 iops : min= 56, max= 146, avg=82.50, stdev=29.05, samples=8 write: IOPS=91, BW=1457KiB/s (1492kB/s)(6256KiB/4295msec) clat (usec): min=2492, max=25473, avg=6276.44, stdev=3876.40 lat (usec): min=2493, max=25475, avg=6277.57, stdev=3876.35 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2966], 10.00th=[ 3195], 20.00th=[ 3490], | 30.00th=[ 3851], 40.00th=[ 4146], 50.00th=[ 4621], 60.00th=[ 5669], | 70.00th=[ 7046], 80.00th=[ 8586], 90.00th=[11469], 95.00th=[14091], | 99.00th=[20841], 99.50th=[23987], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 800, max= 2560, per=23.58%, avg=1383.25, stdev=561.58, samples=8 iops : min= 50, max= 160, avg=86.25, stdev=35.18, samples=8 lat (msec) : 2=1.43%, 4=48.18%, 10=38.02%, 20=11.33%, 50=1.04% cpu : usr=0.19%, sys=11.57%, ctx=899, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5577KiB/s (5711kB/s), 1342KiB/s-1468KiB/s (1374kB/s-1503kB/s), io=23.4MiB (24.5MB), run=4202-4295msec WRITE: bw=5867KiB/s (6008kB/s), 1416KiB/s-1529KiB/s (1450kB/s-1565kB/s), io=24.6MiB (25.8MB), run=4202-4295msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2624: Wed May 21 01:08:50 2025 read: IOPS=16, BW=16.4MiB/s (17.2MB/s)(7168KiB/427msec) clat (usec): min=17646, max=54780, avg=33051.12, stdev=15264.09 lat (usec): min=17646, max=54780, avg=33052.21, stdev=15263.87 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[19792], | 30.00th=[22938], 40.00th=[22938], 50.00th=[29230], 60.00th=[34341], | 70.00th=[34341], 80.00th=[52691], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] write: IOPS=11, BW=11.7MiB/s (12.3MB/s)(5120KiB/427msec) clat (usec): min=19850, max=44596, avg=32847.55, stdev=10659.55 lat (usec): min=19940, max=44615, avg=32899.29, stdev=10647.20 clat percentiles (usec): | 1.00th=[19792], 5.00th=[19792], 10.00th=[19792], 20.00th=[19792], | 30.00th=[27919], 40.00th=[27919], 50.00th=[28705], 60.00th=[28705], | 70.00th=[43254], 80.00th=[43254], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] lat (msec) : 20=25.00%, 50=58.33%, 100=16.67% cpu : usr=0.00%, sys=19.25%, ctx=74, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2625: Wed May 21 01:08:50 2025 read: IOPS=3, BW=3908KiB/s (4002kB/s)(2048KiB/524msec) clat (usec): min=25656, max=49274, avg=37465.47, stdev=16700.06 lat (usec): min=25657, max=49274, avg=37466.37, stdev=16699.73 clat percentiles (usec): | 1.00th=[25560], 5.00th=[25560], 10.00th=[25560], 20.00th=[25560], | 30.00th=[25560], 40.00th=[25560], 50.00th=[25560], 60.00th=[49021], | 70.00th=[49021], 80.00th=[49021], 90.00th=[49021], 95.00th=[49021], | 99.00th=[49021], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 4096, max= 4096, per=11.60%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=19, BW=19.1MiB/s (20.0MB/s)(10.0MiB/524msec) clat (usec): min=16386, max=84784, avg=42575.53, stdev=24590.59 lat (usec): min=16468, max=84813, avg=42657.17, stdev=24582.51 clat percentiles (usec): | 1.00th=[16450], 5.00th=[16450], 10.00th=[16450], 20.00th=[17695], | 30.00th=[17957], 40.00th=[32637], 50.00th=[34341], 60.00th=[35914], | 70.00th=[55313], 80.00th=[55313], 90.00th=[78119], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min=18432, max=18432, per=37.29%, avg=18432.00, stdev= 0.00, samples=1 iops : min= 18, max= 18, avg=18.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=41.67%, 100=33.33% cpu : usr=0.19%, sys=16.63%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2626: Wed May 21 01:08:50 2025 read: IOPS=10, BW=10.3MiB/s (10.8MB/s)(6144KiB/580msec) clat (usec): min=12769, max=83508, avg=43279.14, stdev=30707.56 lat (usec): min=12770, max=83509, avg=43279.81, stdev=30707.56 clat percentiles (usec): | 1.00th=[12780], 5.00th=[12780], 10.00th=[12780], 20.00th=[14746], | 30.00th=[14746], 40.00th=[20841], 50.00th=[20841], 60.00th=[63177], | 70.00th=[64750], 80.00th=[64750], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 8192, max= 8192, per=23.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.3MiB/s (10.8MB/s)(6144KiB/580msec) clat (usec): min=30871, max=69948, avg=46670.66, stdev=14682.78 lat (usec): min=30957, max=69967, avg=46720.54, stdev=14660.03 clat percentiles (usec): | 1.00th=[30802], 5.00th=[30802], 10.00th=[30802], 20.00th=[38011], | 30.00th=[38011], 40.00th=[38536], 50.00th=[38536], 60.00th=[43779], | 70.00th=[58459], 80.00th=[58459], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min=10240, max=10240, per=20.71%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=41.67% cpu : usr=0.00%, sys=14.85%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2627: Wed May 21 01:08:50 2025 read: IOPS=9, BW=9.90MiB/s (10.4MB/s)(5120KiB/505msec) clat (usec): min=15078, max=58697, avg=33390.94, stdev=16955.47 lat (usec): min=15078, max=58699, avg=33391.75, stdev=16955.72 clat percentiles (usec): | 1.00th=[15139], 5.00th=[15139], 10.00th=[15139], 20.00th=[15139], | 30.00th=[23200], 40.00th=[23200], 50.00th=[29492], 60.00th=[29492], | 70.00th=[40633], 80.00th=[40633], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 8175, max= 8175, per=23.15%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.9MiB/s (14.5MB/s)(7168KiB/505msec) clat (msec): min=27, max=104, avg=47.44, stdev=28.32 lat (msec): min=27, max=104, avg=47.49, stdev=28.33 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 33], 60.00th=[ 42], | 70.00th=[ 42], 80.00th=[ 66], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min=14307, max=14307, per=28.94%, avg=14307.00, stdev= 0.00, samples=1 iops : min= 13, max= 13, avg=13.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=66.67%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=15.28%, ctx=37, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=34.5MiB/s (36.2MB/s), 3908KiB/s-16.4MiB/s (4002kB/s-17.2MB/s), io=20.0MiB (20.0MB), run=427-580msec WRITE: bw=48.3MiB/s (50.6MB/s), 10.3MiB/s-19.1MiB/s (10.8MB/s-20.0MB/s), io=28.0MiB (29.4MB), run=427-580msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2629: Wed May 21 01:08:50 2025 read: IOPS=12, BW=12.2MiB/s (12.8MB/s)(7168KiB/572msec) clat (usec): min=13780, max=42666, avg=30253.09, stdev=10214.73 lat (usec): min=13781, max=42668, avg=30253.86, stdev=10214.90 clat percentiles (usec): | 1.00th=[13829], 5.00th=[13829], 10.00th=[13829], 20.00th=[21890], | 30.00th=[26870], 40.00th=[26870], 50.00th=[32900], 60.00th=[33424], | 70.00th=[33424], 80.00th=[40109], 90.00th=[42730], 95.00th=[42730], | 99.00th=[42730], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min=10240, max=10240, per=30.10%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8951KiB/s (9166kB/s)(5120KiB/572msec) clat (msec): min=33, max=127, avg=60.56, stdev=39.59 lat (msec): min=33, max=127, avg=60.60, stdev=39.61 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 43], 60.00th=[ 43], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 8192, max= 8192, per=17.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=75.00%, 100=8.33%, 250=8.33% cpu : usr=0.18%, sys=36.08%, ctx=856, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2630: Wed May 21 01:08:50 2025 read: IOPS=3, BW=3402KiB/s (3484kB/s)(2048KiB/602msec) clat (usec): min=44261, max=51378, avg=47819.82, stdev=5032.80 lat (usec): min=44262, max=51379, avg=47820.77, stdev=5032.45 clat percentiles (usec): | 1.00th=[44303], 5.00th=[44303], 10.00th=[44303], 20.00th=[44303], | 30.00th=[44303], 40.00th=[44303], 50.00th=[44303], 60.00th=[51119], | 70.00th=[51119], 80.00th=[51119], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 4096, max= 4096, per=12.04%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.6MiB/s (17.4MB/s)(10.0MiB/602msec) clat (msec): min=16, max=122, avg=46.28, stdev=33.70 lat (msec): min=16, max=122, avg=46.32, stdev=33.71 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 22], 40.00th=[ 26], 50.00th=[ 34], 60.00th=[ 34], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 83], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min=12288, max=12288, per=25.80%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=39.43%, ctx=209, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2631: Wed May 21 01:08:50 2025 read: IOPS=10, BW=10.8MiB/s (11.3MB/s)(6144KiB/556msec) clat (usec): min=323, max=40133, avg=22039.60, stdev=17985.41 lat (usec): min=324, max=40134, avg=22040.29, stdev=17985.46 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 322], 10.00th=[ 322], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[23200], 50.00th=[23200], 60.00th=[28967], | 70.00th=[39060], 80.00th=[39060], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min=10240, max=10240, per=30.10%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.8MiB/s (11.3MB/s)(6144KiB/556msec) clat (msec): min=28, max=115, avg=67.56, stdev=39.07 lat (msec): min=28, max=115, avg=67.60, stdev=39.05 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 91], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min=10240, max=10240, per=21.50%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=58.33%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=33.33%, ctx=523, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2632: Wed May 21 01:08:50 2025 read: IOPS=9, BW=9570KiB/s (9800kB/s)(5120KiB/535msec) clat (usec): min=461, max=30367, avg=16578.15, stdev=14872.62 lat (usec): min=461, max=30368, avg=16578.92, stdev=14872.70 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 461], 10.00th=[ 461], 20.00th=[ 461], | 30.00th=[ 586], 40.00th=[ 586], 50.00th=[23462], 60.00th=[23462], | 70.00th=[28181], 80.00th=[28181], 90.00th=[30278], 95.00th=[30278], | 99.00th=[30278], 99.50th=[30278], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 8192, max= 8192, per=24.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.1MiB/s (13.7MB/s)(7168KiB/535msec) clat (usec): min=32954, max=83625, avg=56304.81, stdev=17761.37 lat (usec): min=33044, max=83660, avg=56365.11, stdev=17745.70 clat percentiles (usec): | 1.00th=[32900], 5.00th=[32900], 10.00th=[32900], 20.00th=[37487], | 30.00th=[52167], 40.00th=[52167], 50.00th=[53740], 60.00th=[64750], | 70.00th=[64750], 80.00th=[68682], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min=12288, max=12288, per=25.80%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=41.67%, 100=41.67% cpu : usr=0.00%, sys=33.90%, ctx=284, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=33.2MiB/s (34.8MB/s), 3402KiB/s-12.2MiB/s (3484kB/s-12.8MB/s), io=20.0MiB (20.0MB), run=535-602msec WRITE: bw=46.5MiB/s (48.8MB/s), 8951KiB/s-16.6MiB/s (9166kB/s-17.4MB/s), io=28.0MiB (29.4MB), run=535-602msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2639: Wed May 21 01:08:52 2025 read: IOPS=27, BW=110MiB/s (115MB/s)(12.0MiB/109msec) clat (usec): min=1509, max=55469, avg=27178.73, stdev=27075.15 lat (usec): min=1510, max=55470, avg=27179.52, stdev=27075.33 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1516], 10.00th=[ 1516], 20.00th=[ 1516], | 30.00th=[ 1516], 40.00th=[24511], 50.00th=[24511], 60.00th=[24511], | 70.00th=[55313], 80.00th=[55313], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] lat (msec) : 2=33.33%, 50=33.33%, 100=33.33% cpu : usr=0.00%, sys=33.33%, ctx=386, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2640: Wed May 21 01:08:52 2025 read: IOPS=1, BW=7099KiB/s (7269kB/s)(4096KiB/577msec) clat (nsec): min=53949k, max=53949k, avg=53948799.00, stdev= 0.00 lat (nsec): min=53950k, max=53950k, avg=53949798.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[53740], 5.00th=[53740], 10.00th=[53740], 20.00th=[53740], | 30.00th=[53740], 40.00th=[53740], 50.00th=[53740], 60.00th=[53740], | 70.00th=[53740], 80.00th=[53740], 90.00th=[53740], 95.00th=[53740], | 99.00th=[53740], 99.50th=[53740], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] write: IOPS=3, BW=13.9MiB/s (14.5MB/s)(8192KiB/577msec) clat (msec): min=178, max=307, avg=242.96, stdev=91.69 lat (msec): min=178, max=307, avg=243.19, stdev=91.60 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=35.94%, ctx=295, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2641: Wed May 21 01:08:52 2025 write: IOPS=5, BW=20.3MiB/s (21.3MB/s)(12.0MiB/592msec) clat (msec): min=95, max=361, avg=188.41, stdev=150.14 lat (msec): min=95, max=361, avg=188.73, stdev=150.16 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 109], 50.00th=[ 109], 60.00th=[ 109], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=43.82%, ctx=45, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2643: Wed May 21 01:08:52 2025 write: IOPS=7, BW=28.0MiB/s (30.4MB/s)(12.0MiB/414msec) clat (msec): min=80, max=181, avg=127.51, stdev=50.77 lat (msec): min=80, max=181, avg=127.87, stdev=50.67 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 81], | 30.00th=[ 81], 40.00th=[ 122], 50.00th=[ 122], 60.00th=[ 122], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=55.93%, ctx=58, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.7MiB/s (29.1MB/s), 7099KiB/s-110MiB/s (7269kB/s-115MB/s), io=16.0MiB (16.8MB), run=109-577msec WRITE: bw=54.1MiB/s (56.7MB/s), 13.9MiB/s-28.0MiB/s (14.5MB/s-30.4MB/s), io=32.0MiB (33.6MB), run=414-592msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2642: Wed May 21 01:08:52 2025 read: IOPS=4, BW=16.9MiB/s (17.7MB/s)(12.0MiB/712msec) clat (msec): min=118, max=305, avg=231.15, stdev=99.21 lat (msec): min=118, max=305, avg=231.15, stdev=99.21 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min=16384, max=16384, per=90.60%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=11.53%, ctx=36, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2644: Wed May 21 01:08:52 2025 read: IOPS=1, BW=4521KiB/s (4629kB/s)(4096KiB/906msec) clat (nsec): min=246763k, max=246763k, avg=246763466.00, stdev= 0.00 lat (nsec): min=246764k, max=246764k, avg=246764298.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 247], 95.00th=[ 247], | 99.00th=[ 247], 99.50th=[ 247], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 8192, max= 8192, per=45.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9042KiB/s (9259kB/s)(8192KiB/906msec) clat (msec): min=101, max=548, avg=325.19, stdev=315.67 lat (msec): min=102, max=548, avg=325.50, stdev=315.54 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=16.69%, ctx=113, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2645: Wed May 21 01:08:52 2025 write: IOPS=4, BW=16.3MiB/s (17.1MB/s)(12.0MiB/738msec) clat (msec): min=154, max=279, avg=237.71, stdev=71.69 lat (msec): min=155, max=279, avg=238.06, stdev=71.55 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min=16351, max=16351, per=45.21%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=8.82%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2646: Wed May 21 01:08:52 2025 write: IOPS=5, BW=20.4MiB/s (21.4MB/s)(12.0MiB/588msec) clat (msec): min=140, max=252, avg=188.52, stdev=57.83 lat (msec): min=140, max=252, avg=188.74, stdev=57.83 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min=16384, max=16384, per=45.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.34%, sys=22.32%, ctx=150, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.7MiB/s (18.5MB/s), 4521KiB/s-16.9MiB/s (4629kB/s-17.7MB/s), io=16.0MiB (16.8MB), run=712-906msec WRITE: bw=35.3MiB/s (37.0MB/s), 9042KiB/s-20.4MiB/s (9259kB/s-21.4MB/s), io=32.0MiB (33.6MB), run=588-906msec