== sanity test 398b: DIO and buffer IO race ============== 09:31:56 (1740493916) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.40552 s, 35.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28068: Tue Feb 25 09:32:13 2025 read: IOPS=103, BW=416KiB/s (426kB/s)(5992KiB/14409msec) clat (usec): min=67, max=241910, avg=5502.44, stdev=6855.83 lat (usec): min=68, max=241911, avg=5503.82, stdev=6856.01 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 2114], 10.00th=[ 2573], 20.00th=[ 3097], | 30.00th=[ 3556], 40.00th=[ 4146], 50.00th=[ 4817], 60.00th=[ 5473], | 70.00th=[ 6325], 80.00th=[ 7373], 90.00th=[ 8717], 95.00th=[ 10683], | 99.00th=[ 17695], 99.50th=[ 19268], 99.90th=[ 31065], 99.95th=[242222], | 99.99th=[242222] bw ( KiB/s): min= 216, max= 568, per=25.12%, avg=411.25, stdev=68.05, samples=28 iops : min= 54, max= 142, avg=102.68, stdev=17.00, samples=28 write: IOPS=109, BW=437KiB/s (447kB/s)(6296KiB/14409msec) clat (usec): min=500, max=22239, avg=3870.91, stdev=3322.90 lat (usec): min=500, max=22240, avg=3872.30, stdev=3323.13 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 635], 10.00th=[ 717], 20.00th=[ 832], | 30.00th=[ 1037], 40.00th=[ 2245], 50.00th=[ 3228], 60.00th=[ 3949], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 8586], 95.00th=[10421], | 99.00th=[14091], 99.50th=[15664], 99.90th=[19792], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 256, max= 600, per=25.13%, avg=430.68, stdev=71.24, samples=28 iops : min= 64, max= 150, avg=107.54, stdev=17.79, samples=28 lat (usec) : 100=0.94%, 250=1.24%, 500=0.03%, 750=6.58%, 1000=8.43% lat (msec) : 2=5.21%, 4=27.08%, 10=44.34%, 20=5.89%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.28%, sys=22.86%, ctx=6076, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28069: Tue Feb 25 09:32:13 2025 read: IOPS=101, BW=405KiB/s (415kB/s)(5940KiB/14667msec) clat (usec): min=64, max=23846, avg=5216.31, stdev=2945.18 lat (usec): min=64, max=23846, avg=5217.94, stdev=2946.33 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 1975], 10.00th=[ 2540], 20.00th=[ 3032], | 30.00th=[ 3458], 40.00th=[ 3949], 50.00th=[ 4686], 60.00th=[ 5473], | 70.00th=[ 6325], 80.00th=[ 7177], 90.00th=[ 8717], 95.00th=[ 9896], | 99.00th=[16581], 99.50th=[17957], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 176, max= 560, per=24.51%, avg=401.21, stdev=76.21, samples=29 iops : min= 44, max= 140, avg=100.24, stdev=19.05, samples=29 write: IOPS=108, BW=433KiB/s (443kB/s)(6348KiB/14667msec) clat (usec): min=454, max=238060, avg=4308.70, stdev=6866.28 lat (usec): min=455, max=238061, avg=4311.34, stdev=6866.26 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 627], 10.00th=[ 701], 20.00th=[ 857], | 30.00th=[ 1106], 40.00th=[ 2573], 50.00th=[ 3392], 60.00th=[ 4359], | 70.00th=[ 5538], 80.00th=[ 6980], 90.00th=[ 8979], 95.00th=[ 11076], | 99.00th=[ 16188], 99.50th=[ 17695], 99.90th=[ 22676], 99.95th=[238027], | 99.99th=[238027] bw ( KiB/s): min= 224, max= 544, per=24.76%, avg=424.41, stdev=68.10, samples=29 iops : min= 56, max= 136, avg=106.03, stdev=17.03, samples=29 lat (usec) : 100=0.59%, 250=1.56%, 500=0.13%, 750=6.61%, 1000=7.42% lat (msec) : 2=5.11%, 4=28.03%, 10=44.40%, 20=5.86%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.72%, ctx=5977, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28070: Tue Feb 25 09:32:13 2025 read: IOPS=104, BW=418KiB/s (429kB/s)(6028KiB/14405msec) clat (usec): min=59, max=21674, avg=5162.58, stdev=2854.65 lat (usec): min=59, max=21675, avg=5163.60, stdev=2854.49 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 1713], 10.00th=[ 2540], 20.00th=[ 3032], | 30.00th=[ 3458], 40.00th=[ 4015], 50.00th=[ 4686], 60.00th=[ 5407], | 70.00th=[ 6194], 80.00th=[ 7111], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[14877], 99.50th=[16581], 99.90th=[20055], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 192, max= 592, per=25.25%, avg=413.29, stdev=74.44, samples=28 iops : min= 48, max= 148, avg=103.25, stdev=18.61, samples=28 write: IOPS=108, BW=435KiB/s (445kB/s)(6260KiB/14405msec) clat (usec): min=475, max=237961, avg=4181.92, stdev=6885.82 lat (usec): min=475, max=237962, avg=4183.66, stdev=6885.46 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 619], 10.00th=[ 693], 20.00th=[ 824], | 30.00th=[ 996], 40.00th=[ 2147], 50.00th=[ 3359], 60.00th=[ 4146], | 70.00th=[ 5080], 80.00th=[ 6849], 90.00th=[ 9241], 95.00th=[ 10814], | 99.00th=[ 14615], 99.50th=[ 16712], 99.90th=[ 24511], 99.95th=[238027], | 99.99th=[238027] bw ( KiB/s): min= 248, max= 552, per=24.86%, avg=426.14, stdev=71.40, samples=28 iops : min= 62, max= 138, avg=106.46, stdev=17.84, samples=28 lat (usec) : 100=0.85%, 250=1.37%, 500=0.26%, 750=7.10%, 1000=8.20% lat (msec) : 2=4.82%, 4=26.79%, 10=44.40%, 20=6.09%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.74%, ctx=6105, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28071: Tue Feb 25 09:32:13 2025 read: IOPS=104, BW=418KiB/s (428kB/s)(6052KiB/14491msec) clat (usec): min=60, max=239630, avg=5355.41, stdev=6735.75 lat (usec): min=60, max=239630, avg=5356.52, stdev=6735.75 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 2278], 10.00th=[ 2573], 20.00th=[ 3032], | 30.00th=[ 3490], 40.00th=[ 4047], 50.00th=[ 4686], 60.00th=[ 5342], | 70.00th=[ 5997], 80.00th=[ 7046], 90.00th=[ 8455], 95.00th=[ 10159], | 99.00th=[ 15270], 99.50th=[ 19006], 99.90th=[ 33817], 99.95th=[240124], | 99.99th=[240124] bw ( KiB/s): min= 216, max= 526, per=25.10%, avg=410.86, stdev=64.82, samples=28 iops : min= 54, max= 131, avg=102.61, stdev=16.18, samples=28 write: IOPS=107, BW=430KiB/s (441kB/s)(6236KiB/14491msec) clat (usec): min=463, max=24118, avg=4051.93, stdev=3440.84 lat (usec): min=464, max=24120, avg=4054.76, stdev=3440.65 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 619], 10.00th=[ 701], 20.00th=[ 840], | 30.00th=[ 1090], 40.00th=[ 2376], 50.00th=[ 3326], 60.00th=[ 4146], | 70.00th=[ 5276], 80.00th=[ 6980], 90.00th=[ 9110], 95.00th=[10552], | 99.00th=[14091], 99.50th=[16319], 99.90th=[18482], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 248, max= 608, per=24.81%, avg=425.18, stdev=76.43, samples=28 iops : min= 62, max= 152, avg=106.18, stdev=19.15, samples=28 lat (usec) : 100=0.91%, 250=1.01%, 500=0.07%, 750=6.74%, 1000=7.45% lat (msec) : 2=5.14%, 4=27.44%, 10=45.51%, 20=5.50%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.67%, ctx=6127, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1637KiB/s (1676kB/s), 405KiB/s-418KiB/s (415kB/s-429kB/s), io=23.4MiB (24.6MB), run=14405-14667msec WRITE: bw=1714KiB/s (1755kB/s), 430KiB/s-437KiB/s (441kB/s-447kB/s), io=24.6MiB (25.7MB), run=14405-14667msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28073: Tue Feb 25 09:32:18 2025 read: IOPS=79, BW=317KiB/s (324kB/s)(5992KiB/18925msec) clat (usec): min=1771, max=231353, avg=5126.77, stdev=7017.31 lat (usec): min=1771, max=231356, avg=5127.57, stdev=7017.34 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2638], 40.00th=[ 2999], 50.00th=[ 3490], 60.00th=[ 4080], | 70.00th=[ 4948], 80.00th=[ 6587], 90.00th=[ 10683], 95.00th=[ 13566], | 99.00th=[ 19006], 99.50th=[ 23725], 99.90th=[ 31065], 99.95th=[231736], | 99.99th=[231736] bw ( KiB/s): min= 136, max= 712, per=24.54%, avg=311.22, stdev=148.87, samples=37 iops : min= 34, max= 178, avg=77.70, stdev=37.23, samples=37 write: IOPS=83, BW=333KiB/s (341kB/s)(6296KiB/18925msec) clat (usec): min=2370, max=31752, avg=7108.98, stdev=4268.12 lat (usec): min=2371, max=31754, avg=7110.05, stdev=4268.07 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3195], 10.00th=[ 3458], 20.00th=[ 3785], | 30.00th=[ 4113], 40.00th=[ 4752], 50.00th=[ 5538], 60.00th=[ 6652], | 70.00th=[ 8356], 80.00th=[10159], 90.00th=[13042], 95.00th=[15533], | 99.00th=[22152], 99.50th=[25560], 99.90th=[31327], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 144, max= 696, per=24.59%, avg=326.57, stdev=147.55, samples=37 iops : min= 36, max= 174, avg=81.54, stdev=36.91, samples=37 lat (msec) : 2=1.14%, 4=41.31%, 10=41.57%, 20=14.78%, 50=1.17% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.17%, ctx=3684, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28074: Tue Feb 25 09:32:18 2025 read: IOPS=78, BW=314KiB/s (321kB/s)(5940KiB/18923msec) clat (usec): min=1702, max=230643, avg=5108.06, stdev=6984.03 lat (usec): min=1702, max=230644, avg=5108.88, stdev=6984.04 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2900], 50.00th=[ 3392], 60.00th=[ 4113], | 70.00th=[ 5014], 80.00th=[ 6915], 90.00th=[ 10945], 95.00th=[ 13304], | 99.00th=[ 19006], 99.50th=[ 20841], 99.90th=[ 25560], 99.95th=[229639], | 99.99th=[229639] bw ( KiB/s): min= 128, max= 704, per=24.50%, avg=310.70, stdev=154.09, samples=37 iops : min= 32, max= 176, avg=77.54, stdev=38.56, samples=37 write: IOPS=83, BW=335KiB/s (344kB/s)(6348KiB/18923msec) clat (usec): min=2517, max=35980, avg=7112.78, stdev=4255.95 lat (usec): min=2517, max=35981, avg=7113.81, stdev=4256.02 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3130], 10.00th=[ 3458], 20.00th=[ 3752], | 30.00th=[ 4146], 40.00th=[ 4752], 50.00th=[ 5669], 60.00th=[ 6652], | 70.00th=[ 8160], 80.00th=[10159], 90.00th=[13173], 95.00th=[15533], | 99.00th=[20841], 99.50th=[25035], 99.90th=[30802], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 200, max= 656, per=24.75%, avg=328.65, stdev=141.22, samples=37 iops : min= 50, max= 164, avg=82.03, stdev=35.34, samples=37 lat (msec) : 2=1.24%, 4=40.98%, 10=41.70%, 20=15.17%, 50=0.88% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.14%, ctx=3646, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28075: Tue Feb 25 09:32:18 2025 read: IOPS=80, BW=321KiB/s (329kB/s)(6028KiB/18772msec) clat (usec): min=1727, max=30536, avg=5086.83, stdev=3889.61 lat (usec): min=1728, max=30537, avg=5087.64, stdev=3889.65 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2409], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3490], 60.00th=[ 4228], | 70.00th=[ 5211], 80.00th=[ 7111], 90.00th=[11076], 95.00th=[13566], | 99.00th=[18220], 99.50th=[21627], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 120, max= 656, per=25.16%, avg=319.05, stdev=141.93, samples=37 iops : min= 30, max= 164, avg=79.70, stdev=35.48, samples=37 write: IOPS=83, BW=333KiB/s (341kB/s)(6260KiB/18772msec) clat (msec): min=2, max=252, avg= 7.08, stdev= 7.42 lat (msec): min=2, max=252, avg= 7.08, stdev= 7.42 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 22], 99.50th=[ 25], 99.90th=[ 32], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 152, max= 768, per=24.87%, avg=330.30, stdev=149.03, samples=37 iops : min= 38, max= 192, avg=82.51, stdev=37.26, samples=37 lat (msec) : 2=1.14%, 4=40.04%, 10=43.33%, 20=14.49%, 50=0.98% lat (msec) : 500=0.03% cpu : usr=0.19%, sys=11.33%, ctx=3701, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28076: Tue Feb 25 09:32:18 2025 read: IOPS=80, BW=321KiB/s (329kB/s)(6052KiB/18839msec) clat (usec): min=1644, max=25172, avg=4957.99, stdev=3722.03 lat (usec): min=1644, max=25172, avg=4958.75, stdev=3722.03 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2606], 40.00th=[ 2966], 50.00th=[ 3490], 60.00th=[ 4178], | 70.00th=[ 4948], 80.00th=[ 6652], 90.00th=[10552], 95.00th=[13173], | 99.00th=[17957], 99.50th=[20317], 99.90th=[24249], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 120, max= 688, per=25.02%, avg=317.24, stdev=148.51, samples=37 iops : min= 30, max= 172, avg=79.19, stdev=37.20, samples=37 write: IOPS=82, BW=331KiB/s (339kB/s)(6236KiB/18839msec) clat (msec): min=2, max=223, avg= 7.23, stdev= 6.93 lat (msec): min=2, max=223, avg= 7.23, stdev= 6.93 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 24], 99.50th=[ 27], 99.90th=[ 36], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 160, max= 688, per=24.62%, avg=326.97, stdev=145.92, samples=37 iops : min= 40, max= 172, avg=81.62, stdev=36.54, samples=37 lat (msec) : 2=1.30%, 4=40.17%, 10=42.12%, 20=15.23%, 50=1.14% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.24%, ctx=3737, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1269KiB/s (1299kB/s), 314KiB/s-321KiB/s (321kB/s-329kB/s), io=23.4MiB (24.6MB), run=18772-18925msec WRITE: bw=1328KiB/s (1360kB/s), 331KiB/s-335KiB/s (339kB/s-344kB/s), io=24.6MiB (25.7MB), run=18772-18925msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28107: Tue Feb 25 09:32:23 2025 read: IOPS=91, BW=1461KiB/s (1496kB/s)(5920KiB/4051msec) clat (usec): min=59, max=20993, avg=5681.31, stdev=3184.29 lat (usec): min=59, max=20993, avg=5682.12, stdev=3184.26 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 121], 10.00th=[ 2737], 20.00th=[ 3556], | 30.00th=[ 3982], 40.00th=[ 4621], 50.00th=[ 5145], 60.00th=[ 6063], | 70.00th=[ 6718], 80.00th=[ 7832], 90.00th=[ 9503], 95.00th=[10945], | 99.00th=[17171], 99.50th=[17695], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1248, max= 1856, per=24.56%, avg=1452.00, stdev=200.16, samples=8 iops : min= 78, max= 116, avg=90.75, stdev=12.51, samples=8 write: IOPS=98, BW=1572KiB/s (1610kB/s)(6368KiB/4051msec) clat (usec): min=752, max=19045, avg=4764.81, stdev=3690.45 lat (usec): min=753, max=19045, avg=4765.92, stdev=3690.45 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 955], 10.00th=[ 1057], 20.00th=[ 1287], | 30.00th=[ 1778], 40.00th=[ 2999], 50.00th=[ 3851], 60.00th=[ 4686], | 70.00th=[ 6325], 80.00th=[ 8029], 90.00th=[ 9765], 95.00th=[12387], | 99.00th=[17171], 99.50th=[17695], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1120, max= 1920, per=25.27%, avg=1572.00, stdev=260.22, samples=8 iops : min= 70, max= 120, avg=98.25, stdev=16.26, samples=8 lat (usec) : 100=1.69%, 250=2.21%, 1000=3.78% lat (msec) : 2=13.80%, 4=20.05%, 10=50.39%, 20=7.94%, 50=0.13% cpu : usr=0.15%, sys=25.06%, ctx=1788, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28108: Tue Feb 25 09:32:23 2025 read: IOPS=90, BW=1448KiB/s (1482kB/s)(5744KiB/3968msec) clat (usec): min=67, max=21059, avg=5647.52, stdev=3278.73 lat (usec): min=68, max=21060, avg=5648.27, stdev=3278.79 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 135], 10.00th=[ 2737], 20.00th=[ 3359], | 30.00th=[ 3785], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 5800], | 70.00th=[ 6652], 80.00th=[ 7898], 90.00th=[ 9896], 95.00th=[11731], | 99.00th=[18482], 99.50th=[19268], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 960, max= 1660, per=23.65%, avg=1398.29, stdev=245.58, samples=7 iops : min= 60, max= 103, avg=87.29, stdev=15.22, samples=7 write: IOPS=103, BW=1649KiB/s (1689kB/s)(6544KiB/3968msec) clat (usec): min=718, max=21647, avg=4600.95, stdev=3803.58 lat (usec): min=719, max=21648, avg=4602.04, stdev=3803.56 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 988], 10.00th=[ 1074], 20.00th=[ 1254], | 30.00th=[ 1549], 40.00th=[ 2835], 50.00th=[ 3589], 60.00th=[ 4359], | 70.00th=[ 5538], 80.00th=[ 8029], 90.00th=[10159], 95.00th=[11731], | 99.00th=[16057], 99.50th=[17433], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 1344, max= 1856, per=25.42%, avg=1581.14, stdev=189.47, samples=7 iops : min= 84, max= 116, avg=98.71, stdev=11.90, samples=7 lat (usec) : 100=0.65%, 250=2.60%, 500=0.13%, 750=0.39%, 1000=2.73% lat (msec) : 2=16.41%, 4=22.79%, 10=44.27%, 20=9.77%, 50=0.26% cpu : usr=0.15%, sys=24.63%, ctx=1933, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28109: Tue Feb 25 09:32:23 2025 read: IOPS=99, BW=1593KiB/s (1631kB/s)(6256KiB/3927msec) clat (usec): min=59, max=22211, avg=5632.61, stdev=3285.05 lat (usec): min=60, max=22212, avg=5633.33, stdev=3285.06 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 110], 10.00th=[ 2311], 20.00th=[ 3294], | 30.00th=[ 3916], 40.00th=[ 4555], 50.00th=[ 5211], 60.00th=[ 5997], | 70.00th=[ 7111], 80.00th=[ 8225], 90.00th=[ 9503], 95.00th=[11338], | 99.00th=[15270], 99.50th=[17171], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 1312, max= 1696, per=26.21%, avg=1549.71, stdev=153.31, samples=7 iops : min= 82, max= 106, avg=96.86, stdev= 9.58, samples=7 write: IOPS=96, BW=1536KiB/s (1573kB/s)(6032KiB/3927msec) clat (usec): min=799, max=18229, avg=4538.17, stdev=3542.05 lat (usec): min=800, max=18229, avg=4539.32, stdev=3542.05 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1020], 10.00th=[ 1123], 20.00th=[ 1303], | 30.00th=[ 1663], 40.00th=[ 2507], 50.00th=[ 3490], 60.00th=[ 4490], | 70.00th=[ 5932], 80.00th=[ 7832], 90.00th=[ 9634], 95.00th=[11338], | 99.00th=[16188], 99.50th=[17433], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1024, max= 1760, per=24.33%, avg=1513.14, stdev=257.14, samples=7 iops : min= 64, max= 110, avg=94.57, stdev=16.07, samples=7 lat (usec) : 100=1.82%, 250=3.12%, 1000=1.69% lat (msec) : 2=15.36%, 4=21.35%, 10=48.70%, 20=7.81%, 50=0.13% cpu : usr=0.18%, sys=24.73%, ctx=1865, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28111: Tue Feb 25 09:32:23 2025 read: IOPS=95, BW=1526KiB/s (1563kB/s)(6032KiB/3953msec) clat (usec): min=58, max=21628, avg=5448.35, stdev=3226.59 lat (usec): min=58, max=21629, avg=5449.12, stdev=3226.53 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 111], 10.00th=[ 143], 20.00th=[ 3261], | 30.00th=[ 3884], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 5866], | 70.00th=[ 6718], 80.00th=[ 7963], 90.00th=[ 9241], 95.00th=[10814], | 99.00th=[16057], 99.50th=[16319], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 1184, max= 1888, per=25.12%, avg=1485.29, stdev=248.06, samples=7 iops : min= 74, max= 118, avg=92.71, stdev=15.59, samples=7 write: IOPS=98, BW=1583KiB/s (1621kB/s)(6256KiB/3953msec) clat (usec): min=709, max=17615, avg=4734.78, stdev=3746.50 lat (usec): min=710, max=17616, avg=4735.89, stdev=3746.51 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 979], 10.00th=[ 1090], 20.00th=[ 1303], | 30.00th=[ 1696], 40.00th=[ 2868], 50.00th=[ 3752], 60.00th=[ 4817], | 70.00th=[ 6128], 80.00th=[ 7635], 90.00th=[10159], 95.00th=[12649], | 99.00th=[15926], 99.50th=[17433], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1376, max= 1760, per=24.76%, avg=1540.00, stdev=128.44, samples=7 iops : min= 86, max= 110, avg=96.14, stdev= 8.01, samples=7 lat (usec) : 100=1.95%, 250=3.65%, 500=0.13%, 750=0.39%, 1000=2.73% lat (msec) : 2=13.93%, 4=20.57%, 10=47.92%, 20=8.59%, 50=0.13% cpu : usr=0.25%, sys=24.60%, ctx=1838, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5913KiB/s (6055kB/s), 1448KiB/s-1593KiB/s (1482kB/s-1631kB/s), io=23.4MiB (24.5MB), run=3927-4051msec WRITE: bw=6221KiB/s (6370kB/s), 1536KiB/s-1649KiB/s (1573kB/s-1689kB/s), io=24.6MiB (25.8MB), run=3927-4051msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28110: Tue Feb 25 09:32:24 2025 read: IOPS=74, BW=1195KiB/s (1224kB/s)(5920KiB/4954msec) clat (usec): min=2058, max=24198, avg=5210.96, stdev=3958.24 lat (usec): min=2059, max=24199, avg=5211.79, stdev=3958.44 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2343], 10.00th=[ 2474], 20.00th=[ 2638], | 30.00th=[ 2835], 40.00th=[ 3130], 50.00th=[ 3589], 60.00th=[ 4228], | 70.00th=[ 5145], 80.00th=[ 6980], 90.00th=[11207], 95.00th=[14222], | 99.00th=[19006], 99.50th=[20579], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 736, max= 2176, per=22.58%, avg=1077.33, stdev=435.25, samples=9 iops : min= 46, max= 136, avg=67.33, stdev=27.20, samples=9 write: IOPS=80, BW=1285KiB/s (1316kB/s)(6368KiB/4954msec) clat (usec): min=2901, max=29676, avg=7461.73, stdev=4260.11 lat (usec): min=2902, max=29677, avg=7463.09, stdev=4260.20 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3523], 10.00th=[ 3720], 20.00th=[ 3982], | 30.00th=[ 4293], 40.00th=[ 4883], 50.00th=[ 5735], 60.00th=[ 7046], | 70.00th=[ 8979], 80.00th=[11207], 90.00th=[13698], 95.00th=[15533], | 99.00th=[20841], 99.50th=[24511], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 768, max= 2112, per=23.51%, avg=1180.44, stdev=384.81, samples=9 iops : min= 48, max= 132, avg=73.78, stdev=24.05, samples=9 lat (msec) : 4=38.41%, 10=42.84%, 20=17.84%, 50=0.91% cpu : usr=0.18%, sys=12.21%, ctx=1015, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28112: Tue Feb 25 09:32:24 2025 read: IOPS=71, BW=1145KiB/s (1172kB/s)(5744KiB/5017msec) clat (usec): min=1932, max=21983, avg=5241.65, stdev=3970.60 lat (usec): min=1932, max=21984, avg=5242.35, stdev=3970.61 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2278], 10.00th=[ 2376], 20.00th=[ 2573], | 30.00th=[ 2802], 40.00th=[ 3130], 50.00th=[ 3556], 60.00th=[ 4047], | 70.00th=[ 5145], 80.00th=[ 7177], 90.00th=[11600], 95.00th=[14877], | 99.00th=[19268], 99.50th=[20055], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 640, max= 2208, per=23.81%, avg=1136.00, stdev=518.46, samples=10 iops : min= 40, max= 138, avg=71.00, stdev=32.40, samples=10 write: IOPS=81, BW=1304KiB/s (1336kB/s)(6544KiB/5017msec) clat (usec): min=2947, max=31340, avg=7508.87, stdev=4844.52 lat (usec): min=2948, max=31342, avg=7510.48, stdev=4844.94 clat percentiles (usec): | 1.00th=[ 3130], 5.00th=[ 3425], 10.00th=[ 3589], 20.00th=[ 3982], | 30.00th=[ 4359], 40.00th=[ 4686], 50.00th=[ 5538], 60.00th=[ 6915], | 70.00th=[ 8225], 80.00th=[10552], 90.00th=[14353], 95.00th=[17171], | 99.00th=[24773], 99.50th=[29230], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 928, max= 2688, per=25.94%, avg=1302.40, stdev=592.18, samples=10 iops : min= 58, max= 168, avg=81.40, stdev=37.01, samples=10 lat (msec) : 2=0.26%, 4=37.89%, 10=44.14%, 20=15.76%, 50=1.95% cpu : usr=0.10%, sys=12.24%, ctx=978, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28113: Tue Feb 25 09:32:24 2025 read: IOPS=77, BW=1246KiB/s (1276kB/s)(6256KiB/5019msec) clat (usec): min=1966, max=23268, avg=5375.39, stdev=4080.43 lat (usec): min=1967, max=23268, avg=5376.21, stdev=4080.40 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2278], 10.00th=[ 2376], 20.00th=[ 2573], | 30.00th=[ 2802], 40.00th=[ 3097], 50.00th=[ 3589], 60.00th=[ 4146], | 70.00th=[ 5342], 80.00th=[ 8094], 90.00th=[11600], 95.00th=[14484], | 99.00th=[19006], 99.50th=[21627], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 704, max= 2656, per=26.15%, avg=1248.00, stdev=636.97, samples=10 iops : min= 44, max= 166, avg=78.00, stdev=39.81, samples=10 write: IOPS=75, BW=1202KiB/s (1231kB/s)(6032KiB/5019msec) clat (usec): min=2850, max=29226, avg=7691.43, stdev=4550.73 lat (usec): min=2851, max=29227, avg=7693.05, stdev=4550.65 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3294], 10.00th=[ 3556], 20.00th=[ 3949], | 30.00th=[ 4293], 40.00th=[ 4883], 50.00th=[ 5735], 60.00th=[ 7504], | 70.00th=[ 9372], 80.00th=[11731], 90.00th=[14615], 95.00th=[15795], | 99.00th=[23987], 99.50th=[24511], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 736, max= 2336, per=23.78%, avg=1193.60, stdev=567.05, samples=10 iops : min= 46, max= 146, avg=74.60, stdev=35.44, samples=10 lat (msec) : 2=0.13%, 4=40.23%, 10=38.80%, 20=19.79%, 50=1.04% cpu : usr=0.20%, sys=11.88%, ctx=976, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28114: Tue Feb 25 09:32:24 2025 read: IOPS=75, BW=1213KiB/s (1242kB/s)(6032KiB/4972msec) clat (usec): min=2143, max=26627, avg=5386.98, stdev=3901.30 lat (usec): min=2144, max=26628, avg=5387.69, stdev=3901.31 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2376], 10.00th=[ 2474], 20.00th=[ 2671], | 30.00th=[ 2900], 40.00th=[ 3195], 50.00th=[ 3687], 60.00th=[ 4359], | 70.00th=[ 5538], 80.00th=[ 7439], 90.00th=[12125], 95.00th=[14484], | 99.00th=[17695], 99.50th=[18744], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 768, max= 1920, per=22.58%, avg=1077.33, stdev=335.24, samples=9 iops : min= 48, max= 120, avg=67.33, stdev=20.95, samples=9 write: IOPS=78, BW=1258KiB/s (1288kB/s)(6256KiB/4972msec) clat (usec): min=2844, max=30965, avg=7368.62, stdev=4421.26 lat (usec): min=2845, max=30967, avg=7370.17, stdev=4421.60 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3392], 10.00th=[ 3654], 20.00th=[ 3949], | 30.00th=[ 4293], 40.00th=[ 4752], 50.00th=[ 5538], 60.00th=[ 6587], | 70.00th=[ 8586], 80.00th=[11076], 90.00th=[13435], 95.00th=[16581], | 99.00th=[21365], 99.50th=[26346], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 800, max= 2176, per=22.88%, avg=1148.44, stdev=411.19, samples=9 iops : min= 50, max= 136, avg=71.78, stdev=25.70, samples=9 lat (msec) : 4=38.28%, 10=43.49%, 20=17.32%, 50=0.91% cpu : usr=0.08%, sys=12.53%, ctx=972, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4772KiB/s (4887kB/s), 1145KiB/s-1246KiB/s (1172kB/s-1276kB/s), io=23.4MiB (24.5MB), run=4954-5019msec WRITE: bw=5021KiB/s (5141kB/s), 1202KiB/s-1304KiB/s (1231kB/s-1336kB/s), io=24.6MiB (25.8MB), run=4954-5019msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28126: Tue Feb 25 09:32:25 2025 read: IOPS=9, BW=9766KiB/s (10.0MB/s)(7168KiB/734msec) clat (usec): min=614, max=74780, avg=37285.47, stdev=23001.89 lat (usec): min=616, max=74782, avg=37286.95, stdev=23001.81 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 619], 10.00th=[ 619], 20.00th=[30016], | 30.00th=[30802], 40.00th=[30802], 50.00th=[31851], 60.00th=[38536], | 70.00th=[38536], 80.00th=[54264], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min=10240, max=10240, per=40.60%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6975KiB/s (7143kB/s)(5120KiB/734msec) clat (msec): min=35, max=137, avg=81.96, stdev=43.08 lat (msec): min=35, max=137, avg=82.01, stdev=43.10 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 94], 60.00th=[ 94], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 4096, max= 4096, per=11.60%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=30.42%, ctx=1009, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28128: Tue Feb 25 09:32:25 2025 read: IOPS=2, BW=2522KiB/s (2583kB/s)(2048KiB/812msec) clat (msec): min=48, max=107, avg=77.90, stdev=41.91 lat (msec): min=48, max=107, avg=77.90, stdev=41.91 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 49], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 108], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 2048, max= 2048, per=8.12%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.3MiB/s (12.9MB/s)(10.0MiB/812msec) clat (msec): min=36, max=173, avg=63.49, stdev=44.30 lat (msec): min=36, max=173, avg=63.56, stdev=44.31 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 40], 50.00th=[ 45], 60.00th=[ 49], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 109], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 8192, max= 8192, per=23.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=16.67%, 250=25.00% cpu : usr=0.00%, sys=41.68%, ctx=278, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28130: Tue Feb 25 09:32:25 2025 read: IOPS=8, BW=8904KiB/s (9118kB/s)(6144KiB/690msec) clat (usec): min=429, max=75794, avg=39420.56, stdev=32610.44 lat (usec): min=429, max=75795, avg=39421.57, stdev=32610.32 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 469], | 30.00th=[ 469], 40.00th=[40633], 50.00th=[40633], 60.00th=[51643], | 70.00th=[67634], 80.00th=[67634], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 8192, max= 8192, per=32.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8904KiB/s (9118kB/s)(6144KiB/690msec) clat (msec): min=34, max=116, avg=65.99, stdev=30.97 lat (msec): min=34, max=116, avg=66.04, stdev=30.98 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 61], | 70.00th=[ 87], 80.00th=[ 87], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 8192, max= 8192, per=23.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=25.00%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=37.01%, ctx=834, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28131: Tue Feb 25 09:32:25 2025 read: IOPS=6, BW=6481KiB/s (6637kB/s)(5120KiB/790msec) clat (usec): min=613, max=44994, avg=25455.69, stdev=17679.83 lat (usec): min=614, max=44996, avg=25457.09, stdev=17679.92 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 611], 10.00th=[ 611], 20.00th=[ 611], | 30.00th=[20317], 40.00th=[20317], 50.00th=[21627], 60.00th=[21627], | 70.00th=[40109], 80.00th=[40109], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 2048, max= 2048, per=8.12%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9073KiB/s (9291kB/s)(7168KiB/790msec) clat (msec): min=40, max=149, avg=84.26, stdev=39.30 lat (msec): min=40, max=149, avg=84.34, stdev=39.31 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 82], 60.00th=[ 93], | 70.00th=[ 93], 80.00th=[ 116], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 8192, max= 8192, per=23.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=25.00%, 250=16.67% cpu : usr=0.25%, sys=33.59%, ctx=330, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.6MiB/s (25.8MB/s), 2522KiB/s-9766KiB/s (2583kB/s-10.0MB/s), io=20.0MiB (20.0MB), run=690-812msec WRITE: bw=34.5MiB/s (36.2MB/s), 6975KiB/s-12.3MiB/s (7143kB/s-12.9MB/s), io=28.0MiB (29.4MB), run=690-812msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28124: Tue Feb 25 09:32:26 2025 read: IOPS=7, BW=7337KiB/s (7513kB/s)(7168KiB/977msec) clat (msec): min=16, max=164, avg=70.08, stdev=53.45 lat (msec): min=16, max=164, avg=70.09, stdev=53.46 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 29], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 53], 60.00th=[ 96], | 70.00th=[ 96], 80.00th=[ 105], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 6144, max= 6144, per=29.31%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5241KiB/s (5366kB/s)(5120KiB/977msec) clat (msec): min=23, max=160, avg=89.12, stdev=59.50 lat (msec): min=23, max=160, avg=89.19, stdev=59.49 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 109], 60.00th=[ 109], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 2048, max= 2048, per=6.98%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 250=41.67% cpu : usr=0.00%, sys=12.91%, ctx=60, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28125: Tue Feb 25 09:32:26 2025 read: IOPS=2, BW=2860KiB/s (2929kB/s)(2048KiB/716msec) clat (usec): min=17288, max=72256, avg=44772.67, stdev=38868.12 lat (usec): min=17291, max=72258, avg=44774.70, stdev=38867.40 clat percentiles (usec): | 1.00th=[17171], 5.00th=[17171], 10.00th=[17171], 20.00th=[17171], | 30.00th=[17171], 40.00th=[17171], 50.00th=[17171], 60.00th=[71828], | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 4096, max= 4096, per=19.54%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.0MiB/s (14.6MB/s)(10.0MiB/716msec) clat (msec): min=25, max=160, avg=61.04, stdev=42.17 lat (msec): min=26, max=160, avg=61.11, stdev=42.17 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 29], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 46], 60.00th=[ 61], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 103], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min=10240, max=10240, per=34.89%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=17.76%, ctx=92, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28127: Tue Feb 25 09:32:26 2025 read: IOPS=6, BW=6606KiB/s (6765kB/s)(6144KiB/930msec) clat (msec): min=15, max=219, avg=87.37, stdev=73.48 lat (msec): min=15, max=219, avg=87.37, stdev=73.48 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 91], | 70.00th=[ 109], 80.00th=[ 109], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 4096, max= 4096, per=19.54%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6606KiB/s (6765kB/s)(6144KiB/930msec) clat (msec): min=19, max=135, avg=64.72, stdev=42.70 lat (msec): min=19, max=135, avg=64.79, stdev=42.70 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 71], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 8192, max= 8192, per=27.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=16.67%, 100=41.67%, 250=25.00% cpu : usr=0.00%, sys=11.19%, ctx=66, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28129: Tue Feb 25 09:32:26 2025 read: IOPS=5, BW=6052KiB/s (6197kB/s)(5120KiB/846msec) clat (msec): min=15, max=107, avg=76.69, stdev=36.91 lat (msec): min=15, max=107, avg=76.69, stdev=36.91 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 89], 60.00th=[ 89], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 6144, max= 6144, per=29.31%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8473KiB/s (8676kB/s)(7168KiB/846msec) clat (msec): min=21, max=154, avg=61.91, stdev=46.96 lat (msec): min=21, max=154, avg=61.98, stdev=46.96 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 25], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 53], 60.00th=[ 70], | 70.00th=[ 70], 80.00th=[ 83], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 8192, max= 8192, per=27.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=25.00%, 100=41.67%, 250=25.00% cpu : usr=0.12%, sys=15.15%, ctx=128, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.5MiB/s (21.5MB/s), 2860KiB/s-7337KiB/s (2929kB/s-7513kB/s), io=20.0MiB (20.0MB), run=716-977msec WRITE: bw=28.7MiB/s (30.1MB/s), 5241KiB/s-13.0MiB/s (5366kB/s-14.6MB/s), io=28.0MiB (29.4MB), run=716-977msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28138: Tue Feb 25 09:32:28 2025 read: IOPS=4, BW=16.7MiB/s (17.5MB/s)(12.0MiB/720msec) clat (msec): min=31, max=539, avg=235.47, stdev=268.33 lat (msec): min=31, max=539, avg=235.48, stdev=268.33 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 50=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.14%, sys=26.84%, ctx=1351, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28139: Tue Feb 25 09:32:28 2025 read: IOPS=0, BW=3064KiB/s (3137kB/s)(4096KiB/1337msec) clat (nsec): min=524649k, max=524649k, avg=524649157.00, stdev= 0.00 lat (nsec): min=524652k, max=524652k, avg=524651500.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 8192, max= 8192, per=66.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6127KiB/s (6274kB/s)(8192KiB/1337msec) clat (msec): min=275, max=507, avg=391.26, stdev=163.87 lat (msec): min=275, max=507, avg=391.50, stdev=163.92 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 8192, max= 8192, per=33.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.07%, sys=19.99%, ctx=289, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28140: Tue Feb 25 09:32:28 2025 write: IOPS=2, BW=10.0MiB/s (10.5MB/s)(12.0MiB/1194msec) clat (msec): min=257, max=618, avg=382.92, stdev=204.23 lat (msec): min=257, max=618, avg=383.29, stdev=204.28 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min=16384, max=16384, per=66.85%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=30.18%, ctx=37, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28142: Tue Feb 25 09:32:28 2025 write: IOPS=2, BW=10.2MiB/s (10.7MB/s)(12.0MiB/1178msec) clat (msec): min=126, max=672, avg=381.77, stdev=274.69 lat (msec): min=126, max=672, avg=382.07, stdev=274.74 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 127], 10.00th=[ 127], 20.00th=[ 127], | 30.00th=[ 127], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 8192, max= 8192, per=33.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=29.37%, ctx=29, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.0MiB/s (12.5MB/s), 3064KiB/s-16.7MiB/s (3137kB/s-17.5MB/s), io=16.0MiB (16.8MB), run=720-1337msec WRITE: bw=23.9MiB/s (25.1MB/s), 6127KiB/s-10.2MiB/s (6274kB/s-10.7MB/s), io=32.0MiB (33.6MB), run=1178-1337msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28141: Tue Feb 25 09:32:28 2025 read: IOPS=2, BW=9600KiB/s (9830kB/s)(12.0MiB/1280msec) clat (msec): min=83, max=740, avg=413.85, stdev=328.65 lat (msec): min=83, max=740, avg=413.85, stdev=328.65 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 418], 50.00th=[ 418], 60.00th=[ 418], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 8192, max= 8192, per=75.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=10.32%, ctx=78, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28143: Tue Feb 25 09:32:28 2025 read: IOPS=0, BW=2711KiB/s (2776kB/s)(4096KiB/1511msec) clat (nsec): min=808978k, max=808978k, avg=808978182.00, stdev= 0.00 lat (nsec): min=808980k, max=808980k, avg=808980479.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 810], 5.00th=[ 810], 10.00th=[ 810], 20.00th=[ 810], | 30.00th=[ 810], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 810], | 70.00th=[ 810], 80.00th=[ 810], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 5520, max= 5520, per=50.91%, avg=5520.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5422KiB/s (5552kB/s)(8192KiB/1511msec) clat (msec): min=315, max=340, avg=328.21, stdev=17.73 lat (msec): min=315, max=341, avg=328.52, stdev=17.80 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 342], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 5520, max= 5520, per=25.45%, avg=5520.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=9.87%, ctx=104, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28144: Tue Feb 25 09:32:28 2025 write: IOPS=2, BW=8208KiB/s (8405kB/s)(12.0MiB/1497msec) clat (msec): min=47, max=788, avg=491.50, stdev=392.18 lat (msec): min=47, max=789, avg=491.83, stdev=392.21 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 793], 80.00th=[ 793], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 5587, max= 5587, per=25.76%, avg=5587.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.55%, ctx=190, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28145: Tue Feb 25 09:32:28 2025 write: IOPS=3, BW=13.3MiB/s (13.0MB/s)(12.0MiB/901msec) clat (msec): min=161, max=519, avg=292.93, stdev=197.12 lat (msec): min=161, max=519, avg=293.19, stdev=197.21 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=12.22%, ctx=66, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.6MiB/s (11.1MB/s), 2711KiB/s-9600KiB/s (2776kB/s-9830kB/s), io=16.0MiB (16.8MB), run=1280-1511msec WRITE: bw=21.2MiB/s (22.2MB/s), 5422KiB/s-13.3MiB/s (5552kB/s-13.0MB/s), io=32.0MiB (33.6MB), run=901-1511msec