== sanity test 398b: DIO and buffer IO race ============== 13:48:45 (1740682125) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.30939 s, 38.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30411: Thu Feb 27 13:48:58 2025 read: IOPS=139, BW=556KiB/s (570kB/s)(5992KiB/10768msec) clat (usec): min=55, max=154026, avg=4073.15, stdev=4438.61 lat (usec): min=56, max=154028, avg=4073.98, stdev=4438.57 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 202], 10.00th=[ 1975], 20.00th=[ 2311], | 30.00th=[ 2638], 40.00th=[ 2999], 50.00th=[ 3556], 60.00th=[ 4178], | 70.00th=[ 4817], 80.00th=[ 5538], 90.00th=[ 6783], 95.00th=[ 7898], | 99.00th=[ 10552], 99.50th=[ 11863], 99.90th=[ 13829], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 320, max= 744, per=25.05%, avg=553.14, stdev=77.95, samples=21 iops : min= 80, max= 186, avg=138.19, stdev=19.47, samples=21 write: IOPS=146, BW=585KiB/s (599kB/s)(6296KiB/10768msec) clat (usec): min=407, max=18913, avg=2904.23, stdev=2619.99 lat (usec): min=407, max=18913, avg=2906.16, stdev=2619.20 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 510], 10.00th=[ 523], 20.00th=[ 553], | 30.00th=[ 644], 40.00th=[ 1205], 50.00th=[ 2343], 60.00th=[ 2900], | 70.00th=[ 3621], 80.00th=[ 5080], 90.00th=[ 6718], 95.00th=[ 7963], | 99.00th=[10683], 99.50th=[11994], 99.90th=[14615], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 384, max= 768, per=25.17%, avg=581.71, stdev=93.41, samples=21 iops : min= 96, max= 192, avg=145.33, stdev=23.35, samples=21 lat (usec) : 100=2.28%, 250=0.16%, 500=1.20%, 750=16.15%, 1000=1.86% lat (msec) : 2=6.58%, 4=37.30%, 10=32.68%, 20=1.76%, 250=0.03% cpu : usr=0.19%, sys=22.10%, ctx=5356, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30413: Thu Feb 27 13:48:58 2025 read: IOPS=138, BW=553KiB/s (566kB/s)(5940KiB/10747msec) clat (usec): min=50, max=20023, avg=3921.67, stdev=2279.76 lat (usec): min=50, max=20024, avg=3922.23, stdev=2279.75 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 95], 10.00th=[ 1893], 20.00th=[ 2311], | 30.00th=[ 2606], 40.00th=[ 2999], 50.00th=[ 3490], 60.00th=[ 4047], | 70.00th=[ 4621], 80.00th=[ 5473], 90.00th=[ 6587], 95.00th=[ 7963], | 99.00th=[11994], 99.50th=[13698], 99.90th=[17957], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 336, max= 704, per=24.86%, avg=548.95, stdev=83.35, samples=21 iops : min= 84, max= 176, avg=137.24, stdev=20.84, samples=21 write: IOPS=147, BW=591KiB/s (605kB/s)(6348KiB/10747msec) clat (usec): min=402, max=164794, avg=3056.23, stdev=4825.70 lat (usec): min=403, max=164795, avg=3056.98, stdev=4825.74 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 570], | 30.00th=[ 701], 40.00th=[ 1450], 50.00th=[ 2442], 60.00th=[ 2933], | 70.00th=[ 3818], 80.00th=[ 5014], 90.00th=[ 6849], 95.00th=[ 8094], | 99.00th=[ 10945], 99.50th=[ 12649], 99.90th=[ 15795], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 368, max= 736, per=25.14%, avg=580.95, stdev=90.70, samples=21 iops : min= 92, max= 184, avg=145.24, stdev=22.68, samples=21 lat (usec) : 100=2.54%, 250=0.13%, 500=1.50%, 750=14.91%, 1000=2.41% lat (msec) : 2=7.00%, 4=37.21%, 10=32.42%, 20=1.82%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.38%, ctx=5348, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30415: Thu Feb 27 13:48:58 2025 read: IOPS=139, BW=560KiB/s (573kB/s)(6028KiB/10773msec) clat (usec): min=52, max=17358, avg=3900.64, stdev=2246.12 lat (usec): min=52, max=17358, avg=3901.52, stdev=2246.09 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 92], 10.00th=[ 1876], 20.00th=[ 2245], | 30.00th=[ 2606], 40.00th=[ 2999], 50.00th=[ 3556], 60.00th=[ 4113], | 70.00th=[ 4752], 80.00th=[ 5276], 90.00th=[ 6521], 95.00th=[ 7701], | 99.00th=[11469], 99.50th=[13566], 99.90th=[16188], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 288, max= 664, per=25.17%, avg=555.81, stdev=80.74, samples=21 iops : min= 72, max= 166, avg=138.95, stdev=20.19, samples=21 write: IOPS=145, BW=581KiB/s (595kB/s)(6260KiB/10773msec) clat (usec): min=424, max=173614, avg=3083.20, stdev=5069.80 lat (usec): min=424, max=173616, avg=3083.89, stdev=5069.81 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 570], | 30.00th=[ 701], 40.00th=[ 1614], 50.00th=[ 2442], 60.00th=[ 2999], | 70.00th=[ 3720], 80.00th=[ 5080], 90.00th=[ 6718], 95.00th=[ 8029], | 99.00th=[ 11994], 99.50th=[ 13304], 99.90th=[ 17957], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 344, max= 728, per=24.89%, avg=575.24, stdev=88.00, samples=21 iops : min= 86, max= 182, avg=143.81, stdev=22.00, samples=21 lat (usec) : 100=2.64%, 250=0.16%, 500=1.66%, 750=14.78%, 1000=1.86% lat (msec) : 2=7.36%, 4=37.08%, 10=32.36%, 20=2.08%, 250=0.03% cpu : usr=0.21%, sys=22.39%, ctx=5461, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30417: Thu Feb 27 13:48:58 2025 read: IOPS=139, BW=557KiB/s (570kB/s)(6052KiB/10874msec) clat (usec): min=50, max=19100, avg=3916.11, stdev=2245.61 lat (usec): min=50, max=19100, avg=3916.77, stdev=2245.51 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 82], 10.00th=[ 1909], 20.00th=[ 2278], | 30.00th=[ 2606], 40.00th=[ 2999], 50.00th=[ 3556], 60.00th=[ 4228], | 70.00th=[ 4752], 80.00th=[ 5342], 90.00th=[ 6587], 95.00th=[ 7767], | 99.00th=[11207], 99.50th=[13173], 99.90th=[17695], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 256, max= 752, per=25.03%, avg=552.71, stdev=95.56, samples=21 iops : min= 64, max= 188, avg=138.14, stdev=23.91, samples=21 write: IOPS=143, BW=573KiB/s (587kB/s)(6236KiB/10874msec) clat (usec): min=408, max=171829, avg=3129.81, stdev=5045.88 lat (usec): min=409, max=171831, avg=3130.64, stdev=5045.93 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 506], 10.00th=[ 523], 20.00th=[ 562], | 30.00th=[ 701], 40.00th=[ 1582], 50.00th=[ 2442], 60.00th=[ 3032], | 70.00th=[ 3916], 80.00th=[ 5080], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[ 11731], 99.50th=[ 12518], 99.90th=[ 15664], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 336, max= 928, per=24.71%, avg=570.95, stdev=127.21, samples=21 iops : min= 84, max= 232, avg=142.71, stdev=31.81, samples=21 lat (usec) : 100=2.77%, 250=0.26%, 500=1.86%, 750=14.19%, 1000=2.38% lat (msec) : 2=6.67%, 4=35.77%, 10=34.21%, 20=1.86%, 250=0.03% cpu : usr=0.29%, sys=21.62%, ctx=5428, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2208KiB/s (2261kB/s), 553KiB/s-560KiB/s (566kB/s-573kB/s), io=23.4MiB (24.6MB), run=10747-10874msec WRITE: bw=2312KiB/s (2367kB/s), 573KiB/s-591KiB/s (587kB/s-605kB/s), io=24.6MiB (25.7MB), run=10747-10874msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30412: Thu Feb 27 13:49:05 2025 read: IOPS=85, BW=341KiB/s (349kB/s)(5992KiB/17579msec) clat (usec): min=1467, max=163568, avg=4757.24, stdev=4851.06 lat (usec): min=1468, max=163570, avg=4758.04, stdev=4851.06 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 2073], 10.00th=[ 2278], 20.00th=[ 2737], | 30.00th=[ 3228], 40.00th=[ 3621], 50.00th=[ 3916], 60.00th=[ 4228], | 70.00th=[ 5014], 80.00th=[ 6128], 90.00th=[ 8094], 95.00th=[ 9765], | 99.00th=[ 13960], 99.50th=[ 16450], 99.90th=[ 27395], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 184, max= 480, per=25.17%, avg=339.77, stdev=69.79, samples=35 iops : min= 46, max= 120, avg=84.86, stdev=17.42, samples=35 write: IOPS=89, BW=358KiB/s (367kB/s)(6296KiB/17579msec) clat (usec): min=2358, max=25110, avg=6589.29, stdev=2803.15 lat (usec): min=2359, max=25111, avg=6590.19, stdev=2803.18 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3261], 10.00th=[ 3654], 20.00th=[ 4228], | 30.00th=[ 4948], 40.00th=[ 5604], 50.00th=[ 6194], 60.00th=[ 6718], | 70.00th=[ 7242], 80.00th=[ 8455], 90.00th=[10159], 95.00th=[11338], | 99.00th=[16712], 99.50th=[20055], 99.90th=[22938], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 208, max= 550, per=25.29%, avg=357.34, stdev=61.03, samples=35 iops : min= 52, max= 137, avg=89.26, stdev=15.23, samples=35 lat (msec) : 2=1.76%, 4=32.71%, 10=57.55%, 20=7.65%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=11.81%, ctx=3597, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30414: Thu Feb 27 13:49:05 2025 read: IOPS=84, BW=337KiB/s (345kB/s)(5940KiB/17646msec) clat (usec): min=1582, max=166058, avg=4738.11, stdev=4914.58 lat (usec): min=1582, max=166060, avg=4738.95, stdev=4914.59 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2073], 10.00th=[ 2311], 20.00th=[ 2671], | 30.00th=[ 3130], 40.00th=[ 3589], 50.00th=[ 3884], 60.00th=[ 4228], | 70.00th=[ 4817], 80.00th=[ 6063], 90.00th=[ 8356], 95.00th=[ 10028], | 99.00th=[ 13829], 99.50th=[ 15401], 99.90th=[ 21890], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 176, max= 496, per=24.90%, avg=336.20, stdev=71.64, samples=35 iops : min= 44, max= 124, avg=84.03, stdev=17.93, samples=35 write: IOPS=89, BW=360KiB/s (368kB/s)(6348KiB/17646msec) clat (usec): min=2220, max=25397, avg=6639.65, stdev=2923.22 lat (usec): min=2221, max=25397, avg=6640.63, stdev=2923.17 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3261], 10.00th=[ 3589], 20.00th=[ 4228], | 30.00th=[ 4948], 40.00th=[ 5538], 50.00th=[ 6259], 60.00th=[ 6652], | 70.00th=[ 7242], 80.00th=[ 8586], 90.00th=[10290], 95.00th=[11731], | 99.00th=[17433], 99.50th=[20579], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 232, max= 584, per=25.31%, avg=357.69, stdev=66.46, samples=35 iops : min= 58, max= 146, avg=89.40, stdev=16.61, samples=35 lat (msec) : 2=1.76%, 4=32.88%, 10=56.80%, 20=8.20%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.95%, ctx=3655, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30416: Thu Feb 27 13:49:05 2025 read: IOPS=84, BW=339KiB/s (347kB/s)(6028KiB/17786msec) clat (usec): min=1580, max=22415, avg=4760.14, stdev=2696.83 lat (usec): min=1580, max=22416, avg=4760.90, stdev=2696.81 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 2040], 10.00th=[ 2245], 20.00th=[ 2638], | 30.00th=[ 3195], 40.00th=[ 3621], 50.00th=[ 3884], 60.00th=[ 4293], | 70.00th=[ 5211], 80.00th=[ 6718], 90.00th=[ 8717], 95.00th=[10290], | 99.00th=[12911], 99.50th=[15795], 99.90th=[20317], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 168, max= 526, per=25.00%, avg=337.46, stdev=62.98, samples=35 iops : min= 42, max= 131, avg=84.29, stdev=15.72, samples=35 write: IOPS=87, BW=352KiB/s (360kB/s)(6260KiB/17786msec) clat (msec): min=2, max=173, avg= 6.73, stdev= 5.10 lat (msec): min=2, max=173, avg= 6.74, stdev= 5.10 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 18], 99.50th=[ 20], 99.90th=[ 26], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 208, max= 550, per=24.71%, avg=349.11, stdev=74.87, samples=35 iops : min= 52, max= 137, avg=87.20, stdev=18.67, samples=35 lat (msec) : 2=2.08%, 4=32.85%, 10=56.18%, 20=8.63%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.83%, ctx=3628, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30418: Thu Feb 27 13:49:05 2025 read: IOPS=85, BW=342KiB/s (350kB/s)(6052KiB/17715msec) clat (usec): min=1484, max=22044, avg=4623.04, stdev=2547.06 lat (usec): min=1484, max=22044, avg=4623.83, stdev=2547.07 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1991], 10.00th=[ 2245], 20.00th=[ 2606], | 30.00th=[ 3097], 40.00th=[ 3589], 50.00th=[ 3916], 60.00th=[ 4178], | 70.00th=[ 4883], 80.00th=[ 6325], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[12649], 99.50th=[13566], 99.90th=[16712], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 152, max= 488, per=25.10%, avg=338.80, stdev=69.50, samples=35 iops : min= 38, max= 122, avg=84.57, stdev=17.43, samples=35 write: IOPS=88, BW=352KiB/s (360kB/s)(6236KiB/17715msec) clat (msec): min=2, max=172, avg= 6.83, stdev= 5.15 lat (msec): min=2, max=172, avg= 6.83, stdev= 5.15 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 19], 99.50th=[ 21], 99.90th=[ 27], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 176, max= 544, per=24.71%, avg=349.09, stdev=68.77, samples=35 iops : min= 44, max= 136, avg=87.14, stdev=17.18, samples=35 lat (msec) : 2=2.51%, 4=32.49%, 10=56.28%, 20=8.43%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.82%, ctx=3611, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1350KiB/s (1382kB/s), 337KiB/s-342KiB/s (345kB/s-350kB/s), io=23.4MiB (24.6MB), run=17579-17786msec WRITE: bw=1413KiB/s (1447kB/s), 352KiB/s-360KiB/s (360kB/s-368kB/s), io=24.6MiB (25.7MB), run=17579-17786msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30447: Thu Feb 27 13:49:09 2025 read: IOPS=121, BW=1945KiB/s (1991kB/s)(5920KiB/3044msec) clat (usec): min=55, max=13165, avg=4062.61, stdev=2323.26 lat (usec): min=55, max=13166, avg=4064.10, stdev=2322.64 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 72], 10.00th=[ 89], 20.00th=[ 2409], | 30.00th=[ 2900], 40.00th=[ 3458], 50.00th=[ 4015], 60.00th=[ 4555], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 6849], 95.00th=[ 7767], | 99.00th=[10945], 99.50th=[12518], 99.90th=[13173], 99.95th=[13173], | 99.99th=[13173] bw ( KiB/s): min= 1472, max= 2368, per=25.45%, avg=1940.50, stdev=371.36, samples=6 iops : min= 92, max= 148, avg=121.17, stdev=23.14, samples=6 write: IOPS=130, BW=2092KiB/s (2142kB/s)(6368KiB/3044msec) clat (usec): min=626, max=20017, avg=3810.24, stdev=3334.50 lat (usec): min=626, max=20019, avg=3813.66, stdev=3335.23 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 865], | 30.00th=[ 1106], 40.00th=[ 2147], 50.00th=[ 2802], 60.00th=[ 3720], | 70.00th=[ 4817], 80.00th=[ 6456], 90.00th=[ 8291], 95.00th=[10683], | 99.00th=[14746], 99.50th=[17433], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1664, max= 2464, per=26.12%, avg=2095.17, stdev=325.41, samples=6 iops : min= 104, max= 154, avg=130.83, stdev=20.32, samples=6 lat (usec) : 100=5.47%, 250=0.39%, 750=2.73%, 1000=11.07% lat (msec) : 2=6.77%, 4=30.08%, 10=39.84%, 20=3.52%, 50=0.13% cpu : usr=0.10%, sys=24.19%, ctx=1572, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30449: Thu Feb 27 13:49:09 2025 read: IOPS=114, BW=1829KiB/s (1873kB/s)(5744KiB/3141msec) clat (usec): min=56, max=14984, avg=4233.68, stdev=2493.68 lat (usec): min=57, max=14985, avg=4234.24, stdev=2493.69 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 77], 10.00th=[ 84], 20.00th=[ 2442], | 30.00th=[ 2933], 40.00th=[ 3458], 50.00th=[ 4047], 60.00th=[ 4752], | 70.00th=[ 5669], 80.00th=[ 6063], 90.00th=[ 7242], 95.00th=[ 8455], | 99.00th=[10421], 99.50th=[11731], 99.90th=[15008], 99.95th=[15008], | 99.99th=[15008] bw ( KiB/s): min= 1024, max= 2336, per=23.70%, avg=1807.33, stdev=447.69, samples=6 iops : min= 64, max= 146, avg=112.83, stdev=28.03, samples=6 write: IOPS=130, BW=2083KiB/s (2133kB/s)(6544KiB/3141msec) clat (usec): min=626, max=21396, avg=3758.64, stdev=3188.55 lat (usec): min=627, max=21397, avg=3760.42, stdev=3189.66 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 832], | 30.00th=[ 971], 40.00th=[ 2245], 50.00th=[ 2999], 60.00th=[ 3785], | 70.00th=[ 5145], 80.00th=[ 6390], 90.00th=[ 8225], 95.00th=[ 9503], | 99.00th=[14222], 99.50th=[16319], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1344, max= 2592, per=25.39%, avg=2036.50, stdev=451.08, samples=6 iops : min= 84, max= 162, avg=127.17, stdev=28.07, samples=6 lat (usec) : 100=5.21%, 250=0.52%, 500=0.13%, 750=3.91%, 1000=12.24% lat (msec) : 2=4.30%, 4=29.82%, 10=40.89%, 20=2.86%, 50=0.13% cpu : usr=0.16%, sys=23.57%, ctx=1528, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30450: Thu Feb 27 13:49:09 2025 read: IOPS=125, BW=2004KiB/s (2053kB/s)(6256KiB/3121msec) clat (usec): min=54, max=16656, avg=4254.50, stdev=2635.93 lat (usec): min=54, max=16656, avg=4255.16, stdev=2636.21 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 79], 10.00th=[ 117], 20.00th=[ 2507], | 30.00th=[ 2966], 40.00th=[ 3392], 50.00th=[ 3916], 60.00th=[ 4621], | 70.00th=[ 5145], 80.00th=[ 5932], 90.00th=[ 7308], 95.00th=[ 9110], | 99.00th=[12387], 99.50th=[15401], 99.90th=[16712], 99.95th=[16712], | 99.99th=[16712] bw ( KiB/s): min= 1344, max= 2784, per=26.43%, avg=2015.17, stdev=466.37, samples=6 iops : min= 84, max= 174, avg=125.83, stdev=29.15, samples=6 write: IOPS=120, BW=1933KiB/s (1979kB/s)(6032KiB/3121msec) clat (usec): min=605, max=15956, avg=3723.33, stdev=2992.72 lat (usec): min=606, max=15957, avg=3724.26, stdev=2992.67 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 791], 20.00th=[ 857], | 30.00th=[ 1270], 40.00th=[ 2278], 50.00th=[ 2966], 60.00th=[ 3818], | 70.00th=[ 4686], 80.00th=[ 6194], 90.00th=[ 8094], 95.00th=[ 9765], | 99.00th=[12518], 99.50th=[12780], 99.90th=[15926], 99.95th=[15926], | 99.99th=[15926] bw ( KiB/s): min= 1692, max= 2112, per=23.46%, avg=1882.00, stdev=169.51, samples=6 iops : min= 105, max= 132, avg=117.50, stdev=10.77, samples=6 lat (usec) : 100=4.56%, 250=1.17%, 750=2.99%, 1000=9.90% lat (msec) : 2=5.60%, 4=32.03%, 10=39.84%, 20=3.91% cpu : usr=0.29%, sys=23.30%, ctx=1570, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30452: Thu Feb 27 13:49:09 2025 read: IOPS=121, BW=1946KiB/s (1993kB/s)(6032KiB/3099msec) clat (usec): min=62, max=18470, avg=4273.88, stdev=2666.80 lat (usec): min=62, max=18471, avg=4274.44, stdev=2666.81 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 72], 10.00th=[ 83], 20.00th=[ 2442], | 30.00th=[ 3130], 40.00th=[ 3490], 50.00th=[ 4080], 60.00th=[ 4686], | 70.00th=[ 5276], 80.00th=[ 6063], 90.00th=[ 7242], 95.00th=[ 8225], | 99.00th=[13173], 99.50th=[15139], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1216, max= 2240, per=25.09%, avg=1913.17, stdev=375.60, samples=6 iops : min= 76, max= 140, avg=119.33, stdev=23.49, samples=6 write: IOPS=126, BW=2019KiB/s (2067kB/s)(6256KiB/3099msec) clat (usec): min=621, max=17751, avg=3650.61, stdev=3167.22 lat (usec): min=622, max=17752, avg=3651.52, stdev=3167.22 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 758], 10.00th=[ 791], 20.00th=[ 848], | 30.00th=[ 1074], 40.00th=[ 1860], 50.00th=[ 2638], 60.00th=[ 3523], | 70.00th=[ 4686], 80.00th=[ 6259], 90.00th=[ 8356], 95.00th=[ 9896], | 99.00th=[13698], 99.50th=[15008], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1632, max= 2464, per=24.98%, avg=2003.83, stdev=326.29, samples=6 iops : min= 102, max= 154, avg=125.00, stdev=20.36, samples=6 lat (usec) : 100=5.47%, 250=0.52%, 750=2.21%, 1000=12.11% lat (msec) : 2=7.16%, 4=29.04%, 10=39.58%, 20=3.91% cpu : usr=0.13%, sys=23.76%, ctx=1554, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7626KiB/s (7809kB/s), 1829KiB/s-2004KiB/s (1873kB/s-2053kB/s), io=23.4MiB (24.5MB), run=3044-3141msec WRITE: bw=8023KiB/s (8215kB/s), 1933KiB/s-2092KiB/s (1979kB/s-2142kB/s), io=24.6MiB (25.8MB), run=3044-3141msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30446: Thu Feb 27 13:49:10 2025 read: IOPS=93, BW=1489KiB/s (1525kB/s)(5920KiB/3976msec) clat (usec): min=1726, max=23251, avg=4279.89, stdev=3275.03 lat (usec): min=1726, max=23252, avg=4280.51, stdev=3275.09 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1942], 10.00th=[ 2008], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2540], 50.00th=[ 2933], 60.00th=[ 3359], | 70.00th=[ 4228], 80.00th=[ 5669], 90.00th=[ 9372], 95.00th=[11731], | 99.00th=[16057], 99.50th=[21103], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 928, max= 2368, per=22.45%, avg=1339.43, stdev=498.98, samples=7 iops : min= 58, max= 148, avg=83.71, stdev=31.19, samples=7 write: IOPS=100, BW=1602KiB/s (1640kB/s)(6368KiB/3976msec) clat (usec): min=2345, max=20654, avg=5982.02, stdev=3709.63 lat (usec): min=2346, max=20655, avg=5983.21, stdev=3709.67 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2900], 10.00th=[ 3064], 20.00th=[ 3326], | 30.00th=[ 3556], 40.00th=[ 3818], 50.00th=[ 4146], 60.00th=[ 5080], | 70.00th=[ 6587], 80.00th=[ 8586], 90.00th=[11994], 95.00th=[13829], | 99.00th=[19006], 99.50th=[20055], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 1152, max= 2432, per=23.02%, avg=1444.57, stdev=443.95, samples=7 iops : min= 72, max= 152, avg=90.29, stdev=27.75, samples=7 lat (msec) : 2=4.69%, 4=52.47%, 10=30.86%, 20=11.46%, 50=0.52% cpu : usr=0.30%, sys=11.55%, ctx=928, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30448: Thu Feb 27 13:49:10 2025 read: IOPS=89, BW=1433KiB/s (1468kB/s)(5744KiB/4008msec) clat (usec): min=1717, max=21184, avg=4228.15, stdev=3117.37 lat (usec): min=1717, max=21185, avg=4228.72, stdev=3117.38 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2008], 10.00th=[ 2089], 20.00th=[ 2212], | 30.00th=[ 2442], 40.00th=[ 2671], 50.00th=[ 2966], 60.00th=[ 3425], | 70.00th=[ 4228], 80.00th=[ 5407], 90.00th=[ 8455], 95.00th=[11731], | 99.00th=[14746], 99.50th=[17957], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 896, max= 2368, per=24.07%, avg=1435.62, stdev=546.09, samples=8 iops : min= 56, max= 148, avg=89.62, stdev=34.19, samples=8 write: IOPS=102, BW=1633KiB/s (1672kB/s)(6544KiB/4008msec) clat (usec): min=2553, max=21441, avg=5988.65, stdev=3427.98 lat (usec): min=2554, max=21442, avg=5989.91, stdev=3428.16 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2868], 10.00th=[ 3130], 20.00th=[ 3392], | 30.00th=[ 3687], 40.00th=[ 3949], 50.00th=[ 4424], 60.00th=[ 5604], | 70.00th=[ 6783], 80.00th=[ 8455], 90.00th=[10683], 95.00th=[13042], | 99.00th=[19268], 99.50th=[20317], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1152, max= 2944, per=25.93%, avg=1627.62, stdev=624.88, samples=8 iops : min= 72, max= 184, avg=101.62, stdev=39.13, samples=8 lat (msec) : 2=2.21%, 4=50.78%, 10=36.33%, 20=10.16%, 50=0.52% cpu : usr=0.12%, sys=11.83%, ctx=918, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30451: Thu Feb 27 13:49:10 2025 read: IOPS=97, BW=1558KiB/s (1596kB/s)(6256KiB/4015msec) clat (usec): min=1718, max=17555, avg=4347.14, stdev=3133.13 lat (usec): min=1718, max=17556, avg=4347.71, stdev=3133.12 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1942], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2606], 50.00th=[ 2868], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5997], 90.00th=[ 9634], 95.00th=[11469], | 99.00th=[14091], 99.50th=[15926], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 830, max= 2176, per=22.68%, avg=1352.86, stdev=425.22, samples=7 iops : min= 51, max= 136, avg=84.43, stdev=26.76, samples=7 write: IOPS=93, BW=1502KiB/s (1538kB/s)(6032KiB/4015msec) clat (usec): min=2378, max=22055, avg=5966.68, stdev=3591.65 lat (usec): min=2379, max=22059, avg=5967.87, stdev=3591.70 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2737], 10.00th=[ 3032], 20.00th=[ 3359], | 30.00th=[ 3621], 40.00th=[ 3818], 50.00th=[ 4293], 60.00th=[ 5604], | 70.00th=[ 6652], 80.00th=[ 8455], 90.00th=[11338], 95.00th=[13566], | 99.00th=[17695], 99.50th=[21627], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 992, max= 2272, per=21.55%, avg=1352.71, stdev=426.98, samples=7 iops : min= 62, max= 142, avg=84.43, stdev=26.73, samples=7 lat (msec) : 2=4.04%, 4=52.21%, 10=32.16%, 20=11.33%, 50=0.26% cpu : usr=0.12%, sys=11.98%, ctx=895, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30453: Thu Feb 27 13:49:10 2025 read: IOPS=94, BW=1519KiB/s (1555kB/s)(6032KiB/3971msec) clat (usec): min=1737, max=23165, avg=4204.30, stdev=3179.26 lat (usec): min=1738, max=23166, avg=4205.39, stdev=3178.99 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2638], 50.00th=[ 2900], 60.00th=[ 3195], | 70.00th=[ 3916], 80.00th=[ 5800], 90.00th=[ 8586], 95.00th=[11338], | 99.00th=[14615], 99.50th=[21103], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 832, max= 2144, per=22.45%, avg=1339.43, stdev=412.88, samples=7 iops : min= 52, max= 134, avg=83.71, stdev=25.81, samples=7 write: IOPS=98, BW=1575KiB/s (1613kB/s)(6256KiB/3971msec) clat (usec): min=2501, max=21940, avg=5885.80, stdev=3399.14 lat (usec): min=2502, max=21941, avg=5887.07, stdev=3399.14 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 2966], 10.00th=[ 3163], 20.00th=[ 3359], | 30.00th=[ 3654], 40.00th=[ 3884], 50.00th=[ 4293], 60.00th=[ 5342], | 70.00th=[ 6587], 80.00th=[ 8356], 90.00th=[10814], 95.00th=[12125], | 99.00th=[19268], 99.50th=[20317], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 960, max= 2592, per=23.16%, avg=1453.71, stdev=524.79, samples=7 iops : min= 60, max= 162, avg=90.86, stdev=32.80, samples=7 lat (msec) : 2=3.91%, 4=52.86%, 10=33.20%, 20=9.51%, 50=0.52% cpu : usr=0.28%, sys=11.61%, ctx=915, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5966KiB/s (6109kB/s), 1433KiB/s-1558KiB/s (1468kB/s-1596kB/s), io=23.4MiB (24.5MB), run=3971-4015msec WRITE: bw=6276KiB/s (6427kB/s), 1502KiB/s-1633KiB/s (1538kB/s-1672kB/s), io=24.6MiB (25.8MB), run=3971-4015msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30462: Thu Feb 27 13:49:12 2025 read: IOPS=12, BW=12.6MiB/s (13.2MB/s)(7168KiB/557msec) clat (usec): min=412, max=57550, avg=29460.63, stdev=18428.41 lat (usec): min=413, max=57551, avg=29461.57, stdev=18428.64 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[16712], | 30.00th=[25822], 40.00th=[25822], 50.00th=[26608], 60.00th=[36963], | 70.00th=[36963], 80.00th=[42206], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min=10199, max=10199, per=36.40%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9192KiB/s (9413kB/s)(5120KiB/557msec) clat (msec): min=28, max=119, avg=50.36, stdev=39.16 lat (msec): min=28, max=120, avg=50.39, stdev=39.18 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 37], 60.00th=[ 37], | 70.00th=[ 39], 80.00th=[ 39], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 8159, max= 8159, per=20.80%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=66.67%, 100=8.33%, 250=8.33% cpu : usr=0.18%, sys=29.50%, ctx=587, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30464: Thu Feb 27 13:49:12 2025 read: IOPS=2, BW=2802KiB/s (2869kB/s)(2048KiB/731msec) clat (usec): min=22745, max=90797, avg=56771.68, stdev=48120.19 lat (usec): min=22746, max=90799, avg=56772.78, stdev=48120.72 clat percentiles (usec): | 1.00th=[22676], 5.00th=[22676], 10.00th=[22676], 20.00th=[22676], | 30.00th=[22676], 40.00th=[22676], 50.00th=[22676], 60.00th=[90702], | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 2043, max= 2043, per=7.29%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.7MiB/s (14.3MB/s)(10.0MiB/731msec) clat (msec): min=17, max=112, avg=55.44, stdev=31.61 lat (msec): min=17, max=112, avg=55.48, stdev=31.62 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 20], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 44], 60.00th=[ 44], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 90], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 6131, max= 6131, per=15.63%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=30.82%, ctx=556, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30466: Thu Feb 27 13:49:12 2025 read: IOPS=11, BW=11.6MiB/s (12.2MB/s)(6144KiB/516msec) clat (usec): min=588, max=41549, avg=29878.54, stdev=15086.84 lat (usec): min=589, max=41550, avg=29879.28, stdev=15086.76 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[29230], | 30.00th=[29230], 40.00th=[31327], 50.00th=[31327], 60.00th=[38011], | 70.00th=[38536], 80.00th=[38536], 90.00th=[41681], 95.00th=[41681], | 99.00th=[41681], 99.50th=[41681], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min=10240, max=10240, per=36.55%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.6MiB/s (12.2MB/s)(6144KiB/516msec) clat (usec): min=32612, max=80060, avg=55148.44, stdev=21334.92 lat (usec): min=32630, max=80076, avg=55201.39, stdev=21305.39 clat percentiles (usec): | 1.00th=[32637], 5.00th=[32637], 10.00th=[32637], 20.00th=[35914], | 30.00th=[35914], 40.00th=[42730], 50.00th=[42730], 60.00th=[60031], | 70.00th=[79168], 80.00th=[79168], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min=10240, max=10240, per=26.11%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=66.67%, 100=25.00% cpu : usr=0.00%, sys=35.53%, ctx=401, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30469: Thu Feb 27 13:49:12 2025 read: IOPS=9, BW=9377KiB/s (9602kB/s)(5120KiB/546msec) clat (usec): min=400, max=40923, avg=19120.55, stdev=16650.02 lat (usec): min=400, max=40924, avg=19121.18, stdev=16650.00 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 400], 10.00th=[ 400], 20.00th=[ 400], | 30.00th=[ 4424], 40.00th=[ 4424], 50.00th=[24249], 60.00th=[24249], | 70.00th=[25560], 80.00th=[25560], 90.00th=[41157], 95.00th=[41157], | 99.00th=[41157], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 8159, max= 8159, per=29.12%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.8MiB/s (13.4MB/s)(7168KiB/546msec) clat (msec): min=20, max=155, avg=59.10, stdev=45.46 lat (msec): min=20, max=155, avg=59.14, stdev=45.47 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 32], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 47], 60.00th=[ 52], | 70.00th=[ 52], 80.00th=[ 71], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min=10199, max=10199, per=26.00%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 10=8.33%, 50=58.33%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=31.93%, ctx=617, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.4MiB/s (28.7MB/s), 2802KiB/s-12.6MiB/s (2869kB/s-13.2MB/s), io=20.0MiB (20.0MB), run=516-731msec WRITE: bw=38.3MiB/s (40.2MB/s), 9192KiB/s-13.7MiB/s (9413kB/s-14.3MB/s), io=28.0MiB (29.4MB), run=516-731msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30463: Thu Feb 27 13:49:12 2025 read: IOPS=13, BW=13.7MiB/s (14.4MB/s)(7168KiB/510msec) clat (usec): min=12812, max=67932, avg=26211.76, stdev=18901.64 lat (usec): min=12813, max=67933, avg=26212.51, stdev=18901.75 clat percentiles (usec): | 1.00th=[12780], 5.00th=[12780], 10.00th=[12780], 20.00th=[15139], | 30.00th=[19268], 40.00th=[19268], 50.00th=[19530], 60.00th=[23987], | 70.00th=[23987], 80.00th=[25035], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] write: IOPS=9, BW=9.80MiB/s (10.3MB/s)(5120KiB/510msec) clat (usec): min=45693, max=71759, avg=59685.97, stdev=9959.82 lat (usec): min=45714, max=71770, avg=59721.58, stdev=9948.28 clat percentiles (usec): | 1.00th=[45876], 5.00th=[45876], 10.00th=[45876], 20.00th=[45876], | 30.00th=[55313], 40.00th=[55313], 50.00th=[60031], 60.00th=[60031], | 70.00th=[65799], 80.00th=[65799], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] lat (msec) : 20=33.33%, 50=25.00%, 100=41.67% cpu : usr=0.00%, sys=12.97%, ctx=41, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30465: Thu Feb 27 13:49:12 2025 read: IOPS=2, BW=2456KiB/s (2515kB/s)(2048KiB/834msec) clat (usec): min=55239, max=79374, avg=67307.09, stdev=17065.52 lat (usec): min=55240, max=79375, avg=67308.01, stdev=17065.89 clat percentiles (usec): | 1.00th=[55313], 5.00th=[55313], 10.00th=[55313], 20.00th=[55313], | 30.00th=[55313], 40.00th=[55313], 50.00th=[55313], 60.00th=[79168], | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 2043, max= 2043, per=8.32%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.0MiB/s (12.6MB/s)(10.0MiB/834msec) clat (msec): min=16, max=124, avg=59.03, stdev=43.60 lat (msec): min=16, max=124, avg=59.08, stdev=43.59 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 25], 50.00th=[ 42], 60.00th=[ 44], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 118], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 6131, max= 6131, per=17.83%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=25.00%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=14.17%, ctx=65, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30467: Thu Feb 27 13:49:12 2025 read: IOPS=8, BW=9184KiB/s (9404kB/s)(6144KiB/669msec) clat (usec): min=14688, max=83883, avg=40374.03, stdev=27875.69 lat (usec): min=14690, max=83883, avg=40374.87, stdev=27875.49 clat percentiles (usec): | 1.00th=[14746], 5.00th=[14746], 10.00th=[14746], 20.00th=[15533], | 30.00th=[15533], 40.00th=[21103], 50.00th=[21103], 60.00th=[52691], | 70.00th=[54264], 80.00th=[54264], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 6095, max= 6095, per=24.82%, avg=6095.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9184KiB/s (9404kB/s)(6144KiB/669msec) clat (msec): min=24, max=146, avg=67.92, stdev=46.48 lat (msec): min=24, max=146, avg=67.98, stdev=46.49 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 60], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 8126, max= 8126, per=23.64%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=16.92%, ctx=139, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30468: Thu Feb 27 13:49:12 2025 read: IOPS=6, BW=6755KiB/s (6917kB/s)(5120KiB/758msec) clat (usec): min=12492, max=71700, avg=30083.42, stdev=25879.04 lat (usec): min=12492, max=71701, avg=30084.15, stdev=25879.07 clat percentiles (usec): | 1.00th=[12518], 5.00th=[12518], 10.00th=[12518], 20.00th=[12518], | 30.00th=[12780], 40.00th=[12780], 50.00th=[14222], 60.00th=[14222], | 70.00th=[39060], 80.00th=[39060], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] write: IOPS=9, BW=9456KiB/s (9683kB/s)(7168KiB/758msec) clat (msec): min=13, max=147, avg=80.02, stdev=55.29 lat (msec): min=13, max=147, avg=80.08, stdev=55.30 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 85], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 134], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 6095, max= 6095, per=17.73%, avg=6095.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=41.67%, 50=16.67%, 100=16.67%, 250=25.00% cpu : usr=0.00%, sys=12.02%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.0MiB/s (25.1MB/s), 2456KiB/s-13.7MiB/s (2515kB/s-14.4MB/s), io=20.0MiB (20.0MB), run=510-834msec WRITE: bw=33.6MiB/s (35.2MB/s), 9184KiB/s-11.0MiB/s (9404kB/s-12.6MB/s), io=28.0MiB (29.4MB), run=510-834msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30474: Thu Feb 27 13:49:15 2025 read: IOPS=1, BW=6853KiB/s (7018kB/s)(12.0MiB/1793msec) clat (msec): min=58, max=1017, avg=595.42, stdev=489.63 lat (msec): min=58, max=1017, avg=595.42, stdev=489.64 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 709], 50.00th=[ 709], 60.00th=[ 709], | 70.00th=[ 1020], 80.00th=[ 1020], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.22%, sys=20.93%, ctx=1856, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30475: Thu Feb 27 13:49:15 2025 read: IOPS=0, BW=1923KiB/s (1969kB/s)(4096KiB/2130msec) clat (nsec): min=977565k, max=977565k, avg=977564643.00, stdev= 0.00 lat (nsec): min=977567k, max=977567k, avg=977567222.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 978], 5.00th=[ 978], 10.00th=[ 978], 20.00th=[ 978], | 30.00th=[ 978], 40.00th=[ 978], 50.00th=[ 978], 60.00th=[ 978], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3846KiB/s (3938kB/s)(8192KiB/2130msec) clat (msec): min=284, max=851, avg=567.82, stdev=400.87 lat (msec): min=284, max=851, avg=568.06, stdev=400.87 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 852], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 7922, max= 7922, per=51.50%, avg=7922.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 1000=66.67% cpu : usr=0.14%, sys=11.79%, ctx=715, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30476: Thu Feb 27 13:49:15 2025 write: IOPS=1, BW=6273KiB/s (6423kB/s)(12.0MiB/1959msec) clat (msec): min=104, max=1701, avg=647.43, stdev=913.17 lat (msec): min=104, max=1701, avg=647.74, stdev=913.01 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 1703], 80.00th=[ 1703], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] lat (msec) : 250=66.67% cpu : usr=0.00%, sys=16.75%, ctx=54, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30477: Thu Feb 27 13:49:15 2025 write: IOPS=1, BW=6104KiB/s (6251kB/s)(12.0MiB/2013msec) clat (msec): min=118, max=1716, avg=667.43, stdev=909.20 lat (msec): min=118, max=1717, avg=667.67, stdev=909.17 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 1720], 80.00th=[ 1720], 90.00th=[ 1720], 95.00th=[ 1720], | 99.00th=[ 1720], 99.50th=[ 1720], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=13.47%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7692KiB/s (7877kB/s), 1923KiB/s-6853KiB/s (1969kB/s-7018kB/s), io=16.0MiB (16.8MB), run=1793-2130msec WRITE: bw=15.0MiB/s (15.8MB/s), 3846KiB/s-6273KiB/s (3938kB/s-6423kB/s), io=32.0MiB (33.6MB), run=1959-2130msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30480: Thu Feb 27 13:49:15 2025 read: IOPS=1, BW=5220KiB/s (5345kB/s)(12.0MiB/2354msec) clat (msec): min=71, max=1832, avg=782.32, stdev=928.43 lat (msec): min=71, max=1832, avg=782.32, stdev=928.43 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 1838], 80.00th=[ 1838], 90.00th=[ 1838], 95.00th=[ 1838], | 99.00th=[ 1838], 99.50th=[ 1838], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7757, max= 7757, per=100.00%, avg=7757.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.27%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30481: Thu Feb 27 13:49:15 2025 read: IOPS=0, BW=1735KiB/s (1776kB/s)(4096KiB/2361msec) clat (nsec): min=1827.9M, max=1827.9M, avg=1827944945.00, stdev= 0.00 lat (nsec): min=1827.9M, max=1827.9M, avg=1827946087.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1821], 5.00th=[ 1821], 10.00th=[ 1821], 20.00th=[ 1821], | 30.00th=[ 1821], 40.00th=[ 1821], 50.00th=[ 1821], 60.00th=[ 1821], | 70.00th=[ 1821], 80.00th=[ 1821], 90.00th=[ 1821], 95.00th=[ 1821], | 99.00th=[ 1821], 99.50th=[ 1821], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7447, max= 7447, per=100.00%, avg=7447.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3470KiB/s (3553kB/s)(8192KiB/2361msec) clat (msec): min=188, max=337, avg=263.09, stdev=104.91 lat (msec): min=189, max=337, avg=263.43, stdev=104.90 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 7447, max= 7447, per=53.66%, avg=7447.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.08%, sys=3.56%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30482: Thu Feb 27 13:49:15 2025 write: IOPS=1, BW=5948KiB/s (6090kB/s)(12.0MiB/2066msec) clat (msec): min=32, max=1797, avg=684.38, stdev=968.48 lat (msec): min=33, max=1797, avg=684.57, stdev=968.45 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 1804], 80.00th=[ 1804], 90.00th=[ 1804], 95.00th=[ 1804], | 99.00th=[ 1804], 99.50th=[ 1804], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min=14576, max=14576, per=100.00%, avg=14576.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.00%, sys=6.97%, ctx=71, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30483: Thu Feb 27 13:49:15 2025 write: IOPS=1, BW=6430KiB/s (6584kB/s)(12.0MiB/1911msec) clat (msec): min=243, max=955, avg=634.69, stdev=360.90 lat (msec): min=244, max=955, avg=635.00, stdev=360.72 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 245], 10.00th=[ 245], 20.00th=[ 245], | 30.00th=[ 245], 40.00th=[ 709], 50.00th=[ 709], 60.00th=[ 709], | 70.00th=[ 953], 80.00th=[ 953], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 8192, max= 8192, per=59.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.05%, sys=5.29%, ctx=44, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6939KiB/s (7106kB/s), 1735KiB/s-5220KiB/s (1776kB/s-5345kB/s), io=16.0MiB (16.8MB), run=2354-2361msec WRITE: bw=13.6MiB/s (14.2MB/s), 3470KiB/s-6430KiB/s (3553kB/s-6584kB/s), io=32.0MiB (33.6MB), run=1911-2361msec