== sanity test 398c: run fio to test AIO ================= 00:22:54 (1769059374) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.338373 s, 124 MB/s osc.lustre-OST0000-osc-ffff95bab3cab000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147056: Thu Jan 22 00:25:10 2026 write: IOPS=20, BW=80.1KiB/s (82.1kB/s)(10.0MiB/127765msec); 0 zone resets slat (usec): min=22, max=9022, avg=169.34, stdev=538.73 clat (msec): min=180, max=2036, avg=797.73, stdev=199.05 lat (msec): min=181, max=2036, avg=797.90, stdev=199.05 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 514], 10.00th=[ 592], 20.00th=[ 667], | 30.00th=[ 718], 40.00th=[ 760], 50.00th=[ 802], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 936], 90.00th=[ 1003], 95.00th=[ 1070], | 99.00th=[ 1401], 99.50th=[ 1636], 99.90th=[ 1871], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 8, max= 256, per=25.01%, avg=80.04, stdev=29.67, samples=252 iops : min= 2, max= 64, avg=19.58, stdev= 7.44, samples=252 lat (msec) : 250=1.25%, 500=3.67%, 750=33.55%, 1000=50.90%, 2000=10.55% lat (msec) : >=2000=0.08% cpu : usr=0.06%, sys=0.26%, ctx=1974, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147057: Thu Jan 22 00:25:10 2026 write: IOPS=20, BW=80.0KiB/s (81.9kB/s)(10.0MiB/127959msec); 0 zone resets slat (usec): min=21, max=19876, avg=161.97, stdev=592.09 clat (msec): min=213, max=1668, avg=798.39, stdev=189.60 lat (msec): min=213, max=1668, avg=798.55, stdev=189.60 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 518], 10.00th=[ 584], 20.00th=[ 667], | 30.00th=[ 718], 40.00th=[ 751], 50.00th=[ 802], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 936], 90.00th=[ 1011], 95.00th=[ 1083], | 99.00th=[ 1301], 99.50th=[ 1519], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 15, max= 224, per=24.84%, avg=79.47, stdev=28.40, samples=254 iops : min= 3, max= 56, avg=19.44, stdev= 7.17, samples=254 lat (msec) : 250=0.62%, 500=3.75%, 750=34.92%, 1000=49.02%, 2000=11.68% cpu : usr=0.06%, sys=0.26%, ctx=1967, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147058: Thu Jan 22 00:25:10 2026 write: IOPS=20, BW=80.1KiB/s (81.0kB/s)(10.0MiB/127917msec); 0 zone resets slat (usec): min=22, max=19906, avg=170.48, stdev=608.60 clat (msec): min=59, max=1726, avg=798.55, stdev=195.32 lat (msec): min=59, max=1727, avg=798.72, stdev=195.31 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 510], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 718], 40.00th=[ 760], 50.00th=[ 802], 60.00th=[ 835], | 70.00th=[ 877], 80.00th=[ 936], 90.00th=[ 1011], 95.00th=[ 1083], | 99.00th=[ 1401], 99.50th=[ 1552], 99.90th=[ 1703], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 16, max= 208, per=24.82%, avg=79.44, stdev=27.90, samples=254 iops : min= 4, max= 52, avg=19.43, stdev= 7.01, samples=254 lat (msec) : 100=0.04%, 250=0.70%, 500=4.06%, 750=33.20%, 1000=50.94% lat (msec) : 2000=11.05% cpu : usr=0.05%, sys=0.28%, ctx=2331, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147059: Thu Jan 22 00:25:10 2026 write: IOPS=20, BW=80.0KiB/s (81.9kB/s)(10.0MiB/127955msec); 0 zone resets slat (usec): min=21, max=16168, avg=196.14, stdev=779.56 clat (msec): min=197, max=1703, avg=798.42, stdev=192.14 lat (msec): min=197, max=1704, avg=798.62, stdev=192.16 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 518], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 718], 40.00th=[ 760], 50.00th=[ 802], 60.00th=[ 844], | 70.00th=[ 885], 80.00th=[ 944], 90.00th=[ 1011], 95.00th=[ 1070], | 99.00th=[ 1351], 99.50th=[ 1519], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 256, per=25.12%, avg=80.38, stdev=30.48, samples=251 iops : min= 1, max= 64, avg=19.66, stdev= 7.69, samples=251 lat (msec) : 250=0.62%, 500=3.95%, 750=33.09%, 1000=51.48%, 2000=10.86% cpu : usr=0.06%, sys=0.25%, ctx=1978, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=320KiB/s (328kB/s), 80.0KiB/s-80.1KiB/s (81.9kB/s-82.1kB/s), io=40.0MiB (41.9MB), run=127765-127959msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147087: Thu Jan 22 00:26:37 2026 read: IOPS=15, BW=63.0KiB/s (64.6kB/s)(5048KiB/80074msec) slat (usec): min=2368, max=43676, avg=11480.78, stdev=6267.14 clat (msec): min=14, max=622, avg=311.72, stdev=77.68 lat (msec): min=28, max=629, avg=323.20, stdev=78.65 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 203], 10.00th=[ 222], 20.00th=[ 249], | 30.00th=[ 268], 40.00th=[ 288], 50.00th=[ 305], 60.00th=[ 330], | 70.00th=[ 351], 80.00th=[ 376], 90.00th=[ 409], 95.00th=[ 439], | 99.00th=[ 518], 99.50th=[ 567], 99.90th=[ 617], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8, max= 143, per=24.98%, avg=62.45, stdev=27.11, samples=159 iops : min= 2, max= 35, avg=15.14, stdev= 6.78, samples=159 write: IOPS=16, BW=64.8KiB/s (66.4kB/s)(5192KiB/80074msec); 0 zone resets slat (usec): min=27, max=19192, avg=267.27, stdev=940.88 clat (msec): min=208, max=1174, avg=670.16, stdev=171.88 lat (msec): min=208, max=1174, avg=670.43, stdev=171.91 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 409], 10.00th=[ 468], 20.00th=[ 527], | 30.00th=[ 575], 40.00th=[ 617], 50.00th=[ 667], 60.00th=[ 701], | 70.00th=[ 751], 80.00th=[ 810], 90.00th=[ 894], 95.00th=[ 978], | 99.00th=[ 1099], 99.50th=[ 1116], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 143, per=24.70%, avg=64.22, stdev=23.83, samples=159 iops : min= 1, max= 35, avg=15.58, stdev= 5.96, samples=159 lat (msec) : 20=0.04%, 50=0.12%, 100=0.20%, 250=10.12%, 500=46.05% lat (msec) : 750=28.12%, 1000=13.28%, 2000=2.07% cpu : usr=0.13%, sys=0.81%, ctx=4764, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147088: Thu Jan 22 00:26:37 2026 read: IOPS=15, BW=62.1KiB/s (63.6kB/s)(4976KiB/80075msec) slat (usec): min=2325, max=53324, avg=11591.18, stdev=6299.01 clat (msec): min=16, max=612, avg=310.31, stdev=77.82 lat (msec): min=26, max=622, avg=321.92, stdev=78.89 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 197], 10.00th=[ 215], 20.00th=[ 249], | 30.00th=[ 271], 40.00th=[ 288], 50.00th=[ 305], 60.00th=[ 326], | 70.00th=[ 347], 80.00th=[ 372], 90.00th=[ 409], 95.00th=[ 443], | 99.00th=[ 527], 99.50th=[ 550], 99.90th=[ 609], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8, max= 183, per=24.62%, avg=61.55, stdev=27.39, samples=159 iops : min= 2, max= 45, avg=14.92, stdev= 6.86, samples=159 write: IOPS=16, BW=65.7KiB/s (67.3kB/s)(5264KiB/80075msec); 0 zone resets slat (usec): min=27, max=8999, avg=231.07, stdev=549.10 clat (msec): min=246, max=1149, avg=667.17, stdev=170.06 lat (msec): min=246, max=1149, avg=667.40, stdev=170.03 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 401], 10.00th=[ 460], 20.00th=[ 523], | 30.00th=[ 575], 40.00th=[ 617], 50.00th=[ 659], 60.00th=[ 701], | 70.00th=[ 743], 80.00th=[ 810], 90.00th=[ 894], 95.00th=[ 961], | 99.00th=[ 1099], 99.50th=[ 1116], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 15, max= 127, per=25.04%, avg=65.09, stdev=20.57, samples=159 iops : min= 3, max= 31, avg=15.81, stdev= 5.14, samples=159 lat (msec) : 20=0.04%, 50=0.16%, 100=0.12%, 250=10.16%, 500=45.59% lat (msec) : 750=28.95%, 1000=13.12%, 2000=1.88% cpu : usr=0.16%, sys=0.85%, ctx=4983, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147089: Thu Jan 22 00:26:37 2026 read: IOPS=15, BW=62.3KiB/s (63.8kB/s)(4984KiB/80017msec) slat (usec): min=2562, max=50224, avg=11617.56, stdev=6431.01 clat (msec): min=13, max=615, avg=310.75, stdev=78.61 lat (msec): min=22, max=623, avg=322.37, stdev=79.29 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 199], 10.00th=[ 215], 20.00th=[ 247], | 30.00th=[ 266], 40.00th=[ 288], 50.00th=[ 305], 60.00th=[ 326], | 70.00th=[ 347], 80.00th=[ 376], 90.00th=[ 409], 95.00th=[ 447], | 99.00th=[ 527], 99.50th=[ 558], 99.90th=[ 609], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8, max= 143, per=24.72%, avg=61.80, stdev=26.36, samples=159 iops : min= 2, max= 35, avg=14.98, stdev= 6.63, samples=159 write: IOPS=16, BW=65.7KiB/s (67.3kB/s)(5256KiB/80017msec); 0 zone resets slat (usec): min=24, max=14423, avg=261.14, stdev=800.87 clat (msec): min=204, max=1193, avg=666.10, stdev=173.21 lat (msec): min=204, max=1194, avg=666.36, stdev=173.23 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 409], 10.00th=[ 456], 20.00th=[ 518], | 30.00th=[ 567], 40.00th=[ 617], 50.00th=[ 659], 60.00th=[ 693], | 70.00th=[ 743], 80.00th=[ 810], 90.00th=[ 894], 95.00th=[ 969], | 99.00th=[ 1099], 99.50th=[ 1133], 99.90th=[ 1167], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 23, max= 143, per=24.95%, avg=64.86, stdev=22.35, samples=159 iops : min= 5, max= 35, avg=15.75, stdev= 5.59, samples=159 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=10.55%, 500=45.98% lat (msec) : 750=28.59%, 1000=12.54%, 2000=2.07% cpu : usr=0.11%, sys=0.83%, ctx=4715, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147090: Thu Jan 22 00:26:37 2026 read: IOPS=15, BW=63.6KiB/s (65.2kB/s)(5060KiB/79505msec) slat (usec): min=80, max=47976, avg=11713.70, stdev=6320.71 clat (msec): min=15, max=764, avg=311.35, stdev=82.01 lat (msec): min=25, max=769, avg=323.07, stdev=82.87 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 194], 10.00th=[ 218], 20.00th=[ 249], | 30.00th=[ 268], 40.00th=[ 288], 50.00th=[ 305], 60.00th=[ 330], | 70.00th=[ 351], 80.00th=[ 380], 90.00th=[ 414], 95.00th=[ 439], | 99.00th=[ 514], 99.50th=[ 550], 99.90th=[ 751], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 7, max= 167, per=25.61%, avg=64.01, stdev=23.94, samples=156 iops : min= 1, max= 41, avg=15.54, stdev= 5.99, samples=156 write: IOPS=16, BW=65.2KiB/s (66.7kB/s)(5180KiB/79505msec); 0 zone resets slat (usec): min=23, max=9105, avg=274.35, stdev=715.59 clat (msec): min=81, max=1234, avg=665.62, stdev=178.90 lat (msec): min=82, max=1234, avg=665.89, stdev=178.89 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 405], 10.00th=[ 456], 20.00th=[ 518], | 30.00th=[ 567], 40.00th=[ 609], 50.00th=[ 659], 60.00th=[ 701], | 70.00th=[ 743], 80.00th=[ 810], 90.00th=[ 894], 95.00th=[ 986], | 99.00th=[ 1133], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 135, per=24.70%, avg=64.23, stdev=23.07, samples=158 iops : min= 1, max= 33, avg=15.59, stdev= 5.80, samples=158 lat (msec) : 20=0.04%, 50=0.39%, 100=0.27%, 250=10.20%, 500=45.98% lat (msec) : 750=28.40%, 1000=12.66%, 2000=2.07% cpu : usr=0.11%, sys=0.80%, ctx=4477, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=251KiB/s (257kB/s), 62.1KiB/s-63.6KiB/s (63.6kB/s-65.2kB/s), io=19.6MiB (20.5MB), run=79505-80075msec WRITE: bw=261KiB/s (267kB/s), 64.8KiB/s-65.7KiB/s (66.4kB/s-67.3kB/s), io=20.4MiB (21.4MB), run=79505-80075msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147105: Thu Jan 22 00:26:45 2026 read: IOPS=2, BW=113MiB/s (119MB/s)(40.0MiB/353msec) slat (nsec): min=17949k, max=17949k, avg=17949017.00, stdev= 0.00 clat (nsec): min=320034k, max=320034k, avg=320034011.00, stdev= 0.00 lat (nsec): min=337987k, max=337987k, avg=337987035.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.41%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=113MiB/s (119MB/s), 113MiB/s-113MiB/s (119MB/s-119MB/s), io=40.0MiB (41.9MB), run=353-353msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout