== sanity test 398c: run fio to test AIO ================= 21:20:28 (1784942428) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.308839 s, 136 MB/s osc.lustre-OST0000-osc-ffff90e7c4d48800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145636: Fri Jul 24 21:22:17 2026 write: IOPS=25, BW=100KiB/s (103kB/s)(10.0MiB/102139msec); 0 zone resets slat (usec): min=36, max=8416, avg=253.60, stdev=528.70 clat (msec): min=71, max=1195, avg=637.29, stdev=149.03 lat (msec): min=71, max=1195, avg=637.55, stdev=149.06 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 393], 10.00th=[ 477], 20.00th=[ 523], | 30.00th=[ 567], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 751], 90.00th=[ 818], 95.00th=[ 877], | 99.00th=[ 1053], 99.50th=[ 1116], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1200] bw ( KiB/s): min= 31, max= 255, per=24.79%, avg=99.42, stdev=30.45, samples=203 iops : min= 7, max= 63, avg=24.43, stdev= 7.60, samples=203 lat (msec) : 100=0.04%, 250=1.25%, 500=13.16%, 750=65.27%, 1000=18.71% lat (msec) : 2000=1.56% cpu : usr=0.06%, sys=0.42%, ctx=3055, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145637: Fri Jul 24 21:22:17 2026 write: IOPS=25, BW=100KiB/s (103kB/s)(10.0MiB/102102msec); 0 zone resets slat (usec): min=37, max=12792, avg=283.53, stdev=723.45 clat (msec): min=66, max=1195, avg=637.38, stdev=150.45 lat (msec): min=66, max=1195, avg=637.66, stdev=150.48 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 384], 10.00th=[ 472], 20.00th=[ 527], | 30.00th=[ 567], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 760], 90.00th=[ 818], 95.00th=[ 877], | 99.00th=[ 1036], 99.50th=[ 1133], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1200] bw ( KiB/s): min= 24, max= 254, per=24.80%, avg=99.43, stdev=31.22, samples=203 iops : min= 6, max= 63, avg=24.43, stdev= 7.76, samples=203 lat (msec) : 100=0.04%, 250=1.29%, 500=13.16%, 750=64.45%, 1000=19.41% lat (msec) : 2000=1.64% cpu : usr=0.06%, sys=0.40%, ctx=3110, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145638: Fri Jul 24 21:22:17 2026 write: IOPS=25, BW=100KiB/s (103kB/s)(10.0MiB/102067msec); 0 zone resets slat (usec): min=37, max=13599, avg=282.00, stdev=653.88 clat (msec): min=129, max=1216, avg=637.34, stdev=150.77 lat (msec): min=129, max=1216, avg=637.62, stdev=150.76 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 380], 10.00th=[ 468], 20.00th=[ 531], | 30.00th=[ 567], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 760], 90.00th=[ 818], 95.00th=[ 877], | 99.00th=[ 1070], 99.50th=[ 1099], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1217] bw ( KiB/s): min= 31, max= 238, per=24.79%, avg=99.40, stdev=29.66, samples=203 iops : min= 7, max= 59, avg=24.43, stdev= 7.37, samples=203 lat (msec) : 250=1.21%, 500=12.46%, 750=65.27%, 1000=19.30%, 2000=1.76% cpu : usr=0.07%, sys=0.41%, ctx=3255, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145639: Fri Jul 24 21:22:17 2026 write: IOPS=25, BW=100KiB/s (103kB/s)(10.0MiB/102138msec); 0 zone resets slat (usec): min=37, max=17762, avg=262.89, stdev=687.21 clat (msec): min=146, max=1183, avg=637.42, stdev=145.04 lat (msec): min=146, max=1183, avg=637.68, stdev=145.03 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 409], 10.00th=[ 477], 20.00th=[ 527], | 30.00th=[ 575], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 659], | 70.00th=[ 701], 80.00th=[ 760], 90.00th=[ 810], 95.00th=[ 860], | 99.00th=[ 995], 99.50th=[ 1099], 99.90th=[ 1150], 99.95th=[ 1167], | 99.99th=[ 1183] bw ( KiB/s): min= 31, max= 255, per=24.79%, avg=99.42, stdev=28.18, samples=203 iops : min= 7, max= 63, avg=24.44, stdev= 7.05, samples=203 lat (msec) : 250=1.29%, 500=13.32%, 750=63.79%, 1000=20.66%, 2000=0.94% cpu : usr=0.05%, sys=0.39%, ctx=2896, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=401KiB/s (411kB/s), 100KiB/s-100KiB/s (103kB/s-103kB/s), io=40.0MiB (41.9MB), run=102067-102139msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145664: Fri Jul 24 21:23:20 2026 read: IOPS=21, BW=86.8KiB/s (88.9kB/s)(5048KiB/58146msec) slat (usec): min=2061, max=32301, avg=8097.37, stdev=4011.48 clat (msec): min=13, max=469, avg=225.66, stdev=55.51 lat (msec): min=21, max=477, avg=233.77, stdev=56.41 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 146], 10.00th=[ 161], 20.00th=[ 178], | 30.00th=[ 194], 40.00th=[ 209], 50.00th=[ 224], 60.00th=[ 239], | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 313], | 99.00th=[ 380], 99.50th=[ 388], 99.90th=[ 426], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 31, max= 184, per=25.13%, avg=86.43, stdev=30.05, samples=115 iops : min= 7, max= 46, avg=21.23, stdev= 7.53, samples=115 write: IOPS=22, BW=89.3KiB/s (91.4kB/s)(5192KiB/58146msec); 0 zone resets slat (usec): min=42, max=7046, avg=290.86, stdev=555.30 clat (msec): min=128, max=873, avg=487.21, stdev=114.77 lat (msec): min=129, max=873, avg=487.51, stdev=114.77 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 313], 10.00th=[ 342], 20.00th=[ 384], | 30.00th=[ 418], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 510], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 651], 95.00th=[ 693], | 99.00th=[ 785], 99.50th=[ 810], 99.90th=[ 869], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 42, max= 175, per=24.79%, avg=88.73, stdev=22.60, samples=115 iops : min= 10, max= 43, avg=21.80, stdev= 5.74, samples=115 lat (msec) : 20=0.08%, 50=0.16%, 100=0.16%, 250=33.55%, 500=44.26% lat (msec) : 750=21.05%, 1000=0.74% cpu : usr=0.13%, sys=1.02%, ctx=5306, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145665: Fri Jul 24 21:23:20 2026 read: IOPS=21, BW=85.4KiB/s (87.4kB/s)(4976KiB/58277msec) slat (usec): min=2154, max=26315, avg=7999.17, stdev=3842.08 clat (msec): min=36, max=416, avg=227.01, stdev=53.33 lat (msec): min=43, max=423, avg=235.01, stdev=54.11 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 150], 10.00th=[ 167], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 253], 80.00th=[ 268], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 380], 99.50th=[ 397], 99.90th=[ 409], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 8, max= 168, per=24.59%, avg=84.60, stdev=30.60, samples=116 iops : min= 2, max= 42, avg=20.77, stdev= 7.63, samples=116 write: IOPS=22, BW=90.3KiB/s (92.5kB/s)(5264KiB/58277msec); 0 zone resets slat (usec): min=40, max=6464, avg=276.20, stdev=471.72 clat (msec): min=68, max=866, avg=484.87, stdev=121.94 lat (msec): min=68, max=867, avg=485.15, stdev=121.96 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 296], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 506], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 651], 95.00th=[ 701], | 99.00th=[ 776], 99.50th=[ 802], 99.90th=[ 860], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 40, max= 168, per=25.06%, avg=89.72, stdev=25.29, samples=116 iops : min= 10, max= 42, avg=22.05, stdev= 6.42, samples=116 lat (msec) : 50=0.08%, 100=0.35%, 250=33.24%, 500=44.65%, 750=20.90% lat (msec) : 1000=0.78% cpu : usr=0.13%, sys=1.03%, ctx=5247, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145666: Fri Jul 24 21:23:20 2026 read: IOPS=21, BW=85.6KiB/s (87.7kB/s)(4984KiB/58191msec) slat (usec): min=2021, max=30665, avg=8150.97, stdev=4137.20 clat (msec): min=4, max=423, avg=227.88, stdev=55.64 lat (msec): min=8, max=434, avg=236.03, stdev=56.39 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 144], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 241], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 380], 99.50th=[ 397], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 7, max= 191, per=24.85%, avg=85.47, stdev=28.65, samples=115 iops : min= 1, max= 47, avg=20.98, stdev= 7.23, samples=115 write: IOPS=22, BW=90.3KiB/s (92.5kB/s)(5256KiB/58191msec); 0 zone resets slat (usec): min=53, max=8786, avg=296.73, stdev=556.86 clat (msec): min=108, max=826, avg=482.52, stdev=116.45 lat (msec): min=108, max=827, avg=482.82, stdev=116.45 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 309], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 642], 95.00th=[ 684], | 99.00th=[ 776], 99.50th=[ 802], 99.90th=[ 818], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 23, max= 167, per=24.99%, avg=89.47, stdev=25.11, samples=115 iops : min= 5, max= 41, avg=21.98, stdev= 6.35, samples=115 lat (msec) : 10=0.04%, 50=0.20%, 100=0.20%, 250=32.15%, 500=46.21% lat (msec) : 750=20.39%, 1000=0.82% cpu : usr=0.10%, sys=1.00%, ctx=5054, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145667: Fri Jul 24 21:23:20 2026 read: IOPS=21, BW=87.3KiB/s (89.4kB/s)(5060KiB/57969msec) slat (usec): min=99, max=33226, avg=8006.70, stdev=4170.71 clat (msec): min=10, max=460, avg=226.37, stdev=54.65 lat (msec): min=12, max=468, avg=234.38, stdev=55.54 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 148], 10.00th=[ 167], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 209], 50.00th=[ 224], 60.00th=[ 239], | 70.00th=[ 251], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 380], 99.50th=[ 388], 99.90th=[ 426], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 32, max= 199, per=25.32%, avg=87.10, stdev=28.88, samples=115 iops : min= 8, max= 49, avg=21.39, stdev= 7.25, samples=115 write: IOPS=22, BW=89.4KiB/s (91.5kB/s)(5180KiB/57969msec); 0 zone resets slat (usec): min=54, max=8357, avg=299.54, stdev=555.43 clat (msec): min=86, max=831, avg=486.41, stdev=118.17 lat (msec): min=86, max=831, avg=486.71, stdev=118.16 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 305], 10.00th=[ 342], 20.00th=[ 384], | 30.00th=[ 418], 40.00th=[ 451], 50.00th=[ 477], 60.00th=[ 510], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 651], 95.00th=[ 693], | 99.00th=[ 768], 99.50th=[ 793], 99.90th=[ 827], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 32, max= 151, per=24.77%, avg=88.67, stdev=25.10, samples=115 iops : min= 8, max= 37, avg=21.78, stdev= 6.31, samples=115 lat (msec) : 20=0.08%, 50=0.27%, 100=0.16%, 250=34.26%, 500=43.59% lat (msec) : 750=21.05%, 1000=0.59% cpu : usr=0.13%, sys=1.01%, ctx=5299, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=344KiB/s (353kB/s), 85.4KiB/s-87.3KiB/s (87.4kB/s-89.4kB/s), io=19.6MiB (20.5MB), run=57969-58277msec WRITE: bw=358KiB/s (367kB/s), 89.3KiB/s-90.3KiB/s (91.4kB/s-92.5kB/s), io=20.4MiB (21.4MB), run=57969-58277msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145679: Fri Jul 24 21:23:25 2026 read: IOPS=3, BW=131MiB/s (138MB/s)(40.0MiB/305msec) slat (nsec): min=23766k, max=23766k, avg=23765643.00, stdev= 0.00 clat (nsec): min=272242k, max=272242k, avg=272242476.00, stdev= 0.00 lat (nsec): min=296013k, max=296013k, avg=296012831.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.28%, ctx=19, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=131MiB/s (138MB/s), 131MiB/s-131MiB/s (138MB/s-138MB/s), io=40.0MiB (41.9MB), run=305-305msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout