== sanity test 398c: run fio to test AIO ================= 05:00:36 (1788512436) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.4361 s, 96.2 MB/s osc.lustre-OST0000-osc-ffffa00a14caf000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181533: Fri Sep 4 05:01:12 2026 write: IOPS=95, BW=380KiB/s (389kB/s)(10.0MiB/26941msec); 0 zone resets slat (usec): min=28, max=9806, avg=215.55, stdev=654.52 clat (msec): min=18, max=273, avg=167.31, stdev=37.13 lat (msec): min=18, max=274, avg=167.53, stdev=37.14 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 105], 10.00th=[ 122], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 199], 90.00th=[ 220], 95.00th=[ 232], | 99.00th=[ 253], 99.50th=[ 255], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 275] bw ( KiB/s): min= 253, max= 630, per=24.97%, avg=379.51, stdev=74.37, samples=53 iops : min= 63, max= 157, avg=94.49, stdev=18.63, samples=53 lat (msec) : 20=0.08%, 50=0.04%, 100=4.30%, 250=93.98%, 500=1.60% cpu : usr=0.18%, sys=1.16%, ctx=2628, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181534: Fri Sep 4 05:01:12 2026 write: IOPS=95, BW=381KiB/s (390kB/s)(10.0MiB/26887msec); 0 zone resets slat (usec): min=25, max=27923, avg=233.89, stdev=840.22 clat (msec): min=23, max=281, avg=166.92, stdev=37.32 lat (msec): min=23, max=281, avg=167.16, stdev=37.30 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 100], 10.00th=[ 120], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 215], 95.00th=[ 228], | 99.00th=[ 259], 99.50th=[ 264], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 253, max= 638, per=25.06%, avg=380.94, stdev=74.39, samples=53 iops : min= 63, max= 159, avg=94.87, stdev=18.62, samples=53 lat (msec) : 50=0.16%, 100=5.31%, 250=92.50%, 500=2.03% cpu : usr=0.20%, sys=1.18%, ctx=2655, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181535: Fri Sep 4 05:01:12 2026 write: IOPS=95, BW=383KiB/s (392kB/s)(10.0MiB/26743msec); 0 zone resets slat (usec): min=31, max=28378, avg=266.48, stdev=946.98 clat (msec): min=16, max=282, avg=166.69, stdev=38.13 lat (msec): min=16, max=282, avg=166.95, stdev=38.12 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 100], 10.00th=[ 117], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 218], 95.00th=[ 228], | 99.00th=[ 255], 99.50th=[ 264], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 284] bw ( KiB/s): min= 253, max= 629, per=25.07%, avg=381.09, stdev=75.45, samples=53 iops : min= 63, max= 157, avg=94.91, stdev=18.90, samples=53 lat (msec) : 20=0.08%, 50=0.35%, 100=4.65%, 250=93.20%, 500=1.72% cpu : usr=0.19%, sys=1.17%, ctx=2886, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181536: Fri Sep 4 05:01:12 2026 write: IOPS=95, BW=380KiB/s (389kB/s)(10.0MiB/26938msec); 0 zone resets slat (usec): min=29, max=17521, avg=193.49, stdev=728.82 clat (msec): min=74, max=268, avg=167.19, stdev=36.67 lat (msec): min=74, max=268, avg=167.38, stdev=36.70 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 213], 95.00th=[ 224], | 99.00th=[ 259], 99.50th=[ 264], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 268] bw ( KiB/s): min= 253, max= 637, per=24.81%, avg=377.06, stdev=72.56, samples=53 iops : min= 63, max= 159, avg=93.89, stdev=18.18, samples=53 lat (msec) : 100=4.10%, 250=94.10%, 500=1.80% cpu : usr=0.18%, sys=0.90%, ctx=2160, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1520KiB/s (1557kB/s), 380KiB/s-383KiB/s (389kB/s-392kB/s), io=40.0MiB (41.9MB), run=26743-26941msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181561: Fri Sep 4 05:01:56 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(5048KiB/35850msec) slat (usec): min=3003, max=84617, avg=15939.36, stdev=9827.77 clat (msec): min=64, max=462, avg=209.56, stdev=57.50 lat (msec): min=83, max=476, avg=225.51, stdev=60.40 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 138], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 321], | 99.00th=[ 414], 99.50th=[ 422], 99.90th=[ 439], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 62, max= 215, per=25.09%, avg=139.75, stdev=34.89, samples=71 iops : min= 15, max= 53, avg=34.45, stdev= 8.73, samples=71 write: IOPS=36, BW=145KiB/s (148kB/s)(5192KiB/35850msec); 0 zone resets slat (usec): min=29, max=21704, avg=280.33, stdev=1038.80 clat (msec): min=96, max=465, avg=219.33, stdev=55.67 lat (msec): min=100, max=465, avg=219.61, stdev=55.68 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 153], 10.00th=[ 163], 20.00th=[ 176], | 30.00th=[ 188], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 234], 80.00th=[ 253], 90.00th=[ 292], 95.00th=[ 321], | 99.00th=[ 414], 99.50th=[ 430], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 55, max= 216, per=24.84%, avg=144.06, stdev=37.13, samples=71 iops : min= 13, max= 54, avg=35.55, stdev= 9.28, samples=71 lat (msec) : 100=0.20%, 250=80.31%, 500=19.49% cpu : usr=0.25%, sys=1.92%, ctx=4639, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181562: Fri Sep 4 05:01:56 2026 read: IOPS=34, BW=138KiB/s (142kB/s)(4976KiB/35978msec) slat (usec): min=2867, max=72217, avg=15894.61, stdev=9645.60 clat (msec): min=24, max=463, avg=211.75, stdev=59.37 lat (msec): min=44, max=503, avg=227.65, stdev=61.45 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 251], 90.00th=[ 292], 95.00th=[ 334], | 99.00th=[ 384], 99.50th=[ 414], 99.90th=[ 460], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 47, max= 222, per=24.66%, avg=137.38, stdev=36.76, samples=71 iops : min= 11, max= 55, avg=33.85, stdev= 9.23, samples=71 write: IOPS=36, BW=146KiB/s (150kB/s)(5264KiB/35978msec); 0 zone resets slat (usec): min=30, max=17936, avg=306.45, stdev=1155.01 clat (msec): min=36, max=434, avg=219.04, stdev=55.38 lat (msec): min=37, max=435, avg=219.35, stdev=55.37 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 146], 10.00th=[ 159], 20.00th=[ 176], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 405], 99.50th=[ 414], 99.90th=[ 426], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 48, max= 247, per=25.15%, avg=145.86, stdev=38.10, samples=71 iops : min= 12, max= 61, avg=35.99, stdev= 9.53, samples=71 lat (msec) : 50=0.12%, 100=0.31%, 250=78.01%, 500=21.56% cpu : usr=0.37%, sys=1.88%, ctx=4688, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181563: Fri Sep 4 05:01:56 2026 read: IOPS=34, BW=139KiB/s (143kB/s)(4984KiB/35799msec) slat (usec): min=43, max=90572, avg=15989.10, stdev=9955.20 clat (msec): min=26, max=488, avg=213.14, stdev=61.26 lat (msec): min=35, max=508, avg=229.14, stdev=63.56 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 253], 90.00th=[ 292], 95.00th=[ 330], | 99.00th=[ 409], 99.50th=[ 460], 99.90th=[ 481], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 70, max= 215, per=24.94%, avg=138.93, stdev=31.93, samples=70 iops : min= 17, max= 53, avg=34.21, stdev= 7.99, samples=70 write: IOPS=36, BW=147KiB/s (150kB/s)(5256KiB/35799msec); 0 zone resets slat (usec): min=30, max=12290, avg=261.04, stdev=737.21 clat (msec): min=84, max=479, avg=217.90, stdev=55.23 lat (msec): min=85, max=479, avg=218.17, stdev=55.26 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 146], 10.00th=[ 161], 20.00th=[ 176], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 334], | 99.00th=[ 409], 99.50th=[ 422], 99.90th=[ 477], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 55, max= 246, per=25.19%, avg=146.10, stdev=41.46, samples=70 iops : min= 13, max= 61, avg=36.04, stdev=10.37, samples=70 lat (msec) : 50=0.04%, 100=0.39%, 250=78.87%, 500=20.70% cpu : usr=0.29%, sys=1.79%, ctx=4330, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181564: Fri Sep 4 05:01:56 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(5060KiB/36014msec) slat (usec): min=167, max=75362, avg=15792.70, stdev=9669.40 clat (msec): min=32, max=522, avg=212.68, stdev=62.88 lat (msec): min=32, max=534, avg=228.48, stdev=65.47 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 136], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 253], 90.00th=[ 292], 95.00th=[ 334], | 99.00th=[ 426], 99.50th=[ 439], 99.90th=[ 489], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 56, max= 247, per=24.96%, avg=139.01, stdev=36.43, samples=71 iops : min= 14, max= 61, avg=34.25, stdev= 9.09, samples=71 write: IOPS=35, BW=144KiB/s (147kB/s)(5180KiB/36014msec); 0 zone resets slat (usec): min=31, max=11767, avg=243.89, stdev=727.56 clat (msec): min=24, max=507, avg=218.83, stdev=58.46 lat (msec): min=25, max=507, avg=219.08, stdev=58.48 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 146], 10.00th=[ 157], 20.00th=[ 176], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 288], 95.00th=[ 334], | 99.00th=[ 405], 99.50th=[ 422], 99.90th=[ 481], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 69, max= 214, per=24.44%, avg=141.73, stdev=36.23, samples=71 iops : min= 17, max= 53, avg=34.96, stdev= 9.08, samples=71 lat (msec) : 50=0.55%, 100=0.31%, 250=76.88%, 500=22.19%, 750=0.08% cpu : usr=0.26%, sys=1.76%, ctx=4067, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=557KiB/s (571kB/s), 138KiB/s-141KiB/s (142kB/s-144kB/s), io=19.6MiB (20.5MB), run=35799-36014msec WRITE: bw=580KiB/s (594kB/s), 144KiB/s-147KiB/s (147kB/s-150kB/s), io=20.4MiB (21.4MB), run=35799-36014msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181573: Fri Sep 4 05:02:05 2026 read: IOPS=1, BW=79.2MiB/s (83.1MB/s)(40.0MiB/505msec) slat (nsec): min=44329k, max=44329k, avg=44329351.00, stdev= 0.00 clat (nsec): min=435352k, max=435352k, avg=435351547.00, stdev= 0.00 lat (nsec): min=479686k, max=479686k, avg=479686063.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.58%, ctx=8, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=79.2MiB/s (83.1MB/s), 79.2MiB/s-79.2MiB/s (83.1MB/s-83.1MB/s), io=40.0MiB (41.9MB), run=505-505msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout