== sanity test 398c: run fio to test AIO ================= 22:03:14 (1788919394) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.388121 s, 108 MB/s osc.lustre-OST0000-osc-ffff9b54ead44800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=153378: Tue Sep 8 22:05:12 2026 write: IOPS=23, BW=92.1KiB/s (94.3kB/s)(10.0MiB/111198msec); 0 zone resets slat (usec): min=20, max=9276, avg=213.24, stdev=609.06 clat (msec): min=121, max=1240, avg=694.46, stdev=153.19 lat (msec): min=121, max=1240, avg=694.68, stdev=153.21 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 472], 10.00th=[ 510], 20.00th=[ 567], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 693], 60.00th=[ 726], | 70.00th=[ 760], 80.00th=[ 810], 90.00th=[ 877], 95.00th=[ 953], | 99.00th=[ 1116], 99.50th=[ 1133], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1234] bw ( KiB/s): min= 16, max= 215, per=24.89%, avg=91.36, stdev=28.88, samples=221 iops : min= 4, max= 53, avg=22.44, stdev= 7.24, samples=221 lat (msec) : 250=0.62%, 500=8.95%, 750=56.80%, 1000=30.12%, 2000=3.52% cpu : usr=0.06%, sys=0.30%, ctx=2086, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153379: Tue Sep 8 22:05:12 2026 write: IOPS=22, BW=91.8KiB/s (94.0kB/s)(10.0MiB/111521msec); 0 zone resets slat (usec): min=21, max=13047, avg=214.58, stdev=656.39 clat (msec): min=216, max=1225, avg=696.07, stdev=148.55 lat (msec): min=216, max=1226, avg=696.29, stdev=148.58 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 468], 10.00th=[ 518], 20.00th=[ 584], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 693], 60.00th=[ 718], | 70.00th=[ 760], 80.00th=[ 818], 90.00th=[ 885], 95.00th=[ 953], | 99.00th=[ 1099], 99.50th=[ 1150], 99.90th=[ 1183], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 8, max= 224, per=24.92%, avg=91.45, stdev=29.83, samples=221 iops : min= 2, max= 56, avg=22.46, stdev= 7.53, samples=221 lat (msec) : 250=0.62%, 500=7.66%, 750=59.26%, 1000=29.65%, 2000=2.81% cpu : usr=0.06%, sys=0.33%, ctx=2374, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153380: Tue Sep 8 22:05:12 2026 write: IOPS=22, BW=91.9KiB/s (94.1kB/s)(10.0MiB/111400msec); 0 zone resets slat (usec): min=20, max=11113, avg=207.75, stdev=595.59 clat (msec): min=126, max=1239, avg=695.43, stdev=152.21 lat (msec): min=126, max=1239, avg=695.64, stdev=152.22 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 460], 10.00th=[ 514], 20.00th=[ 584], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 684], 60.00th=[ 726], | 70.00th=[ 760], 80.00th=[ 818], 90.00th=[ 894], 95.00th=[ 961], | 99.00th=[ 1083], 99.50th=[ 1133], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1234] bw ( KiB/s): min= 15, max= 232, per=24.91%, avg=91.43, stdev=33.60, samples=221 iops : min= 3, max= 58, avg=22.46, stdev= 8.43, samples=221 lat (msec) : 250=0.82%, 500=7.23%, 750=58.79%, 1000=30.78%, 2000=2.38% cpu : usr=0.07%, sys=0.32%, ctx=2221, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153381: Tue Sep 8 22:05:12 2026 write: IOPS=22, BW=91.8KiB/s (94.0kB/s)(10.0MiB/111519msec); 0 zone resets slat (usec): min=21, max=15221, avg=225.28, stdev=652.83 clat (msec): min=59, max=1192, avg=695.99, stdev=148.27 lat (msec): min=59, max=1192, avg=696.22, stdev=148.29 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 472], 10.00th=[ 510], 20.00th=[ 575], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 684], 60.00th=[ 718], | 70.00th=[ 760], 80.00th=[ 818], 90.00th=[ 885], 95.00th=[ 953], | 99.00th=[ 1099], 99.50th=[ 1133], 99.90th=[ 1183], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 8, max= 159, per=24.80%, avg=91.03, stdev=31.03, samples=222 iops : min= 2, max= 39, avg=22.36, stdev= 7.80, samples=222 lat (msec) : 100=0.04%, 250=0.08%, 500=8.28%, 750=59.73%, 1000=28.52% lat (msec) : 2000=3.36% cpu : usr=0.06%, sys=0.34%, ctx=2354, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=367KiB/s (376kB/s), 91.8KiB/s-92.1KiB/s (94.0kB/s-94.3kB/s), io=40.0MiB (41.9MB), run=111198-111521msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=153406: Tue Sep 8 22:06:18 2026 read: IOPS=20, BW=82.6KiB/s (84.5kB/s)(5048KiB/61144msec) slat (usec): min=19, max=44369, avg=8393.93, stdev=4466.90 clat (msec): min=10, max=693, avg=241.78, stdev=64.44 lat (msec): min=16, max=715, avg=250.19, stdev=65.55 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 153], 10.00th=[ 169], 20.00th=[ 190], | 30.00th=[ 207], 40.00th=[ 222], 50.00th=[ 236], 60.00th=[ 253], | 70.00th=[ 271], 80.00th=[ 292], 90.00th=[ 317], 95.00th=[ 342], | 99.00th=[ 426], 99.50th=[ 472], 99.90th=[ 651], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 23, max= 176, per=25.22%, avg=82.21, stdev=31.46, samples=121 iops : min= 5, max= 44, avg=20.08, stdev= 7.93, samples=121 write: IOPS=21, BW=84.9KiB/s (86.0kB/s)(5192KiB/61144msec); 0 zone resets slat (usec): min=31, max=9608, avg=295.34, stdev=636.89 clat (msec): min=101, max=1003, avg=509.53, stdev=141.56 lat (msec): min=101, max=1004, avg=509.83, stdev=141.58 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 300], 10.00th=[ 342], 20.00th=[ 393], | 30.00th=[ 435], 40.00th=[ 460], 50.00th=[ 498], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 625], 90.00th=[ 701], 95.00th=[ 751], | 99.00th=[ 885], 99.50th=[ 911], 99.90th=[ 961], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 39, max= 135, per=24.90%, avg=84.40, stdev=23.63, samples=121 iops : min= 9, max= 33, avg=20.64, stdev= 5.91, samples=121 lat (msec) : 20=0.08%, 100=0.23%, 250=29.34%, 500=44.96%, 750=22.66% lat (msec) : 1000=2.70%, 2000=0.04% cpu : usr=0.14%, sys=1.06%, ctx=5187, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=153407: Tue Sep 8 22:06:18 2026 read: IOPS=20, BW=80.9KiB/s (82.8kB/s)(4976KiB/61518msec) slat (usec): min=2175, max=40650, avg=8497.67, stdev=4407.66 clat (msec): min=11, max=467, avg=242.17, stdev=60.02 lat (msec): min=22, max=479, avg=250.67, stdev=60.84 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 155], 10.00th=[ 174], 20.00th=[ 194], | 30.00th=[ 209], 40.00th=[ 222], 50.00th=[ 236], 60.00th=[ 253], | 70.00th=[ 271], 80.00th=[ 292], 90.00th=[ 317], 95.00th=[ 347], | 99.00th=[ 414], 99.50th=[ 426], 99.90th=[ 460], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 23, max= 167, per=24.57%, avg=80.08, stdev=31.25, samples=122 iops : min= 5, max= 41, avg=19.55, stdev= 7.79, samples=122 write: IOPS=21, BW=85.6KiB/s (87.6kB/s)(5264KiB/61518msec); 0 zone resets slat (usec): min=25, max=14140, avg=274.98, stdev=728.69 clat (msec): min=131, max=984, avg=509.51, stdev=136.91 lat (msec): min=131, max=984, avg=509.79, stdev=136.94 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 309], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 430], 40.00th=[ 468], 50.00th=[ 502], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 625], 90.00th=[ 693], 95.00th=[ 743], | 99.00th=[ 877], 99.50th=[ 919], 99.90th=[ 969], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 32, max= 152, per=25.00%, avg=84.75, stdev=23.81, samples=122 iops : min= 8, max= 38, avg=20.72, stdev= 6.00, samples=122 lat (msec) : 20=0.08%, 50=0.08%, 100=0.16%, 250=28.63%, 500=45.66% lat (msec) : 750=23.12%, 1000=2.27% cpu : usr=0.14%, sys=1.07%, ctx=5007, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=153408: Tue Sep 8 22:06:18 2026 read: IOPS=20, BW=81.1KiB/s (83.0kB/s)(4984KiB/61471msec) slat (usec): min=2312, max=28721, avg=8367.39, stdev=4176.46 clat (msec): min=8, max=478, avg=241.51, stdev=63.54 lat (msec): min=13, max=506, avg=249.88, stdev=64.35 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 150], 10.00th=[ 167], 20.00th=[ 192], | 30.00th=[ 209], 40.00th=[ 224], 50.00th=[ 239], 60.00th=[ 255], | 70.00th=[ 271], 80.00th=[ 292], 90.00th=[ 326], 95.00th=[ 351], | 99.00th=[ 409], 99.50th=[ 439], 99.90th=[ 468], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 31, max= 168, per=24.74%, avg=80.66, stdev=30.04, samples=122 iops : min= 7, max= 42, avg=19.70, stdev= 7.52, samples=122 write: IOPS=21, BW=85.5KiB/s (87.6kB/s)(5256KiB/61471msec); 0 zone resets slat (usec): min=29, max=7378, avg=261.58, stdev=504.13 clat (msec): min=131, max=1007, avg=509.69, stdev=135.35 lat (msec): min=131, max=1007, avg=509.95, stdev=135.38 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 305], 10.00th=[ 351], 20.00th=[ 397], | 30.00th=[ 430], 40.00th=[ 464], 50.00th=[ 498], 60.00th=[ 531], | 70.00th=[ 575], 80.00th=[ 625], 90.00th=[ 684], 95.00th=[ 743], | 99.00th=[ 877], 99.50th=[ 902], 99.90th=[ 986], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 39, max= 136, per=24.98%, avg=84.67, stdev=23.38, samples=122 iops : min= 9, max= 34, avg=20.70, stdev= 5.87, samples=122 lat (msec) : 10=0.04%, 20=0.12%, 50=0.20%, 100=0.16%, 250=27.97% lat (msec) : 500=46.29%, 750=22.81%, 1000=2.38%, 2000=0.04% cpu : usr=0.13%, sys=1.09%, ctx=4957, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=153409: Tue Sep 8 22:06:18 2026 read: IOPS=20, BW=82.8KiB/s (84.8kB/s)(5060KiB/61074msec) slat (usec): min=2107, max=28726, avg=8476.68, stdev=4158.54 clat (msec): min=6, max=781, avg=242.47, stdev=70.02 lat (msec): min=15, max=793, avg=250.95, stdev=70.92 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 155], 10.00th=[ 171], 20.00th=[ 188], | 30.00th=[ 207], 40.00th=[ 222], 50.00th=[ 234], 60.00th=[ 253], | 70.00th=[ 271], 80.00th=[ 292], 90.00th=[ 321], 95.00th=[ 347], | 99.00th=[ 443], 99.50th=[ 514], 99.90th=[ 776], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 7, max= 191, per=25.28%, avg=82.40, stdev=31.28, samples=121 iops : min= 1, max= 47, avg=20.13, stdev= 7.83, samples=121 write: IOPS=21, BW=84.8KiB/s (86.8kB/s)(5180KiB/61074msec); 0 zone resets slat (usec): min=25, max=8971, avg=298.40, stdev=701.93 clat (msec): min=115, max=978, avg=507.59, stdev=138.52 lat (msec): min=115, max=979, avg=507.89, stdev=138.52 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 305], 10.00th=[ 342], 20.00th=[ 384], | 30.00th=[ 426], 40.00th=[ 464], 50.00th=[ 493], 60.00th=[ 531], | 70.00th=[ 575], 80.00th=[ 617], 90.00th=[ 693], 95.00th=[ 743], | 99.00th=[ 885], 99.50th=[ 911], 99.90th=[ 936], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 31, max= 184, per=24.87%, avg=84.31, stdev=25.29, samples=121 iops : min= 7, max= 46, avg=20.61, stdev= 6.38, samples=121 lat (msec) : 10=0.08%, 20=0.16%, 50=0.16%, 100=0.04%, 250=28.79% lat (msec) : 500=46.68%, 750=21.72%, 1000=2.38% cpu : usr=0.18%, sys=1.07%, ctx=5174, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=326KiB/s (334kB/s), 80.9KiB/s-82.8KiB/s (82.8kB/s-84.8kB/s), io=19.6MiB (20.5MB), run=61074-61518msec WRITE: bw=340KiB/s (348kB/s), 84.8KiB/s-85.6KiB/s (86.8kB/s-87.6kB/s), io=20.4MiB (21.4MB), run=61074-61518msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=153424: Tue Sep 8 22:06:24 2026 read: IOPS=3, BW=153MiB/s (160MB/s)(40.0MiB/262msec) slat (nsec): min=14571k, max=14571k, avg=14571260.00, stdev= 0.00 clat (nsec): min=233040k, max=233040k, avg=233039515.00, stdev= 0.00 lat (nsec): min=247615k, max=247615k, avg=247614849.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.82%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=40.0MiB (41.9MB), run=262-262msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout