== sanity test 398c: run fio to test AIO ================= 08:59:22 (1775221162) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.346297 s, 121 MB/s osc.lustre-OST0000-osc-ffff987883414000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148803: Fri Apr 3 09:01:02 2026 write: IOPS=27, BW=110KiB/s (112kB/s)(10.0MiB/93458msec); 0 zone resets slat (usec): min=35, max=10359, avg=288.07, stdev=521.98 clat (msec): min=93, max=1011, avg=583.34, stdev=142.19 lat (msec): min=93, max=1011, avg=583.63, stdev=142.20 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 384], 10.00th=[ 418], 20.00th=[ 472], | 30.00th=[ 506], 40.00th=[ 535], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 693], 90.00th=[ 785], 95.00th=[ 844], | 99.00th=[ 961], 99.50th=[ 969], 99.90th=[ 1003], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 40, max= 254, per=24.87%, avg=108.67, stdev=30.64, samples=186 iops : min= 10, max= 63, avg=26.78, stdev= 7.66, samples=186 lat (msec) : 100=0.62%, 250=0.04%, 500=27.81%, 750=57.73%, 1000=13.63% lat (msec) : 2000=0.16% cpu : usr=0.05%, sys=0.44%, ctx=3143, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148804: Fri Apr 3 09:01:02 2026 write: IOPS=27, BW=110KiB/s (112kB/s)(10.0MiB/93273msec); 0 zone resets slat (usec): min=38, max=11028, avg=266.76, stdev=502.61 clat (msec): min=85, max=1063, avg=582.48, stdev=141.26 lat (msec): min=85, max=1063, avg=582.75, stdev=141.28 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 380], 10.00th=[ 409], 20.00th=[ 468], | 30.00th=[ 510], 40.00th=[ 535], 50.00th=[ 567], 60.00th=[ 609], | 70.00th=[ 642], 80.00th=[ 701], 90.00th=[ 776], 95.00th=[ 827], | 99.00th=[ 936], 99.50th=[ 969], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1062] bw ( KiB/s): min= 8, max= 254, per=25.10%, avg=109.68, stdev=34.05, samples=184 iops : min= 2, max= 63, avg=27.03, stdev= 8.52, samples=184 lat (msec) : 100=0.62%, 250=0.08%, 500=27.23%, 750=58.83%, 1000=13.12% lat (msec) : 2000=0.12% cpu : usr=0.04%, sys=0.43%, ctx=2627, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148805: Fri Apr 3 09:01:02 2026 write: IOPS=27, BW=109KiB/s (112kB/s)(10.0MiB/93572msec); 0 zone resets slat (usec): min=41, max=11125, avg=290.26, stdev=548.64 clat (msec): min=25, max=1064, avg=584.00, stdev=138.48 lat (msec): min=25, max=1064, avg=584.29, stdev=138.50 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 376], 10.00th=[ 414], 20.00th=[ 468], | 30.00th=[ 506], 40.00th=[ 542], 50.00th=[ 575], 60.00th=[ 609], | 70.00th=[ 642], 80.00th=[ 693], 90.00th=[ 760], 95.00th=[ 844], | 99.00th=[ 969], 99.50th=[ 978], 99.90th=[ 1028], 99.95th=[ 1036], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 256, per=24.87%, avg=108.67, stdev=31.76, samples=186 iops : min= 1, max= 64, avg=26.77, stdev= 7.96, samples=186 lat (msec) : 50=0.08%, 250=0.20%, 500=28.98%, 750=59.10%, 1000=11.29% lat (msec) : 2000=0.35% cpu : usr=0.05%, sys=0.42%, ctx=3299, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148806: Fri Apr 3 09:01:02 2026 write: IOPS=27, BW=109KiB/s (112kB/s)(10.0MiB/93552msec); 0 zone resets slat (usec): min=37, max=8132, avg=260.62, stdev=429.94 clat (msec): min=116, max=1028, avg=583.90, stdev=140.84 lat (msec): min=118, max=1028, avg=584.16, stdev=140.85 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 380], 10.00th=[ 414], 20.00th=[ 468], | 30.00th=[ 510], 40.00th=[ 535], 50.00th=[ 575], 60.00th=[ 609], | 70.00th=[ 642], 80.00th=[ 693], 90.00th=[ 776], 95.00th=[ 835], | 99.00th=[ 953], 99.50th=[ 969], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 248, per=25.14%, avg=109.87, stdev=33.81, samples=184 iops : min= 1, max= 62, avg=27.08, stdev= 8.47, samples=184 lat (msec) : 250=0.66%, 500=26.91%, 750=60.78%, 1000=11.29%, 2000=0.35% cpu : usr=0.05%, sys=0.40%, ctx=3096, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=438KiB/s (448kB/s), 109KiB/s-110KiB/s (112kB/s-112kB/s), io=40.0MiB (41.9MB), run=93273-93572msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148840: Fri Apr 3 09:02:09 2026 read: IOPS=20, BW=81.2KiB/s (83.1kB/s)(5048KiB/62167msec) slat (usec): min=2247, max=25870, avg=7608.16, stdev=3291.54 clat (msec): min=14, max=559, avg=245.56, stdev=55.21 lat (msec): min=20, max=566, avg=253.17, stdev=55.67 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 174], 10.00th=[ 188], 20.00th=[ 203], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 241], 60.00th=[ 255], | 70.00th=[ 271], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 338], | 99.00th=[ 405], 99.50th=[ 418], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 15, max= 175, per=25.11%, avg=80.87, stdev=31.21, samples=123 iops : min= 3, max= 43, avg=19.89, stdev= 7.82, samples=123 write: IOPS=20, BW=83.5KiB/s (85.5kB/s)(5192KiB/62167msec); 0 zone resets slat (usec): min=53, max=9514, avg=291.41, stdev=483.53 clat (msec): min=218, max=1052, avg=518.28, stdev=135.93 lat (msec): min=218, max=1052, avg=518.58, stdev=135.93 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 309], 10.00th=[ 347], 20.00th=[ 401], | 30.00th=[ 443], 40.00th=[ 481], 50.00th=[ 514], 60.00th=[ 550], | 70.00th=[ 584], 80.00th=[ 625], 90.00th=[ 693], 95.00th=[ 751], | 99.00th=[ 877], 99.50th=[ 978], 99.90th=[ 1045], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 31, max= 167, per=24.74%, avg=83.14, stdev=23.50, samples=123 iops : min= 7, max= 41, avg=20.46, stdev= 5.88, samples=123 lat (msec) : 20=0.08%, 50=0.23%, 100=0.08%, 250=28.05%, 500=43.91% lat (msec) : 750=25.08%, 1000=2.46%, 2000=0.12% cpu : usr=0.10%, sys=0.92%, ctx=5299, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148841: Fri Apr 3 09:02:09 2026 read: IOPS=20, BW=80.3KiB/s (82.2kB/s)(4976KiB/61995msec) slat (usec): min=83, max=25862, avg=7608.80, stdev=3239.56 clat (msec): min=8, max=529, avg=246.11, stdev=52.85 lat (msec): min=9, max=534, avg=253.72, stdev=53.47 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 174], 10.00th=[ 188], 20.00th=[ 205], | 30.00th=[ 218], 40.00th=[ 232], 50.00th=[ 245], 60.00th=[ 257], | 70.00th=[ 271], 80.00th=[ 284], 90.00th=[ 305], 95.00th=[ 338], | 99.00th=[ 388], 99.50th=[ 422], 99.90th=[ 518], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 23, max= 152, per=24.75%, avg=79.71, stdev=27.57, samples=123 iops : min= 5, max= 38, avg=19.59, stdev= 6.92, samples=123 write: IOPS=21, BW=84.9KiB/s (86.9kB/s)(5264KiB/61995msec); 0 zone resets slat (usec): min=60, max=5002, avg=291.36, stdev=372.68 clat (msec): min=63, max=1052, avg=513.02, stdev=139.55 lat (msec): min=64, max=1052, avg=513.31, stdev=139.56 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 292], 10.00th=[ 338], 20.00th=[ 393], | 30.00th=[ 439], 40.00th=[ 481], 50.00th=[ 506], 60.00th=[ 550], | 70.00th=[ 584], 80.00th=[ 625], 90.00th=[ 684], 95.00th=[ 751], | 99.00th=[ 869], 99.50th=[ 978], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 24, max= 168, per=25.15%, avg=84.50, stdev=26.34, samples=123 iops : min= 6, max= 42, avg=20.80, stdev= 6.56, samples=123 lat (msec) : 10=0.04%, 20=0.08%, 50=0.08%, 100=0.16%, 250=26.99% lat (msec) : 500=45.47%, 750=24.69%, 1000=2.38%, 2000=0.12% cpu : usr=0.12%, sys=0.92%, ctx=5409, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148842: Fri Apr 3 09:02:09 2026 read: IOPS=20, BW=80.2KiB/s (82.1kB/s)(4984KiB/62169msec) slat (usec): min=2085, max=27810, avg=7733.15, stdev=3324.20 clat (msec): min=12, max=427, avg=244.60, stdev=53.39 lat (msec): min=19, max=448, avg=252.34, stdev=53.97 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 167], 10.00th=[ 188], 20.00th=[ 203], | 30.00th=[ 218], 40.00th=[ 230], 50.00th=[ 241], 60.00th=[ 253], | 70.00th=[ 268], 80.00th=[ 284], 90.00th=[ 309], 95.00th=[ 338], | 99.00th=[ 401], 99.50th=[ 418], 99.90th=[ 426], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 7, max= 175, per=24.82%, avg=79.92, stdev=32.25, samples=123 iops : min= 1, max= 43, avg=19.65, stdev= 8.06, samples=123 write: IOPS=21, BW=84.5KiB/s (86.6kB/s)(5256KiB/62169msec); 0 zone resets slat (usec): min=59, max=5292, avg=281.53, stdev=400.10 clat (msec): min=201, max=1044, avg=516.00, stdev=137.36 lat (msec): min=201, max=1044, avg=516.28, stdev=137.37 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 296], 10.00th=[ 342], 20.00th=[ 397], | 30.00th=[ 439], 40.00th=[ 477], 50.00th=[ 514], 60.00th=[ 550], | 70.00th=[ 584], 80.00th=[ 625], 90.00th=[ 693], 95.00th=[ 751], | 99.00th=[ 869], 99.50th=[ 953], 99.90th=[ 1036], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 31, max= 136, per=24.91%, avg=83.71, stdev=21.67, samples=123 iops : min= 7, max= 34, avg=20.59, stdev= 5.45, samples=123 lat (msec) : 20=0.04%, 50=0.23%, 100=0.12%, 250=28.20%, 500=43.59% lat (msec) : 750=25.20%, 1000=2.50%, 2000=0.12% cpu : usr=0.10%, sys=0.91%, ctx=5138, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148843: Fri Apr 3 09:02:09 2026 read: IOPS=20, BW=82.0KiB/s (84.0kB/s)(5060KiB/61677msec) slat (usec): min=2470, max=23744, avg=7670.45, stdev=3289.84 clat (msec): min=9, max=474, avg=243.73, stdev=53.12 lat (msec): min=17, max=482, avg=251.41, stdev=53.64 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 169], 10.00th=[ 186], 20.00th=[ 203], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 243], 60.00th=[ 255], | 70.00th=[ 268], 80.00th=[ 284], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 380], 99.50th=[ 405], 99.90th=[ 468], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 23, max= 175, per=25.49%, avg=82.07, stdev=28.49, samples=122 iops : min= 5, max= 43, avg=20.19, stdev= 7.13, samples=122 write: IOPS=20, BW=83.0KiB/s (86.0kB/s)(5180KiB/61677msec); 0 zone resets slat (usec): min=57, max=5571, avg=302.17, stdev=457.05 clat (msec): min=195, max=976, avg=514.76, stdev=137.77 lat (msec): min=195, max=976, avg=515.07, stdev=137.77 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 305], 10.00th=[ 342], 20.00th=[ 397], | 30.00th=[ 439], 40.00th=[ 472], 50.00th=[ 510], 60.00th=[ 542], | 70.00th=[ 584], 80.00th=[ 625], 90.00th=[ 693], 95.00th=[ 751], | 99.00th=[ 885], 99.50th=[ 911], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 31, max= 159, per=24.84%, avg=83.46, stdev=24.88, samples=122 iops : min= 7, max= 39, avg=20.53, stdev= 6.20, samples=122 lat (msec) : 10=0.04%, 20=0.08%, 50=0.23%, 100=0.04%, 250=28.63% lat (msec) : 500=44.06%, 750=24.41%, 1000=2.50% cpu : usr=0.11%, sys=0.91%, ctx=5424, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=323KiB/s (331kB/s), 80.2KiB/s-82.0KiB/s (82.1kB/s-84.0kB/s), io=19.6MiB (20.5MB), run=61677-62169msec WRITE: bw=336KiB/s (344kB/s), 83.5KiB/s-84.9KiB/s (85.5kB/s-86.9kB/s), io=20.4MiB (21.4MB), run=61677-62169msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148856: Fri Apr 3 09:02:14 2026 read: IOPS=3, BW=149MiB/s (157MB/s)(40.0MiB/268msec) slat (nsec): min=18639k, max=18639k, avg=18639382.00, stdev= 0.00 clat (nsec): min=234309k, max=234309k, avg=234309376.00, stdev= 0.00 lat (nsec): min=252953k, max=252953k, avg=252952559.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.75%, ctx=43, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=149MiB/s (157MB/s), 149MiB/s-149MiB/s (157MB/s-157MB/s), io=40.0MiB (41.9MB), run=268-268msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout