== sanity test 398c: run fio to test AIO ================= 00:02:30 (1787371350) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.414055 s, 101 MB/s osc.lustre-OST0000-osc-ffff9408c30e1000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145265: Sat Aug 22 00:05:45 2026 write: IOPS=13, BW=55.6KiB/s (56.0kB/s)(10.0MiB/184062msec); 0 zone resets slat (usec): min=23, max=19188, avg=304.76, stdev=1182.43 clat (msec): min=317, max=2963, avg=1148.74, stdev=412.40 lat (msec): min=318, max=2963, avg=1149.05, stdev=412.47 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 642], 10.00th=[ 709], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 986], 50.00th=[ 1083], 60.00th=[ 1183], | 70.00th=[ 1301], 80.00th=[ 1452], 90.00th=[ 1670], 95.00th=[ 1955], | 99.00th=[ 2500], 99.50th=[ 2635], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 128, per=25.54%, avg=56.70, stdev=30.79, samples=354 iops : min= 1, max= 32, avg=13.64, stdev= 7.75, samples=354 lat (msec) : 500=2.58%, 750=10.86%, 1000=28.36%, 2000=53.52%, >=2000=4.69% cpu : usr=0.06%, sys=0.25%, ctx=1773, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145266: Sat Aug 22 00:05:45 2026 write: IOPS=13, BW=55.6KiB/s (56.0kB/s)(10.0MiB/184010msec); 0 zone resets slat (usec): min=23, max=22530, avg=294.77, stdev=1207.59 clat (msec): min=329, max=2930, avg=1148.73, stdev=409.07 lat (msec): min=329, max=2930, avg=1149.03, stdev=409.15 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 651], 10.00th=[ 709], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 986], 50.00th=[ 1070], 60.00th=[ 1183], | 70.00th=[ 1301], 80.00th=[ 1452], 90.00th=[ 1653], 95.00th=[ 1938], | 99.00th=[ 2534], 99.50th=[ 2635], 99.90th=[ 2903], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 128, per=25.47%, avg=56.54, stdev=27.72, samples=355 iops : min= 1, max= 32, avg=13.59, stdev= 7.01, samples=355 lat (msec) : 500=2.62%, 750=9.77%, 1000=29.22%, 2000=54.30%, >=2000=4.10% cpu : usr=0.06%, sys=0.24%, ctx=1912, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145267: Sat Aug 22 00:05:45 2026 write: IOPS=13, BW=55.8KiB/s (57.2kB/s)(10.0MiB/183464msec); 0 zone resets slat (usec): min=21, max=25860, avg=351.29, stdev=1517.71 clat (msec): min=126, max=3044, avg=1145.79, stdev=414.30 lat (msec): min=126, max=3044, avg=1146.14, stdev=414.33 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 625], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 902], 40.00th=[ 995], 50.00th=[ 1083], 60.00th=[ 1183], | 70.00th=[ 1301], 80.00th=[ 1452], 90.00th=[ 1636], 95.00th=[ 1888], | 99.00th=[ 2567], 99.50th=[ 2735], 99.90th=[ 2970], 99.95th=[ 2970], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 240, per=25.54%, avg=56.69, stdev=30.29, samples=354 iops : min= 1, max= 60, avg=13.63, stdev= 7.63, samples=354 lat (msec) : 250=0.12%, 500=3.63%, 750=8.48%, 1000=28.48%, 2000=55.12% lat (msec) : >=2000=4.18% cpu : usr=0.04%, sys=0.28%, ctx=1936, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145268: Sat Aug 22 00:05:45 2026 write: IOPS=13, BW=55.7KiB/s (57.1kB/s)(10.0MiB/183762msec); 0 zone resets slat (usec): min=21, max=15108, avg=235.76, stdev=831.43 clat (msec): min=101, max=2969, avg=1147.42, stdev=414.52 lat (msec): min=101, max=2969, avg=1147.66, stdev=414.53 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 634], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 978], 50.00th=[ 1083], 60.00th=[ 1200], | 70.00th=[ 1318], 80.00th=[ 1452], 90.00th=[ 1670], 95.00th=[ 1905], | 99.00th=[ 2500], 99.50th=[ 2668], 99.90th=[ 2970], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 152, per=25.53%, avg=56.68, stdev=30.24, samples=354 iops : min= 1, max= 38, avg=13.64, stdev= 7.60, samples=354 lat (msec) : 250=0.08%, 500=3.12%, 750=9.41%, 1000=30.43%, 2000=52.81% lat (msec) : >=2000=4.14% cpu : usr=0.05%, sys=0.24%, ctx=1857, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=223KiB/s (228kB/s), 55.6KiB/s-55.8KiB/s (56.0kB/s-57.2kB/s), io=40.0MiB (41.9MB), run=183464-184062msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145304: Sat Aug 22 00:07:53 2026 read: IOPS=10, BW=43.2KiB/s (44.3kB/s)(5048KiB/116762msec) slat (usec): min=33, max=95669, avg=15929.99, stdev=9630.20 clat (msec): min=36, max=1060, avg=463.48, stdev=134.02 lat (msec): min=52, max=1081, avg=479.42, stdev=135.81 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 447], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 567], 90.00th=[ 634], 95.00th=[ 693], | 99.00th=[ 844], 99.50th=[ 911], 99.90th=[ 986], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 119, per=25.12%, avg=42.95, stdev=20.41, samples=230 iops : min= 1, max= 29, avg=10.21, stdev= 5.11, samples=230 write: IOPS=11, BW=44.5KiB/s (45.5kB/s)(5192KiB/116762msec); 0 zone resets slat (usec): min=25, max=20546, avg=343.21, stdev=1108.72 clat (msec): min=152, max=1969, avg=972.10, stdev=278.97 lat (msec): min=153, max=1969, avg=972.45, stdev=278.94 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1452], | 99.00th=[ 1703], 99.50th=[ 1754], 99.90th=[ 1955], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 103, per=24.63%, avg=43.84, stdev=19.62, samples=231 iops : min= 1, max= 25, avg=10.44, stdev= 4.94, samples=231 lat (msec) : 50=0.04%, 100=0.31%, 250=1.21%, 500=31.72%, 750=24.61% lat (msec) : 1000=20.74%, 2000=21.37% cpu : usr=0.12%, sys=0.75%, ctx=4382, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145305: Sat Aug 22 00:07:53 2026 read: IOPS=10, BW=42.6KiB/s (43.6kB/s)(4976KiB/116899msec) slat (usec): min=3055, max=89476, avg=15274.01, stdev=8883.09 clat (msec): min=39, max=1068, avg=458.10, stdev=135.37 lat (msec): min=49, max=1091, avg=473.38, stdev=137.44 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 472], | 70.00th=[ 510], 80.00th=[ 558], 90.00th=[ 634], 95.00th=[ 701], | 99.00th=[ 852], 99.50th=[ 919], 99.90th=[ 1062], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7, max= 110, per=24.93%, avg=42.63, stdev=20.40, samples=228 iops : min= 1, max= 27, avg=10.14, stdev= 5.09, samples=228 write: IOPS=11, BW=45.0KiB/s (46.1kB/s)(5264KiB/116899msec); 0 zone resets slat (usec): min=29, max=25189, avg=380.20, stdev=1519.74 clat (msec): min=277, max=1960, avg=971.30, stdev=281.68 lat (msec): min=279, max=1960, avg=971.68, stdev=281.62 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 527], 10.00th=[ 609], 20.00th=[ 726], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1636], 99.50th=[ 1821], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 102, per=25.05%, avg=44.58, stdev=21.18, samples=231 iops : min= 1, max= 25, avg=10.62, stdev= 5.34, samples=231 lat (msec) : 50=0.08%, 100=0.20%, 250=0.90%, 500=33.48%, 750=23.28% lat (msec) : 1000=20.51%, 2000=21.56% cpu : usr=0.12%, sys=0.71%, ctx=4018, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145306: Sat Aug 22 00:07:53 2026 read: IOPS=10, BW=42.7KiB/s (43.7kB/s)(4984KiB/116853msec) slat (usec): min=2678, max=79891, avg=16164.66, stdev=9037.90 clat (msec): min=30, max=1067, avg=461.62, stdev=135.70 lat (msec): min=37, max=1076, avg=477.79, stdev=137.43 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 447], 60.00th=[ 481], | 70.00th=[ 514], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 902], 99.50th=[ 936], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7, max= 95, per=25.08%, avg=42.88, stdev=19.64, samples=228 iops : min= 1, max= 23, avg=10.19, stdev= 4.91, samples=228 write: IOPS=11, BW=44.0KiB/s (46.1kB/s)(5256KiB/116853msec); 0 zone resets slat (usec): min=31, max=47556, avg=329.35, stdev=1663.51 clat (msec): min=203, max=1895, avg=966.72, stdev=278.44 lat (msec): min=204, max=1895, avg=967.05, stdev=278.39 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 535], 10.00th=[ 609], 20.00th=[ 735], | 30.00th=[ 818], 40.00th=[ 869], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1720], 99.50th=[ 1821], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 104, per=25.13%, avg=44.73, stdev=20.50, samples=229 iops : min= 1, max= 26, avg=10.66, stdev= 5.14, samples=229 lat (msec) : 50=0.12%, 100=0.23%, 250=0.94%, 500=32.46%, 750=24.22% lat (msec) : 1000=20.35%, 2000=21.68% cpu : usr=0.11%, sys=0.73%, ctx=4369, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145307: Sat Aug 22 00:07:53 2026 read: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5060KiB/116440msec) slat (usec): min=2826, max=81468, avg=15738.62, stdev=8806.56 clat (msec): min=13, max=1080, avg=460.84, stdev=135.28 lat (msec): min=19, max=1089, avg=476.59, stdev=136.84 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 447], 60.00th=[ 481], | 70.00th=[ 510], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 693], | 99.00th=[ 877], 99.50th=[ 953], 99.90th=[ 1070], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 119, per=25.13%, avg=42.97, stdev=19.77, samples=231 iops : min= 1, max= 29, avg=10.22, stdev= 5.00, samples=231 write: IOPS=11, BW=44.5KiB/s (45.6kB/s)(5180KiB/116440msec); 0 zone resets slat (usec): min=25, max=31039, avg=323.07, stdev=1215.75 clat (msec): min=265, max=1966, avg=970.27, stdev=282.23 lat (msec): min=266, max=1966, avg=970.59, stdev=282.21 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 531], 10.00th=[ 600], 20.00th=[ 735], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1754], 99.50th=[ 1871], 99.90th=[ 1955], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 119, per=24.76%, avg=44.07, stdev=19.55, samples=229 iops : min= 1, max= 29, avg=10.50, stdev= 4.89, samples=229 lat (msec) : 20=0.04%, 50=0.20%, 100=0.16%, 250=0.62%, 500=33.12% lat (msec) : 750=24.57%, 1000=19.18%, 2000=22.11% cpu : usr=0.12%, sys=0.70%, ctx=4317, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172KiB/s (176kB/s), 42.6KiB/s-43.5KiB/s (43.6kB/s-44.5kB/s), io=19.6MiB (20.5MB), run=116440-116899msec WRITE: bw=179KiB/s (183kB/s), 44.5KiB/s-45.0KiB/s (45.5kB/s-46.1kB/s), io=20.4MiB (21.4MB), run=116440-116899msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145329: Sat Aug 22 00:08:03 2026 read: IOPS=1, BW=71.8MiB/s (75.3MB/s)(40.0MiB/557msec) slat (nsec): min=37672k, max=37672k, avg=37672344.00, stdev= 0.00 clat (nsec): min=501798k, max=501798k, avg=501798240.00, stdev= 0.00 lat (nsec): min=539478k, max=539478k, avg=539478412.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.52%, ctx=33, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=71.8MiB/s (75.3MB/s), 71.8MiB/s-71.8MiB/s (75.3MB/s-75.3MB/s), io=40.0MiB (41.9MB), run=557-557msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout