== sanity test 398c: run fio to test AIO ================= 06:27:21 (1788258441) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.54539 s, 76.9 MB/s osc.lustre-OST0000-osc-ffff953c030e2000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180450: Tue Sep 1 06:27:59 2026 write: IOPS=93, BW=373KiB/s (381kB/s)(10.0MiB/27487msec); 0 zone resets slat (usec): min=25, max=19494, avg=203.21, stdev=755.65 clat (msec): min=31, max=381, avg=171.43, stdev=44.83 lat (msec): min=31, max=381, avg=171.63, stdev=44.81 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 99], 10.00th=[ 117], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 230], 95.00th=[ 251], | 99.00th=[ 275], 99.50th=[ 292], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 380] bw ( KiB/s): min= 254, max= 582, per=25.21%, avg=373.17, stdev=83.27, samples=54 iops : min= 63, max= 145, avg=92.93, stdev=20.81, samples=54 lat (msec) : 50=0.16%, 100=5.08%, 250=89.69%, 500=5.08% cpu : usr=0.15%, sys=0.89%, ctx=2680, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180451: Tue Sep 1 06:27:59 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27662msec); 0 zone resets slat (usec): min=28, max=12653, avg=149.74, stdev=493.82 clat (msec): min=16, max=298, avg=171.92, stdev=42.96 lat (msec): min=16, max=298, avg=172.08, stdev=42.96 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 108], 10.00th=[ 118], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 230], 95.00th=[ 247], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 191, max= 640, per=24.88%, avg=368.30, stdev=89.60, samples=54 iops : min= 47, max= 160, avg=91.70, stdev=22.45, samples=54 lat (msec) : 20=0.12%, 50=0.12%, 100=3.91%, 250=91.52%, 500=4.34% cpu : usr=0.11%, sys=0.78%, ctx=2155, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180452: Tue Sep 1 06:27:59 2026 write: IOPS=92, BW=371KiB/s (380kB/s)(10.0MiB/27592msec); 0 zone resets slat (usec): min=28, max=11001, avg=183.45, stdev=634.52 clat (msec): min=11, max=292, avg=171.32, stdev=43.07 lat (msec): min=11, max=292, avg=171.51, stdev=43.07 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 102], 10.00th=[ 117], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 292] bw ( KiB/s): min= 254, max= 637, per=24.89%, avg=368.31, stdev=80.71, samples=54 iops : min= 63, max= 159, avg=91.72, stdev=20.19, samples=54 lat (msec) : 20=0.08%, 50=0.39%, 100=4.34%, 250=90.70%, 500=4.49% cpu : usr=0.18%, sys=0.84%, ctx=2601, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180453: Tue Sep 1 06:27:59 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27658msec); 0 zone resets slat (usec): min=26, max=15122, avg=193.91, stdev=748.25 clat (msec): min=32, max=291, avg=171.56, stdev=42.77 lat (msec): min=32, max=291, avg=171.76, stdev=42.75 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 106], 10.00th=[ 118], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 180], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 234], 95.00th=[ 249], | 99.00th=[ 268], 99.50th=[ 271], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 292] bw ( KiB/s): min= 254, max= 517, per=24.86%, avg=367.87, stdev=74.66, samples=54 iops : min= 63, max= 129, avg=91.59, stdev=18.71, samples=54 lat (msec) : 50=0.39%, 100=3.87%, 250=92.03%, 500=3.71% cpu : usr=0.17%, sys=0.82%, ctx=2523, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1481KiB/s (1516kB/s), 370KiB/s-373KiB/s (379kB/s-381kB/s), io=40.0MiB (41.9MB), run=27487-27662msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180467: Tue Sep 1 06:28:39 2026 read: IOPS=40, BW=162KiB/s (166kB/s)(5048KiB/31083msec) slat (usec): min=2523, max=70950, avg=13648.74, stdev=7512.39 clat (msec): min=23, max=384, avg=183.21, stdev=43.05 lat (msec): min=29, max=411, avg=196.86, stdev=44.65 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 125], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 243], 95.00th=[ 262], | 99.00th=[ 305], 99.50th=[ 334], 99.90th=[ 368], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 73, max= 232, per=25.16%, avg=162.28, stdev=37.38, samples=61 iops : min= 18, max= 58, avg=40.08, stdev= 9.39, samples=61 write: IOPS=41, BW=167KiB/s (171kB/s)(5192KiB/31083msec); 0 zone resets slat (usec): min=33, max=13484, avg=240.89, stdev=810.66 clat (msec): min=96, max=341, avg=188.25, stdev=41.24 lat (msec): min=97, max=341, avg=188.49, stdev=41.27 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 131], 10.00th=[ 142], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 247], 95.00th=[ 268], | 99.00th=[ 305], 99.50th=[ 326], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 59, max= 247, per=24.89%, avg=167.03, stdev=42.21, samples=61 iops : min= 14, max= 61, avg=41.26, stdev=10.56, samples=61 lat (msec) : 50=0.04%, 100=0.39%, 250=91.09%, 500=8.48% cpu : usr=0.22%, sys=1.66%, ctx=4458, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180468: Tue Sep 1 06:28:39 2026 read: IOPS=40, BW=161KiB/s (165kB/s)(4976KiB/30943msec) slat (usec): min=58, max=59845, avg=13618.08, stdev=7576.05 clat (msec): min=15, max=384, avg=183.57, stdev=45.17 lat (msec): min=20, max=401, avg=197.20, stdev=46.42 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 243], 95.00th=[ 268], | 99.00th=[ 305], 99.50th=[ 330], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 95, max= 238, per=24.87%, avg=160.44, stdev=33.24, samples=61 iops : min= 23, max= 59, avg=39.61, stdev= 8.31, samples=61 write: IOPS=42, BW=170KiB/s (174kB/s)(5264KiB/30943msec); 0 zone resets slat (usec): min=31, max=11628, avg=189.95, stdev=605.67 clat (msec): min=30, max=384, avg=189.09, stdev=41.29 lat (msec): min=30, max=384, avg=189.28, stdev=41.28 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 131], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 220], 90.00th=[ 245], 95.00th=[ 266], | 99.00th=[ 309], 99.50th=[ 342], 99.90th=[ 372], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 95, max= 255, per=25.40%, avg=170.41, stdev=38.11, samples=61 iops : min= 23, max= 63, avg=42.11, stdev= 9.58, samples=61 lat (msec) : 20=0.12%, 50=0.12%, 100=0.31%, 250=91.45%, 500=8.01% cpu : usr=0.27%, sys=1.66%, ctx=4483, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180469: Tue Sep 1 06:28:39 2026 read: IOPS=40, BW=160KiB/s (164kB/s)(4984KiB/31058msec) slat (usec): min=2466, max=68804, avg=13496.26, stdev=7374.74 clat (msec): min=29, max=370, avg=182.65, stdev=42.56 lat (msec): min=36, max=397, avg=196.15, stdev=44.11 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 123], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 239], 95.00th=[ 266], | 99.00th=[ 300], 99.50th=[ 321], 99.90th=[ 368], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 81, max= 262, per=24.90%, avg=160.57, stdev=37.78, samples=61 iops : min= 20, max= 65, avg=39.66, stdev= 9.54, samples=61 write: IOPS=42, BW=169KiB/s (173kB/s)(5256KiB/31058msec); 0 zone resets slat (usec): min=37, max=10719, avg=184.22, stdev=489.62 clat (msec): min=26, max=351, avg=189.23, stdev=43.15 lat (msec): min=26, max=351, avg=189.42, stdev=43.14 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 130], 10.00th=[ 140], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 186], 60.00th=[ 197], | 70.00th=[ 207], 80.00th=[ 220], 90.00th=[ 247], 95.00th=[ 268], | 99.00th=[ 317], 99.50th=[ 338], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 88, max= 262, per=25.16%, avg=168.84, stdev=38.64, samples=61 iops : min= 22, max= 65, avg=41.72, stdev= 9.57, samples=61 lat (msec) : 50=0.08%, 100=0.47%, 250=91.37%, 500=8.09% cpu : usr=0.21%, sys=1.50%, ctx=4438, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180470: Tue Sep 1 06:28:39 2026 read: IOPS=40, BW=163KiB/s (167kB/s)(5060KiB/31092msec) slat (usec): min=2636, max=80749, avg=13815.86, stdev=8067.69 clat (msec): min=53, max=395, avg=182.96, stdev=42.52 lat (msec): min=59, max=416, avg=196.78, stdev=43.76 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 124], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 239], 95.00th=[ 259], | 99.00th=[ 292], 99.50th=[ 313], 99.90th=[ 388], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 71, max= 237, per=25.12%, avg=162.00, stdev=32.31, samples=61 iops : min= 17, max= 59, avg=40.02, stdev= 8.09, samples=61 write: IOPS=41, BW=167KiB/s (171kB/s)(5180KiB/31092msec); 0 zone resets slat (usec): min=30, max=26962, avg=204.40, stdev=912.67 clat (msec): min=59, max=332, avg=189.25, stdev=40.04 lat (msec): min=59, max=332, avg=189.45, stdev=40.01 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 130], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 197], | 70.00th=[ 207], 80.00th=[ 222], 90.00th=[ 243], 95.00th=[ 262], | 99.00th=[ 292], 99.50th=[ 296], 99.90th=[ 321], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 55, max= 303, per=24.69%, avg=165.69, stdev=39.57, samples=61 iops : min= 13, max= 75, avg=40.93, stdev= 9.89, samples=61 lat (msec) : 100=0.86%, 250=91.52%, 500=7.62% cpu : usr=0.23%, sys=1.68%, ctx=4497, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=645KiB/s (661kB/s), 160KiB/s-163KiB/s (164kB/s-167kB/s), io=19.6MiB (20.5MB), run=30943-31092msec WRITE: bw=672KiB/s (688kB/s), 167KiB/s-170KiB/s (171kB/s-174kB/s), io=20.4MiB (21.4MB), run=30943-31092msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180479: Tue Sep 1 06:28:51 2026 read: IOPS=1, BW=65.0MiB/s (68.2MB/s)(40.0MiB/615msec) slat (nsec): min=22632k, max=22632k, avg=22631719.00, stdev= 0.00 clat (nsec): min=576747k, max=576747k, avg=576746651.00, stdev= 0.00 lat (nsec): min=599421k, max=599421k, avg=599421295.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.95%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.0MiB/s (68.2MB/s), 65.0MiB/s-65.0MiB/s (68.2MB/s-68.2MB/s), io=40.0MiB (41.9MB), run=615-615msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout