== sanity test 398c: run fio to test AIO ================= 03:32:07 (1775201527) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.598949 s, 70.0 MB/s osc.lustre-OST0000-osc-ffff89dac8cba000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183653: Fri Apr 3 03:32:50 2026 write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31151msec); 0 zone resets slat (usec): min=52, max=16269, avg=468.53, stdev=1105.14 clat (msec): min=30, max=346, avg=194.08, stdev=40.01 lat (msec): min=30, max=346, avg=194.55, stdev=40.01 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 142], 10.00th=[ 148], 20.00th=[ 161], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 226], 90.00th=[ 247], 95.00th=[ 271], | 99.00th=[ 300], 99.50th=[ 305], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 347] bw ( KiB/s): min= 208, max= 429, per=25.05%, avg=328.18, stdev=56.92, samples=61 iops : min= 52, max= 107, avg=81.64, stdev=14.22, samples=61 lat (msec) : 50=0.35%, 100=0.16%, 250=90.74%, 500=8.75% cpu : usr=0.20%, sys=1.54%, ctx=2779, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183654: Fri Apr 3 03:32:50 2026 write: IOPS=81, BW=328KiB/s (336kB/s)(10.0MiB/31244msec); 0 zone resets slat (usec): min=49, max=17779, avg=460.04, stdev=1133.48 clat (msec): min=107, max=315, avg=194.58, stdev=39.14 lat (msec): min=107, max=315, avg=195.05, stdev=39.13 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 140], 10.00th=[ 148], 20.00th=[ 161], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 226], 90.00th=[ 247], 95.00th=[ 275], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 317] bw ( KiB/s): min= 183, max= 512, per=24.91%, avg=326.34, stdev=61.71, samples=62 iops : min= 45, max= 128, avg=81.15, stdev=15.45, samples=62 lat (msec) : 250=91.29%, 500=8.71% cpu : usr=0.23%, sys=1.44%, ctx=2629, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183655: Fri Apr 3 03:32:50 2026 write: IOPS=81, BW=328KiB/s (336kB/s)(10.0MiB/31237msec); 0 zone resets slat (usec): min=51, max=16040, avg=418.06, stdev=974.03 clat (msec): min=35, max=318, avg=194.69, stdev=39.37 lat (msec): min=37, max=318, avg=195.11, stdev=39.35 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 140], 10.00th=[ 148], 20.00th=[ 161], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 226], 90.00th=[ 247], 95.00th=[ 271], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 197, max= 447, per=24.89%, avg=326.08, stdev=61.25, samples=62 iops : min= 49, max= 111, avg=81.10, stdev=15.30, samples=62 lat (msec) : 50=0.08%, 250=91.05%, 500=8.87% cpu : usr=0.22%, sys=1.37%, ctx=2379, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183656: Fri Apr 3 03:32:50 2026 write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31136msec); 0 zone resets slat (usec): min=50, max=15415, avg=441.10, stdev=988.49 clat (msec): min=26, max=367, avg=194.04, stdev=39.18 lat (msec): min=26, max=368, avg=194.49, stdev=39.17 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 142], 10.00th=[ 148], 20.00th=[ 161], | 30.00th=[ 171], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 199], | 70.00th=[ 213], 80.00th=[ 224], 90.00th=[ 249], 95.00th=[ 266], | 99.00th=[ 300], 99.50th=[ 305], 99.90th=[ 317], 99.95th=[ 334], | 99.99th=[ 368] bw ( KiB/s): min= 205, max= 453, per=25.07%, avg=328.46, stdev=57.74, samples=61 iops : min= 51, max= 113, avg=81.69, stdev=14.40, samples=61 lat (msec) : 50=0.20%, 250=89.92%, 500=9.88% cpu : usr=0.20%, sys=1.57%, ctx=2864, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1311KiB/s (1342kB/s), 328KiB/s-329KiB/s (336kB/s-337kB/s), io=40.0MiB (41.9MB), run=31136-31244msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183672: Fri Apr 3 03:33:35 2026 read: IOPS=35, BW=142KiB/s (145kB/s)(5048KiB/35633msec) slat (usec): min=3766, max=61885, avg=15784.27, stdev=8058.08 clat (msec): min=17, max=386, avg=211.22, stdev=46.08 lat (msec): min=32, max=405, avg=227.02, stdev=47.66 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 220], | 70.00th=[ 232], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 288], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 359], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 95, max= 236, per=24.96%, avg=140.51, stdev=29.88, samples=70 iops : min= 23, max= 59, avg=34.64, stdev= 7.59, samples=70 write: IOPS=36, BW=146KiB/s (149kB/s)(5192KiB/35633msec); 0 zone resets slat (usec): min=54, max=18222, avg=439.53, stdev=1062.37 clat (msec): min=74, max=386, avg=215.43, stdev=44.49 lat (msec): min=77, max=386, avg=215.88, stdev=44.51 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 148], 10.00th=[ 163], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 213], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 251], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 372], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 63, max= 247, per=24.77%, avg=145.14, stdev=38.33, samples=70 iops : min= 15, max= 61, avg=35.77, stdev= 9.60, samples=70 lat (msec) : 20=0.04%, 100=0.20%, 250=79.53%, 500=20.23% cpu : usr=0.32%, sys=2.13%, ctx=4561, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183673: Fri Apr 3 03:33:35 2026 read: IOPS=35, BW=140KiB/s (143kB/s)(4976KiB/35540msec) slat (usec): min=2897, max=55596, avg=15416.85, stdev=7700.06 clat (msec): min=33, max=475, avg=209.70, stdev=48.20 lat (msec): min=49, max=497, avg=225.13, stdev=49.94 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 136], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 268], 95.00th=[ 296], | 99.00th=[ 355], 99.50th=[ 388], 99.90th=[ 439], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 71, max= 212, per=24.78%, avg=139.49, stdev=30.88, samples=70 iops : min= 17, max= 53, avg=34.37, stdev= 7.83, samples=70 write: IOPS=37, BW=148KiB/s (152kB/s)(5264KiB/35540msec); 0 zone resets slat (usec): min=55, max=21445, avg=487.43, stdev=1309.26 clat (msec): min=100, max=461, avg=217.33, stdev=44.28 lat (msec): min=101, max=461, avg=217.82, stdev=44.26 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 153], 10.00th=[ 165], 20.00th=[ 184], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 213], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 359], 99.50th=[ 376], 99.90th=[ 414], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 71, max= 247, per=25.27%, avg=148.07, stdev=35.50, samples=70 iops : min= 17, max= 61, avg=36.50, stdev= 8.89, samples=70 lat (msec) : 50=0.04%, 100=0.16%, 250=81.25%, 500=18.55% cpu : usr=0.26%, sys=2.23%, ctx=4601, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183674: Fri Apr 3 03:33:35 2026 read: IOPS=34, BW=140KiB/s (143kB/s)(4984KiB/35603msec) slat (usec): min=4075, max=58537, avg=15836.89, stdev=7994.11 clat (msec): min=83, max=421, avg=209.05, stdev=47.51 lat (msec): min=99, max=471, avg=224.90, stdev=49.07 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 142], 10.00th=[ 153], 20.00th=[ 169], | 30.00th=[ 182], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 405], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 71, max= 224, per=24.88%, avg=140.07, stdev=34.86, samples=70 iops : min= 17, max= 56, avg=34.49, stdev= 8.80, samples=70 write: IOPS=36, BW=148KiB/s (151kB/s)(5256KiB/35603msec); 0 zone resets slat (usec): min=59, max=12846, avg=421.37, stdev=927.37 clat (msec): min=102, max=405, avg=217.37, stdev=43.96 lat (msec): min=102, max=405, avg=217.80, stdev=43.95 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 150], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 215], 60.00th=[ 226], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 380], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 87, max= 207, per=25.10%, avg=147.07, stdev=28.84, samples=70 iops : min= 21, max= 51, avg=36.27, stdev= 7.25, samples=70 lat (msec) : 100=0.08%, 250=80.66%, 500=19.26% cpu : usr=0.25%, sys=2.15%, ctx=4500, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183675: Fri Apr 3 03:33:35 2026 read: IOPS=35, BW=142KiB/s (146kB/s)(5060KiB/35543msec) slat (usec): min=78, max=57378, avg=15774.46, stdev=8194.17 clat (msec): min=18, max=362, avg=210.50, stdev=47.66 lat (msec): min=18, max=383, avg=226.29, stdev=49.30 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 138], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 222], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 300], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 351], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 79, max= 199, per=25.25%, avg=142.16, stdev=28.03, samples=70 iops : min= 19, max= 49, avg=35.04, stdev= 7.03, samples=70 write: IOPS=36, BW=146KiB/s (149kB/s)(5180KiB/35543msec); 0 zone resets slat (usec): min=59, max=22316, avg=431.31, stdev=1155.63 clat (msec): min=48, max=461, avg=217.02, stdev=44.70 lat (msec): min=48, max=461, avg=217.45, stdev=44.78 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 153], 10.00th=[ 165], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 215], 60.00th=[ 226], | 70.00th=[ 236], 80.00th=[ 251], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 430], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 71, max= 244, per=24.73%, avg=144.94, stdev=34.56, samples=70 iops : min= 17, max= 61, avg=35.73, stdev= 8.77, samples=70 lat (msec) : 20=0.04%, 50=0.20%, 100=0.35%, 250=80.31%, 500=19.10% cpu : usr=0.31%, sys=2.29%, ctx=4506, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=563KiB/s (577kB/s), 140KiB/s-142KiB/s (143kB/s-146kB/s), io=19.6MiB (20.5MB), run=35540-35633msec WRITE: bw=586KiB/s (600kB/s), 146KiB/s-148KiB/s (149kB/s-152kB/s), io=20.4MiB (21.4MB), run=35540-35633msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183684: Fri Apr 3 03:33:46 2026 read: IOPS=1, BW=75.0MiB/s (78.7MB/s)(40.0MiB/533msec) slat (nsec): min=38567k, max=38567k, avg=38566795.00, stdev= 0.00 clat (nsec): min=462397k, max=462397k, avg=462396520.00, stdev= 0.00 lat (nsec): min=500969k, max=500969k, avg=500969058.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.63%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=75.0MiB/s (78.7MB/s), 75.0MiB/s-75.0MiB/s (78.7MB/s-78.7MB/s), io=40.0MiB (41.9MB), run=533-533msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout