== sanity test 398c: run fio to test AIO ================= 12:54:28 (1788540868) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.453924 s, 92.4 MB/s osc.lustre-OST0000-osc-ffff9d7ec2c90000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179660: Fri Sep 4 12:55:08 2026 write: IOPS=85, BW=341KiB/s (350kB/s)(10.0MiB/29993msec); 0 zone resets slat (usec): min=22, max=14419, avg=231.89, stdev=790.35 clat (msec): min=47, max=330, avg=187.05, stdev=49.14 lat (msec): min=48, max=330, avg=187.29, stdev=49.13 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 118], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 182], 60.00th=[ 197], | 70.00th=[ 211], 80.00th=[ 232], 90.00th=[ 251], 95.00th=[ 275], | 99.00th=[ 313], 99.50th=[ 317], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 151, max= 637, per=24.95%, avg=340.37, stdev=87.23, samples=59 iops : min= 37, max= 159, avg=84.73, stdev=21.85, samples=59 lat (msec) : 50=0.08%, 100=2.34%, 250=86.56%, 500=11.02% cpu : usr=0.21%, sys=1.04%, ctx=1989, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179661: Fri Sep 4 12:55:08 2026 write: IOPS=85, BW=341KiB/s (349kB/s)(10.0MiB/30016msec); 0 zone resets slat (usec): min=22, max=19013, avg=244.83, stdev=930.99 clat (msec): min=16, max=351, avg=186.37, stdev=49.68 lat (msec): min=18, max=351, avg=186.62, stdev=49.66 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 121], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 182], 60.00th=[ 197], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 255], 95.00th=[ 275], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 233, max= 565, per=24.85%, avg=338.98, stdev=77.47, samples=59 iops : min= 58, max= 141, avg=84.37, stdev=19.39, samples=59 lat (msec) : 20=0.04%, 50=0.08%, 100=2.70%, 250=85.66%, 500=11.52% cpu : usr=0.23%, sys=0.90%, ctx=2120, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179662: Fri Sep 4 12:55:08 2026 write: IOPS=85, BW=341KiB/s (350kB/s)(10.0MiB/29991msec); 0 zone resets slat (usec): min=21, max=14221, avg=225.88, stdev=803.59 clat (msec): min=19, max=352, avg=186.50, stdev=50.57 lat (msec): min=19, max=352, avg=186.73, stdev=50.55 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 117], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 228], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 355] bw ( KiB/s): min= 183, max= 589, per=24.99%, avg=340.90, stdev=80.85, samples=59 iops : min= 45, max= 147, avg=84.85, stdev=20.25, samples=59 lat (msec) : 20=0.04%, 50=0.08%, 100=2.66%, 250=86.41%, 500=10.82% cpu : usr=0.21%, sys=0.94%, ctx=2097, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179663: Fri Sep 4 12:55:08 2026 write: IOPS=85, BW=341KiB/s (350kB/s)(10.0MiB/29989msec); 0 zone resets slat (usec): min=22, max=14146, avg=210.25, stdev=771.82 clat (msec): min=17, max=352, avg=186.54, stdev=50.41 lat (msec): min=18, max=352, avg=186.75, stdev=50.41 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 117], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 207], 80.00th=[ 226], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 347], 99.50th=[ 351], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 223, max= 557, per=25.01%, avg=341.12, stdev=81.70, samples=59 iops : min= 55, max= 139, avg=84.92, stdev=20.49, samples=59 lat (msec) : 20=0.04%, 50=0.12%, 100=2.70%, 250=86.52%, 500=10.62% cpu : usr=0.16%, sys=0.92%, ctx=1817, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1365KiB/s (1397kB/s), 341KiB/s-341KiB/s (349kB/s-350kB/s), io=40.0MiB (41.9MB), run=29989-30016msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179678: Fri Sep 4 12:55:54 2026 read: IOPS=36, BW=144KiB/s (148kB/s)(5048KiB/34991msec) slat (usec): min=3172, max=72576, avg=15685.21, stdev=9105.95 clat (msec): min=68, max=393, avg=207.38, stdev=59.45 lat (msec): min=86, max=413, avg=223.07, stdev=61.74 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 117], 10.00th=[ 133], 20.00th=[ 155], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 236], 80.00th=[ 257], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 355], 99.50th=[ 376], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 71, max= 254, per=25.12%, avg=143.70, stdev=38.85, samples=69 iops : min= 17, max= 63, avg=35.35, stdev= 9.76, samples=69 write: IOPS=37, BW=148KiB/s (152kB/s)(5192KiB/34991msec); 0 zone resets slat (usec): min=22, max=17629, avg=346.44, stdev=1217.48 clat (msec): min=37, max=372, avg=211.32, stdev=56.61 lat (msec): min=38, max=372, avg=211.68, stdev=56.60 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 121], 10.00th=[ 138], 20.00th=[ 163], | 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 351], 99.50th=[ 355], 99.90th=[ 363], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 55, max= 287, per=24.91%, avg=148.23, stdev=45.96, samples=69 iops : min= 13, max= 71, avg=36.48, stdev=11.53, samples=69 lat (msec) : 50=0.08%, 100=1.56%, 250=74.92%, 500=23.44% cpu : usr=0.27%, sys=2.01%, ctx=4377, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179679: Fri Sep 4 12:55:54 2026 read: IOPS=35, BW=142KiB/s (146kB/s)(4976KiB/34948msec) slat (usec): min=32, max=74051, avg=15983.15, stdev=8880.46 clat (msec): min=84, max=390, avg=207.20, stdev=55.41 lat (msec): min=91, max=415, avg=223.20, stdev=57.61 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 125], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 190], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 380], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 80, max= 270, per=24.76%, avg=141.65, stdev=39.33, samples=69 iops : min= 20, max= 67, avg=34.86, stdev= 9.87, samples=69 write: IOPS=37, BW=151KiB/s (154kB/s)(5264KiB/34948msec); 0 zone resets slat (usec): min=23, max=18763, avg=314.83, stdev=1032.38 clat (msec): min=50, max=380, avg=212.71, stdev=57.27 lat (msec): min=51, max=380, avg=213.02, stdev=57.25 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 123], 10.00th=[ 134], 20.00th=[ 163], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 228], | 70.00th=[ 247], 80.00th=[ 266], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 372], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 62, max= 271, per=25.29%, avg=150.46, stdev=41.13, samples=69 iops : min= 15, max= 67, avg=37.06, stdev=10.30, samples=69 lat (msec) : 100=1.09%, 250=74.02%, 500=24.88% cpu : usr=0.32%, sys=1.89%, ctx=4160, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179680: Fri Sep 4 12:55:54 2026 read: IOPS=35, BW=142KiB/s (146kB/s)(4984KiB/35056msec) slat (usec): min=2913, max=89609, avg=15694.42, stdev=9438.70 clat (msec): min=25, max=403, avg=208.08, stdev=58.21 lat (msec): min=35, max=428, avg=223.79, stdev=60.42 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 120], 10.00th=[ 136], 20.00th=[ 157], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 355], 99.50th=[ 368], 99.90th=[ 388], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 53, max= 294, per=24.84%, avg=142.07, stdev=39.78, samples=69 iops : min= 13, max= 73, avg=34.96, stdev= 9.96, samples=69 write: IOPS=37, BW=150KiB/s (154kB/s)(5256KiB/35056msec); 0 zone resets slat (usec): min=32, max=13108, avg=325.27, stdev=978.13 clat (msec): min=73, max=377, avg=211.71, stdev=55.37 lat (msec): min=73, max=377, avg=212.04, stdev=55.37 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 167], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 355], 99.50th=[ 359], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 55, max= 243, per=25.05%, avg=149.07, stdev=44.06, samples=69 iops : min= 13, max= 60, avg=36.71, stdev=11.06, samples=69 lat (msec) : 50=0.12%, 100=0.82%, 250=75.70%, 500=23.36% cpu : usr=0.31%, sys=1.91%, ctx=4227, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179681: Fri Sep 4 12:55:54 2026 read: IOPS=36, BW=144KiB/s (148kB/s)(5060KiB/35057msec) slat (usec): min=3160, max=69961, avg=15472.29, stdev=8932.22 clat (msec): min=75, max=440, avg=207.94, stdev=58.01 lat (msec): min=86, max=458, avg=223.42, stdev=59.98 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 118], 10.00th=[ 133], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 220], | 70.00th=[ 236], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 435], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 71, max= 302, per=25.14%, avg=143.83, stdev=40.93, samples=69 iops : min= 17, max= 75, avg=35.43, stdev=10.25, samples=69 write: IOPS=36, BW=148KiB/s (151kB/s)(5180KiB/35057msec); 0 zone resets slat (usec): min=27, max=11597, avg=276.06, stdev=775.96 clat (msec): min=38, max=423, avg=211.92, stdev=54.23 lat (msec): min=38, max=423, avg=212.20, stdev=54.24 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 127], 10.00th=[ 142], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 226], | 70.00th=[ 243], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 342], 99.50th=[ 355], 99.90th=[ 393], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 71, max= 271, per=24.74%, avg=147.22, stdev=40.56, samples=69 iops : min= 17, max= 67, avg=36.25, stdev=10.13, samples=69 lat (msec) : 50=0.04%, 100=1.13%, 250=74.45%, 500=24.38% cpu : usr=0.34%, sys=2.00%, ctx=4166, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=572KiB/s (586kB/s), 142KiB/s-144KiB/s (146kB/s-148kB/s), io=19.6MiB (20.5MB), run=34948-35057msec WRITE: bw=596KiB/s (610kB/s), 148KiB/s-151KiB/s (151kB/s-154kB/s), io=20.4MiB (21.4MB), run=34948-35057msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179690: Fri Sep 4 12:56:08 2026 read: IOPS=1, BW=57.3MiB/s (60.1MB/s)(40.0MiB/698msec) slat (nsec): min=25678k, max=25678k, avg=25678112.00, stdev= 0.00 clat (nsec): min=649967k, max=649967k, avg=649967084.00, stdev= 0.00 lat (nsec): min=675652k, max=675652k, avg=675651656.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 651], 5.00th=[ 651], 10.00th=[ 651], 20.00th=[ 651], | 30.00th=[ 651], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 651], | 70.00th=[ 651], 80.00th=[ 651], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.87%, ctx=9, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=57.3MiB/s (60.1MB/s), 57.3MiB/s-57.3MiB/s (60.1MB/s-60.1MB/s), io=40.0MiB (41.9MB), run=698-698msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout