== sanity test 398c: run fio to test AIO ================= 06:29:25 (1787912965) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.576405 s, 72.8 MB/s osc.lustre-OST0000-osc-ffff8a5f89dba000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=178990: Fri Aug 28 06:30:02 2026 write: IOPS=100, BW=401KiB/s (411kB/s)(10.0MiB/25539msec); 0 zone resets slat (usec): min=23, max=11927, avg=178.95, stdev=579.25 clat (msec): min=24, max=293, avg=159.32, stdev=36.07 lat (msec): min=24, max=293, avg=159.50, stdev=36.06 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 112], 10.00th=[ 123], 20.00th=[ 134], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 174], 80.00th=[ 188], 90.00th=[ 209], 95.00th=[ 226], | 99.00th=[ 271], 99.50th=[ 288], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 253, max= 630, per=24.95%, avg=399.72, stdev=77.65, samples=50 iops : min= 63, max= 157, avg=99.46, stdev=19.44, samples=50 lat (msec) : 50=0.23%, 100=1.99%, 250=96.29%, 500=1.48% cpu : usr=0.20%, sys=0.99%, ctx=2819, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178991: Fri Aug 28 06:30:02 2026 write: IOPS=100, BW=401KiB/s (410kB/s)(10.0MiB/25567msec); 0 zone resets slat (usec): min=22, max=31780, avg=193.07, stdev=956.21 clat (msec): min=36, max=313, avg=159.54, stdev=34.90 lat (msec): min=36, max=313, avg=159.73, stdev=34.89 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 111], 10.00th=[ 124], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 148], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 188], 90.00th=[ 207], 95.00th=[ 226], | 99.00th=[ 266], 99.50th=[ 275], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 313] bw ( KiB/s): min= 253, max= 614, per=24.87%, avg=398.42, stdev=75.05, samples=50 iops : min= 63, max= 153, avg=99.12, stdev=18.79, samples=50 lat (msec) : 50=0.08%, 100=1.84%, 250=96.17%, 500=1.91% cpu : usr=0.17%, sys=1.00%, ctx=2514, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178992: Fri Aug 28 06:30:02 2026 write: IOPS=100, BW=401KiB/s (410kB/s)(10.0MiB/25558msec); 0 zone resets slat (usec): min=22, max=21228, avg=177.12, stdev=854.64 clat (msec): min=80, max=312, avg=159.49, stdev=35.57 lat (msec): min=81, max=312, avg=159.67, stdev=35.59 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 134], | 30.00th=[ 140], 40.00th=[ 146], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 188], 90.00th=[ 205], 95.00th=[ 230], | 99.00th=[ 268], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 238, max= 638, per=24.84%, avg=397.94, stdev=77.18, samples=50 iops : min= 59, max= 159, avg=99.00, stdev=19.32, samples=50 lat (msec) : 100=2.70%, 250=95.43%, 500=1.88% cpu : usr=0.13%, sys=1.00%, ctx=2684, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178993: Fri Aug 28 06:30:02 2026 write: IOPS=100, BW=401KiB/s (410kB/s)(10.0MiB/25564msec); 0 zone resets slat (usec): min=22, max=12040, avg=183.16, stdev=660.97 clat (msec): min=13, max=294, avg=159.50, stdev=35.40 lat (msec): min=15, max=299, avg=159.68, stdev=35.38 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 108], 10.00th=[ 122], 20.00th=[ 134], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 155], 60.00th=[ 163], | 70.00th=[ 174], 80.00th=[ 188], 90.00th=[ 205], 95.00th=[ 220], | 99.00th=[ 275], 99.50th=[ 279], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 296] bw ( KiB/s): min= 253, max= 646, per=24.84%, avg=398.00, stdev=78.65, samples=50 iops : min= 63, max= 161, avg=99.02, stdev=19.69, samples=50 lat (msec) : 20=0.04%, 100=2.15%, 250=95.70%, 500=2.11% cpu : usr=0.13%, sys=0.94%, ctx=2382, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1602KiB/s (1641kB/s), 401KiB/s-401KiB/s (410kB/s-411kB/s), io=40.0MiB (41.9MB), run=25539-25567msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179007: Fri Aug 28 06:30:45 2026 read: IOPS=38, BW=155KiB/s (159kB/s)(5048KiB/32583msec) slat (usec): min=2828, max=59151, avg=14758.49, stdev=8249.60 clat (msec): min=80, max=359, avg=192.40, stdev=45.44 lat (msec): min=92, max=397, avg=207.17, stdev=47.14 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 317], 99.50th=[ 334], 99.90th=[ 351], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 71, max= 246, per=25.08%, avg=154.22, stdev=34.48, samples=64 iops : min= 17, max= 61, avg=38.00, stdev= 8.68, samples=64 write: IOPS=39, BW=159KiB/s (163kB/s)(5192KiB/32583msec); 0 zone resets slat (usec): min=30, max=23003, avg=227.40, stdev=997.70 clat (msec): min=92, max=370, avg=197.13, stdev=44.52 lat (msec): min=92, max=370, avg=197.36, stdev=44.56 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 234], 90.00th=[ 264], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 321], 99.90th=[ 330], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 63, max= 239, per=24.80%, avg=158.94, stdev=41.94, samples=64 iops : min= 15, max= 59, avg=39.16, stdev=10.46, samples=64 lat (msec) : 100=0.43%, 250=87.46%, 500=12.11% cpu : usr=0.26%, sys=1.64%, ctx=4050, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179008: Fri Aug 28 06:30:45 2026 read: IOPS=38, BW=153KiB/s (157kB/s)(4976KiB/32541msec) slat (usec): min=2375, max=66812, avg=14561.23, stdev=8145.61 clat (msec): min=78, max=380, avg=191.09, stdev=44.61 lat (msec): min=98, max=391, avg=205.66, stdev=46.06 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 122], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 188], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 226], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 313], 99.50th=[ 330], 99.90th=[ 359], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 47, max= 262, per=24.73%, avg=152.09, stdev=41.97, samples=64 iops : min= 11, max= 65, avg=37.47, stdev=10.53, samples=64 write: IOPS=40, BW=162KiB/s (166kB/s)(5264KiB/32541msec); 0 zone resets slat (usec): min=30, max=11514, avg=194.80, stdev=628.62 clat (msec): min=89, max=383, avg=199.09, stdev=45.29 lat (msec): min=89, max=383, avg=199.29, stdev=45.29 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 317], 99.50th=[ 355], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 103, max= 271, per=25.25%, avg=161.84, stdev=34.50, samples=64 iops : min= 25, max= 67, avg=39.91, stdev= 8.60, samples=64 lat (msec) : 100=0.55%, 250=87.77%, 500=11.68% cpu : usr=0.24%, sys=1.44%, ctx=4241, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179009: Fri Aug 28 06:30:45 2026 read: IOPS=38, BW=153KiB/s (157kB/s)(4984KiB/32584msec) slat (usec): min=55, max=65017, avg=14823.54, stdev=8455.28 clat (msec): min=34, max=397, avg=191.92, stdev=48.02 lat (msec): min=35, max=406, avg=206.75, stdev=49.81 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 123], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 230], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 355], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 71, max= 246, per=24.84%, avg=152.78, stdev=35.91, samples=64 iops : min= 17, max= 61, avg=37.64, stdev= 9.00, samples=64 write: IOPS=40, BW=161KiB/s (165kB/s)(5256KiB/32584msec); 0 zone resets slat (usec): min=23, max=10900, avg=248.40, stdev=815.74 clat (msec): min=45, max=389, avg=199.57, stdev=47.82 lat (msec): min=50, max=389, avg=199.82, stdev=47.83 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 133], 10.00th=[ 146], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 266], 95.00th=[ 292], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 363], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 87, max= 255, per=25.01%, avg=160.30, stdev=39.57, samples=64 iops : min= 21, max= 63, avg=39.50, stdev= 9.89, samples=64 lat (msec) : 50=0.23%, 100=0.90%, 250=85.35%, 500=13.52% cpu : usr=0.28%, sys=1.84%, ctx=4191, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179010: Fri Aug 28 06:30:45 2026 read: IOPS=38, BW=155KiB/s (159kB/s)(5060KiB/32588msec) slat (msec): min=2, max=203, avg=14.62, stdev= 9.73 clat (msec): min=33, max=402, avg=192.24, stdev=47.07 lat (msec): min=53, max=451, avg=206.86, stdev=48.86 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 125], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 255], 95.00th=[ 275], | 99.00th=[ 321], 99.50th=[ 342], 99.90th=[ 388], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 63, max= 239, per=25.20%, avg=155.00, stdev=36.59, samples=64 iops : min= 15, max= 59, avg=38.25, stdev= 9.16, samples=64 write: IOPS=39, BW=159KiB/s (163kB/s)(5180KiB/32588msec); 0 zone resets slat (usec): min=29, max=17291, avg=231.68, stdev=855.06 clat (msec): min=53, max=466, avg=197.26, stdev=46.44 lat (msec): min=53, max=466, avg=197.50, stdev=46.45 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 129], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 393], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 64, max= 295, per=24.74%, avg=158.58, stdev=42.46, samples=64 iops : min= 16, max= 73, avg=39.13, stdev=10.63, samples=64 lat (msec) : 50=0.12%, 100=0.86%, 250=87.23%, 500=11.80% cpu : usr=0.26%, sys=1.59%, ctx=4265, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=616KiB/s (631kB/s), 153KiB/s-155KiB/s (157kB/s-159kB/s), io=19.6MiB (20.5MB), run=32541-32588msec WRITE: bw=641KiB/s (656kB/s), 159KiB/s-162KiB/s (163kB/s-166kB/s), io=20.4MiB (21.4MB), run=32541-32588msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179019: Fri Aug 28 06:30:56 2026 read: IOPS=1, BW=74.3MiB/s (77.0MB/s)(40.0MiB/538msec) slat (nsec): min=21897k, max=21897k, avg=21896531.00, stdev= 0.00 clat (nsec): min=499812k, max=499812k, avg=499811761.00, stdev= 0.00 lat (nsec): min=521712k, max=521712k, avg=521712330.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.05%, ctx=9, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.3MiB/s (77.0MB/s), 74.3MiB/s-74.3MiB/s (77.0MB/s-77.0MB/s), io=40.0MiB (41.9MB), run=538-538msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout