== sanity test 398c: run fio to test AIO ================= 21:14:40 (1787102080) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.190913 s, 220 MB/s osc.lustre-OST0000-osc-ffff8edd456a8000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146450: Tue Aug 18 21:15:43 2026 write: IOPS=43, BW=176KiB/s (180kB/s)(10.0MiB/58327msec); 0 zone resets slat (usec): min=36, max=5569, avg=231.36, stdev=290.89 clat (msec): min=83, max=654, avg=363.68, stdev=85.79 lat (msec): min=83, max=654, avg=363.91, stdev=85.79 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 234], 10.00th=[ 262], 20.00th=[ 296], | 30.00th=[ 313], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 439], 90.00th=[ 485], 95.00th=[ 514], | 99.00th=[ 567], 99.50th=[ 617], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 95, max= 317, per=24.86%, avg=174.53, stdev=41.28, samples=116 iops : min= 23, max= 79, avg=43.35, stdev=10.41, samples=116 lat (msec) : 100=0.66%, 250=6.09%, 500=86.41%, 750=6.84% cpu : usr=0.07%, sys=0.71%, ctx=3185, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146451: Tue Aug 18 21:15:43 2026 write: IOPS=43, BW=176KiB/s (180kB/s)(10.0MiB/58325msec); 0 zone resets slat (usec): min=34, max=4728, avg=232.87, stdev=289.12 clat (msec): min=82, max=652, avg=363.64, stdev=87.72 lat (msec): min=82, max=652, avg=363.87, stdev=87.73 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 236], 10.00th=[ 262], 20.00th=[ 292], | 30.00th=[ 313], 40.00th=[ 338], 50.00th=[ 355], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 443], 90.00th=[ 485], 95.00th=[ 514], | 99.00th=[ 584], 99.50th=[ 617], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 95, max= 318, per=24.86%, avg=174.52, stdev=40.53, samples=116 iops : min= 23, max= 79, avg=43.35, stdev=10.18, samples=116 lat (msec) : 100=0.70%, 250=6.95%, 500=85.23%, 750=7.11% cpu : usr=0.08%, sys=0.70%, ctx=3178, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146452: Tue Aug 18 21:15:43 2026 write: IOPS=43, BW=176KiB/s (180kB/s)(10.0MiB/58330msec); 0 zone resets slat (usec): min=40, max=9406, avg=244.78, stdev=337.90 clat (msec): min=83, max=681, avg=363.87, stdev=86.53 lat (msec): min=83, max=681, avg=364.12, stdev=86.53 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 236], 10.00th=[ 262], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 376], | 70.00th=[ 405], 80.00th=[ 439], 90.00th=[ 481], 95.00th=[ 514], | 99.00th=[ 575], 99.50th=[ 600], 99.90th=[ 667], 99.95th=[ 676], | 99.99th=[ 684] bw ( KiB/s): min= 103, max= 316, per=24.86%, avg=174.52, stdev=39.61, samples=116 iops : min= 25, max= 79, avg=43.35, stdev= 9.98, samples=116 lat (msec) : 100=0.62%, 250=7.07%, 500=85.16%, 750=7.15% cpu : usr=0.08%, sys=0.71%, ctx=3322, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146453: Tue Aug 18 21:15:43 2026 write: IOPS=44, BW=176KiB/s (180kB/s)(10.0MiB/58143msec); 0 zone resets slat (usec): min=36, max=5945, avg=245.54, stdev=346.70 clat (msec): min=31, max=681, avg=362.95, stdev=89.35 lat (msec): min=31, max=682, avg=363.20, stdev=89.35 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 232], 10.00th=[ 262], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 376], | 70.00th=[ 405], 80.00th=[ 443], 90.00th=[ 481], 95.00th=[ 514], | 99.00th=[ 567], 99.50th=[ 625], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 684] bw ( KiB/s): min= 63, max= 375, per=25.00%, avg=175.52, stdev=44.66, samples=115 iops : min= 15, max= 93, avg=43.60, stdev=11.22, samples=115 lat (msec) : 50=0.23%, 100=1.17%, 250=6.52%, 500=85.27%, 750=6.80% cpu : usr=0.08%, sys=0.74%, ctx=3172, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=702KiB/s (719kB/s), 176KiB/s-176KiB/s (180kB/s-180kB/s), io=40.0MiB (41.9MB), run=58143-58330msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146472: Tue Aug 18 21:16:23 2026 read: IOPS=34, BW=136KiB/s (140kB/s)(5048KiB/37050msec) slat (usec): min=1553, max=16881, avg=4911.92, stdev=2184.40 clat (msec): min=3, max=271, avg=144.18, stdev=35.64 lat (msec): min=6, max=279, avg=149.10, stdev=36.09 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 174], 90.00th=[ 190], 95.00th=[ 209], | 99.00th=[ 241], 99.50th=[ 253], 99.90th=[ 259], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 47, max= 224, per=25.02%, avg=135.38, stdev=38.99, samples=73 iops : min= 11, max= 56, avg=33.63, stdev= 9.73, samples=73 write: IOPS=35, BW=140KiB/s (143kB/s)(5192KiB/37050msec); 0 zone resets slat (usec): min=51, max=4556, avg=226.08, stdev=235.77 clat (msec): min=126, max=618, avg=310.28, stdev=84.61 lat (msec): min=127, max=618, avg=310.51, stdev=84.61 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 190], 10.00th=[ 213], 20.00th=[ 241], | 30.00th=[ 262], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 372], 90.00th=[ 435], 95.00th=[ 481], | 99.00th=[ 550], 99.50th=[ 567], 99.90th=[ 600], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 64, max= 232, per=24.72%, avg=139.15, stdev=34.82, samples=73 iops : min= 16, max= 58, avg=34.58, stdev= 8.70, samples=73 lat (msec) : 4=0.04%, 10=0.08%, 20=0.04%, 50=0.12%, 100=3.71% lat (msec) : 250=57.30%, 500=36.84%, 750=1.88% cpu : usr=0.19%, sys=1.40%, ctx=4986, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146473: Tue Aug 18 21:16:23 2026 read: IOPS=33, BW=134KiB/s (138kB/s)(4976KiB/37014msec) slat (usec): min=1390, max=20038, avg=4858.10, stdev=2306.48 clat (msec): min=12, max=270, avg=140.91, stdev=37.23 lat (msec): min=18, max=275, avg=145.77, stdev=37.80 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 88], 10.00th=[ 99], 20.00th=[ 112], | 30.00th=[ 122], 40.00th=[ 130], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 171], 90.00th=[ 190], 95.00th=[ 207], | 99.00th=[ 232], 99.50th=[ 253], 99.90th=[ 268], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 48, max= 272, per=24.76%, avg=133.95, stdev=42.22, samples=73 iops : min= 12, max= 68, avg=33.27, stdev=10.54, samples=73 write: IOPS=35, BW=142KiB/s (146kB/s)(5264KiB/37014msec); 0 zone resets slat (usec): min=54, max=15991, avg=237.87, stdev=509.42 clat (msec): min=129, max=637, avg=310.98, stdev=83.81 lat (msec): min=129, max=638, avg=311.22, stdev=83.81 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 190], 10.00th=[ 215], 20.00th=[ 241], | 30.00th=[ 264], 40.00th=[ 284], 50.00th=[ 300], 60.00th=[ 321], | 70.00th=[ 342], 80.00th=[ 372], 90.00th=[ 426], 95.00th=[ 481], | 99.00th=[ 542], 99.50th=[ 567], 99.90th=[ 584], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 48, max= 208, per=25.21%, avg=141.95, stdev=34.31, samples=73 iops : min= 12, max= 52, avg=35.27, stdev= 8.55, samples=73 lat (msec) : 20=0.20%, 50=0.43%, 100=4.92%, 250=55.16%, 500=37.50% lat (msec) : 750=1.80% cpu : usr=0.14%, sys=1.43%, ctx=4988, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146474: Tue Aug 18 21:16:23 2026 read: IOPS=33, BW=135KiB/s (138kB/s)(4984KiB/37014msec) slat (usec): min=1404, max=14210, avg=4789.48, stdev=2158.18 clat (msec): min=3, max=262, avg=140.21, stdev=36.64 lat (msec): min=5, max=271, avg=145.01, stdev=37.10 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 89], 10.00th=[ 101], 20.00th=[ 112], | 30.00th=[ 121], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 157], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 203], | 99.00th=[ 243], 99.50th=[ 255], 99.90th=[ 259], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 48, max= 272, per=24.88%, avg=134.62, stdev=44.43, samples=73 iops : min= 12, max= 68, avg=33.44, stdev=11.11, samples=73 write: IOPS=35, BW=142KiB/s (145kB/s)(5256KiB/37014msec); 0 zone resets slat (usec): min=52, max=5192, avg=239.94, stdev=297.64 clat (msec): min=137, max=617, avg=311.71, stdev=84.19 lat (msec): min=137, max=619, avg=311.96, stdev=84.20 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 199], 10.00th=[ 218], 20.00th=[ 241], | 30.00th=[ 262], 40.00th=[ 279], 50.00th=[ 300], 60.00th=[ 321], | 70.00th=[ 342], 80.00th=[ 372], 90.00th=[ 426], 95.00th=[ 472], | 99.00th=[ 558], 99.50th=[ 600], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 32, max= 224, per=25.04%, avg=140.99, stdev=34.09, samples=73 iops : min= 8, max= 56, avg=35.04, stdev= 8.51, samples=73 lat (msec) : 4=0.04%, 10=0.08%, 20=0.08%, 50=0.47%, 100=3.95% lat (msec) : 250=56.37%, 500=37.07%, 750=1.95% cpu : usr=0.15%, sys=1.40%, ctx=5032, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146475: Tue Aug 18 21:16:23 2026 read: IOPS=34, BW=137KiB/s (141kB/s)(5060KiB/36852msec) slat (usec): min=85, max=17584, avg=4871.63, stdev=2260.88 clat (msec): min=4, max=343, avg=141.99, stdev=36.82 lat (msec): min=9, max=347, avg=146.86, stdev=37.29 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 90], 10.00th=[ 102], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 133], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 169], 90.00th=[ 186], 95.00th=[ 205], | 99.00th=[ 247], 99.50th=[ 253], 99.90th=[ 317], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 48, max= 232, per=25.34%, avg=137.10, stdev=41.98, samples=73 iops : min= 12, max= 58, avg=34.07, stdev=10.46, samples=73 write: IOPS=35, BW=141KiB/s (144kB/s)(5180KiB/36852msec); 0 zone resets slat (usec): min=37, max=7137, avg=232.72, stdev=318.51 clat (msec): min=71, max=618, avg=311.24, stdev=84.32 lat (msec): min=71, max=619, avg=311.48, stdev=84.33 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 190], 10.00th=[ 215], 20.00th=[ 245], | 30.00th=[ 264], 40.00th=[ 279], 50.00th=[ 300], 60.00th=[ 321], | 70.00th=[ 347], 80.00th=[ 376], 90.00th=[ 435], 95.00th=[ 472], | 99.00th=[ 535], 99.50th=[ 567], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 64, max= 207, per=24.89%, avg=140.14, stdev=28.94, samples=73 iops : min= 16, max= 51, avg=34.82, stdev= 7.25, samples=73 lat (msec) : 10=0.12%, 20=0.08%, 50=0.27%, 100=4.26%, 250=55.70% lat (msec) : 500=38.12%, 750=1.45% cpu : usr=0.15%, sys=1.43%, ctx=4963, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=542KiB/s (555kB/s), 134KiB/s-137KiB/s (138kB/s-141kB/s), io=19.6MiB (20.5MB), run=36852-37050msec WRITE: bw=564KiB/s (577kB/s), 140KiB/s-142KiB/s (143kB/s-146kB/s), io=20.4MiB (21.4MB), run=36852-37050msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146484: Tue Aug 18 21:16:27 2026 read: IOPS=5, BW=220MiB/s (230MB/s)(40.0MiB/182msec) slat (nsec): min=15530k, max=15530k, avg=15529921.00, stdev= 0.00 clat (nsec): min=160037k, max=160037k, avg=160036660.00, stdev= 0.00 lat (nsec): min=175572k, max=175572k, avg=175571704.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.63%, ctx=44, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=220MiB/s (230MB/s), 220MiB/s-220MiB/s (230MB/s-230MB/s), io=40.0MiB (41.9MB), run=182-182msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout