== sanity test 398c: run fio to test AIO ================= 14:46:05 (1790102765) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.569535 s, 73.6 MB/s osc.lustre-OST0000-osc-ffff8c008588d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180764: Tue Sep 22 14:46:42 2026 write: IOPS=100, BW=401KiB/s (410kB/s)(10.0MiB/25556msec); 0 zone resets slat (usec): min=22, max=19032, avg=198.40, stdev=819.35 clat (msec): min=13, max=309, avg=158.76, stdev=42.42 lat (msec): min=13, max=309, avg=158.96, stdev=42.42 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 103], 10.00th=[ 116], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 192], 90.00th=[ 211], 95.00th=[ 239], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 309] bw ( KiB/s): min= 221, max= 720, per=24.83%, avg=397.52, stdev=92.82, samples=50 iops : min= 55, max= 180, avg=98.98, stdev=23.29, samples=50 lat (msec) : 20=0.08%, 50=0.08%, 100=4.61%, 250=90.90%, 500=4.34% cpu : usr=0.15%, sys=1.08%, ctx=2759, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180765: Tue Sep 22 14:46:42 2026 write: IOPS=100, BW=400KiB/s (410kB/s)(10.0MiB/25575msec); 0 zone resets slat (usec): min=22, max=14797, avg=158.01, stdev=597.64 clat (msec): min=20, max=294, avg=158.86, stdev=42.58 lat (msec): min=21, max=294, avg=159.02, stdev=42.60 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 104], 10.00th=[ 117], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 171], 80.00th=[ 190], 90.00th=[ 213], 95.00th=[ 243], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 127, max= 752, per=24.75%, avg=396.32, stdev=101.06, samples=50 iops : min= 31, max= 188, avg=98.68, stdev=25.34, samples=50 lat (msec) : 50=0.12%, 100=4.22%, 250=91.33%, 500=4.34% cpu : usr=0.15%, sys=0.86%, ctx=1737, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180766: Tue Sep 22 14:46:42 2026 write: IOPS=100, BW=402KiB/s (411kB/s)(10.0MiB/25500msec); 0 zone resets slat (usec): min=25, max=15769, avg=171.35, stdev=689.80 clat (msec): min=20, max=306, avg=158.98, stdev=43.29 lat (msec): min=20, max=307, avg=159.15, stdev=43.29 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 102], 10.00th=[ 116], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 174], 80.00th=[ 192], 90.00th=[ 218], 95.00th=[ 247], | 99.00th=[ 284], 99.50th=[ 296], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 309] bw ( KiB/s): min= 207, max= 720, per=24.90%, avg=398.62, stdev=86.33, samples=50 iops : min= 51, max= 180, avg=99.26, stdev=21.64, samples=50 lat (msec) : 50=0.20%, 100=4.45%, 250=90.55%, 500=4.80% cpu : usr=0.18%, sys=0.93%, ctx=2400, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180767: Tue Sep 22 14:46:42 2026 write: IOPS=100, BW=401KiB/s (411kB/s)(10.0MiB/25541msec); 0 zone resets slat (usec): min=23, max=14093, avg=178.22, stdev=684.48 clat (msec): min=14, max=299, avg=158.66, stdev=43.24 lat (msec): min=14, max=299, avg=158.84, stdev=43.23 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 103], 10.00th=[ 116], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 174], 80.00th=[ 192], 90.00th=[ 215], 95.00th=[ 241], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 300] bw ( KiB/s): min= 231, max= 784, per=24.86%, avg=398.02, stdev=94.59, samples=50 iops : min= 57, max= 196, avg=99.10, stdev=23.71, samples=50 lat (msec) : 20=0.08%, 50=0.16%, 100=4.61%, 250=90.74%, 500=4.41% cpu : usr=0.18%, sys=0.98%, ctx=2678, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1602KiB/s (1640kB/s), 400KiB/s-402KiB/s (410kB/s-411kB/s), io=40.0MiB (41.9MB), run=25500-25575msec 0% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180781: Tue Sep 22 14:47:18 2026 read: IOPS=44, BW=177KiB/s (181kB/s)(5048KiB/28496msec) slat (usec): min=2325, max=73742, avg=12253.94, stdev=7573.78 clat (msec): min=56, max=404, avg=166.81, stdev=50.67 lat (msec): min=61, max=418, avg=179.07, stdev=53.22 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 92], 10.00th=[ 105], 20.00th=[ 125], | 30.00th=[ 140], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 205], 90.00th=[ 230], 95.00th=[ 255], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 401], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 79, max= 295, per=25.24%, avg=176.66, stdev=50.15, samples=56 iops : min= 19, max= 73, avg=43.68, stdev=12.58, samples=56 write: IOPS=45, BW=182KiB/s (187kB/s)(5192KiB/28496msec); 0 zone resets slat (usec): min=26, max=57822, avg=248.80, stdev=1829.98 clat (msec): min=79, max=419, avg=173.67, stdev=51.16 lat (msec): min=80, max=419, avg=173.92, stdev=51.09 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 103], 10.00th=[ 113], 20.00th=[ 131], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 239], 95.00th=[ 271], | 99.00th=[ 338], 99.50th=[ 393], 99.90th=[ 409], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 61, max= 336, per=24.98%, avg=182.11, stdev=53.90, samples=56 iops : min= 15, max= 84, avg=45.04, stdev=13.52, samples=56 lat (msec) : 100=6.45%, 250=86.68%, 500=6.88% cpu : usr=0.24%, sys=1.76%, ctx=4664, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180782: Tue Sep 22 14:47:18 2026 read: IOPS=43, BW=175KiB/s (180kB/s)(4976KiB/28381msec) slat (usec): min=54, max=76826, avg=12493.66, stdev=7792.56 clat (msec): min=34, max=405, avg=168.18, stdev=53.51 lat (msec): min=36, max=417, avg=180.68, stdev=55.81 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 122], | 30.00th=[ 140], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 176], | 70.00th=[ 190], 80.00th=[ 209], 90.00th=[ 239], 95.00th=[ 268], | 99.00th=[ 321], 99.50th=[ 326], 99.90th=[ 376], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 71, max= 311, per=24.99%, avg=174.95, stdev=51.42, samples=56 iops : min= 17, max= 77, avg=43.23, stdev=12.94, samples=56 write: IOPS=46, BW=185KiB/s (190kB/s)(5264KiB/28381msec); 0 zone resets slat (usec): min=25, max=16581, avg=205.99, stdev=790.03 clat (msec): min=75, max=359, avg=173.23, stdev=49.45 lat (msec): min=75, max=359, avg=173.44, stdev=49.48 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 103], 10.00th=[ 113], 20.00th=[ 130], | 30.00th=[ 144], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 268], | 99.00th=[ 317], 99.50th=[ 347], 99.90th=[ 355], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 101, max= 343, per=25.47%, avg=185.68, stdev=54.41, samples=56 iops : min= 25, max= 85, avg=45.93, stdev=13.64, samples=56 lat (msec) : 50=0.12%, 100=7.11%, 250=85.66%, 500=7.11% cpu : usr=0.30%, sys=1.73%, ctx=4299, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180783: Tue Sep 22 14:47:18 2026 read: IOPS=43, BW=174KiB/s (178kB/s)(4984KiB/28656msec) slat (usec): min=57, max=88537, avg=12562.37, stdev=8319.85 clat (msec): min=8, max=388, avg=168.95, stdev=52.54 lat (msec): min=31, max=403, avg=181.51, stdev=54.75 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 126], | 30.00th=[ 142], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 207], 90.00th=[ 232], 95.00th=[ 264], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 75, max= 296, per=24.69%, avg=172.80, stdev=45.88, samples=56 iops : min= 18, max= 74, avg=42.70, stdev=11.55, samples=56 write: IOPS=45, BW=183KiB/s (188kB/s)(5256KiB/28656msec); 0 zone resets slat (usec): min=35, max=21659, avg=172.01, stdev=729.73 clat (msec): min=27, max=425, avg=173.59, stdev=53.24 lat (msec): min=28, max=425, avg=173.76, stdev=53.24 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 102], 10.00th=[ 111], 20.00th=[ 129], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 239], 95.00th=[ 271], | 99.00th=[ 338], 99.50th=[ 368], 99.90th=[ 418], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 83, max= 367, per=24.81%, avg=180.88, stdev=56.26, samples=56 iops : min= 20, max= 91, avg=44.73, stdev=14.14, samples=56 lat (msec) : 10=0.04%, 50=0.35%, 100=5.94%, 250=86.21%, 500=7.46% cpu : usr=0.33%, sys=1.97%, ctx=4554, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180784: Tue Sep 22 14:47:18 2026 read: IOPS=44, BW=177KiB/s (181kB/s)(5060KiB/28649msec) slat (usec): min=38, max=193277, avg=12810.18, stdev=9386.81 clat (msec): min=10, max=387, avg=168.21, stdev=53.60 lat (msec): min=14, max=399, avg=181.02, stdev=56.19 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 92], 10.00th=[ 104], 20.00th=[ 122], | 30.00th=[ 138], 40.00th=[ 150], 50.00th=[ 165], 60.00th=[ 178], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 236], 95.00th=[ 271], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 372], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 60, max= 360, per=25.11%, avg=175.77, stdev=55.25, samples=56 iops : min= 15, max= 90, avg=43.46, stdev=13.81, samples=56 write: IOPS=45, BW=181KiB/s (185kB/s)(5180KiB/28649msec); 0 zone resets slat (usec): min=23, max=17011, avg=208.18, stdev=889.19 clat (msec): min=55, max=388, avg=173.56, stdev=52.24 lat (msec): min=55, max=388, avg=173.77, stdev=52.25 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 97], 10.00th=[ 111], 20.00th=[ 128], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 241], 95.00th=[ 268], | 99.00th=[ 334], 99.50th=[ 351], 99.90th=[ 376], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 83, max= 343, per=24.60%, avg=179.30, stdev=52.91, samples=56 iops : min= 20, max= 85, avg=44.34, stdev=13.29, samples=56 lat (msec) : 20=0.04%, 50=0.04%, 100=7.07%, 250=85.08%, 500=7.77% cpu : usr=0.30%, sys=1.99%, ctx=4539, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=700KiB/s (717kB/s), 174KiB/s-177KiB/s (178kB/s-181kB/s), io=19.6MiB (20.5MB), run=28381-28656msec WRITE: bw=729KiB/s (747kB/s), 181KiB/s-185KiB/s (185kB/s-190kB/s), io=20.4MiB (21.4MB), run=28381-28656msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180795: Tue Sep 22 14:47:26 2026 read: IOPS=1, BW=66.4MiB/s (69.7MB/s)(40.0MiB/602msec) slat (nsec): min=43495k, max=43495k, avg=43494961.00, stdev= 0.00 clat (nsec): min=525407k, max=525407k, avg=525407441.00, stdev= 0.00 lat (nsec): min=568907k, max=568907k, avg=568907122.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.33%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=66.4MiB/s (69.7MB/s), 66.4MiB/s-66.4MiB/s (69.7MB/s-69.7MB/s), io=40.0MiB (41.9MB), run=602-602msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout