== sanity test 398c: run fio to test AIO ================= 06:57:48 (1788519468) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.551796 s, 76.0 MB/s osc.lustre-OST0000-osc-ffff97d5c8b52000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179529: Fri Sep 4 06:58:30 2026 write: IOPS=88, BW=353KiB/s (361kB/s)(10.0MiB/29046msec); 0 zone resets slat (usec): min=24, max=14799, avg=211.08, stdev=689.19 clat (msec): min=18, max=405, avg=181.15, stdev=48.37 lat (msec): min=18, max=405, avg=181.36, stdev=48.37 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 241], 95.00th=[ 268], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 372], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 166, max= 566, per=25.03%, avg=352.61, stdev=81.23, samples=57 iops : min= 41, max= 141, avg=87.81, stdev=20.31, samples=57 lat (msec) : 20=0.08%, 50=0.04%, 100=1.09%, 250=89.77%, 500=9.02% cpu : usr=0.20%, sys=1.11%, ctx=2569, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179530: Fri Sep 4 06:58:30 2026 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29067msec); 0 zone resets slat (usec): min=21, max=21224, avg=173.32, stdev=852.21 clat (msec): min=27, max=366, avg=180.54, stdev=48.22 lat (msec): min=27, max=366, avg=180.71, stdev=48.23 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 120], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 245], 95.00th=[ 268], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 198, max= 510, per=24.84%, avg=349.93, stdev=79.26, samples=57 iops : min= 49, max= 127, avg=87.12, stdev=19.80, samples=57 lat (msec) : 50=0.08%, 100=1.91%, 250=89.45%, 500=8.55% cpu : usr=0.14%, sys=0.69%, ctx=961, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179531: Fri Sep 4 06:58:30 2026 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29061msec); 0 zone resets slat (usec): min=26, max=19480, avg=216.46, stdev=807.79 clat (msec): min=37, max=360, avg=180.50, stdev=47.81 lat (msec): min=38, max=370, avg=180.72, stdev=47.83 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 120], 10.00th=[ 130], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 241], 95.00th=[ 271], | 99.00th=[ 321], 99.50th=[ 326], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 363] bw ( KiB/s): min= 240, max= 510, per=24.91%, avg=350.98, stdev=76.74, samples=57 iops : min= 60, max= 127, avg=87.37, stdev=19.21, samples=57 lat (msec) : 50=0.08%, 100=2.30%, 250=89.61%, 500=8.01% cpu : usr=0.18%, sys=0.98%, ctx=2631, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179532: Fri Sep 4 06:58:30 2026 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29057msec); 0 zone resets slat (usec): min=27, max=14329, avg=210.25, stdev=697.80 clat (msec): min=15, max=381, avg=180.76, stdev=46.99 lat (msec): min=15, max=381, avg=180.97, stdev=47.02 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 118], 10.00th=[ 133], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 215], 90.00th=[ 245], 95.00th=[ 268], | 99.00th=[ 326], 99.50th=[ 338], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 380] bw ( KiB/s): min= 174, max= 558, per=25.02%, avg=352.58, stdev=79.50, samples=57 iops : min= 43, max= 139, avg=87.74, stdev=19.89, samples=57 lat (msec) : 20=0.04%, 50=0.04%, 100=1.45%, 250=91.02%, 500=7.46% cpu : usr=0.20%, sys=0.99%, ctx=2677, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1409KiB/s (1443kB/s), 352KiB/s-353KiB/s (361kB/s-361kB/s), io=40.0MiB (41.9MB), run=29046-29067msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179546: Fri Sep 4 06:59:11 2026 read: IOPS=38, BW=154KiB/s (158kB/s)(5048KiB/32707msec) slat (usec): min=2804, max=49714, avg=14433.70, stdev=7862.15 clat (msec): min=38, max=374, avg=195.41, stdev=51.35 lat (msec): min=56, max=391, avg=209.85, stdev=52.95 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 122], 10.00th=[ 134], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 236], 90.00th=[ 266], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 44, max= 240, per=25.09%, avg=153.55, stdev=37.64, samples=64 iops : min= 11, max= 60, avg=37.94, stdev= 9.42, samples=64 write: IOPS=39, BW=159KiB/s (163kB/s)(5192KiB/32707msec); 0 zone resets slat (usec): min=29, max=19595, avg=320.00, stdev=1141.05 clat (msec): min=95, max=381, avg=196.12, stdev=48.53 lat (msec): min=95, max=381, avg=196.44, stdev=48.56 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 127], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 266], 95.00th=[ 292], | 99.00th=[ 330], 99.50th=[ 347], 99.90th=[ 372], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 44, max= 271, per=24.92%, avg=158.72, stdev=46.30, samples=64 iops : min= 11, max= 67, avg=39.22, stdev=11.56, samples=64 lat (msec) : 50=0.04%, 100=0.47%, 250=85.31%, 500=14.18% cpu : usr=0.31%, sys=1.98%, ctx=4444, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179547: Fri Sep 4 06:59:11 2026 read: IOPS=38, BW=153KiB/s (156kB/s)(4976KiB/32588msec) slat (usec): min=2565, max=71063, avg=14758.44, stdev=8034.00 clat (msec): min=56, max=387, avg=193.06, stdev=51.23 lat (msec): min=65, max=410, avg=207.83, stdev=53.09 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 118], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 163], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 201], | 70.00th=[ 215], 80.00th=[ 236], 90.00th=[ 266], 95.00th=[ 288], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 384], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 36, max= 231, per=24.79%, avg=151.73, stdev=39.43, samples=64 iops : min= 9, max= 57, avg=37.47, stdev= 9.87, samples=64 write: IOPS=40, BW=162KiB/s (165kB/s)(5264KiB/32588msec); 0 zone resets slat (usec): min=34, max=19213, avg=286.78, stdev=987.76 clat (msec): min=81, max=397, avg=197.37, stdev=48.19 lat (msec): min=81, max=397, avg=197.66, stdev=48.28 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 128], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 236], 90.00th=[ 264], 95.00th=[ 292], | 99.00th=[ 326], 99.50th=[ 338], 99.90th=[ 393], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 78, max= 303, per=25.37%, avg=161.58, stdev=43.92, samples=64 iops : min= 19, max= 75, avg=39.91, stdev=11.03, samples=64 lat (msec) : 100=0.62%, 250=85.94%, 500=13.44% cpu : usr=0.23%, sys=1.92%, ctx=4411, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179548: Fri Sep 4 06:59:11 2026 read: IOPS=38, BW=153KiB/s (156kB/s)(4984KiB/32679msec) slat (usec): min=38, max=62539, avg=14441.61, stdev=7871.44 clat (msec): min=86, max=426, avg=193.87, stdev=53.98 lat (msec): min=93, max=433, avg=208.32, stdev=56.03 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 121], 10.00th=[ 129], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 215], 80.00th=[ 234], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 418], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 55, max= 279, per=24.86%, avg=152.17, stdev=42.62, samples=64 iops : min= 13, max= 69, avg=37.56, stdev=10.68, samples=64 write: IOPS=40, BW=161KiB/s (165kB/s)(5256KiB/32679msec); 0 zone resets slat (usec): min=35, max=14124, avg=285.59, stdev=882.10 clat (msec): min=88, max=411, avg=199.71, stdev=49.19 lat (msec): min=88, max=411, avg=200.00, stdev=49.25 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 266], 95.00th=[ 292], | 99.00th=[ 334], 99.50th=[ 363], 99.90th=[ 380], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 78, max= 246, per=25.12%, avg=160.03, stdev=43.56, samples=64 iops : min= 19, max= 61, avg=39.53, stdev=10.95, samples=64 lat (msec) : 100=0.59%, 250=84.41%, 500=15.00% cpu : usr=0.28%, sys=2.10%, ctx=4404, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179549: Fri Sep 4 06:59:11 2026 read: IOPS=38, BW=154KiB/s (158kB/s)(5060KiB/32784msec) slat (usec): min=75, max=151508, avg=14605.72, stdev=9447.92 clat (msec): min=32, max=408, avg=193.49, stdev=53.56 lat (msec): min=43, max=419, avg=208.10, stdev=55.63 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 118], 10.00th=[ 130], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 215], 80.00th=[ 236], 90.00th=[ 266], 95.00th=[ 288], | 99.00th=[ 338], 99.50th=[ 363], 99.90th=[ 401], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 44, max= 279, per=25.09%, avg=153.55, stdev=42.62, samples=65 iops : min= 11, max= 69, avg=37.91, stdev=10.56, samples=65 write: IOPS=39, BW=158KiB/s (162kB/s)(5180KiB/32784msec); 0 zone resets slat (usec): min=29, max=24168, avg=282.46, stdev=1082.44 clat (msec): min=43, max=396, avg=199.67, stdev=50.27 lat (msec): min=43, max=396, avg=199.95, stdev=50.23 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 209], | 70.00th=[ 220], 80.00th=[ 234], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 59, max= 256, per=24.65%, avg=157.05, stdev=40.46, samples=65 iops : min= 14, max= 64, avg=38.78, stdev=10.23, samples=65 lat (msec) : 50=0.39%, 100=0.94%, 250=84.49%, 500=14.18% cpu : usr=0.28%, sys=1.91%, ctx=4285, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=612KiB/s (627kB/s), 153KiB/s-154KiB/s (156kB/s-158kB/s), io=19.6MiB (20.5MB), run=32588-32784msec WRITE: bw=637KiB/s (653kB/s), 158KiB/s-162KiB/s (162kB/s-165kB/s), io=20.4MiB (21.4MB), run=32588-32784msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179558: Fri Sep 4 06:59:21 2026 read: IOPS=2, BW=115MiB/s (121MB/s)(40.0MiB/347msec) slat (nsec): min=32038k, max=32038k, avg=32037892.00, stdev= 0.00 clat (nsec): min=300342k, max=300342k, avg=300342165.00, stdev= 0.00 lat (nsec): min=332385k, max=332385k, avg=332385236.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.05%, ctx=13, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=115MiB/s (121MB/s), 115MiB/s-115MiB/s (121MB/s-121MB/s), io=40.0MiB (41.9MB), run=347-347msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout