== sanity test 398c: run fio to test AIO ================= 07:13:12 (1789125192) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.636217 s, 65.9 MB/s osc.lustre-OST0000-osc-ffff957b50857000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179562: Fri Sep 11 07:13:54 2026 write: IOPS=79, BW=319KiB/s (327kB/s)(10.0MiB/32078msec); 0 zone resets slat (usec): min=22, max=13706, avg=228.26, stdev=787.09 clat (msec): min=90, max=361, avg=199.57, stdev=45.52 lat (msec): min=90, max=361, avg=199.80, stdev=45.54 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 129], 10.00th=[ 146], 20.00th=[ 167], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 338], 99.50th=[ 347], 99.90th=[ 355], 99.95th=[ 359], | 99.99th=[ 363] bw ( KiB/s): min= 134, max= 506, per=24.83%, avg=316.86, stdev=67.68, samples=63 iops : min= 33, max= 126, avg=78.78, stdev=16.96, samples=63 lat (msec) : 100=2.27%, 250=85.90%, 500=11.84% cpu : usr=0.16%, sys=0.75%, ctx=1645, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179563: Fri Sep 11 07:13:54 2026 write: IOPS=79, BW=319KiB/s (327kB/s)(10.0MiB/32076msec); 0 zone resets slat (usec): min=29, max=15901, avg=212.81, stdev=719.93 clat (msec): min=64, max=364, avg=199.81, stdev=47.86 lat (msec): min=64, max=364, avg=200.02, stdev=47.85 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 126], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 288], | 99.00th=[ 330], 99.50th=[ 338], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 182, max= 534, per=24.96%, avg=318.44, stdev=67.85, samples=63 iops : min= 45, max= 133, avg=79.16, stdev=16.90, samples=63 lat (msec) : 100=1.37%, 250=83.05%, 500=15.59% cpu : usr=0.15%, sys=0.84%, ctx=1650, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179564: Fri Sep 11 07:13:54 2026 write: IOPS=79, BW=320KiB/s (327kB/s)(10.0MiB/32020msec); 0 zone resets slat (usec): min=29, max=16951, avg=302.67, stdev=977.32 clat (msec): min=33, max=367, avg=199.19, stdev=46.73 lat (msec): min=33, max=367, avg=199.50, stdev=46.71 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 129], 10.00th=[ 150], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 241], 90.00th=[ 259], 95.00th=[ 288], | 99.00th=[ 313], 99.50th=[ 334], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 206, max= 451, per=25.00%, avg=319.06, stdev=59.01, samples=63 iops : min= 51, max= 112, avg=79.32, stdev=14.73, samples=63 lat (msec) : 50=0.27%, 100=1.48%, 250=83.48%, 500=14.77% cpu : usr=0.17%, sys=1.02%, ctx=2906, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179565: Fri Sep 11 07:13:54 2026 write: IOPS=80, BW=320KiB/s (328kB/s)(10.0MiB/31952msec); 0 zone resets slat (usec): min=30, max=18430, avg=245.33, stdev=835.72 clat (msec): min=54, max=364, avg=199.23, stdev=47.84 lat (msec): min=54, max=364, avg=199.47, stdev=47.84 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 122], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 143, max= 535, per=25.04%, avg=319.49, stdev=70.72, samples=63 iops : min= 35, max= 133, avg=79.40, stdev=17.64, samples=63 lat (msec) : 100=1.52%, 250=84.26%, 500=14.22% cpu : usr=0.16%, sys=0.89%, ctx=2316, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1277KiB/s (1308kB/s), 319KiB/s-320KiB/s (327kB/s-328kB/s), io=40.0MiB (41.9MB), run=31952-32078msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179581: Fri Sep 11 07:14:39 2026 read: IOPS=35, BW=141KiB/s (145kB/s)(5048KiB/35772msec) slat (usec): min=3570, max=71813, avg=15341.03, stdev=7917.29 clat (msec): min=19, max=379, avg=210.29, stdev=45.20 lat (msec): min=24, max=415, avg=225.64, stdev=46.74 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 326], 99.50th=[ 338], 99.90th=[ 368], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 79, max= 223, per=24.93%, avg=139.63, stdev=30.69, samples=70 iops : min= 19, max= 55, avg=34.34, stdev= 7.71, samples=70 write: IOPS=36, BW=145KiB/s (149kB/s)(5192KiB/35772msec); 0 zone resets slat (usec): min=35, max=10462, avg=264.63, stdev=721.34 clat (msec): min=43, max=358, avg=218.45, stdev=42.46 lat (msec): min=44, max=358, avg=218.71, stdev=42.45 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 155], 10.00th=[ 165], 20.00th=[ 184], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 253], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 321], 99.50th=[ 326], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 71, max= 229, per=24.67%, avg=144.07, stdev=34.03, samples=70 iops : min= 17, max= 57, avg=35.46, stdev= 8.55, samples=70 lat (msec) : 20=0.04%, 50=0.04%, 100=0.16%, 250=79.26%, 500=20.51% cpu : usr=0.23%, sys=1.57%, ctx=4196, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179582: Fri Sep 11 07:14:39 2026 read: IOPS=34, BW=139KiB/s (143kB/s)(4976KiB/35752msec) slat (usec): min=3313, max=77194, avg=15507.32, stdev=8427.95 clat (msec): min=22, max=400, avg=212.20, stdev=46.48 lat (msec): min=31, max=417, avg=227.71, stdev=47.77 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 146], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 363], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 64, max= 207, per=24.64%, avg=138.00, stdev=33.31, samples=70 iops : min= 16, max= 51, avg=33.94, stdev= 8.33, samples=70 write: IOPS=36, BW=147KiB/s (151kB/s)(5264KiB/35752msec); 0 zone resets slat (usec): min=31, max=18751, avg=290.53, stdev=978.02 clat (msec): min=69, max=354, avg=217.28, stdev=43.39 lat (msec): min=69, max=354, avg=217.57, stdev=43.38 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 153], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 255], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 79, max= 231, per=25.17%, avg=147.00, stdev=33.18, samples=70 iops : min= 19, max= 57, avg=36.20, stdev= 8.38, samples=70 lat (msec) : 50=0.04%, 100=0.27%, 250=78.09%, 500=21.60% cpu : usr=0.27%, sys=1.66%, ctx=4161, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179583: Fri Sep 11 07:14:39 2026 read: IOPS=35, BW=140KiB/s (144kB/s)(4984KiB/35556msec) slat (usec): min=38, max=63930, avg=15241.19, stdev=8305.11 clat (msec): min=63, max=381, avg=211.97, stdev=45.53 lat (msec): min=82, max=400, avg=227.22, stdev=47.15 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 234], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 321], 99.50th=[ 351], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 71, max= 215, per=25.02%, avg=140.13, stdev=34.36, samples=70 iops : min= 17, max= 53, avg=34.47, stdev= 8.65, samples=70 write: IOPS=36, BW=148KiB/s (151kB/s)(5256KiB/35556msec); 0 zone resets slat (usec): min=32, max=19752, avg=285.96, stdev=972.42 clat (msec): min=52, max=382, avg=216.65, stdev=43.68 lat (msec): min=54, max=383, avg=216.94, stdev=43.72 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 150], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 190], 40.00th=[ 205], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 253], 90.00th=[ 275], 95.00th=[ 288], | 99.00th=[ 330], 99.50th=[ 347], 99.90th=[ 380], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 71, max= 255, per=25.16%, avg=146.93, stdev=37.39, samples=70 iops : min= 17, max= 63, avg=36.20, stdev= 9.36, samples=70 lat (msec) : 100=0.39%, 250=79.10%, 500=20.51% cpu : usr=0.24%, sys=1.65%, ctx=4087, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179584: Fri Sep 11 07:14:39 2026 read: IOPS=35, BW=142KiB/s (145kB/s)(5060KiB/35718msec) slat (usec): min=3338, max=70875, avg=15398.75, stdev=8087.60 clat (msec): min=64, max=420, avg=209.53, stdev=44.77 lat (msec): min=72, max=439, avg=224.94, stdev=46.31 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 384], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 71, max= 245, per=25.22%, avg=141.21, stdev=35.68, samples=70 iops : min= 17, max= 61, avg=34.74, stdev= 8.97, samples=70 write: IOPS=36, BW=145KiB/s (149kB/s)(5180KiB/35718msec); 0 zone resets slat (usec): min=35, max=20985, avg=282.21, stdev=1066.00 clat (msec): min=114, max=409, avg=218.83, stdev=43.44 lat (msec): min=114, max=409, avg=219.11, stdev=43.48 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 153], 10.00th=[ 167], 20.00th=[ 182], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 241], 80.00th=[ 253], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 338], 99.50th=[ 363], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 79, max= 207, per=24.66%, avg=144.00, stdev=30.70, samples=70 iops : min= 19, max= 51, avg=35.44, stdev= 7.72, samples=70 lat (msec) : 100=0.12%, 250=80.08%, 500=19.80% cpu : usr=0.22%, sys=1.56%, ctx=4071, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=561KiB/s (574kB/s), 139KiB/s-142KiB/s (143kB/s-145kB/s), io=19.6MiB (20.5MB), run=35556-35772msec WRITE: bw=584KiB/s (598kB/s), 145KiB/s-148KiB/s (149kB/s-151kB/s), io=20.4MiB (21.4MB), run=35556-35772msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179594: Fri Sep 11 07:14:51 2026 read: IOPS=1, BW=64.9MiB/s (68.1MB/s)(40.0MiB/616msec) slat (nsec): min=45927k, max=45927k, avg=45927460.00, stdev= 0.00 clat (nsec): min=553629k, max=553629k, avg=553629380.00, stdev= 0.00 lat (nsec): min=599562k, max=599562k, avg=599561909.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.28%, ctx=11, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=64.9MiB/s (68.1MB/s), 64.9MiB/s-64.9MiB/s (68.1MB/s-68.1MB/s), io=40.0MiB (41.9MB), run=616-616msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout