== sanity test 398c: run fio to test AIO ================= 23:20:11 (1763353211) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.214462 s, 196 MB/s osc.lustre-OST0000-osc-ffff914509760000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148479: Sun Nov 16 23:21:27 2025 write: IOPS=35, BW=143KiB/s (147kB/s)(10.0MiB/71512msec); 0 zone resets slat (usec): min=27, max=2662, avg=147.33, stdev=186.12 clat (msec): min=66, max=794, avg=446.42, stdev=93.94 lat (msec): min=67, max=794, avg=446.57, stdev=93.94 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 305], 10.00th=[ 330], 20.00th=[ 376], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 447], 60.00th=[ 468], | 70.00th=[ 489], 80.00th=[ 514], 90.00th=[ 558], 95.00th=[ 609], | 99.00th=[ 718], 99.50th=[ 760], 99.90th=[ 785], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 31, max= 256, per=24.92%, avg=142.56, stdev=31.88, samples=142 iops : min= 7, max= 64, avg=35.31, stdev= 8.02, samples=142 lat (msec) : 100=0.04%, 250=1.37%, 500=73.95%, 750=24.06%, 1000=0.59% cpu : usr=0.08%, sys=0.44%, ctx=2504, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148480: Sun Nov 16 23:21:27 2025 write: IOPS=35, BW=143KiB/s (147kB/s)(10.0MiB/71511msec); 0 zone resets slat (usec): min=20, max=4239, avg=148.23, stdev=207.00 clat (msec): min=159, max=807, avg=446.32, stdev=93.15 lat (msec): min=159, max=808, avg=446.47, stdev=93.17 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 376], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 447], 60.00th=[ 468], | 70.00th=[ 485], 80.00th=[ 514], 90.00th=[ 558], 95.00th=[ 600], | 99.00th=[ 718], 99.50th=[ 743], 99.90th=[ 776], 99.95th=[ 793], | 99.99th=[ 810] bw ( KiB/s): min= 55, max= 256, per=24.93%, avg=142.59, stdev=32.78, samples=142 iops : min= 13, max= 64, avg=35.32, stdev= 8.27, samples=142 lat (msec) : 250=1.64%, 500=74.34%, 750=23.55%, 1000=0.47% cpu : usr=0.09%, sys=0.43%, ctx=2530, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148481: Sun Nov 16 23:21:27 2025 write: IOPS=35, BW=143KiB/s (147kB/s)(10.0MiB/71362msec); 0 zone resets slat (usec): min=26, max=3664, avg=153.29, stdev=225.43 clat (msec): min=152, max=773, avg=445.66, stdev=94.42 lat (msec): min=153, max=773, avg=445.82, stdev=94.43 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 305], 10.00th=[ 334], 20.00th=[ 372], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 443], 60.00th=[ 468], | 70.00th=[ 489], 80.00th=[ 518], 90.00th=[ 567], 95.00th=[ 609], | 99.00th=[ 701], 99.50th=[ 735], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 776] bw ( KiB/s): min= 71, max= 256, per=24.93%, avg=142.58, stdev=34.55, samples=142 iops : min= 17, max= 64, avg=35.32, stdev= 8.71, samples=142 lat (msec) : 250=2.11%, 500=72.03%, 750=25.59%, 1000=0.27% cpu : usr=0.08%, sys=0.44%, ctx=2473, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148482: Sun Nov 16 23:21:27 2025 write: IOPS=35, BW=143KiB/s (147kB/s)(10.0MiB/71512msec); 0 zone resets slat (usec): min=27, max=3502, avg=150.38, stdev=216.51 clat (msec): min=151, max=780, avg=446.25, stdev=92.90 lat (msec): min=151, max=780, avg=446.41, stdev=92.91 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 368], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 443], 60.00th=[ 464], | 70.00th=[ 489], 80.00th=[ 514], 90.00th=[ 558], 95.00th=[ 609], | 99.00th=[ 726], 99.50th=[ 751], 99.90th=[ 760], 99.95th=[ 776], | 99.99th=[ 785] bw ( KiB/s): min= 7, max= 240, per=24.93%, avg=142.58, stdev=33.58, samples=142 iops : min= 1, max= 60, avg=35.32, stdev= 8.44, samples=142 lat (msec) : 250=1.17%, 500=74.14%, 750=24.18%, 1000=0.51% cpu : usr=0.10%, sys=0.41%, ctx=2572, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=573KiB/s (587kB/s), 143KiB/s-143KiB/s (147kB/s-147kB/s), io=40.0MiB (41.9MB), run=71362-71512msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148502: Sun Nov 16 23:22:06 2025 read: IOPS=35, BW=142KiB/s (146kB/s)(5048KiB/35472msec) slat (usec): min=1631, max=13120, avg=4021.60, stdev=1367.47 clat (msec): min=5, max=244, avg=139.51, stdev=33.38 lat (msec): min=10, max=249, avg=143.54, stdev=33.73 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 93], 10.00th=[ 102], 20.00th=[ 114], | 30.00th=[ 122], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 167], 90.00th=[ 184], 95.00th=[ 201], | 99.00th=[ 228], 99.50th=[ 232], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 48, max= 248, per=25.14%, avg=142.03, stdev=44.08, samples=70 iops : min= 12, max= 62, avg=35.34, stdev=11.10, samples=70 write: IOPS=36, BW=146KiB/s (150kB/s)(5192KiB/35472msec); 0 zone resets slat (usec): min=37, max=2229, avg=156.11, stdev=126.69 clat (msec): min=87, max=541, avg=296.70, stdev=74.24 lat (msec): min=87, max=541, avg=296.85, stdev=74.25 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 174], 10.00th=[ 203], 20.00th=[ 234], | 30.00th=[ 257], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 313], | 70.00th=[ 338], 80.00th=[ 359], 90.00th=[ 388], 95.00th=[ 414], | 99.00th=[ 502], 99.50th=[ 518], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 80, max= 200, per=24.81%, avg=145.87, stdev=27.90, samples=70 iops : min= 20, max= 50, avg=36.30, stdev= 7.08, samples=70 lat (msec) : 10=0.04%, 20=0.08%, 50=0.23%, 100=4.41%, 250=57.81% lat (msec) : 500=36.91%, 750=0.51% cpu : usr=0.18%, sys=1.37%, ctx=5017, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148503: Sun Nov 16 23:22:06 2025 read: IOPS=35, BW=141KiB/s (144kB/s)(4976KiB/35339msec) slat (usec): min=36, max=11787, avg=4008.88, stdev=1395.87 clat (msec): min=5, max=407, avg=140.10, stdev=38.17 lat (msec): min=9, max=411, avg=144.11, stdev=38.50 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 113], | 30.00th=[ 122], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 167], 90.00th=[ 184], 95.00th=[ 201], | 99.00th=[ 245], 99.50th=[ 326], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 48, max= 240, per=24.87%, avg=140.51, stdev=43.26, samples=70 iops : min= 12, max= 60, avg=34.96, stdev=10.84, samples=70 write: IOPS=37, BW=149KiB/s (153kB/s)(5264KiB/35339msec); 0 zone resets slat (usec): min=27, max=1603, avg=159.78, stdev=132.75 clat (msec): min=45, max=533, avg=292.86, stdev=78.05 lat (msec): min=45, max=534, avg=293.03, stdev=78.05 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 176], 10.00th=[ 199], 20.00th=[ 232], | 30.00th=[ 255], 40.00th=[ 271], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 355], 90.00th=[ 388], 95.00th=[ 426], | 99.00th=[ 510], 99.50th=[ 518], 99.90th=[ 527], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 56, max= 240, per=25.29%, avg=148.71, stdev=34.88, samples=70 iops : min= 14, max= 60, avg=37.01, stdev= 8.80, samples=70 lat (msec) : 10=0.12%, 20=0.08%, 50=0.27%, 100=5.23%, 250=56.68% lat (msec) : 500=36.88%, 750=0.74% cpu : usr=0.17%, sys=1.40%, ctx=4961, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148504: Sun Nov 16 23:22:06 2025 read: IOPS=35, BW=141KiB/s (144kB/s)(4984KiB/35442msec) slat (usec): min=1749, max=14213, avg=4056.00, stdev=1339.08 clat (msec): min=5, max=248, avg=138.24, stdev=34.79 lat (msec): min=9, max=251, avg=142.30, stdev=35.12 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 90], 10.00th=[ 100], 20.00th=[ 111], | 30.00th=[ 120], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 182], 95.00th=[ 201], | 99.00th=[ 228], 99.50th=[ 239], 99.90th=[ 247], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 56, max= 256, per=24.93%, avg=140.83, stdev=45.48, samples=70 iops : min= 14, max= 64, avg=35.04, stdev=11.45, samples=70 write: IOPS=37, BW=148KiB/s (152kB/s)(5256KiB/35442msec); 0 zone resets slat (usec): min=32, max=4185, avg=165.61, stdev=184.03 clat (msec): min=51, max=550, avg=295.64, stdev=73.48 lat (msec): min=51, max=550, avg=295.81, stdev=73.48 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 176], 10.00th=[ 201], 20.00th=[ 236], | 30.00th=[ 255], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 334], 80.00th=[ 359], 90.00th=[ 388], 95.00th=[ 414], | 99.00th=[ 502], 99.50th=[ 518], 99.90th=[ 542], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 96, max= 224, per=25.01%, avg=147.06, stdev=32.86, samples=70 iops : min= 24, max= 56, avg=36.60, stdev= 8.27, samples=70 lat (msec) : 10=0.23%, 20=0.16%, 50=0.20%, 100=4.77%, 250=56.88% lat (msec) : 500=37.23%, 750=0.55% cpu : usr=0.17%, sys=1.36%, ctx=4865, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148505: Sun Nov 16 23:22:06 2025 read: IOPS=35, BW=144KiB/s (147kB/s)(5060KiB/35261msec) slat (usec): min=1695, max=14674, avg=3985.39, stdev=1380.04 clat (msec): min=3, max=395, avg=139.32, stdev=36.49 lat (msec): min=8, max=398, avg=143.31, stdev=36.77 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 94], 10.00th=[ 102], 20.00th=[ 111], | 30.00th=[ 121], 40.00th=[ 128], 50.00th=[ 136], 60.00th=[ 142], | 70.00th=[ 153], 80.00th=[ 165], 90.00th=[ 184], 95.00th=[ 203], | 99.00th=[ 234], 99.50th=[ 266], 99.90th=[ 393], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 32, max= 264, per=25.35%, avg=143.23, stdev=42.15, samples=70 iops : min= 8, max= 66, avg=35.64, stdev=10.63, samples=70 write: IOPS=36, BW=147KiB/s (150kB/s)(5180KiB/35261msec); 0 zone resets slat (usec): min=39, max=2242, avg=164.80, stdev=165.15 clat (msec): min=63, max=569, avg=294.66, stdev=75.96 lat (msec): min=63, max=569, avg=294.83, stdev=75.97 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 176], 10.00th=[ 199], 20.00th=[ 234], | 30.00th=[ 257], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 355], 90.00th=[ 388], 95.00th=[ 422], | 99.00th=[ 514], 99.50th=[ 518], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 80, max= 248, per=24.88%, avg=146.27, stdev=32.46, samples=70 iops : min= 20, max= 62, avg=36.40, stdev= 8.18, samples=70 lat (msec) : 4=0.04%, 10=0.04%, 20=0.08%, 50=0.23%, 100=4.14% lat (msec) : 250=57.77%, 500=36.80%, 750=0.90% cpu : usr=0.21%, sys=1.34%, ctx=5172, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=566KiB/s (579kB/s), 141KiB/s-144KiB/s (144kB/s-147kB/s), io=19.6MiB (20.5MB), run=35261-35472msec WRITE: bw=589KiB/s (603kB/s), 146KiB/s-149KiB/s (150kB/s-153kB/s), io=20.4MiB (21.4MB), run=35261-35472msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148515: Sun Nov 16 23:22:11 2025 read: IOPS=5, BW=205MiB/s (215MB/s)(40.0MiB/195msec) slat (nsec): min=14182k, max=14182k, avg=14182040.00, stdev= 0.00 clat (nsec): min=170174k, max=170174k, avg=170174056.00, stdev= 0.00 lat (nsec): min=184362k, max=184362k, avg=184361646.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.70%, ctx=28, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=40.0MiB (41.9MB), run=195-195msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout