== sanity test 398c: run fio to test AIO ================= 14:45:26 (1785782726) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.749101 s, 56.0 MB/s osc.lustre-OST0000-osc-ffff9b73a0013000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=178813: Mon Aug 3 14:46:25 2026 write: IOPS=60, BW=242KiB/s (248kB/s)(10.0MiB/42234msec); 0 zone resets slat (usec): min=48, max=22385, avg=349.39, stdev=1530.36 clat (msec): min=44, max=658, avg=263.19, stdev=78.63 lat (msec): min=44, max=658, avg=263.54, stdev=78.58 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 165], 10.00th=[ 182], 20.00th=[ 199], | 30.00th=[ 218], 40.00th=[ 232], 50.00th=[ 249], 60.00th=[ 271], | 70.00th=[ 288], 80.00th=[ 317], 90.00th=[ 368], 95.00th=[ 414], | 99.00th=[ 542], 99.50th=[ 575], 99.90th=[ 634], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 87, max= 384, per=25.13%, avg=241.25, stdev=69.49, samples=83 iops : min= 21, max= 96, avg=59.86, stdev=17.44, samples=83 lat (msec) : 50=0.04%, 100=0.66%, 250=50.43%, 500=47.46%, 750=1.41% cpu : usr=0.12%, sys=0.66%, ctx=1739, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178814: Mon Aug 3 14:46:25 2026 write: IOPS=60, BW=240KiB/s (246kB/s)(10.0MiB/42635msec); 0 zone resets slat (usec): min=47, max=27982, avg=308.32, stdev=1379.47 clat (msec): min=47, max=658, avg=264.21, stdev=78.20 lat (msec): min=47, max=658, avg=264.52, stdev=78.23 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 165], 10.00th=[ 184], 20.00th=[ 203], | 30.00th=[ 220], 40.00th=[ 236], 50.00th=[ 249], 60.00th=[ 268], | 70.00th=[ 292], 80.00th=[ 317], 90.00th=[ 376], 95.00th=[ 414], | 99.00th=[ 493], 99.50th=[ 634], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 125, max= 384, per=24.94%, avg=239.39, stdev=66.19, samples=84 iops : min= 31, max= 96, avg=59.38, stdev=16.60, samples=84 lat (msec) : 50=0.04%, 100=0.08%, 250=50.51%, 500=48.71%, 750=0.66% cpu : usr=0.09%, sys=0.72%, ctx=1804, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178815: Mon Aug 3 14:46:25 2026 write: IOPS=60, BW=240KiB/s (246kB/s)(10.0MiB/42635msec); 0 zone resets slat (usec): min=48, max=33782, avg=380.98, stdev=1756.64 clat (msec): min=10, max=657, avg=264.48, stdev=79.51 lat (msec): min=11, max=657, avg=264.87, stdev=79.61 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 165], 10.00th=[ 182], 20.00th=[ 199], | 30.00th=[ 213], 40.00th=[ 234], 50.00th=[ 251], 60.00th=[ 271], | 70.00th=[ 296], 80.00th=[ 321], 90.00th=[ 368], 95.00th=[ 405], | 99.00th=[ 498], 99.50th=[ 625], 99.90th=[ 634], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 119, max= 408, per=24.93%, avg=239.32, stdev=70.23, samples=84 iops : min= 29, max= 102, avg=59.37, stdev=17.64, samples=84 lat (msec) : 20=0.04%, 50=0.12%, 100=0.27%, 250=48.36%, 500=50.23% lat (msec) : 750=0.98% cpu : usr=0.13%, sys=0.86%, ctx=2302, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178816: Mon Aug 3 14:46:25 2026 write: IOPS=60, BW=241KiB/s (247kB/s)(10.0MiB/42533msec); 0 zone resets slat (usec): min=47, max=45356, avg=349.61, stdev=1589.39 clat (msec): min=12, max=660, avg=264.38, stdev=77.91 lat (msec): min=13, max=661, avg=264.73, stdev=77.89 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 171], 10.00th=[ 188], 20.00th=[ 203], | 30.00th=[ 220], 40.00th=[ 234], 50.00th=[ 249], 60.00th=[ 268], | 70.00th=[ 292], 80.00th=[ 317], 90.00th=[ 376], 95.00th=[ 414], | 99.00th=[ 489], 99.50th=[ 584], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 659] bw ( KiB/s): min= 87, max= 382, per=24.93%, avg=239.31, stdev=66.98, samples=83 iops : min= 21, max= 95, avg=59.37, stdev=16.81, samples=83 lat (msec) : 20=0.08%, 50=0.08%, 100=0.16%, 250=50.94%, 500=47.85% lat (msec) : 750=0.90% cpu : usr=0.13%, sys=0.96%, ctx=2464, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=961KiB/s (984kB/s), 240KiB/s-242KiB/s (246kB/s-248kB/s), io=40.0MiB (41.9MB), run=42234-42635msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=178833: Mon Aug 3 14:47:21 2026 read: IOPS=28, BW=114KiB/s (117kB/s)(5048KiB/44211msec) slat (usec): min=2475, max=95381, avg=20098.37, stdev=12803.93 clat (msec): min=92, max=847, avg=261.50, stdev=77.42 lat (msec): min=107, max=853, avg=281.61, stdev=80.28 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 153], 10.00th=[ 169], 20.00th=[ 197], | 30.00th=[ 222], 40.00th=[ 239], 50.00th=[ 253], 60.00th=[ 275], | 70.00th=[ 300], 80.00th=[ 321], 90.00th=[ 359], 95.00th=[ 384], | 99.00th=[ 447], 99.50th=[ 550], 99.90th=[ 760], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 63, max= 223, per=25.37%, avg=114.15, stdev=35.84, samples=87 iops : min= 15, max= 55, avg=28.07, stdev= 9.03, samples=87 write: IOPS=29, BW=117KiB/s (120kB/s)(5192KiB/44211msec); 0 zone resets slat (usec): min=50, max=21772, avg=302.37, stdev=1278.17 clat (msec): min=115, max=759, avg=268.15, stdev=72.89 lat (msec): min=115, max=760, avg=268.46, stdev=72.89 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 161], 10.00th=[ 180], 20.00th=[ 207], | 30.00th=[ 226], 40.00th=[ 247], 50.00th=[ 264], 60.00th=[ 279], | 70.00th=[ 300], 80.00th=[ 326], 90.00th=[ 363], 95.00th=[ 388], | 99.00th=[ 447], 99.50th=[ 485], 99.90th=[ 743], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 39, max= 239, per=25.11%, avg=117.75, stdev=32.92, samples=87 iops : min= 9, max= 59, avg=28.97, stdev= 8.27, samples=87 lat (msec) : 100=0.04%, 250=45.00%, 500=54.41%, 750=0.43%, 1000=0.12% cpu : usr=0.18%, sys=1.40%, ctx=4161, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178834: Mon Aug 3 14:47:21 2026 read: IOPS=28, BW=112KiB/s (115kB/s)(4976KiB/44276msec) slat (usec): min=2847, max=92470, avg=20058.79, stdev=12546.90 clat (msec): min=18, max=820, avg=260.82, stdev=77.26 lat (msec): min=46, max=828, avg=280.89, stdev=80.08 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 194], | 30.00th=[ 215], 40.00th=[ 236], 50.00th=[ 259], 60.00th=[ 279], | 70.00th=[ 296], 80.00th=[ 317], 90.00th=[ 351], 95.00th=[ 384], | 99.00th=[ 472], 99.50th=[ 510], 99.90th=[ 709], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 31, max= 199, per=24.94%, avg=112.22, stdev=33.20, samples=87 iops : min= 7, max= 49, avg=27.57, stdev= 8.34, samples=87 write: IOPS=29, BW=119KiB/s (122kB/s)(5264KiB/44276msec); 0 zone resets slat (usec): min=48, max=12338, avg=281.91, stdev=879.62 clat (msec): min=92, max=776, avg=271.29, stdev=75.54 lat (msec): min=92, max=776, avg=271.58, stdev=75.55 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 161], 10.00th=[ 182], 20.00th=[ 205], | 30.00th=[ 228], 40.00th=[ 245], 50.00th=[ 266], 60.00th=[ 288], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 372], 95.00th=[ 397], | 99.00th=[ 456], 99.50th=[ 502], 99.90th=[ 693], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 47, max= 222, per=25.44%, avg=119.30, stdev=34.76, samples=87 iops : min= 11, max= 55, avg=29.36, stdev= 8.74, samples=87 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=44.18%, 500=54.92% lat (msec) : 750=0.55%, 1000=0.08% cpu : usr=0.19%, sys=1.39%, ctx=4123, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178835: Mon Aug 3 14:47:21 2026 read: IOPS=27, BW=112KiB/s (115kB/s)(4984KiB/44542msec) slat (usec): min=55, max=98175, avg=20523.17, stdev=13392.33 clat (msec): min=67, max=585, avg=261.44, stdev=69.21 lat (msec): min=108, max=614, avg=281.97, stdev=72.75 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 197], | 30.00th=[ 222], 40.00th=[ 241], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 321], 90.00th=[ 355], 95.00th=[ 380], | 99.00th=[ 439], 99.50th=[ 460], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 46, max= 216, per=24.81%, avg=111.65, stdev=34.99, samples=88 iops : min= 11, max= 54, avg=27.43, stdev= 8.80, samples=88 write: IOPS=29, BW=118KiB/s (121kB/s)(5256KiB/44542msec); 0 zone resets slat (usec): min=52, max=21854, avg=332.81, stdev=1287.42 clat (msec): min=90, max=554, avg=274.26, stdev=70.08 lat (msec): min=90, max=554, avg=274.59, stdev=70.09 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 171], 10.00th=[ 186], 20.00th=[ 211], | 30.00th=[ 234], 40.00th=[ 253], 50.00th=[ 268], 60.00th=[ 288], | 70.00th=[ 313], 80.00th=[ 338], 90.00th=[ 368], 95.00th=[ 397], | 99.00th=[ 443], 99.50th=[ 460], 99.90th=[ 498], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 55, max= 214, per=24.97%, avg=117.11, stdev=30.98, samples=88 iops : min= 13, max= 53, avg=28.81, stdev= 7.73, samples=88 lat (msec) : 100=0.16%, 250=41.72%, 500=58.01%, 750=0.12% cpu : usr=0.18%, sys=1.35%, ctx=4284, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178836: Mon Aug 3 14:47:21 2026 read: IOPS=28, BW=114KiB/s (116kB/s)(5060KiB/44544msec) slat (msec): min=2, max=112, avg=20.27, stdev=13.26 clat (msec): min=24, max=504, avg=262.64, stdev=72.40 lat (msec): min=34, max=533, avg=282.91, stdev=75.52 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 197], | 30.00th=[ 218], 40.00th=[ 239], 50.00th=[ 257], 60.00th=[ 279], | 70.00th=[ 300], 80.00th=[ 321], 90.00th=[ 359], 95.00th=[ 388], | 99.00th=[ 456], 99.50th=[ 472], 99.90th=[ 489], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 39, max= 191, per=25.16%, avg=113.24, stdev=32.08, samples=88 iops : min= 9, max= 47, avg=27.83, stdev= 8.11, samples=88 write: IOPS=29, BW=116KiB/s (119kB/s)(5180KiB/44544msec); 0 zone resets slat (usec): min=51, max=43956, avg=347.72, stdev=1675.23 clat (msec): min=102, max=513, avg=270.83, stdev=72.98 lat (msec): min=102, max=513, avg=271.18, stdev=73.08 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 165], 10.00th=[ 182], 20.00th=[ 205], | 30.00th=[ 226], 40.00th=[ 247], 50.00th=[ 266], 60.00th=[ 288], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 368], 95.00th=[ 397], | 99.00th=[ 460], 99.50th=[ 477], 99.90th=[ 502], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 47, max= 223, per=24.62%, avg=115.48, stdev=36.08, samples=88 iops : min= 11, max= 55, avg=28.41, stdev= 9.02, samples=88 lat (msec) : 50=0.04%, 100=0.04%, 250=44.22%, 500=55.59%, 750=0.12% cpu : usr=0.20%, sys=1.36%, ctx=4139, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=451KiB/s (461kB/s), 112KiB/s-114KiB/s (115kB/s-117kB/s), io=19.6MiB (20.5MB), run=44211-44544msec WRITE: bw=469KiB/s (480kB/s), 116KiB/s-119KiB/s (119kB/s-122kB/s), io=20.4MiB (21.4MB), run=44211-44544msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=178848: Mon Aug 3 14:47:42 2026 read: IOPS=1, BW=63.2MiB/s (66.3MB/s)(40.0MiB/633msec) slat (nsec): min=28174k, max=28174k, avg=28173630.00, stdev= 0.00 clat (nsec): min=563273k, max=563273k, avg=563273180.00, stdev= 0.00 lat (nsec): min=591450k, max=591450k, avg=591450030.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 750=100.00% cpu : usr=0.16%, sys=2.06%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.2MiB/s (66.3MB/s), 63.2MiB/s-63.2MiB/s (66.3MB/s-66.3MB/s), io=40.0MiB (41.9MB), run=633-633msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout