== sanity test 398c: run fio to test AIO ================= 09:43:57 (1788615837) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.370161 s, 113 MB/s osc.lustre-OST0000-osc-ffff95b3186ff800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179676: Sat Sep 5 09:44:29 2026 write: IOPS=109, BW=438KiB/s (448kB/s)(10.0MiB/23400msec); 0 zone resets slat (usec): min=21, max=8314, avg=134.37, stdev=489.60 clat (msec): min=28, max=283, avg=145.95, stdev=34.45 lat (msec): min=28, max=283, avg=146.09, stdev=34.44 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 99], 10.00th=[ 107], 20.00th=[ 118], | 30.00th=[ 126], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 176], 90.00th=[ 192], 95.00th=[ 207], | 99.00th=[ 245], 99.50th=[ 259], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 250, max= 611, per=25.15%, avg=437.57, stdev=74.97, samples=46 iops : min= 62, max= 152, avg=109.02, stdev=18.71, samples=46 lat (msec) : 50=0.08%, 100=5.59%, 250=93.55%, 500=0.78% cpu : usr=0.14%, sys=0.79%, ctx=1271, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179677: Sat Sep 5 09:44:29 2026 write: IOPS=108, BW=435KiB/s (446kB/s)(10.0MiB/23536msec); 0 zone resets slat (usec): min=21, max=12207, avg=156.95, stdev=585.35 clat (msec): min=19, max=304, avg=146.28, stdev=33.54 lat (msec): min=19, max=304, avg=146.44, stdev=33.51 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 99], 10.00th=[ 108], 20.00th=[ 121], | 30.00th=[ 129], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 174], 90.00th=[ 192], 95.00th=[ 209], | 99.00th=[ 239], 99.50th=[ 262], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 305] bw ( KiB/s): min= 242, max= 576, per=24.93%, avg=433.74, stdev=77.28, samples=46 iops : min= 60, max= 144, avg=108.04, stdev=19.34, samples=46 lat (msec) : 20=0.04%, 100=5.94%, 250=93.40%, 500=0.62% cpu : usr=0.22%, sys=0.88%, ctx=1992, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179678: Sat Sep 5 09:44:29 2026 write: IOPS=108, BW=436KiB/s (446kB/s)(10.0MiB/23504msec); 0 zone resets slat (usec): min=22, max=14622, avg=193.93, stdev=687.43 clat (msec): min=19, max=289, avg=145.99, stdev=33.29 lat (msec): min=20, max=289, avg=146.18, stdev=33.28 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 120], | 30.00th=[ 128], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 174], 90.00th=[ 188], 95.00th=[ 211], | 99.00th=[ 247], 99.50th=[ 253], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 292] bw ( KiB/s): min= 262, max= 592, per=24.97%, avg=434.43, stdev=75.60, samples=46 iops : min= 65, max= 148, avg=108.24, stdev=18.90, samples=46 lat (msec) : 20=0.04%, 50=0.16%, 100=5.82%, 250=93.20%, 500=0.78% cpu : usr=0.28%, sys=1.12%, ctx=2859, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179679: Sat Sep 5 09:44:29 2026 write: IOPS=108, BW=436KiB/s (446kB/s)(10.0MiB/23509msec); 0 zone resets slat (usec): min=20, max=10934, avg=175.94, stdev=634.12 clat (msec): min=18, max=280, avg=145.96, stdev=33.54 lat (msec): min=19, max=281, avg=146.14, stdev=33.56 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 100], 10.00th=[ 108], 20.00th=[ 121], | 30.00th=[ 129], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 174], 90.00th=[ 190], 95.00th=[ 209], | 99.00th=[ 245], 99.50th=[ 251], 99.90th=[ 264], 99.95th=[ 268], | 99.99th=[ 279] bw ( KiB/s): min= 254, max= 568, per=24.87%, avg=432.67, stdev=71.77, samples=46 iops : min= 63, max= 142, avg=107.78, stdev=17.92, samples=46 lat (msec) : 20=0.08%, 50=0.39%, 100=4.96%, 250=93.98%, 500=0.59% cpu : usr=0.18%, sys=1.03%, ctx=2865, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1740KiB/s (1782kB/s), 435KiB/s-438KiB/s (446kB/s-448kB/s), io=40.0MiB (41.9MB), run=23400-23536msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179692: Sat Sep 5 09:45:05 2026 read: IOPS=43, BW=173KiB/s (177kB/s)(5048KiB/29220msec) slat (usec): min=2350, max=65169, avg=12821.43, stdev=8151.78 clat (msec): min=75, max=376, avg=171.14, stdev=49.59 lat (msec): min=81, max=412, avg=183.96, stdev=52.39 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 100], 10.00th=[ 112], 20.00th=[ 129], | 30.00th=[ 142], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 236], 95.00th=[ 264], | 99.00th=[ 317], 99.50th=[ 342], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 87, max= 296, per=25.07%, avg=171.95, stdev=44.97, samples=57 iops : min= 21, max= 74, avg=42.58, stdev=11.39, samples=57 write: IOPS=44, BW=178KiB/s (182kB/s)(5192KiB/29220msec); 0 zone resets slat (usec): min=24, max=20636, avg=206.99, stdev=784.42 clat (msec): min=25, max=394, avg=178.51, stdev=49.88 lat (msec): min=25, max=394, avg=178.72, stdev=49.89 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 104], 10.00th=[ 116], 20.00th=[ 136], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 218], 90.00th=[ 245], 95.00th=[ 268], | 99.00th=[ 321], 99.50th=[ 338], 99.90th=[ 372], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 71, max= 311, per=24.88%, avg=177.67, stdev=52.57, samples=57 iops : min= 17, max= 77, avg=44.04, stdev=13.23, samples=57 lat (msec) : 50=0.04%, 100=4.18%, 250=87.81%, 500=7.97% cpu : usr=0.24%, sys=1.64%, ctx=4301, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179693: Sat Sep 5 09:45:05 2026 read: IOPS=42, BW=171KiB/s (175kB/s)(4976KiB/29146msec) slat (usec): min=51, max=71840, avg=12988.83, stdev=7833.89 clat (msec): min=17, max=392, avg=172.08, stdev=51.97 lat (msec): min=27, max=426, avg=185.08, stdev=54.11 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 100], 10.00th=[ 112], 20.00th=[ 128], | 30.00th=[ 142], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 268], | 99.00th=[ 338], 99.50th=[ 368], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 71, max= 304, per=24.84%, avg=170.39, stdev=51.97, samples=57 iops : min= 17, max= 76, avg=42.19, stdev=13.13, samples=57 write: IOPS=45, BW=181KiB/s (185kB/s)(5264KiB/29146msec); 0 zone resets slat (usec): min=26, max=16736, avg=236.37, stdev=965.61 clat (msec): min=33, max=375, avg=178.36, stdev=49.01 lat (msec): min=33, max=375, avg=178.60, stdev=49.00 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 134], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 241], 95.00th=[ 268], | 99.00th=[ 313], 99.50th=[ 321], 99.90th=[ 363], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 94, max= 288, per=25.35%, avg=180.98, stdev=44.64, samples=57 iops : min= 23, max= 72, avg=44.86, stdev=11.23, samples=57 lat (msec) : 20=0.04%, 50=0.12%, 100=4.18%, 250=88.71%, 500=6.95% cpu : usr=0.27%, sys=1.77%, ctx=4226, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179694: Sat Sep 5 09:45:05 2026 read: IOPS=42, BW=171KiB/s (175kB/s)(4984KiB/29175msec) slat (usec): min=2138, max=69754, avg=12841.49, stdev=7683.82 clat (msec): min=27, max=382, avg=170.34, stdev=48.89 lat (msec): min=34, max=415, avg=183.18, stdev=51.53 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 103], 10.00th=[ 113], 20.00th=[ 129], | 30.00th=[ 142], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 209], 90.00th=[ 239], 95.00th=[ 262], | 99.00th=[ 300], 99.50th=[ 334], 99.90th=[ 376], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 78, max= 296, per=24.95%, avg=171.12, stdev=50.28, samples=57 iops : min= 19, max= 74, avg=42.39, stdev=12.74, samples=57 write: IOPS=45, BW=180KiB/s (184kB/s)(5256KiB/29175msec); 0 zone resets slat (usec): min=21, max=18291, avg=246.13, stdev=1002.63 clat (msec): min=18, max=415, avg=179.02, stdev=49.79 lat (msec): min=18, max=415, avg=179.26, stdev=49.84 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 107], 10.00th=[ 117], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 241], 95.00th=[ 271], | 99.00th=[ 317], 99.50th=[ 342], 99.90th=[ 405], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 103, max= 304, per=25.17%, avg=179.70, stdev=49.31, samples=57 iops : min= 25, max= 76, avg=44.53, stdev=12.39, samples=57 lat (msec) : 20=0.04%, 50=0.04%, 100=3.67%, 250=88.28%, 500=7.97% cpu : usr=0.28%, sys=1.81%, ctx=4573, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179695: Sat Sep 5 09:45:05 2026 read: IOPS=43, BW=173KiB/s (177kB/s)(5060KiB/29224msec) slat (usec): min=2372, max=66518, avg=13127.90, stdev=8081.12 clat (msec): min=51, max=367, avg=169.98, stdev=49.14 lat (msec): min=63, max=398, avg=183.11, stdev=51.90 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 100], 10.00th=[ 111], 20.00th=[ 127], | 30.00th=[ 140], 40.00th=[ 153], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 234], 95.00th=[ 259], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 359], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 62, max= 288, per=25.20%, avg=172.89, stdev=51.48, samples=57 iops : min= 15, max= 72, avg=42.82, stdev=12.97, samples=57 write: IOPS=44, BW=177KiB/s (182kB/s)(5180KiB/29224msec); 0 zone resets slat (usec): min=22, max=17160, avg=195.87, stdev=722.38 clat (msec): min=34, max=384, avg=179.69, stdev=48.53 lat (msec): min=35, max=384, avg=179.89, stdev=48.54 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 105], 10.00th=[ 118], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 239], 95.00th=[ 264], | 99.00th=[ 313], 99.50th=[ 338], 99.90th=[ 376], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 87, max= 304, per=24.74%, avg=176.67, stdev=45.32, samples=57 iops : min= 21, max= 76, avg=43.79, stdev=11.43, samples=57 lat (msec) : 50=0.04%, 100=4.26%, 250=88.83%, 500=6.88% cpu : usr=0.31%, sys=1.68%, ctx=4014, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=687KiB/s (703kB/s), 171KiB/s-173KiB/s (175kB/s-177kB/s), io=19.6MiB (20.5MB), run=29146-29224msec WRITE: bw=715KiB/s (732kB/s), 177KiB/s-181KiB/s (182kB/s-185kB/s), io=20.4MiB (21.4MB), run=29146-29224msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179704: Sat Sep 5 09:45:15 2026 read: IOPS=1, BW=72.3MiB/s (75.8MB/s)(40.0MiB/553msec) slat (nsec): min=63058k, max=63058k, avg=63057897.00, stdev= 0.00 clat (nsec): min=443556k, max=443556k, avg=443555502.00, stdev= 0.00 lat (nsec): min=506618k, max=506618k, avg=506618229.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 443], 10.00th=[ 443], 20.00th=[ 443], | 30.00th=[ 443], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] lat (msec) : 500=100.00% cpu : usr=0.18%, sys=2.36%, ctx=7, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=72.3MiB/s (75.8MB/s), 72.3MiB/s-72.3MiB/s (75.8MB/s-75.8MB/s), io=40.0MiB (41.9MB), run=553-553msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout