== sanity test 398c: run fio to test AIO ================= 10:59:20 (1786373960) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.14275 s, 294 MB/s osc.lustre-OST0000-osc-ffff8a450c671000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183963: Mon Aug 10 10:59:30 2026 write: IOPS=415, BW=1660KiB/s (1700kB/s)(10.0MiB/6168msec); 0 zone resets slat (usec): min=35, max=2105, avg=190.51, stdev=129.67 clat (msec): min=18, max=110, avg=38.01, stdev=11.97 lat (msec): min=18, max=110, avg=38.20, stdev=11.99 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 33], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 37], | 70.00th=[ 39], 80.00th=[ 42], 90.00th=[ 52], 95.00th=[ 64], | 99.00th=[ 88], 99.50th=[ 101], 99.90th=[ 109], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 966, max= 2000, per=24.88%, avg=1651.83, stdev=364.57, samples=12 iops : min= 241, max= 500, avg=412.92, stdev=91.23, samples=12 lat (msec) : 20=0.20%, 50=88.32%, 100=10.98%, 250=0.51% cpu : usr=0.50%, sys=5.79%, ctx=2528, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183964: Mon Aug 10 10:59:30 2026 write: IOPS=415, BW=1661KiB/s (1701kB/s)(10.0MiB/6166msec); 0 zone resets slat (usec): min=42, max=28408, avg=204.46, stdev=575.67 clat (msec): min=4, max=107, avg=38.04, stdev=11.93 lat (msec): min=4, max=107, avg=38.24, stdev=11.94 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 33], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 37], | 70.00th=[ 39], 80.00th=[ 42], 90.00th=[ 53], 95.00th=[ 64], | 99.00th=[ 87], 99.50th=[ 99], 99.90th=[ 107], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 974, max= 1992, per=24.89%, avg=1653.00, stdev=360.38, samples=12 iops : min= 243, max= 498, avg=413.17, stdev=90.15, samples=12 lat (msec) : 10=0.08%, 20=0.23%, 50=87.97%, 100=11.25%, 250=0.47% cpu : usr=0.34%, sys=6.47%, ctx=2576, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183965: Mon Aug 10 10:59:30 2026 write: IOPS=418, BW=1675KiB/s (1715kB/s)(10.0MiB/6115msec); 0 zone resets slat (usec): min=35, max=3096, avg=186.94, stdev=133.53 clat (msec): min=3, max=110, avg=37.98, stdev=12.16 lat (msec): min=4, max=110, avg=38.17, stdev=12.18 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 37], | 70.00th=[ 40], 80.00th=[ 42], 90.00th=[ 53], 95.00th=[ 64], | 99.00th=[ 85], 99.50th=[ 101], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 990, max= 2086, per=24.81%, avg=1647.64, stdev=381.79, samples=11 iops : min= 247, max= 521, avg=411.82, stdev=95.48, samples=11 lat (msec) : 4=0.04%, 10=0.31%, 20=0.43%, 50=87.50%, 100=11.21% lat (msec) : 250=0.51% cpu : usr=0.54%, sys=5.63%, ctx=2653, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183966: Mon Aug 10 10:59:30 2026 write: IOPS=415, BW=1661KiB/s (1701kB/s)(10.0MiB/6166msec); 0 zone resets slat (usec): min=49, max=3212, avg=195.55, stdev=136.95 clat (msec): min=18, max=109, avg=38.02, stdev=11.95 lat (msec): min=19, max=110, avg=38.22, stdev=11.97 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 37], | 70.00th=[ 39], 80.00th=[ 42], 90.00th=[ 52], 95.00th=[ 65], | 99.00th=[ 88], 99.50th=[ 104], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 974, max= 1984, per=24.92%, avg=1654.75, stdev=359.29, samples=12 iops : min= 243, max= 496, avg=413.58, stdev=89.86, samples=12 lat (msec) : 20=0.12%, 50=88.28%, 100=11.05%, 250=0.55% cpu : usr=0.54%, sys=6.37%, ctx=2615, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=6641KiB/s (6800kB/s), 1660KiB/s-1675KiB/s (1700kB/s-1715kB/s), io=40.0MiB (41.9MB), run=6115-6168msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183976: Mon Aug 10 11:00:13 2026 read: IOPS=32, BW=130KiB/s (133kB/s)(5048KiB/38868msec) slat (usec): min=83, max=553056, avg=19465.10, stdev=29574.44 clat (msec): min=11, max=835, avg=228.94, stdev=124.77 lat (msec): min=19, max=857, avg=248.41, stdev=132.30 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 132], | 30.00th=[ 150], 40.00th=[ 169], 50.00th=[ 199], 60.00th=[ 232], | 70.00th=[ 259], 80.00th=[ 305], 90.00th=[ 384], 95.00th=[ 485], | 99.00th=[ 684], 99.50th=[ 701], 99.90th=[ 768], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 40, max= 288, per=24.86%, avg=128.25, stdev=56.34, samples=75 iops : min= 10, max= 72, avg=31.61, stdev=14.16, samples=75 write: IOPS=33, BW=134KiB/s (137kB/s)(5192KiB/38868msec); 0 zone resets slat (usec): min=53, max=22861, avg=407.16, stdev=1307.68 clat (msec): min=11, max=857, avg=235.24, stdev=128.24 lat (msec): min=11, max=858, avg=235.65, stdev=128.31 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 108], 10.00th=[ 120], 20.00th=[ 134], | 30.00th=[ 153], 40.00th=[ 176], 50.00th=[ 201], 60.00th=[ 234], | 70.00th=[ 266], 80.00th=[ 309], 90.00th=[ 409], 95.00th=[ 506], | 99.00th=[ 693], 99.50th=[ 785], 99.90th=[ 852], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 21, max= 288, per=24.77%, avg=133.04, stdev=61.80, samples=75 iops : min= 5, max= 72, avg=32.83, stdev=15.52, samples=75 lat (msec) : 20=0.20%, 50=0.55%, 100=3.95%, 250=61.52%, 500=28.79% lat (msec) : 750=4.49%, 1000=0.51% cpu : usr=0.17%, sys=1.67%, ctx=4309, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183977: Mon Aug 10 11:00:13 2026 read: IOPS=32, BW=129KiB/s (132kB/s)(4976KiB/38642msec) slat (msec): min=2, max=551, avg=19.47, stdev=26.13 clat (msec): min=22, max=765, avg=231.30, stdev=122.96 lat (msec): min=34, max=773, avg=250.78, stdev=130.88 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 101], 10.00th=[ 113], 20.00th=[ 131], | 30.00th=[ 150], 40.00th=[ 174], 50.00th=[ 201], 60.00th=[ 228], | 70.00th=[ 266], 80.00th=[ 313], 90.00th=[ 393], 95.00th=[ 477], | 99.00th=[ 676], 99.50th=[ 709], 99.90th=[ 743], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 36, max= 296, per=24.94%, avg=128.67, stdev=56.33, samples=75 iops : min= 9, max= 74, avg=31.72, stdev=14.18, samples=75 write: IOPS=34, BW=136KiB/s (139kB/s)(5264KiB/38642msec); 0 zone resets slat (usec): min=52, max=36495, avg=481.55, stdev=2097.16 clat (msec): min=29, max=786, avg=231.21, stdev=122.59 lat (msec): min=29, max=786, avg=231.69, stdev=122.76 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 111], 10.00th=[ 120], 20.00th=[ 136], | 30.00th=[ 150], 40.00th=[ 176], 50.00th=[ 203], 60.00th=[ 232], | 70.00th=[ 253], 80.00th=[ 296], 90.00th=[ 388], 95.00th=[ 477], | 99.00th=[ 693], 99.50th=[ 751], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 39, max= 296, per=25.45%, avg=136.64, stdev=62.05, samples=75 iops : min= 9, max= 74, avg=33.71, stdev=15.61, samples=75 lat (msec) : 50=0.08%, 100=3.40%, 250=63.91%, 500=28.20%, 750=4.06% lat (msec) : 1000=0.35% cpu : usr=0.25%, sys=1.89%, ctx=4237, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183978: Mon Aug 10 11:00:13 2026 read: IOPS=32, BW=129KiB/s (132kB/s)(4984KiB/38662msec) slat (usec): min=57, max=524527, avg=19424.53, stdev=26375.49 clat (msec): min=19, max=891, avg=230.83, stdev=126.10 lat (msec): min=22, max=900, avg=250.26, stdev=133.09 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 112], 20.00th=[ 131], | 30.00th=[ 148], 40.00th=[ 171], 50.00th=[ 205], 60.00th=[ 232], | 70.00th=[ 264], 80.00th=[ 313], 90.00th=[ 393], 95.00th=[ 456], | 99.00th=[ 693], 99.50th=[ 776], 99.90th=[ 860], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 288, per=25.04%, avg=129.23, stdev=57.82, samples=75 iops : min= 1, max= 72, avg=31.84, stdev=14.56, samples=75 write: IOPS=33, BW=136KiB/s (139kB/s)(5256KiB/38662msec); 0 zone resets slat (usec): min=38, max=62810, avg=516.40, stdev=2245.70 clat (msec): min=48, max=830, avg=232.54, stdev=121.66 lat (msec): min=49, max=831, avg=233.06, stdev=121.84 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 111], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 153], 40.00th=[ 174], 50.00th=[ 201], 60.00th=[ 230], | 70.00th=[ 259], 80.00th=[ 309], 90.00th=[ 380], 95.00th=[ 481], | 99.00th=[ 676], 99.50th=[ 776], 99.90th=[ 802], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 40, max= 296, per=25.29%, avg=135.83, stdev=65.23, samples=75 iops : min= 10, max= 74, avg=33.51, stdev=16.36, samples=75 lat (msec) : 20=0.04%, 50=0.27%, 100=3.75%, 250=62.73%, 500=29.26% lat (msec) : 750=3.32%, 1000=0.62% cpu : usr=0.20%, sys=1.73%, ctx=4188, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183979: Mon Aug 10 11:00:13 2026 read: IOPS=32, BW=130KiB/s (134kB/s)(5060KiB/38787msec) slat (msec): min=2, max=535, avg=18.15, stdev=22.63 clat (msec): min=15, max=994, avg=231.04, stdev=125.94 lat (msec): min=31, max=1016, avg=249.20, stdev=132.03 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 106], 10.00th=[ 117], 20.00th=[ 136], | 30.00th=[ 153], 40.00th=[ 171], 50.00th=[ 197], 60.00th=[ 226], | 70.00th=[ 259], 80.00th=[ 309], 90.00th=[ 397], 95.00th=[ 477], | 99.00th=[ 701], 99.50th=[ 802], 99.90th=[ 919], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 38, max= 272, per=25.19%, avg=129.96, stdev=55.49, samples=75 iops : min= 9, max= 68, avg=32.03, stdev=14.00, samples=75 write: IOPS=33, BW=134KiB/s (137kB/s)(5180KiB/38787msec); 0 zone resets slat (usec): min=53, max=13466, avg=383.85, stdev=1052.28 clat (msec): min=29, max=993, avg=234.27, stdev=130.69 lat (msec): min=30, max=994, avg=234.66, stdev=130.78 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 109], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 157], 40.00th=[ 176], 50.00th=[ 199], 60.00th=[ 228], | 70.00th=[ 257], 80.00th=[ 296], 90.00th=[ 405], 95.00th=[ 485], | 99.00th=[ 726], 99.50th=[ 793], 99.90th=[ 978], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 15, max= 320, per=24.81%, avg=133.21, stdev=65.29, samples=75 iops : min= 3, max= 80, avg=32.89, stdev=16.40, samples=75 lat (msec) : 20=0.04%, 50=0.27%, 100=2.58%, 250=65.00%, 500=27.42% lat (msec) : 750=3.95%, 1000=0.74% cpu : usr=0.25%, sys=2.15%, ctx=4327, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=516KiB/s (529kB/s), 129KiB/s-130KiB/s (132kB/s-134kB/s), io=19.6MiB (20.5MB), run=38642-38868msec WRITE: bw=538KiB/s (550kB/s), 134KiB/s-136KiB/s (137kB/s-139kB/s), io=20.4MiB (21.4MB), run=38642-38868msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183989: Mon Aug 10 11:00:21 2026 read: IOPS=1, BW=58.5MiB/s (61.3MB/s)(40.0MiB/684msec) slat (nsec): min=22580k, max=22580k, avg=22579959.00, stdev= 0.00 clat (nsec): min=611958k, max=611958k, avg=611957544.00, stdev= 0.00 lat (nsec): min=634543k, max=634543k, avg=634543031.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 609], 10.00th=[ 609], 20.00th=[ 609], | 30.00th=[ 609], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 609], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.61%, ctx=9, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=58.5MiB/s (61.3MB/s), 58.5MiB/s-58.5MiB/s (61.3MB/s-61.3MB/s), io=40.0MiB (41.9MB), run=684-684msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout