== sanity test 398c: run fio to test AIO ================= 13:04:08 (1786467848) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.186478 s, 225 MB/s osc.lustre-OST0000-osc-ffff9e49054be800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=191869: Tue Aug 11 13:04:23 2026 write: IOPS=230, BW=923KiB/s (945kB/s)(10.0MiB/11094msec); 0 zone resets slat (usec): min=35, max=6874, avg=210.09, stdev=334.07 clat (msec): min=7, max=120, avg=68.66, stdev=14.67 lat (msec): min=7, max=120, avg=68.88, stdev=14.68 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 89], 95.00th=[ 97], | 99.00th=[ 106], 99.50th=[ 109], 99.90th=[ 112], 99.95th=[ 118], | 99.99th=[ 121] bw ( KiB/s): min= 694, max= 1137, per=25.00%, avg=923.05, stdev=125.68, samples=21 iops : min= 173, max= 284, avg=230.57, stdev=31.47, samples=21 lat (msec) : 10=0.04%, 50=9.77%, 100=86.41%, 250=3.79% cpu : usr=0.31%, sys=3.25%, ctx=2716, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=191870: Tue Aug 11 13:04:23 2026 write: IOPS=231, BW=927KiB/s (949kB/s)(10.0MiB/11044msec); 0 zone resets slat (usec): min=37, max=5418, avg=203.02, stdev=316.29 clat (msec): min=15, max=118, avg=68.74, stdev=14.74 lat (msec): min=15, max=119, avg=68.94, stdev=14.76 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 58], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 89], 95.00th=[ 96], | 99.00th=[ 106], 99.50th=[ 112], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 118] bw ( KiB/s): min= 662, max= 1137, per=25.14%, avg=928.05, stdev=135.23, samples=21 iops : min= 165, max= 284, avg=231.81, stdev=33.83, samples=21 lat (msec) : 20=0.08%, 50=9.02%, 100=87.77%, 250=3.12% cpu : usr=0.39%, sys=2.85%, ctx=2659, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=191871: Tue Aug 11 13:04:23 2026 write: IOPS=231, BW=928KiB/s (950kB/s)(10.0MiB/11037msec); 0 zone resets slat (usec): min=35, max=7484, avg=208.57, stdev=315.87 clat (msec): min=15, max=115, avg=68.55, stdev=14.79 lat (msec): min=16, max=116, avg=68.76, stdev=14.79 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 46], 10.00th=[ 50], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 69], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 88], 95.00th=[ 95], | 99.00th=[ 105], 99.50th=[ 109], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 640, max= 1152, per=25.11%, avg=926.95, stdev=134.37, samples=21 iops : min= 160, max= 288, avg=231.57, stdev=33.61, samples=21 lat (msec) : 20=0.16%, 50=10.62%, 100=86.25%, 250=2.97% cpu : usr=0.34%, sys=2.96%, ctx=2689, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=191872: Tue Aug 11 13:04:23 2026 write: IOPS=231, BW=924KiB/s (946kB/s)(10.0MiB/11079msec); 0 zone resets slat (usec): min=36, max=30130, avg=226.89, stdev=663.04 clat (msec): min=9, max=122, avg=68.59, stdev=14.85 lat (msec): min=10, max=122, avg=68.82, stdev=14.83 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 89], 95.00th=[ 95], | 99.00th=[ 108], 99.50th=[ 117], 99.90th=[ 122], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 688, max= 1152, per=25.06%, avg=925.24, stdev=126.59, samples=21 iops : min= 172, max= 288, avg=231.14, stdev=31.69, samples=21 lat (msec) : 10=0.04%, 20=0.12%, 50=9.57%, 100=87.62%, 250=2.66% cpu : usr=0.50%, sys=3.17%, ctx=2784, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3692KiB/s (3781kB/s), 923KiB/s-928KiB/s (945kB/s-950kB/s), io=40.0MiB (41.9MB), run=11037-11094msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=191884: Tue Aug 11 13:04:39 2026 read: IOPS=105, BW=421KiB/s (431kB/s)(5048KiB/11985msec) slat (usec): min=1426, max=23370, avg=5026.96, stdev=2478.01 clat (msec): min=20, max=144, avg=70.73, stdev=15.52 lat (msec): min=30, max=147, avg=75.76, stdev=15.90 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 78], 80.00th=[ 83], 90.00th=[ 91], 95.00th=[ 99], | 99.00th=[ 115], 99.50th=[ 124], 99.90th=[ 142], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 287, max= 520, per=25.27%, avg=423.04, stdev=55.71, samples=23 iops : min= 71, max= 130, avg=105.52, stdev=14.06, samples=23 write: IOPS=108, BW=433KiB/s (444kB/s)(5192KiB/11985msec); 0 zone resets slat (usec): min=39, max=5693, avg=194.14, stdev=305.84 clat (msec): min=38, max=122, avg=72.84, stdev=13.96 lat (msec): min=38, max=122, avg=73.03, stdev=13.95 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 92], 95.00th=[ 101], | 99.00th=[ 110], 99.50th=[ 112], 99.90th=[ 122], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 336, max= 544, per=25.09%, avg=437.30, stdev=55.30, samples=23 iops : min= 84, max= 136, avg=109.09, stdev=13.88, samples=23 lat (msec) : 50=4.73%, 100=90.82%, 250=4.45% cpu : usr=0.33%, sys=3.19%, ctx=4669, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=191885: Tue Aug 11 13:04:39 2026 read: IOPS=105, BW=422KiB/s (432kB/s)(4976KiB/11802msec) slat (usec): min=54, max=21440, avg=5033.69, stdev=2442.84 clat (msec): min=7, max=175, avg=69.85, stdev=16.08 lat (msec): min=18, max=183, avg=74.89, stdev=16.53 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 77], 80.00th=[ 82], 90.00th=[ 89], 95.00th=[ 96], | 99.00th=[ 115], 99.50th=[ 138], 99.90th=[ 171], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 311, max= 504, per=25.22%, avg=422.26, stdev=55.67, samples=23 iops : min= 77, max= 126, avg=105.30, stdev=14.04, samples=23 write: IOPS=111, BW=446KiB/s (457kB/s)(5264KiB/11802msec); 0 zone resets slat (usec): min=36, max=5465, avg=198.83, stdev=306.41 clat (msec): min=32, max=160, avg=72.23, stdev=14.06 lat (msec): min=33, max=160, avg=72.43, stdev=14.06 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 75], | 70.00th=[ 79], 80.00th=[ 83], 90.00th=[ 90], 95.00th=[ 96], | 99.00th=[ 113], 99.50th=[ 120], 99.90th=[ 150], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 360, max= 581, per=25.78%, avg=449.39, stdev=56.93, samples=23 iops : min= 90, max= 145, avg=112.09, stdev=14.26, samples=23 lat (msec) : 10=0.04%, 20=0.08%, 50=4.26%, 100=91.80%, 250=3.83% cpu : usr=0.29%, sys=3.30%, ctx=4661, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=191886: Tue Aug 11 13:04:39 2026 read: IOPS=104, BW=417KiB/s (427kB/s)(4984KiB/11956msec) slat (usec): min=1338, max=19855, avg=5034.66, stdev=2351.05 clat (msec): min=25, max=136, avg=70.15, stdev=14.58 lat (msec): min=28, max=143, avg=75.19, stdev=15.06 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 69], 60.00th=[ 73], | 70.00th=[ 78], 80.00th=[ 83], 90.00th=[ 89], 95.00th=[ 96], | 99.00th=[ 110], 99.50th=[ 112], 99.90th=[ 130], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 335, max= 520, per=25.12%, avg=420.43, stdev=60.49, samples=23 iops : min= 83, max= 130, avg=104.87, stdev=15.20, samples=23 write: IOPS=109, BW=440KiB/s (450kB/s)(5256KiB/11956msec); 0 zone resets slat (usec): min=42, max=3475, avg=183.22, stdev=186.29 clat (msec): min=38, max=144, avg=73.04, stdev=14.16 lat (msec): min=39, max=144, avg=73.23, stdev=14.16 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 92], 95.00th=[ 99], | 99.00th=[ 111], 99.50th=[ 115], 99.90th=[ 144], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 360, max= 552, per=25.31%, avg=441.13, stdev=45.53, samples=23 iops : min= 90, max= 138, avg=110.04, stdev=11.42, samples=23 lat (msec) : 50=4.10%, 100=92.30%, 250=3.59% cpu : usr=0.23%, sys=3.37%, ctx=4654, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=191887: Tue Aug 11 13:04:39 2026 read: IOPS=105, BW=423KiB/s (433kB/s)(5060KiB/11966msec) slat (usec): min=1413, max=18582, avg=5041.14, stdev=2308.96 clat (msec): min=19, max=132, avg=71.01, stdev=14.89 lat (msec): min=23, max=141, avg=76.06, stdev=15.31 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 79], 80.00th=[ 83], 90.00th=[ 90], 95.00th=[ 97], | 99.00th=[ 114], 99.50th=[ 123], 99.90th=[ 130], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 352, max= 528, per=25.45%, avg=426.09, stdev=51.51, samples=23 iops : min= 88, max= 132, avg=106.26, stdev=12.97, samples=23 write: IOPS=108, BW=433KiB/s (443kB/s)(5180KiB/11966msec); 0 zone resets slat (usec): min=50, max=5764, avg=193.82, stdev=296.63 clat (msec): min=39, max=165, avg=72.53, stdev=14.60 lat (msec): min=39, max=165, avg=72.73, stdev=14.62 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 79], 80.00th=[ 83], 90.00th=[ 91], 95.00th=[ 97], | 99.00th=[ 118], 99.50th=[ 127], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 295, max= 558, per=24.91%, avg=434.26, stdev=61.18, samples=23 iops : min= 73, max= 139, avg=108.30, stdev=15.38, samples=23 lat (msec) : 20=0.04%, 50=4.26%, 100=91.91%, 250=3.79% cpu : usr=0.32%, sys=3.32%, ctx=4754, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1674KiB/s (1715kB/s), 417KiB/s-423KiB/s (427kB/s-433kB/s), io=19.6MiB (20.5MB), run=11802-11985msec WRITE: bw=1743KiB/s (1785kB/s), 433KiB/s-446KiB/s (443kB/s-457kB/s), io=20.4MiB (21.4MB), run=11802-11985msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=191892: Tue Aug 11 13:04:43 2026 read: IOPS=7, BW=286MiB/s (300MB/s)(40.0MiB/140msec) slat (nsec): min=12154k, max=12154k, avg=12154380.00, stdev= 0.00 clat (nsec): min=121082k, max=121082k, avg=121082452.00, stdev= 0.00 lat (nsec): min=133257k, max=133257k, avg=133257013.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 122], 50.00th=[ 122], 60.00th=[ 122], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.19%, ctx=10, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=286MiB/s (300MB/s), 286MiB/s-286MiB/s (300MB/s-300MB/s), io=40.0MiB (41.9MB), run=140-140msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout