== sanity test 398c: run fio to test AIO ================= 12:26:12 (1781627172) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.161485 s, 260 MB/s osc.lustre-OST0000-osc-ffff96ff20146800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180949: Tue Jun 16 12:26:25 2026 write: IOPS=279, BW=1118KiB/s (1144kB/s)(10.0MiB/9163msec); 0 zone resets slat (usec): min=48, max=27223, avg=229.32, stdev=561.89 clat (msec): min=11, max=114, avg=56.67, stdev=15.84 lat (msec): min=11, max=114, avg=56.91, stdev=15.82 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 42], | 30.00th=[ 47], 40.00th=[ 51], 50.00th=[ 56], 60.00th=[ 61], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 79], 95.00th=[ 85], | 99.00th=[ 97], 99.50th=[ 106], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 115] bw ( KiB/s): min= 816, max= 1572, per=24.85%, avg=1110.33, stdev=220.65, samples=18 iops : min= 204, max= 393, avg=277.56, stdev=55.18, samples=18 lat (msec) : 20=0.16%, 50=39.77%, 100=59.18%, 250=0.90% cpu : usr=0.60%, sys=4.62%, ctx=2677, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180950: Tue Jun 16 12:26:25 2026 write: IOPS=280, BW=1123KiB/s (1150kB/s)(10.0MiB/9118msec); 0 zone resets slat (usec): min=41, max=3576, avg=211.14, stdev=200.88 clat (msec): min=5, max=119, avg=56.59, stdev=15.76 lat (msec): min=5, max=119, avg=56.80, stdev=15.77 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 35], 10.00th=[ 38], 20.00th=[ 42], | 30.00th=[ 47], 40.00th=[ 50], 50.00th=[ 56], 60.00th=[ 61], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 78], 95.00th=[ 84], | 99.00th=[ 97], 99.50th=[ 104], 99.90th=[ 106], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 784, max= 1654, per=25.25%, avg=1128.24, stdev=227.81, samples=17 iops : min= 196, max= 413, avg=282.00, stdev=56.90, samples=17 lat (msec) : 10=0.04%, 50=40.74%, 100=58.40%, 250=0.82% cpu : usr=0.52%, sys=4.14%, ctx=2583, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180951: Tue Jun 16 12:26:25 2026 write: IOPS=280, BW=1123KiB/s (1150kB/s)(10.0MiB/9118msec); 0 zone resets slat (usec): min=46, max=3624, avg=218.28, stdev=209.91 clat (msec): min=11, max=120, avg=56.69, stdev=15.80 lat (msec): min=11, max=120, avg=56.91, stdev=15.80 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 43], | 30.00th=[ 47], 40.00th=[ 50], 50.00th=[ 55], 60.00th=[ 61], | 70.00th=[ 65], 80.00th=[ 70], 90.00th=[ 78], 95.00th=[ 86], | 99.00th=[ 99], 99.50th=[ 106], 99.90th=[ 108], 99.95th=[ 110], | 99.99th=[ 121] bw ( KiB/s): min= 792, max= 1660, per=25.27%, avg=1129.06, stdev=230.99, samples=17 iops : min= 198, max= 415, avg=282.24, stdev=57.77, samples=17 lat (msec) : 20=0.04%, 50=40.62%, 100=58.40%, 250=0.94% cpu : usr=0.38%, sys=4.50%, ctx=2529, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180952: Tue Jun 16 12:26:25 2026 write: IOPS=279, BW=1117KiB/s (1144kB/s)(10.0MiB/9167msec); 0 zone resets slat (usec): min=41, max=4027, avg=220.19, stdev=205.68 clat (msec): min=5, max=107, avg=56.67, stdev=15.44 lat (msec): min=6, max=107, avg=56.89, stdev=15.45 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 43], | 30.00th=[ 47], 40.00th=[ 50], 50.00th=[ 56], 60.00th=[ 61], | 70.00th=[ 65], 80.00th=[ 70], 90.00th=[ 78], 95.00th=[ 85], | 99.00th=[ 96], 99.50th=[ 105], 99.90th=[ 106], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 808, max= 1501, per=24.87%, avg=1111.28, stdev=202.27, samples=18 iops : min= 202, max= 375, avg=277.78, stdev=50.56, samples=18 lat (msec) : 10=0.08%, 20=0.04%, 50=40.47%, 100=58.71%, 250=0.70% cpu : usr=0.39%, sys=4.78%, ctx=2742, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4468KiB/s (4575kB/s), 1117KiB/s-1123KiB/s (1144kB/s-1150kB/s), io=40.0MiB (41.9MB), run=9118-9167msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180963: Tue Jun 16 12:26:39 2026 read: IOPS=120, BW=480KiB/s (492kB/s)(5048KiB/10506msec) slat (usec): min=1314, max=18341, avg=4373.72, stdev=2171.18 clat (msec): min=11, max=128, avg=62.44, stdev=16.71 lat (msec): min=16, max=133, avg=66.81, stdev=17.43 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 62], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 77], 90.00th=[ 85], 95.00th=[ 93], | 99.00th=[ 114], 99.50th=[ 116], 99.90th=[ 123], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 294, max= 694, per=25.21%, avg=481.50, stdev=100.00, samples=20 iops : min= 73, max= 173, avg=120.20, stdev=24.94, samples=20 write: IOPS=123, BW=494KiB/s (506kB/s)(5192KiB/10506msec); 0 zone resets slat (usec): min=57, max=4411, avg=205.89, stdev=221.59 clat (msec): min=10, max=121, avg=63.55, stdev=16.01 lat (msec): min=11, max=122, avg=63.76, stdev=16.01 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 66], | 70.00th=[ 71], 80.00th=[ 77], 90.00th=[ 86], 95.00th=[ 94], | 99.00th=[ 109], 99.50th=[ 114], 99.90th=[ 118], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 360, max= 670, per=24.84%, avg=493.90, stdev=83.41, samples=20 iops : min= 90, max= 167, avg=123.30, stdev=20.78, samples=20 lat (msec) : 20=0.31%, 50=22.62%, 100=74.41%, 250=2.66% cpu : usr=0.47%, sys=4.20%, ctx=4566, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180964: Tue Jun 16 12:26:39 2026 read: IOPS=119, BW=478KiB/s (489kB/s)(4976KiB/10418msec) slat (usec): min=48, max=15546, avg=4341.28, stdev=2126.97 clat (msec): min=6, max=156, avg=62.17, stdev=17.69 lat (msec): min=6, max=159, avg=66.51, stdev=18.42 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 84], 95.00th=[ 91], | 99.00th=[ 116], 99.50th=[ 146], 99.90th=[ 155], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 333, max= 654, per=25.12%, avg=479.70, stdev=99.71, samples=20 iops : min= 83, max= 163, avg=119.75, stdev=24.82, samples=20 write: IOPS=126, BW=505KiB/s (517kB/s)(5264KiB/10418msec); 0 zone resets slat (usec): min=50, max=5221, avg=198.10, stdev=207.51 clat (msec): min=27, max=155, avg=63.33, stdev=16.47 lat (msec): min=27, max=156, avg=63.53, stdev=16.48 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 84], 95.00th=[ 92], | 99.00th=[ 112], 99.50th=[ 136], 99.90th=[ 153], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 328, max= 710, per=25.54%, avg=507.70, stdev=86.54, samples=20 iops : min= 82, max= 177, avg=126.75, stdev=21.56, samples=20 lat (msec) : 10=0.23%, 50=22.58%, 100=74.30%, 250=2.89% cpu : usr=0.45%, sys=4.18%, ctx=4571, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180965: Tue Jun 16 12:26:39 2026 read: IOPS=118, BW=475KiB/s (486kB/s)(4984KiB/10494msec) slat (usec): min=1243, max=14040, avg=4371.43, stdev=2026.79 clat (msec): min=17, max=148, avg=62.65, stdev=16.62 lat (msec): min=21, max=151, avg=67.03, stdev=17.29 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 85], 95.00th=[ 92], | 99.00th=[ 113], 99.50th=[ 120], 99.90th=[ 133], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 344, max= 566, per=25.14%, avg=480.15, stdev=70.82, samples=20 iops : min= 86, max= 141, avg=119.85, stdev=17.66, samples=20 write: IOPS=125, BW=501KiB/s (513kB/s)(5256KiB/10494msec); 0 zone resets slat (usec): min=41, max=6701, avg=202.36, stdev=265.45 clat (msec): min=26, max=166, avg=63.23, stdev=17.10 lat (msec): min=26, max=166, avg=63.43, stdev=17.10 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 75], 90.00th=[ 86], 95.00th=[ 95], | 99.00th=[ 117], 99.50th=[ 128], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 320, max= 734, per=25.26%, avg=502.15, stdev=113.78, samples=20 iops : min= 80, max= 183, avg=125.35, stdev=28.37, samples=20 lat (msec) : 20=0.04%, 50=22.27%, 100=74.41%, 250=3.28% cpu : usr=0.45%, sys=4.22%, ctx=4553, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180966: Tue Jun 16 12:26:39 2026 read: IOPS=120, BW=482KiB/s (493kB/s)(5060KiB/10503msec) slat (usec): min=1189, max=17236, avg=4358.33, stdev=2082.56 clat (msec): min=28, max=127, avg=62.16, stdev=16.09 lat (msec): min=32, max=134, avg=66.52, stdev=16.76 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 83], 95.00th=[ 91], | 99.00th=[ 115], 99.50th=[ 118], 99.90th=[ 127], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 310, max= 654, per=25.37%, avg=484.65, stdev=94.10, samples=20 iops : min= 77, max= 163, avg=121.00, stdev=23.49, samples=20 write: IOPS=123, BW=493KiB/s (505kB/s)(5180KiB/10503msec); 0 zone resets slat (usec): min=59, max=4068, avg=201.18, stdev=205.85 clat (msec): min=29, max=155, avg=63.63, stdev=15.92 lat (msec): min=29, max=156, avg=63.83, stdev=15.93 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 77], 90.00th=[ 86], 95.00th=[ 93], | 99.00th=[ 111], 99.50th=[ 116], 99.90th=[ 140], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 368, max= 630, per=24.91%, avg=495.25, stdev=79.35, samples=20 iops : min= 92, max= 157, avg=123.65, stdev=19.73, samples=20 lat (msec) : 50=22.42%, 100=75.04%, 250=2.54% cpu : usr=0.43%, sys=4.28%, ctx=4535, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1910KiB/s (1956kB/s), 475KiB/s-482KiB/s (486kB/s-493kB/s), io=19.6MiB (20.5MB), run=10418-10506msec WRITE: bw=1989KiB/s (2036kB/s), 493KiB/s-505KiB/s (505kB/s-517kB/s), io=20.4MiB (21.4MB), run=10418-10506msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180971: Tue Jun 16 12:26:43 2026 read: IOPS=6, BW=260MiB/s (272MB/s)(40.0MiB/154msec) slat (nsec): min=9325.9k, max=9325.9k, avg=9325904.00, stdev= 0.00 clat (nsec): min=140172k, max=140172k, avg=140172088.00, stdev= 0.00 lat (nsec): min=149502k, max=149502k, avg=149501691.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.58%, ctx=11, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=260MiB/s (272MB/s), 260MiB/s-260MiB/s (272MB/s-272MB/s), io=40.0MiB (41.9MB), run=154-154msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout