== sanity test 398c: run fio to test AIO ================= 23:02:24 (1781233344) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.565264 s, 74.2 MB/s osc.lustre-OST0000-osc-ffffa0ae02d01000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181186: Thu Jun 11 23:02:59 2026 write: IOPS=109, BW=439KiB/s (449kB/s)(10.0MiB/23344msec); 0 zone resets slat (usec): min=37, max=10059, avg=233.75, stdev=588.03 clat (msec): min=19, max=324, avg=144.43, stdev=38.93 lat (msec): min=19, max=324, avg=144.66, stdev=38.92 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 91], 10.00th=[ 102], 20.00th=[ 111], | 30.00th=[ 122], 40.00th=[ 130], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 161], 80.00th=[ 178], 90.00th=[ 199], 95.00th=[ 209], | 99.00th=[ 247], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 253, max= 654, per=25.08%, avg=439.93, stdev=101.43, samples=45 iops : min= 63, max= 163, avg=109.62, stdev=25.35, samples=45 lat (msec) : 20=0.04%, 50=0.16%, 100=8.36%, 250=90.62%, 500=0.82% cpu : usr=0.18%, sys=1.22%, ctx=2187, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181187: Thu Jun 11 23:02:59 2026 write: IOPS=109, BW=440KiB/s (450kB/s)(10.0MiB/23294msec); 0 zone resets slat (usec): min=36, max=24027, avg=270.81, stdev=884.59 clat (msec): min=24, max=321, avg=145.13, stdev=39.03 lat (msec): min=24, max=322, avg=145.41, stdev=39.04 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 92], 10.00th=[ 102], 20.00th=[ 114], | 30.00th=[ 123], 40.00th=[ 129], 50.00th=[ 140], 60.00th=[ 150], | 70.00th=[ 163], 80.00th=[ 176], 90.00th=[ 194], 95.00th=[ 213], | 99.00th=[ 268], 99.50th=[ 309], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 321] bw ( KiB/s): min= 253, max= 670, per=25.25%, avg=442.87, stdev=100.43, samples=45 iops : min= 63, max= 167, avg=110.38, stdev=25.11, samples=45 lat (msec) : 50=0.12%, 100=9.41%, 250=89.02%, 500=1.45% cpu : usr=0.16%, sys=1.24%, ctx=2524, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181188: Thu Jun 11 23:02:59 2026 write: IOPS=109, BW=439KiB/s (450kB/s)(10.0MiB/23315msec); 0 zone resets slat (usec): min=38, max=16053, avg=281.31, stdev=794.64 clat (msec): min=13, max=344, avg=144.29, stdev=38.91 lat (msec): min=14, max=344, avg=144.57, stdev=38.91 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 91], 10.00th=[ 103], 20.00th=[ 114], | 30.00th=[ 123], 40.00th=[ 130], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 161], 80.00th=[ 176], 90.00th=[ 197], 95.00th=[ 209], | 99.00th=[ 241], 99.50th=[ 317], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 347] bw ( KiB/s): min= 182, max= 670, per=25.17%, avg=441.42, stdev=99.81, samples=45 iops : min= 45, max= 167, avg=110.02, stdev=24.95, samples=45 lat (msec) : 20=0.08%, 50=0.20%, 100=8.67%, 250=90.35%, 500=0.70% cpu : usr=0.20%, sys=1.43%, ctx=2888, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181189: Thu Jun 11 23:02:59 2026 write: IOPS=109, BW=439KiB/s (449kB/s)(10.0MiB/23345msec); 0 zone resets slat (usec): min=40, max=19182, avg=282.09, stdev=1024.03 clat (msec): min=11, max=327, avg=144.61, stdev=38.76 lat (msec): min=12, max=327, avg=144.90, stdev=38.74 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 94], 10.00th=[ 102], 20.00th=[ 114], | 30.00th=[ 124], 40.00th=[ 130], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 161], 80.00th=[ 174], 90.00th=[ 197], 95.00th=[ 213], | 99.00th=[ 262], 99.50th=[ 275], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 253, max= 640, per=25.07%, avg=439.69, stdev=96.69, samples=45 iops : min= 63, max= 160, avg=109.56, stdev=24.17, samples=45 lat (msec) : 20=0.16%, 50=0.16%, 100=9.18%, 250=89.26%, 500=1.25% cpu : usr=0.18%, sys=1.29%, ctx=2173, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1755KiB/s (1797kB/s), 439KiB/s-440KiB/s (449kB/s-450kB/s), io=40.0MiB (41.9MB), run=23294-23345msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181203: Thu Jun 11 23:03:38 2026 read: IOPS=40, BW=161KiB/s (165kB/s)(5048KiB/31407msec) slat (usec): min=2399, max=93808, avg=13832.56, stdev=8735.70 clat (msec): min=61, max=446, avg=183.19, stdev=56.98 lat (msec): min=68, max=453, avg=197.03, stdev=59.78 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 104], 10.00th=[ 117], 20.00th=[ 134], | 30.00th=[ 150], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 228], 90.00th=[ 259], 95.00th=[ 288], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 430], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 21, max= 327, per=25.16%, avg=160.53, stdev=47.46, samples=62 iops : min= 5, max= 81, avg=39.68, stdev=11.94, samples=62 write: IOPS=41, BW=165KiB/s (169kB/s)(5192KiB/31407msec); 0 zone resets slat (usec): min=38, max=15223, avg=343.24, stdev=1005.75 clat (msec): min=67, max=396, avg=190.98, stdev=55.47 lat (msec): min=67, max=396, avg=191.32, stdev=55.47 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 107], 10.00th=[ 122], 20.00th=[ 142], | 30.00th=[ 159], 40.00th=[ 174], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 268], 95.00th=[ 296], | 99.00th=[ 334], 99.50th=[ 363], 99.90th=[ 393], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 7, max= 271, per=24.90%, avg=165.34, stdev=48.81, samples=62 iops : min= 1, max= 67, avg=40.87, stdev=12.26, samples=62 lat (msec) : 100=3.67%, 250=82.93%, 500=13.40% cpu : usr=0.17%, sys=1.85%, ctx=4360, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181204: Thu Jun 11 23:03:38 2026 read: IOPS=39, BW=158KiB/s (162kB/s)(4976KiB/31435msec) slat (usec): min=2061, max=75123, avg=14093.04, stdev=9073.43 clat (msec): min=16, max=459, avg=185.19, stdev=58.91 lat (msec): min=32, max=467, avg=199.29, stdev=61.64 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 104], 10.00th=[ 117], 20.00th=[ 136], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 180], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 228], 90.00th=[ 259], 95.00th=[ 296], | 99.00th=[ 359], 99.50th=[ 388], 99.90th=[ 456], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 14, max= 261, per=24.74%, avg=157.85, stdev=46.51, samples=62 iops : min= 3, max= 65, avg=39.00, stdev=11.67, samples=62 write: IOPS=41, BW=167KiB/s (171kB/s)(5264KiB/31435msec); 0 zone resets slat (usec): min=41, max=22421, avg=311.50, stdev=1122.39 clat (msec): min=64, max=449, avg=190.11, stdev=55.49 lat (msec): min=64, max=449, avg=190.42, stdev=55.51 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 102], 10.00th=[ 123], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 171], 50.00th=[ 188], 60.00th=[ 201], | 70.00th=[ 218], 80.00th=[ 234], 90.00th=[ 268], 95.00th=[ 284], | 99.00th=[ 338], 99.50th=[ 363], 99.90th=[ 430], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 62, max= 312, per=25.21%, avg=167.42, stdev=51.03, samples=62 iops : min= 15, max= 78, avg=41.40, stdev=12.84, samples=62 lat (msec) : 20=0.04%, 50=0.08%, 100=4.14%, 250=82.58%, 500=13.16% cpu : usr=0.22%, sys=1.80%, ctx=4586, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181205: Thu Jun 11 23:03:38 2026 read: IOPS=40, BW=160KiB/s (164kB/s)(4984KiB/31119msec) slat (usec): min=751, max=71120, avg=14178.71, stdev=9184.09 clat (msec): min=20, max=396, avg=183.52, stdev=55.85 lat (msec): min=32, max=461, avg=197.70, stdev=59.09 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 103], 10.00th=[ 117], 20.00th=[ 136], | 30.00th=[ 150], 40.00th=[ 165], 50.00th=[ 180], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 226], 90.00th=[ 257], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 384], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 58, max= 311, per=25.10%, avg=160.11, stdev=45.56, samples=61 iops : min= 14, max= 77, avg=39.57, stdev=11.42, samples=61 write: IOPS=42, BW=169KiB/s (173kB/s)(5256KiB/31119msec); 0 zone resets slat (usec): min=56, max=22970, avg=344.10, stdev=1145.78 clat (msec): min=54, max=429, avg=189.17, stdev=56.98 lat (msec): min=54, max=429, avg=189.52, stdev=57.01 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 104], 10.00th=[ 122], 20.00th=[ 140], | 30.00th=[ 157], 40.00th=[ 171], 50.00th=[ 186], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 232], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 338], 99.50th=[ 393], 99.90th=[ 422], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 72, max= 272, per=25.33%, avg=168.20, stdev=46.62, samples=61 iops : min= 18, max= 68, avg=41.59, stdev=11.73, samples=61 lat (msec) : 50=0.12%, 100=4.14%, 250=82.50%, 500=13.24% cpu : usr=0.24%, sys=1.90%, ctx=4500, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181206: Thu Jun 11 23:03:38 2026 read: IOPS=40, BW=161KiB/s (165kB/s)(5060KiB/31437msec) slat (msec): min=2, max=126, avg=14.16, stdev= 9.72 clat (msec): min=21, max=447, avg=184.33, stdev=57.66 lat (msec): min=30, max=459, avg=198.50, stdev=61.00 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 103], 10.00th=[ 115], 20.00th=[ 134], | 30.00th=[ 150], 40.00th=[ 165], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 209], 80.00th=[ 230], 90.00th=[ 259], 95.00th=[ 292], | 99.00th=[ 347], 99.50th=[ 372], 99.90th=[ 435], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 21, max= 295, per=25.20%, avg=160.81, stdev=45.43, samples=62 iops : min= 5, max= 73, avg=39.74, stdev=11.35, samples=62 write: IOPS=41, BW=165KiB/s (169kB/s)(5180KiB/31437msec); 0 zone resets slat (usec): min=45, max=19774, avg=349.08, stdev=1212.28 clat (msec): min=20, max=419, avg=190.55, stdev=57.95 lat (msec): min=20, max=419, avg=190.90, stdev=58.01 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 104], 10.00th=[ 122], 20.00th=[ 142], | 30.00th=[ 157], 40.00th=[ 171], 50.00th=[ 186], 60.00th=[ 201], | 70.00th=[ 218], 80.00th=[ 236], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 368], 99.50th=[ 388], 99.90th=[ 397], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 28, max= 287, per=24.65%, avg=163.65, stdev=50.58, samples=62 iops : min= 7, max= 71, avg=40.47, stdev=12.72, samples=62 lat (msec) : 50=0.08%, 100=4.14%, 250=81.64%, 500=14.14% cpu : usr=0.30%, sys=2.08%, ctx=4515, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=638KiB/s (654kB/s), 158KiB/s-161KiB/s (162kB/s-165kB/s), io=19.6MiB (20.5MB), run=31119-31437msec WRITE: bw=665KiB/s (681kB/s), 165KiB/s-169KiB/s (169kB/s-173kB/s), io=20.4MiB (21.4MB), run=31119-31437msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181215: Thu Jun 11 23:03:48 2026 read: IOPS=1, BW=62.0MiB/s (66.1MB/s)(40.0MiB/635msec) slat (nsec): min=29101k, max=29101k, avg=29101482.00, stdev= 0.00 clat (nsec): min=582093k, max=582093k, avg=582092867.00, stdev= 0.00 lat (nsec): min=611198k, max=611198k, avg=611198479.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.74%, ctx=9, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.0MiB/s (66.1MB/s), 62.0MiB/s-62.0MiB/s (66.1MB/s-66.1MB/s), io=40.0MiB (41.9MB), run=635-635msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout