== sanity test 398c: run fio to test AIO ================= 15:55:42 (1780343742) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.299602 s, 140 MB/s osc.lustre-OST0000-osc-ffff9515196bc000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182113: Mon Jun 1 15:56:08 2026 write: IOPS=132, BW=530KiB/s (542kB/s)(10.0MiB/19332msec); 0 zone resets slat (usec): min=40, max=21230, avg=276.33, stdev=738.53 clat (msec): min=9, max=209, avg=119.39, stdev=25.99 lat (msec): min=10, max=209, avg=119.67, stdev=25.99 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 79], 10.00th=[ 90], 20.00th=[ 100], | 30.00th=[ 106], 40.00th=[ 112], 50.00th=[ 117], 60.00th=[ 125], | 70.00th=[ 132], 80.00th=[ 140], 90.00th=[ 157], 95.00th=[ 167], | 99.00th=[ 184], 99.50th=[ 201], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 334, max= 792, per=24.83%, avg=525.84, stdev=91.64, samples=38 iops : min= 83, max= 198, avg=131.16, stdev=22.98, samples=38 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=20.74%, 250=79.06% cpu : usr=0.28%, sys=1.77%, ctx=2710, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182114: Mon Jun 1 15:56:08 2026 write: IOPS=132, BW=530KiB/s (543kB/s)(10.0MiB/19326msec); 0 zone resets slat (usec): min=36, max=12159, avg=313.99, stdev=745.05 clat (msec): min=25, max=206, avg=119.45, stdev=25.08 lat (msec): min=25, max=206, avg=119.76, stdev=25.08 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 82], 10.00th=[ 89], 20.00th=[ 99], | 30.00th=[ 107], 40.00th=[ 112], 50.00th=[ 120], 60.00th=[ 125], | 70.00th=[ 133], 80.00th=[ 140], 90.00th=[ 153], 95.00th=[ 163], | 99.00th=[ 186], 99.50th=[ 192], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 335, max= 776, per=24.93%, avg=527.92, stdev=94.79, samples=38 iops : min= 83, max= 194, avg=131.68, stdev=23.76, samples=38 lat (msec) : 50=0.12%, 100=22.03%, 250=77.85% cpu : usr=0.24%, sys=2.10%, ctx=2995, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182115: Mon Jun 1 15:56:08 2026 write: IOPS=132, BW=531KiB/s (544kB/s)(10.0MiB/19290msec); 0 zone resets slat (usec): min=36, max=8407, avg=273.44, stdev=586.27 clat (msec): min=25, max=210, avg=120.10, stdev=25.21 lat (msec): min=26, max=210, avg=120.38, stdev=25.21 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 84], 10.00th=[ 92], 20.00th=[ 101], | 30.00th=[ 106], 40.00th=[ 112], 50.00th=[ 118], 60.00th=[ 125], | 70.00th=[ 132], 80.00th=[ 140], 90.00th=[ 155], 95.00th=[ 165], | 99.00th=[ 186], 99.50th=[ 190], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 382, max= 848, per=25.04%, avg=530.42, stdev=90.66, samples=38 iops : min= 95, max= 212, avg=132.32, stdev=22.73, samples=38 lat (msec) : 50=0.08%, 100=20.70%, 250=79.22% cpu : usr=0.27%, sys=1.80%, ctx=2756, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182116: Mon Jun 1 15:56:08 2026 write: IOPS=132, BW=531KiB/s (543kB/s)(10.0MiB/19300msec); 0 zone resets slat (usec): min=36, max=67340, avg=305.99, stdev=1512.23 clat (msec): min=17, max=203, avg=119.36, stdev=25.42 lat (msec): min=17, max=204, avg=119.67, stdev=25.40 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 83], 10.00th=[ 91], 20.00th=[ 100], | 30.00th=[ 105], 40.00th=[ 111], 50.00th=[ 118], 60.00th=[ 125], | 70.00th=[ 131], 80.00th=[ 142], 90.00th=[ 153], 95.00th=[ 163], | 99.00th=[ 188], 99.50th=[ 190], 99.90th=[ 199], 99.95th=[ 201], | 99.99th=[ 205] bw ( KiB/s): min= 360, max= 840, per=25.01%, avg=529.66, stdev=96.14, samples=38 iops : min= 90, max= 210, avg=132.13, stdev=24.10, samples=38 lat (msec) : 20=0.04%, 50=0.12%, 100=21.84%, 250=78.01% cpu : usr=0.26%, sys=1.80%, ctx=2632, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2119KiB/s (2170kB/s), 530KiB/s-531KiB/s (542kB/s-544kB/s), io=40.0MiB (41.9MB), run=19290-19332msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182129: Mon Jun 1 15:56:36 2026 read: IOPS=57, BW=231KiB/s (237kB/s)(5048KiB/21825msec) slat (usec): min=1792, max=37054, avg=9948.03, stdev=5345.32 clat (msec): min=27, max=248, avg=128.74, stdev=33.29 lat (msec): min=32, max=269, avg=138.70, stdev=34.45 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 85], 10.00th=[ 90], 20.00th=[ 100], | 30.00th=[ 108], 40.00th=[ 117], 50.00th=[ 126], 60.00th=[ 134], | 70.00th=[ 144], 80.00th=[ 155], 90.00th=[ 174], 95.00th=[ 190], | 99.00th=[ 226], 99.50th=[ 230], 99.90th=[ 247], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 149, max= 367, per=25.13%, avg=230.44, stdev=53.00, samples=43 iops : min= 37, max= 91, avg=57.16, stdev=13.20, samples=43 write: IOPS=59, BW=238KiB/s (244kB/s)(5192KiB/21825msec); 0 zone resets slat (usec): min=38, max=16187, avg=331.15, stdev=875.91 clat (msec): min=20, max=243, avg=132.31, stdev=30.78 lat (msec): min=20, max=243, avg=132.65, stdev=30.84 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 87], 10.00th=[ 94], 20.00th=[ 105], | 30.00th=[ 115], 40.00th=[ 123], 50.00th=[ 130], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 174], 95.00th=[ 190], | 99.00th=[ 211], 99.50th=[ 215], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 144, max= 327, per=24.82%, avg=237.07, stdev=49.49, samples=43 iops : min= 36, max= 81, avg=58.84, stdev=12.34, samples=43 lat (msec) : 50=0.20%, 100=17.81%, 250=81.99% cpu : usr=0.34%, sys=2.77%, ctx=4595, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182130: Mon Jun 1 15:56:36 2026 read: IOPS=56, BW=228KiB/s (233kB/s)(4976KiB/21870msec) slat (usec): min=1986, max=41242, avg=10005.08, stdev=5284.44 clat (msec): min=10, max=259, avg=127.96, stdev=33.45 lat (msec): min=18, max=275, avg=137.98, stdev=34.70 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 83], 10.00th=[ 90], 20.00th=[ 101], | 30.00th=[ 108], 40.00th=[ 116], 50.00th=[ 125], 60.00th=[ 133], | 70.00th=[ 142], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 190], | 99.00th=[ 220], 99.50th=[ 234], 99.90th=[ 257], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 119, max= 311, per=24.51%, avg=224.77, stdev=50.23, samples=43 iops : min= 29, max= 77, avg=55.77, stdev=12.50, samples=43 write: IOPS=60, BW=241KiB/s (246kB/s)(5264KiB/21870msec); 0 zone resets slat (usec): min=37, max=14022, avg=270.13, stdev=664.05 clat (msec): min=27, max=243, avg=134.28, stdev=31.88 lat (msec): min=27, max=243, avg=134.55, stdev=31.87 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 88], 10.00th=[ 97], 20.00th=[ 106], | 30.00th=[ 115], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 178], 95.00th=[ 194], | 99.00th=[ 220], 99.50th=[ 224], 99.90th=[ 243], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 118, max= 367, per=24.99%, avg=238.65, stdev=48.38, samples=43 iops : min= 29, max= 91, avg=59.21, stdev=12.08, samples=43 lat (msec) : 20=0.08%, 50=0.20%, 100=16.76%, 250=82.85%, 500=0.12% cpu : usr=0.36%, sys=2.79%, ctx=4616, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182131: Mon Jun 1 15:56:36 2026 read: IOPS=57, BW=228KiB/s (234kB/s)(4984KiB/21824msec) slat (usec): min=54, max=35865, avg=9929.11, stdev=5307.31 clat (msec): min=26, max=253, avg=128.02, stdev=31.84 lat (msec): min=42, max=274, avg=137.95, stdev=33.12 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 84], 10.00th=[ 91], 20.00th=[ 101], | 30.00th=[ 109], 40.00th=[ 117], 50.00th=[ 125], 60.00th=[ 133], | 70.00th=[ 144], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 186], | 99.00th=[ 218], 99.50th=[ 232], 99.90th=[ 243], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 111, max= 335, per=24.87%, avg=228.02, stdev=51.59, samples=43 iops : min= 27, max= 83, avg=56.58, stdev=12.85, samples=43 write: IOPS=60, BW=241KiB/s (247kB/s)(5256KiB/21824msec); 0 zone resets slat (usec): min=43, max=8599, avg=288.75, stdev=582.52 clat (msec): min=51, max=252, avg=134.17, stdev=31.48 lat (msec): min=52, max=253, avg=134.46, stdev=31.49 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 90], 10.00th=[ 96], 20.00th=[ 108], | 30.00th=[ 116], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 226], 99.50th=[ 241], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 174, max= 365, per=25.04%, avg=239.09, stdev=45.38, samples=43 iops : min= 43, max= 91, avg=59.33, stdev=11.35, samples=43 lat (msec) : 50=0.04%, 100=16.33%, 250=83.52%, 500=0.12% cpu : usr=0.33%, sys=2.58%, ctx=4645, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182132: Mon Jun 1 15:56:36 2026 read: IOPS=57, BW=231KiB/s (237kB/s)(5060KiB/21858msec) slat (usec): min=2298, max=41145, avg=9904.79, stdev=5465.68 clat (msec): min=23, max=242, avg=128.67, stdev=32.25 lat (msec): min=31, max=258, avg=138.58, stdev=33.53 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 84], 10.00th=[ 91], 20.00th=[ 101], | 30.00th=[ 110], 40.00th=[ 117], 50.00th=[ 125], 60.00th=[ 136], | 70.00th=[ 144], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 186], | 99.00th=[ 215], 99.50th=[ 230], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 136, max= 341, per=25.15%, avg=230.60, stdev=48.92, samples=43 iops : min= 34, max= 85, avg=57.19, stdev=12.23, samples=43 write: IOPS=59, BW=237KiB/s (243kB/s)(5180KiB/21858msec); 0 zone resets slat (usec): min=38, max=10302, avg=307.30, stdev=680.23 clat (msec): min=68, max=238, avg=133.25, stdev=30.42 lat (msec): min=68, max=239, avg=133.56, stdev=30.43 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 108], | 30.00th=[ 116], 40.00th=[ 123], 50.00th=[ 130], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 174], 95.00th=[ 188], | 99.00th=[ 218], 99.50th=[ 224], 99.90th=[ 234], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 166, max= 335, per=24.63%, avg=235.26, stdev=41.64, samples=43 iops : min= 41, max= 83, avg=58.37, stdev=10.34, samples=43 lat (msec) : 50=0.12%, 100=16.05%, 250=83.83% cpu : usr=0.33%, sys=2.67%, ctx=4628, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=918KiB/s (940kB/s), 228KiB/s-231KiB/s (233kB/s-237kB/s), io=19.6MiB (20.5MB), run=21824-21870msec WRITE: bw=955KiB/s (978kB/s), 237KiB/s-241KiB/s (243kB/s-247kB/s), io=20.4MiB (21.4MB), run=21824-21870msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182139: Mon Jun 1 15:56:42 2026 read: IOPS=2, BW=100MiB/s (105MB/s)(40.0MiB/400msec) slat (nsec): min=33692k, max=33692k, avg=33692439.00, stdev= 0.00 clat (nsec): min=356933k, max=356933k, avg=356932619.00, stdev= 0.00 lat (nsec): min=390631k, max=390631k, avg=390630666.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.76%, ctx=14, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=40.0MiB (41.9MB), run=400-400msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout