== sanity test 398c: run fio to test AIO ================= 00:28:54 (1746160134) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.211696 s, 198 MB/s osc.lustre-OST0000-osc-ffff880078417800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=21266: Fri May 2 00:29:09 2025 write: IOPS=209, BW=839KiB/s (860kB/s)(10.0MiB/12198msec) slat (usec): min=11, max=5367, avg=94.69, stdev=223.45 clat (msec): min=7, max=130, avg=75.80, stdev=13.92 lat (msec): min=7, max=131, avg=75.89, stdev=13.90 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 54], 10.00th=[ 59], 20.00th=[ 65], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 80], | 70.00th=[ 83], 80.00th=[ 87], 90.00th=[ 94], 95.00th=[ 102], | 99.00th=[ 110], 99.50th=[ 112], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 131] bw ( KiB/s): min= 694, max= 1016, per=25.07%, avg=838.58, stdev=85.21, samples=24 iops : min= 173, max= 254, avg=209.54, stdev=21.32, samples=24 lat (msec) : 10=0.04%, 20=0.12%, 50=1.72%, 100=92.54%, 250=5.59% cpu : usr=0.26%, sys=2.94%, ctx=1152, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21267: Fri May 2 00:29:09 2025 write: IOPS=209, BW=837KiB/s (857kB/s)(10.0MiB/12229msec) slat (usec): min=10, max=2027, avg=86.57, stdev=168.87 clat (msec): min=14, max=124, avg=75.99, stdev=13.68 lat (msec): min=14, max=124, avg=76.08, stdev=13.68 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 65], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 78], | 70.00th=[ 82], 80.00th=[ 87], 90.00th=[ 94], 95.00th=[ 102], | 99.00th=[ 114], 99.50th=[ 115], 99.90th=[ 117], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 702, max= 1008, per=25.01%, avg=836.75, stdev=85.81, samples=24 iops : min= 175, max= 252, avg=209.08, stdev=21.48, samples=24 lat (msec) : 20=0.08%, 50=0.82%, 100=93.32%, 250=5.78% cpu : usr=0.00%, sys=3.20%, ctx=1092, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21268: Fri May 2 00:29:09 2025 write: IOPS=209, BW=836KiB/s (857kB/s)(10.0MiB/12242msec) slat (usec): min=11, max=3639, avg=94.90, stdev=206.27 clat (msec): min=8, max=114, avg=75.97, stdev=13.95 lat (msec): min=9, max=114, avg=76.07, stdev=13.94 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 56], 10.00th=[ 60], 20.00th=[ 65], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 87], 90.00th=[ 95], 95.00th=[ 103], | 99.00th=[ 109], 99.50th=[ 112], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 664, max= 992, per=24.94%, avg=834.08, stdev=93.17, samples=24 iops : min= 166, max= 248, avg=208.38, stdev=23.32, samples=24 lat (msec) : 10=0.04%, 20=0.20%, 50=2.15%, 100=90.62%, 250=6.99% cpu : usr=0.59%, sys=2.77%, ctx=1155, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21269: Fri May 2 00:29:09 2025 write: IOPS=209, BW=838KiB/s (858kB/s)(10.0MiB/12221msec) slat (usec): min=11, max=2875, avg=98.89, stdev=213.79 clat (msec): min=16, max=161, avg=75.99, stdev=14.57 lat (msec): min=16, max=161, avg=76.09, stdev=14.57 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 53], 10.00th=[ 59], 20.00th=[ 65], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 87], 90.00th=[ 95], 95.00th=[ 104], | 99.00th=[ 112], 99.50th=[ 115], 99.90th=[ 146], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 728, max= 1024, per=25.09%, avg=839.33, stdev=81.49, samples=24 iops : min= 182, max= 256, avg=209.75, stdev=20.38, samples=24 lat (msec) : 20=0.04%, 50=3.16%, 100=90.27%, 250=6.52% cpu : usr=0.97%, sys=2.27%, ctx=1153, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3346KiB/s (3426kB/s), 836KiB/s-839KiB/s (857kB/s-860kB/s), io=40.0MiB (41.9MB), run=12198-12242msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21289: Fri May 2 00:29:22 2025 read: IOPS=107, BW=431KiB/s (442kB/s)(5048KiB/11704msec) slat (usec): min=1234, max=19464, avg=4775.07, stdev=2101.80 clat (msec): min=30, max=174, avg=69.13, stdev=18.16 lat (msec): min=32, max=183, avg=73.91, stdev=18.96 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 46], 10.00th=[ 49], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 72], | 70.00th=[ 77], 80.00th=[ 84], 90.00th=[ 94], 95.00th=[ 104], | 99.00th=[ 115], 99.50th=[ 125], 99.90th=[ 174], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 279, max= 568, per=25.10%, avg=430.22, stdev=77.75, samples=23 iops : min= 69, max= 142, avg=107.39, stdev=19.50, samples=23 write: IOPS=110, BW=444KiB/s (454kB/s)(5192KiB/11704msec) slat (usec): min=15, max=8644, avg=112.94, stdev=339.26 clat (msec): min=10, max=150, avg=71.56, stdev=17.33 lat (msec): min=10, max=150, avg=71.68, stdev=17.32 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 79], 80.00th=[ 86], 90.00th=[ 96], 95.00th=[ 104], | 99.00th=[ 117], 99.50th=[ 120], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 320, max= 584, per=24.88%, avg=444.13, stdev=82.22, samples=23 iops : min= 80, max= 146, avg=110.87, stdev=20.57, samples=23 lat (msec) : 20=0.04%, 50=11.05%, 100=82.19%, 250=6.72% cpu : usr=0.62%, sys=5.78%, ctx=2367, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21290: Fri May 2 00:29:22 2025 read: IOPS=107, BW=429KiB/s (439kB/s)(4976KiB/11609msec) slat (usec): min=15, max=22545, avg=4786.71, stdev=2109.39 clat (msec): min=4, max=170, avg=68.49, stdev=18.38 lat (msec): min=6, max=175, avg=73.28, stdev=19.14 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 54], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 75], 80.00th=[ 83], 90.00th=[ 92], 95.00th=[ 101], | 99.00th=[ 127], 99.50th=[ 138], 99.90th=[ 165], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 320, max= 552, per=25.00%, avg=428.48, stdev=68.03, samples=23 iops : min= 80, max= 138, avg=106.96, stdev=16.96, samples=23 write: IOPS=113, BW=453KiB/s (464kB/s)(5264KiB/11609msec) slat (usec): min=14, max=2495, avg=99.97, stdev=195.24 clat (msec): min=20, max=170, avg=71.52, stdev=17.70 lat (msec): min=22, max=170, avg=71.63, stdev=17.70 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 49], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 69], 60.00th=[ 73], | 70.00th=[ 79], 80.00th=[ 86], 90.00th=[ 96], 95.00th=[ 106], | 99.00th=[ 125], 99.50th=[ 133], 99.90th=[ 153], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 304, max= 592, per=25.47%, avg=454.57, stdev=83.22, samples=23 iops : min= 76, max= 148, avg=113.48, stdev=20.86, samples=23 lat (msec) : 10=0.04%, 20=0.04%, 50=10.35%, 100=83.32%, 250=6.25% cpu : usr=1.03%, sys=5.21%, ctx=2314, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21291: Fri May 2 00:29:22 2025 read: IOPS=106, BW=427KiB/s (437kB/s)(4984KiB/11673msec) slat (usec): min=1306, max=13850, avg=4684.97, stdev=1988.48 clat (msec): min=19, max=141, avg=68.10, stdev=17.45 lat (msec): min=25, max=144, avg=72.79, stdev=18.11 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 83], 90.00th=[ 92], 95.00th=[ 99], | 99.00th=[ 118], 99.50th=[ 125], 99.90th=[ 138], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 303, max= 608, per=24.97%, avg=428.00, stdev=81.48, samples=23 iops : min= 75, max= 152, avg=106.78, stdev=20.45, samples=23 write: IOPS=112, BW=450KiB/s (461kB/s)(5256KiB/11673msec) slat (usec): min=14, max=3448, avg=101.75, stdev=205.94 clat (msec): min=36, max=142, avg=72.20, stdev=17.79 lat (msec): min=36, max=142, avg=72.30, stdev=17.79 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 49], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 87], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 127], 99.50th=[ 133], 99.90th=[ 140], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 272, max= 598, per=25.17%, avg=449.22, stdev=76.21, samples=23 iops : min= 68, max= 149, avg=112.09, stdev=19.07, samples=23 lat (msec) : 20=0.04%, 50=10.51%, 100=83.32%, 250=6.13% cpu : usr=1.56%, sys=4.70%, ctx=2317, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21292: Fri May 2 00:29:22 2025 read: IOPS=108, BW=433KiB/s (443kB/s)(5060KiB/11699msec) slat (usec): min=1016, max=15021, avg=4736.21, stdev=2041.02 clat (msec): min=13, max=142, avg=68.66, stdev=17.32 lat (msec): min=19, max=144, avg=73.40, stdev=18.00 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 45], 10.00th=[ 49], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 71], | 70.00th=[ 77], 80.00th=[ 84], 90.00th=[ 92], 95.00th=[ 101], | 99.00th=[ 121], 99.50th=[ 124], 99.90th=[ 132], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 328, max= 558, per=25.25%, avg=432.87, stdev=66.83, samples=23 iops : min= 82, max= 139, avg=108.00, stdev=16.71, samples=23 write: IOPS=110, BW=443KiB/s (453kB/s)(5180KiB/11699msec) slat (usec): min=12, max=3336, avg=106.08, stdev=215.03 clat (msec): min=35, max=148, avg=71.81, stdev=17.56 lat (msec): min=35, max=148, avg=71.92, stdev=17.56 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 86], 90.00th=[ 96], 95.00th=[ 105], | 99.00th=[ 124], 99.50th=[ 127], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 280, max= 632, per=24.74%, avg=441.57, stdev=91.53, samples=23 iops : min= 70, max= 158, avg=110.13, stdev=22.89, samples=23 lat (msec) : 20=0.04%, 50=10.23%, 100=83.79%, 250=5.94% cpu : usr=0.57%, sys=5.85%, ctx=2381, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1715KiB/s (1756kB/s), 427KiB/s-433KiB/s (437kB/s-443kB/s), io=19.6MiB (20.5MB), run=11609-11704msec WRITE: bw=1785KiB/s (1828kB/s), 443KiB/s-453KiB/s (453kB/s-464kB/s), io=20.4MiB (21.4MB), run=11609-11704msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=21305: Fri May 2 00:29:24 2025 read: IOPS=4, BW=163MiB/s (171MB/s)(40.0MiB/246msec) slat (nsec): min=5885.8k, max=5885.8k, avg=5885842.00, stdev= 0.00 clat (nsec): min=222902k, max=222902k, avg=222901961.00, stdev= 0.00 lat (nsec): min=228817k, max=228817k, avg=228817251.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] lat (msec) : 250=100.00% cpu : usr=0.41%, sys=0.82%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=40.0MiB (41.9MB), run=246-246msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout