== sanity test 398c: run fio to test AIO ================= 21:44:58 (1746150298) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.23966 s, 175 MB/s osc.lustre-OST0000-osc-ffff88012b4fd000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=24441: Thu May 1 21:45:17 2025 write: IOPS=161, BW=645KiB/s (660kB/s)(10.0MiB/15882msec) slat (usec): min=11, max=4435, avg=134.33, stdev=326.75 clat (msec): min=5, max=264, avg=98.76, stdev=29.46 lat (msec): min=6, max=264, avg=98.89, stdev=29.46 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 75], | 30.00th=[ 83], 40.00th=[ 89], 50.00th=[ 96], 60.00th=[ 102], | 70.00th=[ 109], 80.00th=[ 117], 90.00th=[ 130], 95.00th=[ 148], | 99.00th=[ 215], 99.50th=[ 228], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 266] bw ( KiB/s): min= 328, max= 904, per=25.02%, avg=645.06, stdev=135.00, samples=31 iops : min= 82, max= 226, avg=161.16, stdev=33.80, samples=31 lat (msec) : 10=0.04%, 20=0.04%, 50=0.51%, 100=57.19%, 250=42.19% lat (msec) : 500=0.04% cpu : usr=0.36%, sys=2.89%, ctx=1230, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24442: Thu May 1 21:45:17 2025 write: IOPS=161, BW=645KiB/s (661kB/s)(10.0MiB/15870msec) slat (usec): min=11, max=4292, avg=128.44, stdev=331.73 clat (msec): min=14, max=262, avg=98.69, stdev=29.19 lat (msec): min=14, max=262, avg=98.82, stdev=29.20 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 77], | 30.00th=[ 83], 40.00th=[ 90], 50.00th=[ 96], 60.00th=[ 103], | 70.00th=[ 108], 80.00th=[ 117], 90.00th=[ 129], 95.00th=[ 153], | 99.00th=[ 213], 99.50th=[ 230], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 262] bw ( KiB/s): min= 360, max= 928, per=24.91%, avg=642.16, stdev=130.97, samples=31 iops : min= 90, max= 232, avg=160.35, stdev=32.76, samples=31 lat (msec) : 20=0.12%, 50=0.78%, 100=56.88%, 250=42.19%, 500=0.04% cpu : usr=0.46%, sys=2.68%, ctx=1242, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24443: Thu May 1 21:45:17 2025 write: IOPS=161, BW=647KiB/s (662kB/s)(10.0MiB/15835msec) slat (usec): min=11, max=6251, avg=97.64, stdev=281.31 clat (msec): min=18, max=256, avg=98.54, stdev=29.69 lat (msec): min=18, max=256, avg=98.64, stdev=29.70 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 83], 40.00th=[ 89], 50.00th=[ 95], 60.00th=[ 101], | 70.00th=[ 109], 80.00th=[ 117], 90.00th=[ 130], 95.00th=[ 144], | 99.00th=[ 213], 99.50th=[ 230], 99.90th=[ 253], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 303, max= 904, per=25.03%, avg=645.39, stdev=138.55, samples=31 iops : min= 75, max= 226, avg=161.16, stdev=34.67, samples=31 lat (msec) : 20=0.04%, 50=0.86%, 100=58.20%, 250=40.74%, 500=0.16% cpu : usr=0.26%, sys=2.36%, ctx=921, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24444: Thu May 1 21:45:17 2025 write: IOPS=161, BW=645KiB/s (660kB/s)(10.0MiB/15888msec) slat (usec): min=11, max=6742, avg=98.24, stdev=285.47 clat (msec): min=24, max=255, avg=98.78, stdev=28.92 lat (msec): min=25, max=255, avg=98.88, stdev=28.93 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 77], | 30.00th=[ 84], 40.00th=[ 90], 50.00th=[ 95], 60.00th=[ 102], | 70.00th=[ 109], 80.00th=[ 117], 90.00th=[ 128], 95.00th=[ 153], | 99.00th=[ 224], 99.50th=[ 226], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 288, max= 896, per=24.87%, avg=641.23, stdev=146.23, samples=31 iops : min= 72, max= 224, avg=160.16, stdev=36.57, samples=31 lat (msec) : 50=0.82%, 100=57.54%, 250=41.48%, 500=0.16% cpu : usr=0.30%, sys=2.11%, ctx=761, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2578KiB/s (2640kB/s), 645KiB/s-647KiB/s (660kB/s-662kB/s), io=40.0MiB (41.9MB), run=15835-15888msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24467: Thu May 1 21:45:34 2025 read: IOPS=80, BW=323KiB/s (330kB/s)(5048KiB/15652msec) slat (usec): min=1326, max=22571, avg=6319.61, stdev=2796.56 clat (msec): min=14, max=172, avg=91.84, stdev=23.14 lat (msec): min=22, max=180, avg=98.17, stdev=24.00 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 89], 60.00th=[ 95], | 70.00th=[ 102], 80.00th=[ 110], 90.00th=[ 125], 95.00th=[ 136], | 99.00th=[ 157], 99.50th=[ 163], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 192, max= 422, per=25.06%, avg=321.29, stdev=50.02, samples=31 iops : min= 48, max= 105, avg=80.03, stdev=12.58, samples=31 write: IOPS=82, BW=332KiB/s (340kB/s)(5192KiB/15652msec) slat (usec): min=11, max=7431, avg=130.35, stdev=376.56 clat (msec): min=19, max=168, avg=95.23, stdev=21.09 lat (msec): min=19, max=168, avg=95.36, stdev=21.11 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 65], 10.00th=[ 69], 20.00th=[ 78], | 30.00th=[ 84], 40.00th=[ 89], 50.00th=[ 94], 60.00th=[ 100], | 70.00th=[ 105], 80.00th=[ 112], 90.00th=[ 124], 95.00th=[ 133], | 99.00th=[ 157], 99.50th=[ 161], 99.90th=[ 167], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 207, max= 526, per=24.80%, avg=330.81, stdev=68.46, samples=31 iops : min= 51, max= 131, avg=82.42, stdev=17.17, samples=31 lat (msec) : 20=0.08%, 50=0.55%, 100=64.22%, 250=35.16% cpu : usr=0.69%, sys=5.07%, ctx=2248, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24468: Thu May 1 21:45:34 2025 read: IOPS=80, BW=321KiB/s (329kB/s)(4976KiB/15497msec) slat (usec): min=19, max=21338, avg=6409.82, stdev=2851.22 clat (msec): min=16, max=195, avg=91.29, stdev=23.95 lat (msec): min=18, max=201, avg=97.71, stdev=24.88 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 84], 50.00th=[ 89], 60.00th=[ 95], | 70.00th=[ 102], 80.00th=[ 109], 90.00th=[ 123], 95.00th=[ 136], | 99.00th=[ 163], 99.50th=[ 174], 99.90th=[ 190], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 216, max= 424, per=25.05%, avg=321.13, stdev=56.62, samples=30 iops : min= 54, max= 106, avg=80.03, stdev=14.23, samples=30 write: IOPS=84, BW=340KiB/s (348kB/s)(5264KiB/15497msec) slat (usec): min=16, max=3864, avg=129.15, stdev=295.82 clat (msec): min=29, max=210, avg=95.55, stdev=22.23 lat (msec): min=29, max=211, avg=95.68, stdev=22.22 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 65], 10.00th=[ 70], 20.00th=[ 78], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 94], 60.00th=[ 100], | 70.00th=[ 105], 80.00th=[ 112], 90.00th=[ 125], 95.00th=[ 136], | 99.00th=[ 163], 99.50th=[ 171], 99.90th=[ 201], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 215, max= 472, per=25.51%, avg=340.33, stdev=54.77, samples=30 iops : min= 53, max= 118, avg=84.83, stdev=13.70, samples=30 lat (msec) : 20=0.12%, 50=0.90%, 100=64.38%, 250=34.61% cpu : usr=0.46%, sys=5.22%, ctx=2323, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24469: Thu May 1 21:45:34 2025 read: IOPS=79, BW=320KiB/s (327kB/s)(4984KiB/15598msec) slat (usec): min=1249, max=23562, avg=6467.88, stdev=2852.18 clat (msec): min=38, max=227, avg=92.83, stdev=23.76 lat (msec): min=43, max=240, avg=99.31, stdev=24.55 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 84], 50.00th=[ 89], 60.00th=[ 96], | 70.00th=[ 103], 80.00th=[ 110], 90.00th=[ 126], 95.00th=[ 138], | 99.00th=[ 161], 99.50th=[ 174], 99.90th=[ 222], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 192, max= 464, per=24.94%, avg=319.68, stdev=57.74, samples=31 iops : min= 48, max= 116, avg=79.81, stdev=14.42, samples=31 write: IOPS=84, BW=337KiB/s (345kB/s)(5256KiB/15598msec) slat (usec): min=15, max=2914, avg=124.22, stdev=273.09 clat (msec): min=24, max=221, avg=94.44, stdev=22.44 lat (msec): min=24, max=221, avg=94.57, stdev=22.45 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 92], 60.00th=[ 97], | 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 125], 95.00th=[ 134], | 99.00th=[ 161], 99.50th=[ 163], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 192, max= 496, per=25.22%, avg=336.45, stdev=71.58, samples=31 iops : min= 48, max= 124, avg=84.03, stdev=17.84, samples=31 lat (msec) : 50=0.74%, 100=64.80%, 250=34.45% cpu : usr=0.15%, sys=5.51%, ctx=2263, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24470: Thu May 1 21:45:34 2025 read: IOPS=80, BW=324KiB/s (332kB/s)(5060KiB/15628msec) slat (usec): min=1584, max=27112, avg=6363.66, stdev=2842.60 clat (msec): min=18, max=166, avg=92.46, stdev=22.23 lat (msec): min=26, max=174, avg=98.83, stdev=23.13 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 84], 50.00th=[ 89], 60.00th=[ 95], | 70.00th=[ 103], 80.00th=[ 110], 90.00th=[ 123], 95.00th=[ 138], | 99.00th=[ 153], 99.50th=[ 159], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 216, max= 464, per=25.26%, avg=323.81, stdev=57.11, samples=31 iops : min= 54, max= 116, avg=80.65, stdev=14.37, samples=31 write: IOPS=82, BW=331KiB/s (339kB/s)(5180KiB/15628msec) slat (usec): min=12, max=10558, avg=135.53, stdev=495.80 clat (msec): min=16, max=182, avg=94.81, stdev=21.46 lat (msec): min=16, max=182, avg=94.94, stdev=21.48 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 65], 10.00th=[ 70], 20.00th=[ 78], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 97], | 70.00th=[ 104], 80.00th=[ 111], 90.00th=[ 122], 95.00th=[ 134], | 99.00th=[ 163], 99.50th=[ 169], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 231, max= 464, per=24.72%, avg=329.77, stdev=60.93, samples=31 iops : min= 57, max= 116, avg=82.16, stdev=15.28, samples=31 lat (msec) : 20=0.12%, 50=0.23%, 100=65.27%, 250=34.38% cpu : usr=1.11%, sys=4.51%, ctx=2285, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1282KiB/s (1313kB/s), 320KiB/s-324KiB/s (327kB/s-332kB/s), io=19.6MiB (20.5MB), run=15497-15652msec WRITE: bw=1335KiB/s (1367kB/s), 331KiB/s-340KiB/s (339kB/s-348kB/s), io=20.4MiB (21.4MB), run=15497-15652msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=24487: Thu May 1 21:45:36 2025 read: IOPS=2, BW=105MiB/s (110MB/s)(40.0MiB/382msec) slat (nsec): min=8623.7k, max=8623.7k, avg=8623705.00, stdev= 0.00 clat (nsec): min=350096k, max=350096k, avg=350095735.00, stdev= 0.00 lat (nsec): min=358744k, max=358744k, avg=358744332.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.79%, ctx=14, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=40.0MiB (41.9MB), run=382-382msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout