== sanity test 398c: run fio to test AIO ================= 08:06:18 (1744027578) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0848319 s, 494 MB/s osc.lustre-OST0000-osc-ffff88012ab17800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=10816: Mon Apr 7 08:06:44 2025 write: IOPS=100, BW=403KiB/s (413kB/s)(10.0MiB/25400msec) slat (usec): min=13, max=374, avg=74.84, stdev=42.70 clat (msec): min=59, max=256, avg=158.42, stdev=29.59 lat (msec): min=59, max=256, avg=158.50, stdev=29.59 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 122], 20.00th=[ 134], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 197], 95.00th=[ 209], | 99.00th=[ 236], 99.50th=[ 245], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 257] bw ( KiB/s): min= 272, max= 520, per=24.93%, avg=401.92, stdev=47.68, samples=50 iops : min= 68, max= 130, avg=100.38, stdev=11.92, samples=50 lat (msec) : 100=1.52%, 250=98.16%, 500=0.31% cpu : usr=0.19%, sys=0.82%, ctx=2415, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=10817: Mon Apr 7 08:06:44 2025 write: IOPS=100, BW=403KiB/s (413kB/s)(10.0MiB/25400msec) slat (usec): min=13, max=1082, avg=72.24, stdev=47.70 clat (msec): min=56, max=277, avg=158.45, stdev=30.27 lat (msec): min=56, max=277, avg=158.52, stdev=30.27 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 112], 10.00th=[ 121], 20.00th=[ 134], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 197], 95.00th=[ 209], | 99.00th=[ 243], 99.50th=[ 247], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 279] bw ( KiB/s): min= 280, max= 518, per=24.98%, avg=402.72, stdev=47.11, samples=50 iops : min= 70, max= 129, avg=100.58, stdev=11.80, samples=50 lat (msec) : 100=1.17%, 250=98.44%, 500=0.39% cpu : usr=0.15%, sys=0.83%, ctx=2207, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=10818: Mon Apr 7 08:06:44 2025 write: IOPS=100, BW=403KiB/s (413kB/s)(10.0MiB/25409msec) slat (usec): min=13, max=493, avg=73.26, stdev=46.94 clat (msec): min=75, max=256, avg=158.48, stdev=29.07 lat (msec): min=75, max=256, avg=158.56, stdev=29.07 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 113], 10.00th=[ 122], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 197], 95.00th=[ 209], | 99.00th=[ 232], 99.50th=[ 243], 99.90th=[ 251], 99.95th=[ 255], | 99.99th=[ 257] bw ( KiB/s): min= 256, max= 512, per=24.90%, avg=401.44, stdev=49.62, samples=50 iops : min= 64, max= 128, avg=100.26, stdev=12.42, samples=50 lat (msec) : 100=1.33%, 250=98.55%, 500=0.12% cpu : usr=0.12%, sys=0.85%, ctx=2334, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=10819: Mon Apr 7 08:06:44 2025 write: IOPS=100, BW=403KiB/s (413kB/s)(10.0MiB/25380msec) slat (usec): min=12, max=1084, avg=74.81, stdev=52.57 clat (msec): min=21, max=268, avg=158.41, stdev=30.84 lat (msec): min=21, max=268, avg=158.49, stdev=30.84 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 113], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 197], 95.00th=[ 211], | 99.00th=[ 241], 99.50th=[ 247], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 264, max= 608, per=25.05%, avg=403.78, stdev=56.12, samples=50 iops : min= 66, max= 152, avg=100.90, stdev=14.04, samples=50 lat (msec) : 50=0.43%, 100=1.25%, 250=97.97%, 500=0.35% cpu : usr=0.13%, sys=0.84%, ctx=2484, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1612KiB/s (1651kB/s), 403KiB/s-403KiB/s (413kB/s-413kB/s), io=40.0MiB (41.9MB), run=25380-25409msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10849: Mon Apr 7 08:07:01 2025 read: IOPS=79, BW=317KiB/s (325kB/s)(5048KiB/15903msec) slat (usec): min=722, max=2949, avg=1190.09, stdev=281.71 clat (usec): min=1015, max=158002, avg=65317.49, stdev=12834.41 lat (msec): min=2, max=158, avg=66.51, stdev=12.84 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 97], 99.50th=[ 110], 99.90th=[ 146], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 216, max= 416, per=25.17%, avg=316.87, stdev=55.11, samples=31 iops : min= 54, max= 104, avg=79.19, stdev=13.75, samples=31 write: IOPS=81, BW=326KiB/s (334kB/s)(5192KiB/15903msec) slat (usec): min=14, max=254, avg=60.42, stdev=32.19 clat (msec): min=50, max=215, avg=130.92, stdev=28.47 lat (msec): min=50, max=215, avg=130.98, stdev=28.47 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 87], 10.00th=[ 95], 20.00th=[ 106], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 180], | 99.00th=[ 197], 99.50th=[ 203], 99.90th=[ 209], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 272, max= 400, per=24.90%, avg=326.42, stdev=39.04, samples=31 iops : min= 68, max= 100, avg=81.58, stdev= 9.76, samples=31 lat (msec) : 2=0.04%, 10=0.12%, 20=0.16%, 50=3.36%, 100=52.62% lat (msec) : 250=43.71% cpu : usr=0.23%, sys=1.69%, ctx=2912, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10850: Mon Apr 7 08:07:01 2025 read: IOPS=78, BW=312KiB/s (320kB/s)(4976KiB/15929msec) slat (usec): min=633, max=3053, avg=1209.57, stdev=301.92 clat (usec): min=1461, max=104109, avg=64486.15, stdev=11994.20 lat (msec): min=2, max=104, avg=65.70, stdev=11.99 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 79], 95.00th=[ 83], | 99.00th=[ 90], 99.50th=[ 94], 99.90th=[ 104], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 152, max= 432, per=24.71%, avg=311.16, stdev=68.51, samples=31 iops : min= 38, max= 108, avg=77.74, stdev=17.08, samples=31 write: IOPS=82, BW=330KiB/s (338kB/s)(5264KiB/15929msec) slat (usec): min=12, max=304, avg=61.59, stdev=34.50 clat (msec): min=47, max=217, avg=131.29, stdev=29.50 lat (msec): min=47, max=217, avg=131.35, stdev=29.49 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 84], 10.00th=[ 93], 20.00th=[ 106], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 157], 90.00th=[ 169], 95.00th=[ 180], | 99.00th=[ 197], 99.50th=[ 205], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 240, max= 424, per=25.15%, avg=329.74, stdev=41.13, samples=31 iops : min= 60, max= 106, avg=82.39, stdev=10.27, samples=31 lat (msec) : 2=0.04%, 10=0.12%, 20=0.39%, 50=3.71%, 100=52.34% lat (msec) : 250=43.40% cpu : usr=0.19%, sys=1.76%, ctx=2972, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10851: Mon Apr 7 08:07:01 2025 read: IOPS=78, BW=313KiB/s (320kB/s)(4984KiB/15929msec) slat (usec): min=12, max=3215, avg=1193.91, stdev=322.76 clat (usec): min=1567, max=123943, avg=64741.04, stdev=12123.44 lat (msec): min=2, max=125, avg=65.94, stdev=12.16 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 64], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 83], | 99.00th=[ 91], 99.50th=[ 93], 99.90th=[ 97], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 216, max= 456, per=24.98%, avg=314.55, stdev=64.16, samples=31 iops : min= 54, max= 114, avg=78.61, stdev=16.01, samples=31 write: IOPS=82, BW=330KiB/s (338kB/s)(5256KiB/15929msec) slat (usec): min=12, max=1100, avg=60.75, stdev=43.82 clat (msec): min=16, max=224, avg=131.24, stdev=30.60 lat (msec): min=16, max=224, avg=131.31, stdev=30.60 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 83], 10.00th=[ 94], 20.00th=[ 107], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 182], | 99.00th=[ 197], 99.50th=[ 207], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 256, max= 408, per=25.02%, avg=327.97, stdev=39.51, samples=31 iops : min= 64, max= 102, avg=81.97, stdev= 9.87, samples=31 lat (msec) : 2=0.08%, 4=0.12%, 10=0.04%, 20=0.27%, 50=3.83% lat (msec) : 100=52.07%, 250=43.59% cpu : usr=0.28%, sys=1.64%, ctx=2789, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10852: Mon Apr 7 08:07:01 2025 read: IOPS=79, BW=318KiB/s (325kB/s)(5060KiB/15929msec) slat (usec): min=657, max=3026, avg=1192.50, stdev=285.55 clat (msec): min=2, max=146, avg=65.05, stdev=13.63 lat (msec): min=3, max=148, avg=66.24, stdev=13.65 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 72], 80.00th=[ 75], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 96], 99.50th=[ 103], 99.90th=[ 146], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 200, max= 416, per=25.35%, avg=319.19, stdev=54.26, samples=31 iops : min= 50, max= 104, avg=79.77, stdev=13.53, samples=31 write: IOPS=81, BW=325KiB/s (333kB/s)(5180KiB/15929msec) slat (usec): min=13, max=495, avg=60.20, stdev=34.65 clat (msec): min=43, max=215, avg=131.75, stdev=29.50 lat (msec): min=43, max=216, avg=131.81, stdev=29.50 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 85], 10.00th=[ 93], 20.00th=[ 108], | 30.00th=[ 117], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 169], 95.00th=[ 184], | 99.00th=[ 205], 99.50th=[ 207], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 248, max= 376, per=24.72%, avg=324.10, stdev=32.03, samples=31 iops : min= 62, max= 94, avg=81.00, stdev= 7.99, samples=31 lat (msec) : 4=0.04%, 10=0.04%, 20=0.55%, 50=3.71%, 100=52.07% lat (msec) : 250=43.59% cpu : usr=0.15%, sys=1.76%, ctx=2923, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1260KiB/s (1290kB/s), 312KiB/s-318KiB/s (320kB/s-325kB/s), io=19.6MiB (20.5MB), run=15903-15929msec WRITE: bw=1312KiB/s (1343kB/s), 325KiB/s-330KiB/s (333kB/s-338kB/s), io=20.4MiB (21.4MB), run=15903-15929msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=10868: Mon Apr 7 08:07:03 2025 read: IOPS=10, BW=400MiB/s (419MB/s)(40.0MiB/100msec) slat (nsec): min=7039.6k, max=7039.6k, avg=7039645.00, stdev= 0.00 clat (nsec): min=88486k, max=88486k, avg=88486473.00, stdev= 0.00 lat (nsec): min=95539k, max=95539k, avg=95539495.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[88605], 5.00th=[88605], 10.00th=[88605], 20.00th=[88605], | 30.00th=[88605], 40.00th=[88605], 50.00th=[88605], 60.00th=[88605], | 70.00th=[88605], 80.00th=[88605], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=8.08%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=400MiB/s (419MB/s), 400MiB/s-400MiB/s (419MB/s-419MB/s), io=40.0MiB (41.9MB), run=100-100msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout