== sanity test 398c: run fio to test AIO ================= 09:08:46 (1740492526) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.112126 s, 374 MB/s osc.lustre-OST0000-osc-ffff880099b68000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=6653: Tue Feb 25 09:09:18 2025 write: IOPS=85, BW=343KiB/s (351kB/s)(10.0MiB/29847msec) slat (usec): min=13, max=262, avg=49.25, stdev=24.02 clat (msec): min=98, max=362, avg=186.23, stdev=38.62 lat (msec): min=98, max=362, avg=186.28, stdev=38.62 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 253], | 99.00th=[ 313], 99.50th=[ 338], 99.90th=[ 351], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 200, max= 512, per=24.94%, avg=341.97, stdev=54.47, samples=59 iops : min= 50, max= 128, avg=85.49, stdev=13.62, samples=59 lat (msec) : 100=0.66%, 250=93.63%, 500=5.70% cpu : usr=0.07%, sys=0.58%, ctx=1911, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=6654: Tue Feb 25 09:09:18 2025 write: IOPS=85, BW=344KiB/s (352kB/s)(10.0MiB/29800msec) slat (usec): min=13, max=316, avg=51.33, stdev=27.80 clat (msec): min=24, max=367, avg=186.05, stdev=40.31 lat (msec): min=24, max=368, avg=186.10, stdev=40.31 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 128], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 232], 95.00th=[ 255], | 99.00th=[ 313], 99.50th=[ 330], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 368] bw ( KiB/s): min= 200, max= 496, per=25.03%, avg=343.14, stdev=56.97, samples=59 iops : min= 50, max= 124, avg=85.75, stdev=14.24, samples=59 lat (msec) : 50=0.20%, 100=0.90%, 250=92.50%, 500=6.41% cpu : usr=0.12%, sys=0.52%, ctx=2194, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=6655: Tue Feb 25 09:09:18 2025 write: IOPS=85, BW=343KiB/s (352kB/s)(10.0MiB/29819msec) slat (usec): min=13, max=214, avg=47.15, stdev=21.92 clat (msec): min=62, max=367, avg=186.12, stdev=39.30 lat (msec): min=62, max=368, avg=186.17, stdev=39.30 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 127], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 232], 95.00th=[ 251], | 99.00th=[ 317], 99.50th=[ 334], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 368] bw ( KiB/s): min= 168, max= 512, per=24.97%, avg=342.32, stdev=55.45, samples=59 iops : min= 42, max= 128, avg=85.54, stdev=13.86, samples=59 lat (msec) : 100=0.08%, 250=94.80%, 500=5.12% cpu : usr=0.10%, sys=0.53%, ctx=1773, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=6656: Tue Feb 25 09:09:18 2025 write: IOPS=85, BW=343KiB/s (351kB/s)(10.0MiB/29867msec) slat (usec): min=13, max=298, avg=50.82, stdev=28.40 clat (msec): min=47, max=364, avg=186.36, stdev=39.10 lat (msec): min=47, max=364, avg=186.41, stdev=39.10 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 313], 99.50th=[ 334], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 192, max= 512, per=24.84%, avg=340.59, stdev=53.70, samples=59 iops : min= 48, max= 128, avg=85.03, stdev=13.47, samples=59 lat (msec) : 50=0.04%, 100=0.86%, 250=93.52%, 500=5.59% cpu : usr=0.08%, sys=0.56%, ctx=2048, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1371KiB/s (1404kB/s), 343KiB/s-344KiB/s (351kB/s-352kB/s), io=40.0MiB (41.9MB), run=29800-29867msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6690: Tue Feb 25 09:09:36 2025 read: IOPS=73, BW=296KiB/s (303kB/s)(5048KiB/17056msec) slat (usec): min=19, max=3534, avg=1287.59, stdev=364.86 clat (usec): min=1975, max=180763, avg=69769.23, stdev=14677.93 lat (msec): min=3, max=182, avg=71.06, stdev=14.73 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 51], 10.00th=[ 55], 20.00th=[ 60], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 90], | 99.00th=[ 103], 99.50th=[ 133], 99.90th=[ 180], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 208, max= 383, per=25.12%, avg=295.18, stdev=46.34, samples=34 iops : min= 52, max= 95, avg=73.71, stdev=11.54, samples=34 write: IOPS=76, BW=304KiB/s (312kB/s)(5192KiB/17056msec) slat (usec): min=14, max=409, avg=53.08, stdev=26.14 clat (msec): min=28, max=250, avg=140.93, stdev=30.94 lat (msec): min=28, max=250, avg=140.99, stdev=30.94 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 96], 10.00th=[ 104], 20.00th=[ 116], | 30.00th=[ 124], 40.00th=[ 132], 50.00th=[ 140], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 167], 90.00th=[ 184], 95.00th=[ 194], | 99.00th=[ 224], 99.50th=[ 232], 99.90th=[ 247], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 208, max= 392, per=24.81%, avg=303.41, stdev=34.19, samples=34 iops : min= 52, max= 98, avg=75.76, stdev= 8.61, samples=34 lat (msec) : 2=0.04%, 4=0.04%, 20=0.27%, 50=2.34%, 100=49.30% lat (msec) : 250=47.97%, 500=0.04% cpu : usr=0.21%, sys=1.44%, ctx=3173, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6691: Tue Feb 25 09:09:36 2025 read: IOPS=72, BW=292KiB/s (299kB/s)(4976KiB/17056msec) slat (usec): min=722, max=3648, avg=1293.95, stdev=356.96 clat (usec): min=1500, max=179710, avg=69753.73, stdev=14807.55 lat (msec): min=2, max=180, avg=71.05, stdev=14.83 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 59], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 80], 90.00th=[ 86], 95.00th=[ 92], | 99.00th=[ 104], 99.50th=[ 118], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 184, max= 416, per=24.76%, avg=290.94, stdev=49.73, samples=34 iops : min= 46, max= 104, avg=72.65, stdev=12.51, samples=34 write: IOPS=77, BW=309KiB/s (316kB/s)(5264KiB/17056msec) slat (usec): min=14, max=345, avg=53.57, stdev=26.67 clat (msec): min=36, max=248, avg=139.87, stdev=30.60 lat (msec): min=36, max=248, avg=139.92, stdev=30.60 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 97], 10.00th=[ 104], 20.00th=[ 113], | 30.00th=[ 122], 40.00th=[ 130], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 182], 95.00th=[ 192], | 99.00th=[ 230], 99.50th=[ 232], 99.90th=[ 247], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 200, max= 408, per=25.16%, avg=307.65, stdev=40.22, samples=34 iops : min= 50, max= 102, avg=76.82, stdev=10.01, samples=34 lat (msec) : 2=0.08%, 10=0.08%, 20=0.35%, 50=1.84%, 100=49.22% lat (msec) : 250=48.44% cpu : usr=0.18%, sys=1.48%, ctx=3224, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6692: Tue Feb 25 09:09:36 2025 read: IOPS=73, BW=292KiB/s (299kB/s)(4984KiB/17066msec) slat (usec): min=717, max=5029, avg=1296.49, stdev=377.81 clat (usec): min=1239, max=114882, avg=69790.70, stdev=13064.90 lat (msec): min=2, max=116, avg=71.09, stdev=13.07 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 90], | 99.00th=[ 104], 99.50th=[ 112], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 176, max= 408, per=24.88%, avg=292.35, stdev=55.09, samples=34 iops : min= 44, max= 102, avg=73.00, stdev=13.73, samples=34 write: IOPS=76, BW=308KiB/s (315kB/s)(5256KiB/17066msec) slat (usec): min=14, max=285, avg=54.41, stdev=26.62 clat (msec): min=67, max=255, avg=140.00, stdev=29.95 lat (msec): min=67, max=255, avg=140.05, stdev=29.95 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 95], 10.00th=[ 106], 20.00th=[ 116], | 30.00th=[ 123], 40.00th=[ 130], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 182], 95.00th=[ 190], | 99.00th=[ 220], 99.50th=[ 232], 99.90th=[ 249], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 240, max= 360, per=25.04%, avg=306.24, stdev=32.17, samples=34 iops : min= 60, max= 90, avg=76.47, stdev= 8.06, samples=34 lat (msec) : 2=0.08%, 20=0.31%, 50=2.03%, 100=49.06%, 250=48.48% lat (msec) : 500=0.04% cpu : usr=0.24%, sys=1.38%, ctx=3412, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6693: Tue Feb 25 09:09:36 2025 read: IOPS=74, BW=296KiB/s (303kB/s)(5060KiB/17075msec) slat (usec): min=679, max=4008, avg=1287.90, stdev=334.88 clat (msec): min=2, max=142, avg=70.24, stdev=13.27 lat (msec): min=4, max=144, avg=71.53, stdev=13.29 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 50], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 91], | 99.00th=[ 102], 99.50th=[ 112], 99.90th=[ 116], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 192, max= 400, per=25.22%, avg=296.35, stdev=48.34, samples=34 iops : min= 48, max= 100, avg=74.00, stdev=12.10, samples=34 write: IOPS=75, BW=303KiB/s (311kB/s)(5180KiB/17075msec) slat (usec): min=14, max=321, avg=55.29, stdev=28.76 clat (msec): min=24, max=249, avg=140.73, stdev=31.39 lat (msec): min=24, max=249, avg=140.79, stdev=31.39 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 94], 10.00th=[ 103], 20.00th=[ 113], | 30.00th=[ 124], 40.00th=[ 131], 50.00th=[ 140], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 167], 90.00th=[ 184], 95.00th=[ 194], | 99.00th=[ 224], 99.50th=[ 234], 99.90th=[ 247], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 240, max= 360, per=24.71%, avg=302.24, stdev=29.69, samples=34 iops : min= 60, max= 90, avg=75.47, stdev= 7.41, samples=34 lat (msec) : 4=0.04%, 10=0.04%, 20=0.31%, 50=2.27%, 100=50.59% lat (msec) : 250=46.76% cpu : usr=0.16%, sys=1.46%, ctx=3462, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1175KiB/s (1203kB/s), 292KiB/s-296KiB/s (299kB/s-303kB/s), io=19.6MiB (20.5MB), run=17056-17075msec WRITE: bw=1224KiB/s (1253kB/s), 303KiB/s-309KiB/s (311kB/s-316kB/s), io=20.4MiB (21.4MB), run=17056-17075msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=6710: Tue Feb 25 09:09:38 2025 read: IOPS=5, BW=234MiB/s (245MB/s)(40.0MiB/171msec) slat (nsec): min=5920.6k, max=5920.6k, avg=5920603.00, stdev= 0.00 clat (nsec): min=159174k, max=159174k, avg=159173659.00, stdev= 0.00 lat (nsec): min=165146k, max=165146k, avg=165146262.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.12%, ctx=11, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=40.0MiB (41.9MB), run=171-171msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout