== sanity test 398c: run fio to test AIO ================= 07:16:48 (1734437808) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.152557 s, 275 MB/s osc.lustre-OST0000-osc-ffff8800b63eb800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=5304: Tue Dec 17 07:17:16 2024 write: IOPS=95, BW=382KiB/s (392kB/s)(10.0MiB/26780msec) slat (usec): min=13, max=429, avg=77.01, stdev=41.73 clat (msec): min=74, max=339, avg=167.00, stdev=30.46 lat (msec): min=74, max=339, avg=167.08, stdev=30.46 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 120], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 207], 95.00th=[ 220], | 99.00th=[ 245], 99.50th=[ 255], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 312, max= 512, per=25.00%, avg=382.30, stdev=38.85, samples=53 iops : min= 78, max= 128, avg=95.45, stdev= 9.71, samples=53 lat (msec) : 100=0.98%, 250=98.24%, 500=0.78% cpu : usr=0.21%, sys=0.78%, ctx=2742, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5305: Tue Dec 17 07:17:16 2024 write: IOPS=95, BW=382KiB/s (392kB/s)(10.0MiB/26780msec) slat (usec): min=14, max=582, avg=76.54, stdev=43.73 clat (msec): min=17, max=267, avg=166.91, stdev=28.74 lat (msec): min=17, max=267, avg=166.99, stdev=28.75 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 121], 10.00th=[ 132], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 207], 95.00th=[ 215], | 99.00th=[ 236], 99.50th=[ 243], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 268] bw ( KiB/s): min= 312, max= 512, per=24.92%, avg=380.96, stdev=35.29, samples=53 iops : min= 78, max= 128, avg=95.23, stdev= 8.82, samples=53 lat (msec) : 20=0.08%, 100=0.39%, 250=99.22%, 500=0.31% cpu : usr=0.16%, sys=0.77%, ctx=2902, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5306: Tue Dec 17 07:17:16 2024 write: IOPS=95, BW=384KiB/s (393kB/s)(10.0MiB/26695msec) slat (usec): min=13, max=412, avg=79.30, stdev=46.01 clat (msec): min=29, max=288, avg=166.54, stdev=31.02 lat (msec): min=29, max=288, avg=166.62, stdev=31.02 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 118], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 207], 95.00th=[ 218], | 99.00th=[ 239], 99.50th=[ 247], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 304, max= 552, per=25.07%, avg=383.36, stdev=44.14, samples=53 iops : min= 76, max= 138, avg=95.83, stdev=11.00, samples=53 lat (msec) : 50=0.51%, 100=0.98%, 250=98.12%, 500=0.39% cpu : usr=0.17%, sys=0.78%, ctx=2960, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5307: Tue Dec 17 07:17:16 2024 write: IOPS=95, BW=383KiB/s (392kB/s)(10.0MiB/26770msec) slat (usec): min=13, max=1879, avg=77.08, stdev=55.97 clat (msec): min=74, max=309, avg=166.96, stdev=29.86 lat (msec): min=74, max=309, avg=167.03, stdev=29.86 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 120], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 207], 95.00th=[ 218], | 99.00th=[ 243], 99.50th=[ 249], 99.90th=[ 288], 99.95th=[ 300], | 99.99th=[ 309] bw ( KiB/s): min= 296, max= 512, per=24.99%, avg=382.09, stdev=40.78, samples=53 iops : min= 74, max= 128, avg=95.47, stdev=10.19, samples=53 lat (msec) : 100=0.94%, 250=98.59%, 500=0.47% cpu : usr=0.14%, sys=0.81%, ctx=2759, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1529KiB/s (1566kB/s), 382KiB/s-384KiB/s (392kB/s-393kB/s), io=40.0MiB (41.9MB), run=26695-26780msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5338: Tue Dec 17 07:17:33 2024 read: IOPS=78, BW=316KiB/s (323kB/s)(5048KiB/15989msec) slat (usec): min=735, max=4833, avg=1215.35, stdev=297.08 clat (msec): min=8, max=105, avg=64.31, stdev=11.94 lat (msec): min=9, max=106, avg=65.52, stdev=11.97 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 67], | 70.00th=[ 70], 80.00th=[ 74], 90.00th=[ 79], 95.00th=[ 83], | 99.00th=[ 96], 99.50th=[ 103], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 192, max= 440, per=25.13%, avg=313.90, stdev=56.32, samples=31 iops : min= 48, max= 110, avg=78.35, stdev=14.12, samples=31 write: IOPS=81, BW=325KiB/s (333kB/s)(5192KiB/15989msec) slat (usec): min=14, max=219, avg=60.51, stdev=29.97 clat (msec): min=39, max=219, avg=132.99, stdev=31.05 lat (msec): min=39, max=219, avg=133.05, stdev=31.06 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 82], 10.00th=[ 93], 20.00th=[ 105], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 174], 95.00th=[ 184], | 99.00th=[ 205], 99.50th=[ 209], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 264, max= 376, per=24.72%, avg=321.39, stdev=27.46, samples=31 iops : min= 66, max= 94, avg=80.23, stdev= 6.89, samples=31 lat (msec) : 10=0.12%, 20=0.27%, 50=3.98%, 100=52.77%, 250=42.85% cpu : usr=0.26%, sys=1.58%, ctx=3289, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5339: Tue Dec 17 07:17:33 2024 read: IOPS=77, BW=310KiB/s (318kB/s)(4976KiB/16047msec) slat (usec): min=722, max=4122, avg=1220.38, stdev=286.56 clat (usec): min=1293, max=106642, avg=64630.81, stdev=11718.73 lat (msec): min=2, max=107, avg=65.85, stdev=11.74 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 65], 60.00th=[ 67], | 70.00th=[ 70], 80.00th=[ 73], 90.00th=[ 79], 95.00th=[ 84], | 99.00th=[ 97], 99.50th=[ 103], 99.90th=[ 105], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 216, max= 424, per=24.63%, avg=307.59, stdev=58.86, samples=32 iops : min= 54, max= 106, avg=76.78, stdev=14.72, samples=32 write: IOPS=82, BW=328KiB/s (336kB/s)(5264KiB/16047msec) slat (usec): min=13, max=334, avg=60.47, stdev=32.43 clat (msec): min=23, max=219, avg=132.60, stdev=32.81 lat (msec): min=23, max=219, avg=132.66, stdev=32.81 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 78], 10.00th=[ 90], 20.00th=[ 105], | 30.00th=[ 116], 40.00th=[ 124], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 205], 99.50th=[ 218], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 263, max= 400, per=25.14%, avg=326.84, stdev=40.49, samples=32 iops : min= 65, max= 100, avg=81.59, stdev=10.18, samples=32 lat (msec) : 2=0.04%, 10=0.08%, 20=0.23%, 50=3.63%, 100=52.62% lat (msec) : 250=43.40% cpu : usr=0.26%, sys=1.60%, ctx=3007, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5340: Tue Dec 17 07:17:33 2024 read: IOPS=77, BW=310KiB/s (318kB/s)(4984KiB/16062msec) slat (usec): min=724, max=2977, avg=1216.19, stdev=280.12 clat (usec): min=1660, max=103983, avg=65089.92, stdev=11979.91 lat (msec): min=2, max=105, avg=66.31, stdev=12.01 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 85], | 99.00th=[ 99], 99.50th=[ 102], 99.90th=[ 104], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 184, max= 424, per=24.75%, avg=309.09, stdev=58.05, samples=32 iops : min= 46, max= 106, avg=77.16, stdev=14.50, samples=32 write: IOPS=81, BW=327KiB/s (335kB/s)(5256KiB/16062msec) slat (usec): min=14, max=209, avg=61.01, stdev=31.48 clat (msec): min=39, max=221, avg=132.40, stdev=32.32 lat (msec): min=39, max=221, avg=132.46, stdev=32.32 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 79], 10.00th=[ 90], 20.00th=[ 106], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 201], 99.50th=[ 207], 99.90th=[ 218], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 264, max= 432, per=24.99%, avg=324.84, stdev=39.76, samples=32 iops : min= 66, max= 108, avg=81.09, stdev= 9.95, samples=32 lat (msec) : 2=0.04%, 10=0.08%, 20=0.20%, 50=3.83%, 100=52.54% lat (msec) : 250=43.32% cpu : usr=0.19%, sys=1.59%, ctx=3550, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5341: Tue Dec 17 07:17:33 2024 read: IOPS=79, BW=317KiB/s (325kB/s)(5060KiB/15938msec) slat (usec): min=50, max=3438, avg=1227.91, stdev=327.52 clat (msec): min=2, max=206, avg=65.18, stdev=14.41 lat (msec): min=3, max=207, avg=66.41, stdev=14.43 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 81], 95.00th=[ 85], | 99.00th=[ 99], 99.50th=[ 108], 99.90th=[ 188], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 224, max= 423, per=25.40%, avg=317.26, stdev=52.50, samples=31 iops : min= 56, max= 105, avg=79.19, stdev=13.06, samples=31 write: IOPS=81, BW=325KiB/s (333kB/s)(5180KiB/15938msec) slat (usec): min=14, max=286, avg=61.30, stdev=31.72 clat (msec): min=16, max=238, avg=131.77, stdev=33.21 lat (msec): min=16, max=238, avg=131.84, stdev=33.21 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 78], 10.00th=[ 89], 20.00th=[ 104], | 30.00th=[ 114], 40.00th=[ 125], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 209], 99.50th=[ 218], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 247, max= 424, per=24.90%, avg=323.71, stdev=42.69, samples=31 iops : min= 61, max= 106, avg=80.81, stdev=10.72, samples=31 lat (msec) : 4=0.08%, 10=0.31%, 20=0.35%, 50=4.14%, 100=52.70% lat (msec) : 250=42.42% cpu : usr=0.20%, sys=1.64%, ctx=3580, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1249KiB/s (1279kB/s), 310KiB/s-317KiB/s (318kB/s-325kB/s), io=19.6MiB (20.5MB), run=15938-16062msec WRITE: bw=1301KiB/s (1332kB/s), 325KiB/s-328KiB/s (333kB/s-336kB/s), io=20.4MiB (21.4MB), run=15938-16062msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=5358: Tue Dec 17 07:17:35 2024 read: IOPS=10, BW=440MiB/s (461MB/s)(40.0MiB/91msec) slat (nsec): min=7504.2k, max=7504.2k, avg=7504245.00, stdev= 0.00 clat (nsec): min=78988k, max=78988k, avg=78988157.00, stdev= 0.00 lat (nsec): min=86508k, max=86508k, avg=86508257.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[79168], 5.00th=[79168], 10.00th=[79168], 20.00th=[79168], | 30.00th=[79168], 40.00th=[79168], 50.00th=[79168], 60.00th=[79168], | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.78%, ctx=4, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=440MiB/s (461MB/s), 440MiB/s-440MiB/s (461MB/s-461MB/s), io=40.0MiB (41.9MB), run=91-91msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout