== sanity test 398c: run fio to test AIO ================= 10:23:57 (1724768637) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0842777 s, 498 MB/s osc.lustre-OST0000-osc-ffff88008ab59800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=23332: Tue Aug 27 10:24:25 2024 write: IOPS=95, BW=383KiB/s (392kB/s)(10.0MiB/26736msec) slat (usec): min=13, max=501, avg=76.28, stdev=44.92 clat (msec): min=48, max=293, avg=166.80, stdev=34.21 lat (msec): min=48, max=294, avg=166.88, stdev=34.21 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 111], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 169], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 192], 90.00th=[ 209], 95.00th=[ 222], | 99.00th=[ 247], 99.50th=[ 268], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 296] bw ( KiB/s): min= 264, max= 768, per=24.90%, avg=381.40, stdev=69.98, samples=53 iops : min= 66, max= 192, avg=95.32, stdev=17.53, samples=53 lat (msec) : 50=0.04%, 100=4.02%, 250=94.96%, 500=0.98% cpu : usr=0.19%, sys=0.73%, ctx=2789, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23333: Tue Aug 27 10:24:25 2024 write: IOPS=96, BW=384KiB/s (394kB/s)(10.0MiB/26634msec) slat (usec): min=13, max=509, avg=80.47, stdev=47.89 clat (msec): min=11, max=304, avg=166.23, stdev=36.59 lat (msec): min=11, max=305, avg=166.31, stdev=36.59 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 105], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 192], 90.00th=[ 209], 95.00th=[ 222], | 99.00th=[ 257], 99.50th=[ 268], 99.90th=[ 292], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 264, max= 856, per=25.07%, avg=384.11, stdev=78.20, samples=53 iops : min= 66, max= 214, avg=96.00, stdev=19.58, samples=53 lat (msec) : 20=0.43%, 50=0.62%, 100=3.67%, 250=94.02%, 500=1.25% cpu : usr=0.17%, sys=0.77%, ctx=3040, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23334: Tue Aug 27 10:24:25 2024 write: IOPS=95, BW=383KiB/s (392kB/s)(10.0MiB/26719msec) slat (usec): min=13, max=434, avg=77.99, stdev=44.11 clat (msec): min=38, max=286, avg=166.74, stdev=35.48 lat (msec): min=38, max=286, avg=166.82, stdev=35.48 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 111], 10.00th=[ 127], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 192], 90.00th=[ 209], 95.00th=[ 222], | 99.00th=[ 251], 99.50th=[ 271], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 264, max= 768, per=24.99%, avg=382.87, stdev=69.59, samples=53 iops : min= 66, max= 192, avg=95.66, stdev=17.37, samples=53 lat (msec) : 50=0.62%, 100=3.95%, 250=94.41%, 500=1.02% cpu : usr=0.20%, sys=0.75%, ctx=2908, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23335: Tue Aug 27 10:24:25 2024 write: IOPS=95, BW=383KiB/s (392kB/s)(10.0MiB/26719msec) slat (usec): min=14, max=423, avg=78.00, stdev=43.70 clat (msec): min=40, max=293, avg=166.76, stdev=35.70 lat (msec): min=40, max=294, avg=166.84, stdev=35.70 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 110], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 169], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 192], 90.00th=[ 209], 95.00th=[ 222], | 99.00th=[ 257], 99.50th=[ 275], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 296] bw ( KiB/s): min= 272, max= 768, per=25.00%, avg=383.06, stdev=71.12, samples=53 iops : min= 68, max= 192, avg=95.74, stdev=17.81, samples=53 lat (msec) : 50=0.70%, 100=3.79%, 250=94.18%, 500=1.33% cpu : usr=0.20%, sys=0.73%, ctx=3046, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1532KiB/s (1569kB/s), 383KiB/s-384KiB/s (392kB/s-394kB/s), io=40.0MiB (41.9MB), run=26634-26736msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23366: Tue Aug 27 10:24:42 2024 read: IOPS=79, BW=319KiB/s (327kB/s)(5048KiB/15809msec) slat (usec): min=670, max=3699, avg=1222.32, stdev=289.62 clat (msec): min=2, max=231, avg=65.43, stdev=14.89 lat (msec): min=3, max=232, avg=66.66, stdev=14.90 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 64], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 95], 99.50th=[ 101], 99.90th=[ 230], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 200, max= 440, per=25.29%, avg=320.71, stdev=50.62, samples=31 iops : min= 50, max= 110, avg=80.13, stdev=12.70, samples=31 write: IOPS=82, BW=328KiB/s (336kB/s)(5192KiB/15809msec) slat (usec): min=13, max=1040, avg=62.22, stdev=42.30 clat (msec): min=34, max=297, avg=129.65, stdev=30.05 lat (msec): min=34, max=297, avg=129.71, stdev=30.05 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 105], | 30.00th=[ 114], 40.00th=[ 123], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 144], 80.00th=[ 153], 90.00th=[ 167], 95.00th=[ 178], | 99.00th=[ 203], 99.50th=[ 234], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 264, max= 400, per=24.96%, avg=329.48, stdev=34.82, samples=31 iops : min= 66, max= 100, avg=82.32, stdev= 8.75, samples=31 lat (msec) : 4=0.08%, 10=0.04%, 20=0.31%, 50=3.79%, 100=52.81% lat (msec) : 250=42.73%, 500=0.23% cpu : usr=0.28%, sys=1.54%, ctx=3476, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23367: Tue Aug 27 10:24:42 2024 read: IOPS=78, BW=315KiB/s (323kB/s)(4976KiB/15781msec) slat (usec): min=15, max=3700, avg=1210.57, stdev=306.80 clat (usec): min=1047, max=154927, avg=64901.01, stdev=12877.61 lat (usec): min=1063, max=156407, avg=66113.70, stdev=12912.91 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 92], 99.50th=[ 97], 99.90th=[ 138], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 215, max= 424, per=24.80%, avg=314.52, stdev=56.35, samples=31 iops : min= 53, max= 106, avg=78.58, stdev=14.11, samples=31 write: IOPS=83, BW=334KiB/s (342kB/s)(5264KiB/15781msec) slat (usec): min=13, max=578, avg=63.44, stdev=35.30 clat (msec): min=17, max=252, avg=129.14, stdev=29.67 lat (msec): min=17, max=252, avg=129.20, stdev=29.67 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 106], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 144], 80.00th=[ 153], 90.00th=[ 167], 95.00th=[ 180], | 99.00th=[ 199], 99.50th=[ 213], 99.90th=[ 247], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 272, max= 416, per=25.25%, avg=333.35, stdev=41.53, samples=31 iops : min= 68, max= 104, avg=83.29, stdev=10.42, samples=31 lat (msec) : 2=0.08%, 4=0.12%, 10=0.23%, 20=0.35%, 50=3.20% lat (msec) : 100=51.84%, 250=44.14%, 500=0.04% cpu : usr=0.26%, sys=1.60%, ctx=3593, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23368: Tue Aug 27 10:24:42 2024 read: IOPS=78, BW=315KiB/s (323kB/s)(4984KiB/15820msec) slat (usec): min=709, max=3769, avg=1224.04, stdev=302.76 clat (usec): min=1941, max=234696, avg=65018.17, stdev=12938.22 lat (msec): min=3, max=236, avg=66.24, stdev=12.96 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 49], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 89], 99.50th=[ 95], 99.90th=[ 194], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 208, max= 424, per=25.09%, avg=318.13, stdev=54.44, samples=31 iops : min= 52, max= 106, avg=79.48, stdev=13.61, samples=31 write: IOPS=83, BW=332KiB/s (340kB/s)(5256KiB/15820msec) slat (usec): min=13, max=1280, avg=63.29, stdev=46.93 clat (msec): min=32, max=294, avg=129.40, stdev=30.07 lat (msec): min=32, max=294, avg=129.46, stdev=30.07 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 85], 10.00th=[ 93], 20.00th=[ 104], | 30.00th=[ 113], 40.00th=[ 121], 50.00th=[ 130], 60.00th=[ 136], | 70.00th=[ 144], 80.00th=[ 153], 90.00th=[ 167], 95.00th=[ 178], | 99.00th=[ 203], 99.50th=[ 245], 99.90th=[ 284], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 239, max= 392, per=25.18%, avg=332.32, stdev=37.48, samples=31 iops : min= 59, max= 98, avg=83.03, stdev= 9.43, samples=31 lat (msec) : 2=0.04%, 4=0.04%, 20=0.31%, 50=3.28%, 100=53.09% lat (msec) : 250=43.12%, 500=0.12% cpu : usr=0.30%, sys=1.56%, ctx=3376, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23369: Tue Aug 27 10:24:42 2024 read: IOPS=79, BW=320KiB/s (328kB/s)(5060KiB/15820msec) slat (usec): min=651, max=4163, avg=1205.42, stdev=305.61 clat (usec): min=1225, max=108969, avg=64954.32, stdev=12478.72 lat (msec): min=2, max=110, avg=66.16, stdev=12.50 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 49], 10.00th=[ 54], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 81], 95.00th=[ 85], | 99.00th=[ 97], 99.50th=[ 100], 99.90th=[ 107], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 232, max= 472, per=25.37%, avg=321.74, stdev=57.07, samples=31 iops : min= 58, max= 118, avg=80.39, stdev=14.28, samples=31 write: IOPS=81, BW=327KiB/s (335kB/s)(5180KiB/15820msec) slat (usec): min=13, max=313, avg=63.44, stdev=33.88 clat (msec): min=34, max=295, avg=130.38, stdev=30.32 lat (msec): min=34, max=295, avg=130.44, stdev=30.32 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 89], 10.00th=[ 95], 20.00th=[ 105], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 138], | 70.00th=[ 144], 80.00th=[ 153], 90.00th=[ 167], 95.00th=[ 180], | 99.00th=[ 207], 99.50th=[ 275], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 239, max= 400, per=24.84%, avg=327.94, stdev=31.16, samples=31 iops : min= 59, max= 100, avg=81.94, stdev= 7.87, samples=31 lat (msec) : 2=0.08%, 4=0.12%, 10=0.20%, 20=0.31%, 50=2.62% lat (msec) : 100=53.52%, 250=42.73%, 500=0.43% cpu : usr=0.25%, sys=1.61%, ctx=3450, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1269KiB/s (1299kB/s), 315KiB/s-320KiB/s (323kB/s-328kB/s), io=19.6MiB (20.5MB), run=15781-15820msec WRITE: bw=1321KiB/s (1352kB/s), 327KiB/s-334KiB/s (335kB/s-342kB/s), io=20.4MiB (21.4MB), run=15781-15820msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=23385: Tue Aug 27 10:24:43 2024 read: IOPS=12, BW=506MiB/s (531MB/s)(40.0MiB/79msec) slat (nsec): min=5955.9k, max=5955.9k, avg=5955890.00, stdev= 0.00 clat (nsec): min=68142k, max=68142k, avg=68141767.00, stdev= 0.00 lat (nsec): min=74117k, max=74117k, avg=74116643.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[67634], 5.00th=[67634], 10.00th=[67634], 20.00th=[67634], | 30.00th=[67634], 40.00th=[67634], 50.00th=[67634], 60.00th=[67634], | 70.00th=[67634], 80.00th=[67634], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.69%, ctx=14, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=506MiB/s (531MB/s), 506MiB/s-506MiB/s (531MB/s-531MB/s), io=40.0MiB (41.9MB), run=79-79msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout