== sanity test 398c: run fio to test AIO ================= 13:07:00 (1736273220) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.118757 s, 353 MB/s osc.lustre-OST0000-osc-ffff880137374000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=24759: Tue Jan 7 13:07:25 2025 write: IOPS=107, BW=429KiB/s (440kB/s)(10.0MiB/23854msec) slat (usec): min=12, max=525, avg=70.91, stdev=43.09 clat (msec): min=51, max=242, avg=148.92, stdev=26.95 lat (msec): min=51, max=242, avg=149.00, stdev=26.95 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 103], 10.00th=[ 115], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 184], 95.00th=[ 194], | 99.00th=[ 209], 99.50th=[ 213], 99.90th=[ 222], 99.95th=[ 234], | 99.99th=[ 243] bw ( KiB/s): min= 312, max= 640, per=24.95%, avg=428.15, stdev=57.99, samples=47 iops : min= 78, max= 160, avg=106.96, stdev=14.52, samples=47 lat (msec) : 100=3.95%, 250=96.05% cpu : usr=0.19%, sys=0.88%, ctx=1991, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24760: Tue Jan 7 13:07:25 2025 write: IOPS=107, BW=429KiB/s (440kB/s)(10.0MiB/23854msec) slat (usec): min=12, max=385, avg=68.65, stdev=43.81 clat (msec): min=51, max=228, avg=148.93, stdev=27.05 lat (msec): min=51, max=229, avg=149.00, stdev=27.05 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 106], 10.00th=[ 114], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 194], | 99.00th=[ 209], 99.50th=[ 213], 99.90th=[ 222], 99.95th=[ 226], | 99.99th=[ 230] bw ( KiB/s): min= 320, max= 640, per=24.97%, avg=428.49, stdev=53.02, samples=47 iops : min= 80, max= 160, avg=107.04, stdev=13.27, samples=47 lat (msec) : 100=4.06%, 250=95.94% cpu : usr=0.24%, sys=0.80%, ctx=1948, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24761: Tue Jan 7 13:07:25 2025 write: IOPS=108, BW=432KiB/s (442kB/s)(10.0MiB/23703msec) slat (usec): min=13, max=371, avg=70.44, stdev=42.84 clat (msec): min=9, max=233, avg=148.01, stdev=28.44 lat (msec): min=9, max=233, avg=148.08, stdev=28.44 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 100], 10.00th=[ 113], 20.00th=[ 127], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 155], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 184], 95.00th=[ 194], | 99.00th=[ 207], 99.50th=[ 211], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 234] bw ( KiB/s): min= 352, max= 696, per=25.13%, avg=431.21, stdev=61.47, samples=47 iops : min= 88, max= 174, avg=107.72, stdev=15.37, samples=47 lat (msec) : 10=0.04%, 20=0.04%, 50=0.16%, 100=4.92%, 250=94.84% cpu : usr=0.24%, sys=0.83%, ctx=1895, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24762: Tue Jan 7 13:07:25 2025 write: IOPS=107, BW=429KiB/s (439kB/s)(10.0MiB/23863msec) slat (usec): min=13, max=339, avg=69.19, stdev=41.07 clat (msec): min=51, max=260, avg=148.98, stdev=27.59 lat (msec): min=51, max=260, avg=149.05, stdev=27.59 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 102], 10.00th=[ 113], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 194], | 99.00th=[ 209], 99.50th=[ 215], 99.90th=[ 251], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 328, max= 640, per=24.96%, avg=428.32, stdev=52.41, samples=47 iops : min= 82, max= 160, avg=107.00, stdev=13.11, samples=47 lat (msec) : 100=4.30%, 250=95.59%, 500=0.12% cpu : usr=0.25%, sys=0.82%, ctx=1884, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1716KiB/s (1758kB/s), 429KiB/s-432KiB/s (439kB/s-442kB/s), io=40.0MiB (41.9MB), run=23703-23863msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24791: Tue Jan 7 13:07:41 2025 read: IOPS=88, BW=352KiB/s (361kB/s)(5048KiB/14325msec) slat (usec): min=14, max=3030, avg=829.50, stdev=219.67 clat (usec): min=1294, max=165306, avg=59943.96, stdev=13405.03 lat (usec): min=1366, max=166116, avg=60775.25, stdev=13425.49 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 73], 95.00th=[ 78], | 99.00th=[ 88], 99.50th=[ 125], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 224, max= 456, per=25.34%, avg=350.75, stdev=59.03, samples=28 iops : min= 56, max= 114, avg=87.61, stdev=14.68, samples=28 write: IOPS=90, BW=362KiB/s (371kB/s)(5192KiB/14325msec) slat (usec): min=13, max=1721, avg=62.21, stdev=57.68 clat (msec): min=12, max=216, avg=117.27, stdev=29.27 lat (msec): min=12, max=216, avg=117.34, stdev=29.27 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 72], 10.00th=[ 81], 20.00th=[ 94], | 30.00th=[ 102], 40.00th=[ 108], 50.00th=[ 117], 60.00th=[ 126], | 70.00th=[ 132], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 167], | 99.00th=[ 192], 99.50th=[ 197], 99.90th=[ 213], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 304, max= 440, per=25.19%, avg=363.04, stdev=37.06, samples=28 iops : min= 76, max= 110, avg=90.68, stdev= 9.29, samples=28 lat (msec) : 2=0.12%, 4=0.12%, 10=0.16%, 20=0.35%, 50=7.73% lat (msec) : 100=55.04%, 250=36.48% cpu : usr=0.24%, sys=1.82%, ctx=3392, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24792: Tue Jan 7 13:07:41 2025 read: IOPS=85, BW=343KiB/s (352kB/s)(4976KiB/14495msec) slat (usec): min=500, max=4109, avg=837.33, stdev=229.20 clat (usec): min=9616, max=96774, avg=60737.83, stdev=10120.08 lat (usec): min=10265, max=97529, avg=61577.71, stdev=10126.48 clat percentiles (usec): | 1.00th=[38011], 5.00th=[45351], 10.00th=[49021], 20.00th=[52691], | 30.00th=[55313], 40.00th=[58459], 50.00th=[61080], 60.00th=[63177], | 70.00th=[65274], 80.00th=[68682], 90.00th=[72877], 95.00th=[77071], | 99.00th=[87557], 99.50th=[89654], 99.90th=[92799], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 240, max= 424, per=24.56%, avg=339.89, stdev=49.77, samples=28 iops : min= 60, max= 106, avg=84.89, stdev=12.49, samples=28 write: IOPS=90, BW=363KiB/s (372kB/s)(5264KiB/14495msec) slat (usec): min=13, max=1688, avg=64.64, stdev=64.74 clat (msec): min=37, max=214, avg=117.81, stdev=28.30 lat (msec): min=37, max=214, avg=117.88, stdev=28.30 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 94], | 30.00th=[ 102], 40.00th=[ 110], 50.00th=[ 118], 60.00th=[ 124], | 70.00th=[ 132], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 169], | 99.00th=[ 192], 99.50th=[ 194], 99.90th=[ 213], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 296, max= 432, per=25.05%, avg=361.04, stdev=34.39, samples=28 iops : min= 74, max= 108, avg=90.18, stdev= 8.54, samples=28 lat (msec) : 10=0.04%, 20=0.16%, 50=5.82%, 100=57.15%, 250=36.84% cpu : usr=0.27%, sys=1.75%, ctx=3600, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24793: Tue Jan 7 13:07:41 2025 read: IOPS=86, BW=344KiB/s (353kB/s)(4984KiB/14477msec) slat (usec): min=469, max=3719, avg=827.03, stdev=201.82 clat (msec): min=7, max=128, avg=60.65, stdev=10.74 lat (msec): min=8, max=130, avg=61.48, stdev=10.76 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 46], 10.00th=[ 49], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 64], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 73], 95.00th=[ 78], | 99.00th=[ 87], 99.50th=[ 88], 99.90th=[ 97], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 272, max= 496, per=24.93%, avg=345.04, stdev=63.01, samples=28 iops : min= 68, max= 124, avg=86.18, stdev=15.75, samples=28 write: IOPS=90, BW=363KiB/s (372kB/s)(5256KiB/14477msec) slat (usec): min=12, max=952, avg=61.19, stdev=43.33 clat (msec): min=48, max=212, avg=117.72, stdev=28.13 lat (msec): min=48, max=212, avg=117.79, stdev=28.13 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 75], 10.00th=[ 84], 20.00th=[ 94], | 30.00th=[ 102], 40.00th=[ 109], 50.00th=[ 117], 60.00th=[ 125], | 70.00th=[ 133], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 171], | 99.00th=[ 190], 99.50th=[ 197], 99.90th=[ 207], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 280, max= 408, per=25.05%, avg=361.04, stdev=39.22, samples=28 iops : min= 70, max= 102, avg=90.18, stdev= 9.80, samples=28 lat (msec) : 10=0.16%, 20=0.20%, 50=5.94%, 100=57.03%, 250=36.68% cpu : usr=0.24%, sys=1.79%, ctx=3590, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24794: Tue Jan 7 13:07:41 2025 read: IOPS=87, BW=350KiB/s (358kB/s)(5060KiB/14465msec) slat (usec): min=489, max=3816, avg=837.74, stdev=226.82 clat (msec): min=2, max=106, avg=60.72, stdev=10.97 lat (msec): min=3, max=107, avg=61.55, stdev=10.98 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 46], 10.00th=[ 49], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 64], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 73], 95.00th=[ 78], | 99.00th=[ 88], 99.50th=[ 89], 99.90th=[ 97], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 216, max= 472, per=25.22%, avg=349.07, stdev=61.41, samples=28 iops : min= 54, max= 118, avg=87.21, stdev=15.30, samples=28 write: IOPS=89, BW=358KiB/s (367kB/s)(5180KiB/14465msec) slat (usec): min=14, max=307, avg=62.22, stdev=36.41 clat (msec): min=47, max=209, avg=118.33, stdev=28.33 lat (msec): min=48, max=209, avg=118.40, stdev=28.33 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 84], 20.00th=[ 95], | 30.00th=[ 103], 40.00th=[ 109], 50.00th=[ 116], 60.00th=[ 126], | 70.00th=[ 133], 80.00th=[ 142], 90.00th=[ 157], 95.00th=[ 169], | 99.00th=[ 190], 99.50th=[ 194], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 312, max= 424, per=24.72%, avg=356.21, stdev=28.18, samples=28 iops : min= 78, max= 106, avg=89.00, stdev= 7.05, samples=28 lat (msec) : 4=0.04%, 10=0.08%, 20=0.35%, 50=6.37%, 100=56.41% lat (msec) : 250=36.76% cpu : usr=0.32%, sys=1.69%, ctx=3589, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1384KiB/s (1418kB/s), 343KiB/s-352KiB/s (352kB/s-361kB/s), io=19.6MiB (20.5MB), run=14325-14495msec WRITE: bw=1441KiB/s (1476kB/s), 358KiB/s-363KiB/s (367kB/s-372kB/s), io=20.4MiB (21.4MB), run=14325-14495msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=24810: Tue Jan 7 13:07:42 2025 read: IOPS=12, BW=488MiB/s (512MB/s)(40.0MiB/82msec) slat (nsec): min=7769.6k, max=7769.6k, avg=7769633.00, stdev= 0.00 clat (nsec): min=71185k, max=71185k, avg=71185245.00, stdev= 0.00 lat (nsec): min=78973k, max=78973k, avg=78972834.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[70779], 5.00th=[70779], 10.00th=[70779], 20.00th=[70779], | 30.00th=[70779], 40.00th=[70779], 50.00th=[70779], 60.00th=[70779], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=8.64%, ctx=15, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=488MiB/s (512MB/s), 488MiB/s-488MiB/s (512MB/s-512MB/s), io=40.0MiB (41.9MB), run=82-82msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck