== sanity test 398c: run fio to test AIO ================= 18:12:48 (1748470368) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0998682 s, 420 MB/s osc.lustre-OST0000-osc-ffff8800aa53a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=19372: Wed May 28 18:13:24 2025 write: IOPS=75, BW=300KiB/s (307kB/s)(10.0MiB/34110msec) slat (usec): min=12, max=490, avg=74.35, stdev=46.90 clat (msec): min=13, max=351, avg=212.97, stdev=38.72 lat (msec): min=13, max=351, avg=213.04, stdev=38.72 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 150], 10.00th=[ 165], 20.00th=[ 188], | 30.00th=[ 199], 40.00th=[ 207], 50.00th=[ 215], 60.00th=[ 222], | 70.00th=[ 230], 80.00th=[ 241], 90.00th=[ 257], 95.00th=[ 275], | 99.00th=[ 305], 99.50th=[ 321], 99.90th=[ 342], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 223, max= 479, per=24.98%, avg=299.31, stdev=44.23, samples=68 iops : min= 55, max= 119, avg=74.76, stdev=11.03, samples=68 lat (msec) : 20=0.08%, 50=0.43%, 100=0.35%, 250=85.59%, 500=13.55% cpu : usr=0.15%, sys=0.64%, ctx=2224, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19373: Wed May 28 18:13:24 2025 write: IOPS=74, BW=300KiB/s (307kB/s)(10.0MiB/34156msec) slat (usec): min=13, max=441, avg=71.54, stdev=47.54 clat (msec): min=53, max=341, avg=213.24, stdev=36.73 lat (msec): min=53, max=342, avg=213.31, stdev=36.73 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 155], 10.00th=[ 167], 20.00th=[ 188], | 30.00th=[ 199], 40.00th=[ 207], 50.00th=[ 215], 60.00th=[ 222], | 70.00th=[ 230], 80.00th=[ 241], 90.00th=[ 259], 95.00th=[ 275], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 224, max= 512, per=24.93%, avg=298.62, stdev=42.93, samples=68 iops : min= 56, max= 128, avg=74.60, stdev=10.71, samples=68 lat (msec) : 100=0.62%, 250=86.45%, 500=12.93% cpu : usr=0.10%, sys=0.66%, ctx=2065, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19374: Wed May 28 18:13:24 2025 write: IOPS=74, BW=300KiB/s (307kB/s)(10.0MiB/34165msec) slat (usec): min=13, max=1106, avg=71.00, stdev=49.84 clat (msec): min=34, max=346, avg=213.22, stdev=36.07 lat (msec): min=34, max=346, avg=213.29, stdev=36.07 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 153], 10.00th=[ 167], 20.00th=[ 188], | 30.00th=[ 199], 40.00th=[ 207], 50.00th=[ 213], 60.00th=[ 222], | 70.00th=[ 230], 80.00th=[ 239], 90.00th=[ 257], 95.00th=[ 271], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[ 334], 99.95th=[ 338], | 99.99th=[ 347] bw ( KiB/s): min= 232, max= 400, per=24.93%, avg=298.69, stdev=39.87, samples=68 iops : min= 58, max= 100, avg=74.66, stdev= 9.97, samples=68 lat (msec) : 50=0.08%, 100=0.27%, 250=85.74%, 500=13.91% cpu : usr=0.17%, sys=0.62%, ctx=2030, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19375: Wed May 28 18:13:24 2025 write: IOPS=74, BW=300KiB/s (307kB/s)(10.0MiB/34165msec) slat (usec): min=13, max=359, avg=72.16, stdev=46.24 clat (msec): min=42, max=351, avg=213.20, stdev=36.07 lat (msec): min=42, max=351, avg=213.27, stdev=36.07 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 153], 10.00th=[ 167], 20.00th=[ 186], | 30.00th=[ 199], 40.00th=[ 209], 50.00th=[ 215], 60.00th=[ 222], | 70.00th=[ 230], 80.00th=[ 239], 90.00th=[ 257], 95.00th=[ 275], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 199, max= 392, per=24.92%, avg=298.49, stdev=39.05, samples=68 iops : min= 49, max= 98, avg=74.56, stdev= 9.75, samples=68 lat (msec) : 50=0.04%, 100=0.35%, 250=86.76%, 500=12.85% cpu : usr=0.14%, sys=0.63%, ctx=2178, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1199KiB/s (1228kB/s), 300KiB/s-300KiB/s (307kB/s-307kB/s), io=40.0MiB (41.9MB), run=34110-34165msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19413: Wed May 28 18:13:44 2025 read: IOPS=67, BW=270KiB/s (276kB/s)(5048KiB/18705msec) slat (usec): min=11, max=2748, avg=1223.97, stdev=316.33 clat (usec): min=1525, max=161157, avg=75852.58, stdev=18181.48 lat (msec): min=2, max=162, avg=77.08, stdev=18.21 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 51], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 89], 90.00th=[ 97], 95.00th=[ 103], | 99.00th=[ 126], 99.50th=[ 132], 99.90th=[ 148], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 176, max= 376, per=25.31%, avg=269.81, stdev=47.88, samples=37 iops : min= 44, max= 94, avg=67.43, stdev=11.95, samples=37 write: IOPS=69, BW=278KiB/s (284kB/s)(5192KiB/18705msec) slat (usec): min=13, max=962, avg=63.16, stdev=51.42 clat (msec): min=21, max=276, avg=155.39, stdev=34.70 lat (msec): min=21, max=276, avg=155.45, stdev=34.70 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 96], 10.00th=[ 110], 20.00th=[ 126], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 199], 95.00th=[ 207], | 99.00th=[ 234], 99.50th=[ 243], 99.90th=[ 257], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 216, max= 360, per=24.99%, avg=277.38, stdev=39.02, samples=37 iops : min= 54, max= 90, avg=69.32, stdev= 9.76, samples=37 lat (msec) : 2=0.12%, 4=0.20%, 10=0.20%, 20=0.39%, 50=1.84% lat (msec) : 100=46.21%, 250=50.90%, 500=0.16% cpu : usr=0.21%, sys=1.46%, ctx=3289, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19414: Wed May 28 18:13:44 2025 read: IOPS=66, BW=264KiB/s (271kB/s)(4976KiB/18820msec) slat (usec): min=689, max=2950, avg=1234.38, stdev=300.36 clat (usec): min=1504, max=139215, avg=75666.10, stdev=15753.63 lat (msec): min=2, max=140, avg=76.90, stdev=15.77 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 53], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 88], 90.00th=[ 94], 95.00th=[ 101], | 99.00th=[ 117], 99.50th=[ 128], 99.90th=[ 133], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 175, max= 440, per=24.59%, avg=262.11, stdev=54.33, samples=37 iops : min= 43, max= 110, avg=65.41, stdev=13.61, samples=37 write: IOPS=69, BW=280KiB/s (286kB/s)(5264KiB/18820msec) slat (usec): min=13, max=339, avg=62.58, stdev=37.53 clat (msec): min=15, max=257, avg=155.71, stdev=34.21 lat (msec): min=15, max=257, avg=155.78, stdev=34.21 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 199], 95.00th=[ 209], | 99.00th=[ 230], 99.50th=[ 236], 99.90th=[ 255], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 216, max= 360, per=24.96%, avg=277.03, stdev=34.28, samples=37 iops : min= 54, max= 90, avg=69.14, stdev= 8.59, samples=37 lat (msec) : 2=0.04%, 4=0.08%, 10=0.04%, 20=0.31%, 50=1.29% lat (msec) : 100=47.11%, 250=51.02%, 500=0.12% cpu : usr=0.22%, sys=1.48%, ctx=3176, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19415: Wed May 28 18:13:44 2025 read: IOPS=66, BW=265KiB/s (271kB/s)(4984KiB/18821msec) slat (usec): min=655, max=4521, avg=1240.25, stdev=310.17 clat (msec): min=2, max=139, avg=76.27, stdev=15.70 lat (msec): min=4, max=140, avg=77.51, stdev=15.71 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 53], 10.00th=[ 59], 20.00th=[ 65], | 30.00th=[ 69], 40.00th=[ 73], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 95], 95.00th=[ 103], | 99.00th=[ 117], 99.50th=[ 131], 99.90th=[ 138], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 184, max= 392, per=24.76%, avg=263.92, stdev=52.72, samples=37 iops : min= 46, max= 98, avg=65.92, stdev=13.18, samples=37 write: IOPS=69, BW=279KiB/s (286kB/s)(5256KiB/18821msec) slat (usec): min=14, max=525, avg=64.39, stdev=41.81 clat (msec): min=56, max=289, avg=155.34, stdev=33.55 lat (msec): min=56, max=289, avg=155.40, stdev=33.56 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 101], 10.00th=[ 110], 20.00th=[ 126], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 184], 90.00th=[ 199], 95.00th=[ 207], | 99.00th=[ 228], 99.50th=[ 241], 99.90th=[ 271], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 208, max= 368, per=24.89%, avg=276.24, stdev=38.38, samples=37 iops : min= 52, max= 92, avg=69.00, stdev= 9.59, samples=37 lat (msec) : 4=0.04%, 20=0.35%, 50=1.05%, 100=46.95%, 250=51.45% lat (msec) : 500=0.16% cpu : usr=0.22%, sys=1.50%, ctx=3301, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19416: Wed May 28 18:13:44 2025 read: IOPS=67, BW=270KiB/s (276kB/s)(5060KiB/18763msec) slat (usec): min=731, max=5144, avg=1215.66, stdev=297.02 clat (msec): min=2, max=176, avg=76.76, stdev=16.04 lat (msec): min=3, max=177, avg=77.97, stdev=16.06 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 53], 10.00th=[ 59], 20.00th=[ 65], | 30.00th=[ 69], 40.00th=[ 73], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 102], | 99.00th=[ 124], 99.50th=[ 131], 99.90th=[ 144], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 144, max= 424, per=25.34%, avg=270.14, stdev=56.12, samples=37 iops : min= 36, max= 106, avg=67.43, stdev=14.02, samples=37 write: IOPS=69, BW=276KiB/s (283kB/s)(5180KiB/18763msec) slat (usec): min=13, max=1545, avg=64.87, stdev=67.72 clat (msec): min=69, max=276, avg=155.22, stdev=33.50 lat (msec): min=69, max=276, avg=155.29, stdev=33.50 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 97], 10.00th=[ 110], 20.00th=[ 126], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 157], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 184], 90.00th=[ 199], 95.00th=[ 207], | 99.00th=[ 232], 99.50th=[ 243], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 224, max= 352, per=24.78%, avg=275.11, stdev=32.99, samples=37 iops : min= 56, max= 88, avg=68.68, stdev= 8.28, samples=37 lat (msec) : 4=0.12%, 20=0.12%, 50=1.33%, 100=47.30%, 250=51.02% lat (msec) : 500=0.12% cpu : usr=0.24%, sys=1.46%, ctx=3236, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1066KiB/s (1092kB/s), 264KiB/s-270KiB/s (271kB/s-276kB/s), io=19.6MiB (20.5MB), run=18705-18821msec WRITE: bw=1110KiB/s (1137kB/s), 276KiB/s-280KiB/s (283kB/s-286kB/s), io=20.4MiB (21.4MB), run=18705-18821msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=19434: Wed May 28 18:13:45 2025 read: IOPS=9, BW=388MiB/s (407MB/s)(40.0MiB/103msec) slat (nsec): min=9024.2k, max=9024.2k, avg=9024172.00, stdev= 0.00 clat (nsec): min=87186k, max=87186k, avg=87185957.00, stdev= 0.00 lat (nsec): min=96235k, max=96235k, avg=96235357.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[87557], 5.00th=[87557], 10.00th=[87557], 20.00th=[87557], | 30.00th=[87557], 40.00th=[87557], 50.00th=[87557], 60.00th=[87557], | 70.00th=[87557], 80.00th=[87557], 90.00th=[87557], 95.00th=[87557], | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=8.82%, ctx=15, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=40.0MiB (41.9MB), run=103-103msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout