== sanity test 398c: run fio to test AIO ================= 08:52:09 (1748868729) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.087535 s, 479 MB/s osc.lustre-OST0000-osc-ffff8800ad11e000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=4178: Mon Jun 2 08:52:43 2025 write: IOPS=79, BW=317KiB/s (325kB/s)(10.0MiB/32292msec) slat (usec): min=12, max=616, avg=80.56, stdev=57.76 clat (msec): min=52, max=382, avg=201.50, stdev=47.09 lat (msec): min=53, max=382, avg=201.58, stdev=47.09 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 140], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 384] bw ( KiB/s): min= 192, max= 448, per=24.92%, avg=315.94, stdev=58.37, samples=64 iops : min= 48, max= 112, avg=78.94, stdev=14.59, samples=64 lat (msec) : 100=0.78%, 250=86.45%, 500=12.77% cpu : usr=0.19%, sys=0.68%, ctx=2400, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=4179: Mon Jun 2 08:52:43 2025 write: IOPS=79, BW=317KiB/s (325kB/s)(10.0MiB/32292msec) slat (usec): min=11, max=715, avg=79.68, stdev=57.24 clat (msec): min=69, max=378, avg=201.56, stdev=46.65 lat (msec): min=70, max=378, avg=201.64, stdev=46.66 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 138], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 288], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 380] bw ( KiB/s): min= 184, max= 440, per=24.94%, avg=316.19, stdev=58.08, samples=64 iops : min= 46, max= 110, avg=79.00, stdev=14.53, samples=64 lat (msec) : 100=0.90%, 250=86.25%, 500=12.85% cpu : usr=0.13%, sys=0.74%, ctx=2280, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=4180: Mon Jun 2 08:52:43 2025 write: IOPS=79, BW=317KiB/s (325kB/s)(10.0MiB/32292msec) slat (usec): min=12, max=848, avg=76.64, stdev=58.45 clat (msec): min=59, max=383, avg=201.51, stdev=46.58 lat (msec): min=59, max=383, avg=201.59, stdev=46.58 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 142], 10.00th=[ 153], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 288], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 384] bw ( KiB/s): min= 184, max= 496, per=24.89%, avg=315.56, stdev=59.67, samples=64 iops : min= 46, max= 124, avg=78.84, stdev=14.93, samples=64 lat (msec) : 100=0.78%, 250=86.05%, 500=13.16% cpu : usr=0.16%, sys=0.68%, ctx=2151, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=4181: Mon Jun 2 08:52:43 2025 write: IOPS=79, BW=318KiB/s (325kB/s)(10.0MiB/32232msec) slat (usec): min=13, max=2118, avg=81.90, stdev=73.23 clat (msec): min=14, max=383, avg=201.20, stdev=48.07 lat (msec): min=14, max=383, avg=201.28, stdev=48.08 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 142], 10.00th=[ 150], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 351], 99.90th=[ 380], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 184, max= 504, per=25.00%, avg=316.94, stdev=60.84, samples=64 iops : min= 46, max= 126, avg=79.19, stdev=15.22, samples=64 lat (msec) : 20=0.08%, 50=0.43%, 100=0.82%, 250=84.73%, 500=13.95% cpu : usr=0.12%, sys=0.76%, ctx=2326, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1268KiB/s (1299kB/s), 317KiB/s-318KiB/s (325kB/s-325kB/s), io=40.0MiB (41.9MB), run=32232-32292msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4217: Mon Jun 2 08:53:05 2025 read: IOPS=60, BW=242KiB/s (248kB/s)(5048KiB/20829msec) slat (usec): min=742, max=4989, avg=1427.81, stdev=397.35 clat (usec): min=1326, max=189519, avg=82679.15, stdev=17583.40 lat (msec): min=2, max=191, avg=84.11, stdev=17.67 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 69], | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 104], 95.00th=[ 112], | 99.00th=[ 132], 99.50th=[ 144], 99.90th=[ 188], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 136, max= 344, per=25.18%, avg=242.22, stdev=54.07, samples=41 iops : min= 34, max= 86, avg=60.46, stdev=13.53, samples=41 write: IOPS=62, BW=249KiB/s (255kB/s)(5192KiB/20829msec) slat (usec): min=15, max=1501, avg=69.88, stdev=62.79 clat (msec): min=45, max=336, avg=174.49, stdev=42.33 lat (msec): min=45, max=336, avg=174.56, stdev=42.33 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 111], 10.00th=[ 125], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 245], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 168, max= 328, per=24.82%, avg=248.66, stdev=36.67, samples=41 iops : min= 42, max= 82, avg=62.07, stdev= 9.18, samples=41 lat (msec) : 2=0.04%, 20=0.23%, 50=0.23%, 100=43.40%, 250=53.87% lat (msec) : 500=2.23% cpu : usr=0.31%, sys=1.50%, ctx=2963, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4218: Mon Jun 2 08:53:05 2025 read: IOPS=59, BW=239KiB/s (244kB/s)(4976KiB/20847msec) slat (usec): min=719, max=6730, avg=1452.43, stdev=436.15 clat (usec): min=1664, max=153180, avg=83021.12, stdev=18154.58 lat (msec): min=2, max=154, avg=84.48, stdev=18.26 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 58], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 91], 80.00th=[ 96], 90.00th=[ 106], 95.00th=[ 113], | 99.00th=[ 133], 99.50th=[ 138], 99.90th=[ 144], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 104, max= 368, per=24.71%, avg=237.73, stdev=54.47, samples=41 iops : min= 26, max= 92, avg=59.34, stdev=13.64, samples=41 write: IOPS=63, BW=253KiB/s (259kB/s)(5264KiB/20847msec) slat (usec): min=16, max=898, avg=69.49, stdev=57.53 clat (msec): min=65, max=333, avg=173.29, stdev=41.95 lat (msec): min=65, max=333, avg=173.36, stdev=41.95 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 110], 10.00th=[ 123], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 228], 95.00th=[ 243], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 321], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 192, max= 320, per=25.07%, avg=251.20, stdev=35.50, samples=41 iops : min= 48, max= 80, avg=62.71, stdev= 8.87, samples=41 lat (msec) : 2=0.04%, 4=0.04%, 10=0.12%, 20=0.35%, 50=0.47% lat (msec) : 100=41.76%, 250=55.51%, 500=1.72% cpu : usr=0.37%, sys=1.46%, ctx=2925, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4219: Mon Jun 2 08:53:05 2025 read: IOPS=59, BW=240KiB/s (245kB/s)(4984KiB/20797msec) slat (usec): min=12, max=3334, avg=1444.61, stdev=410.27 clat (usec): min=1821, max=182238, avg=82806.71, stdev=19007.70 lat (msec): min=3, max=183, avg=84.25, stdev=19.13 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 96], 90.00th=[ 106], 95.00th=[ 114], | 99.00th=[ 138], 99.50th=[ 142], 99.90th=[ 153], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 160, max= 392, per=25.04%, avg=240.90, stdev=57.61, samples=41 iops : min= 40, max= 98, avg=60.17, stdev=14.38, samples=41 write: IOPS=63, BW=253KiB/s (259kB/s)(5256KiB/20797msec) slat (usec): min=11, max=2383, avg=72.86, stdev=92.13 clat (msec): min=18, max=336, avg=173.06, stdev=44.02 lat (msec): min=18, max=336, avg=173.13, stdev=44.03 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 109], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 209], 90.00th=[ 228], 95.00th=[ 245], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 326], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 143, max= 392, per=25.17%, avg=252.22, stdev=45.62, samples=41 iops : min= 35, max= 98, avg=63.00, stdev=11.44, samples=41 lat (msec) : 2=0.04%, 4=0.12%, 10=0.12%, 20=0.51%, 50=0.90% lat (msec) : 100=40.82%, 250=55.51%, 500=1.99% cpu : usr=0.27%, sys=1.55%, ctx=2923, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4220: Mon Jun 2 08:53:05 2025 read: IOPS=60, BW=243KiB/s (249kB/s)(5060KiB/20838msec) slat (usec): min=713, max=4620, avg=1439.73, stdev=430.60 clat (usec): min=1381, max=150672, avg=82409.23, stdev=18179.09 lat (msec): min=2, max=152, avg=83.85, stdev=18.27 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 57], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 83], 60.00th=[ 87], | 70.00th=[ 91], 80.00th=[ 96], 90.00th=[ 105], 95.00th=[ 113], | 99.00th=[ 127], 99.50th=[ 134], 99.90th=[ 142], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 120, max= 392, per=25.30%, avg=243.39, stdev=56.97, samples=41 iops : min= 30, max= 98, avg=60.76, stdev=14.23, samples=41 write: IOPS=62, BW=249KiB/s (255kB/s)(5180KiB/20838msec) slat (usec): min=12, max=1001, avg=68.73, stdev=54.61 clat (msec): min=58, max=338, avg=175.07, stdev=42.04 lat (msec): min=58, max=339, avg=175.14, stdev=42.04 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 112], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 209], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 334], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 136, max= 312, per=24.72%, avg=247.68, stdev=35.47, samples=41 iops : min= 34, max= 78, avg=61.83, stdev= 8.89, samples=41 lat (msec) : 2=0.08%, 4=0.04%, 10=0.04%, 20=0.39%, 50=0.55% lat (msec) : 100=42.34%, 250=54.41%, 500=2.15% cpu : usr=0.31%, sys=1.50%, ctx=2980, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=963KiB/s (986kB/s), 239KiB/s-243KiB/s (244kB/s-249kB/s), io=19.6MiB (20.5MB), run=20797-20847msec WRITE: bw=1002KiB/s (1026kB/s), 249KiB/s-253KiB/s (255kB/s-259kB/s), io=20.4MiB (21.4MB), run=20797-20847msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=4240: Mon Jun 2 08:53:06 2025 read: IOPS=10, BW=408MiB/s (428MB/s)(40.0MiB/98msec) slat (nsec): min=6621.2k, max=6621.2k, avg=6621243.00, stdev= 0.00 clat (nsec): min=86619k, max=86619k, avg=86619053.00, stdev= 0.00 lat (nsec): min=93263k, max=93263k, avg=93262629.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[86508], 5.00th=[86508], 10.00th=[86508], 20.00th=[86508], | 30.00th=[86508], 40.00th=[86508], 50.00th=[86508], 60.00th=[86508], | 70.00th=[86508], 80.00th=[86508], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] lat (msec) : 100=100.00% cpu : usr=1.03%, sys=6.19%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=408MiB/s (428MB/s), 408MiB/s-408MiB/s (428MB/s-428MB/s), io=40.0MiB (41.9MB), run=98-98msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout