== sanity test 398c: run fio to test AIO ================= 02:33:35 (1739518415) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.16504 s, 254 MB/s osc.lustre-OST0000-osc-ffff880085ada800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=17932: Fri Feb 14 02:33:50 2025 write: IOPS=197, BW=791KiB/s (810kB/s)(10.0MiB/12938msec) slat (usec): min=11, max=3127, avg=92.82, stdev=210.04 clat (msec): min=24, max=148, avg=80.54, stdev=16.43 lat (msec): min=24, max=148, avg=80.64, stdev=16.43 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 67], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 84], | 70.00th=[ 88], 80.00th=[ 92], 90.00th=[ 104], 95.00th=[ 109], | 99.00th=[ 128], 99.50th=[ 132], 99.90th=[ 142], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 630, max= 904, per=24.90%, avg=787.44, stdev=82.85, samples=25 iops : min= 157, max= 226, avg=196.72, stdev=20.85, samples=25 lat (msec) : 50=1.33%, 100=86.25%, 250=12.42% cpu : usr=0.00%, sys=3.01%, ctx=1022, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=17933: Fri Feb 14 02:33:50 2025 write: IOPS=197, BW=791KiB/s (810kB/s)(10.0MiB/12945msec) slat (usec): min=12, max=4345, avg=103.74, stdev=245.83 clat (msec): min=10, max=134, avg=80.50, stdev=15.57 lat (msec): min=10, max=134, avg=80.61, stdev=15.58 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 57], 10.00th=[ 64], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 88], 80.00th=[ 93], 90.00th=[ 102], 95.00th=[ 109], | 99.00th=[ 118], 99.50th=[ 123], 99.90th=[ 129], 99.95th=[ 133], | 99.99th=[ 136] bw ( KiB/s): min= 638, max= 928, per=24.76%, avg=782.96, stdev=86.88, samples=25 iops : min= 159, max= 232, avg=195.60, stdev=21.84, samples=25 lat (msec) : 20=0.23%, 50=0.82%, 100=87.85%, 250=11.09% cpu : usr=0.43%, sys=2.63%, ctx=1157, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=17934: Fri Feb 14 02:33:50 2025 write: IOPS=197, BW=791KiB/s (810kB/s)(10.0MiB/12953msec) slat (usec): min=11, max=3020, avg=104.52, stdev=228.99 clat (msec): min=12, max=132, avg=80.50, stdev=15.83 lat (msec): min=12, max=132, avg=80.61, stdev=15.83 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 56], 10.00th=[ 63], 20.00th=[ 67], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 88], 80.00th=[ 92], 90.00th=[ 103], 95.00th=[ 111], | 99.00th=[ 124], 99.50th=[ 130], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 558, max= 912, per=24.87%, avg=786.40, stdev=82.70, samples=25 iops : min= 139, max= 228, avg=196.52, stdev=20.77, samples=25 lat (msec) : 20=0.08%, 50=1.64%, 100=86.88%, 250=11.41% cpu : usr=0.63%, sys=2.55%, ctx=1307, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=17935: Fri Feb 14 02:33:50 2025 write: IOPS=197, BW=791KiB/s (810kB/s)(10.0MiB/12945msec) slat (usec): min=11, max=4801, avg=104.09, stdev=235.95 clat (msec): min=9, max=132, avg=80.55, stdev=15.60 lat (msec): min=9, max=132, avg=80.65, stdev=15.60 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 57], 10.00th=[ 64], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 88], 80.00th=[ 92], 90.00th=[ 103], 95.00th=[ 109], | 99.00th=[ 122], 99.50th=[ 127], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 133] bw ( KiB/s): min= 582, max= 984, per=24.87%, avg=786.40, stdev=91.41, samples=25 iops : min= 145, max= 246, avg=196.52, stdev=22.95, samples=25 lat (msec) : 10=0.04%, 20=0.12%, 50=1.33%, 100=86.52%, 250=11.99% cpu : usr=0.25%, sys=2.89%, ctx=1188, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3162KiB/s (3238kB/s), 791KiB/s-791KiB/s (810kB/s-810kB/s), io=40.0MiB (41.9MB), run=12938-12953msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17956: Fri Feb 14 02:34:02 2025 read: IOPS=114, BW=457KiB/s (468kB/s)(5048KiB/11052msec) slat (usec): min=1303, max=14997, avg=4319.05, stdev=1832.06 clat (msec): min=6, max=115, avg=64.64, stdev=15.16 lat (msec): min=14, max=122, avg=68.96, stdev=15.77 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 53], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 72], 80.00th=[ 78], 90.00th=[ 87], 95.00th=[ 92], | 99.00th=[ 104], 99.50th=[ 107], 99.90th=[ 112], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 280, max= 640, per=24.51%, avg=444.81, stdev=80.46, samples=21 iops : min= 70, max= 160, avg=111.10, stdev=20.24, samples=21 write: IOPS=117, BW=470KiB/s (481kB/s)(5192KiB/11052msec) slat (usec): min=13, max=4679, avg=102.20, stdev=248.42 clat (msec): min=17, max=113, avg=67.81, stdev=14.65 lat (msec): min=17, max=113, avg=67.91, stdev=14.64 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 45], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 60], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 88], 95.00th=[ 95], | 99.00th=[ 105], 99.50th=[ 110], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 319, max= 600, per=24.48%, avg=462.67, stdev=73.11, samples=21 iops : min= 79, max= 150, avg=115.57, stdev=18.38, samples=21 lat (msec) : 10=0.04%, 20=0.08%, 50=13.01%, 100=84.92%, 250=1.95% cpu : usr=1.36%, sys=4.87%, ctx=2517, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17957: Fri Feb 14 02:34:02 2025 read: IOPS=112, BW=451KiB/s (461kB/s)(4976KiB/11042msec) slat (usec): min=1242, max=16162, avg=4353.89, stdev=1820.45 clat (msec): min=27, max=118, avg=65.17, stdev=15.71 lat (msec): min=29, max=127, avg=69.53, stdev=16.28 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 52], | 30.00th=[ 56], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 68], | 70.00th=[ 73], 80.00th=[ 80], 90.00th=[ 86], 95.00th=[ 93], | 99.00th=[ 110], 99.50th=[ 112], 99.90th=[ 118], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 296, max= 640, per=24.66%, avg=447.64, stdev=81.24, samples=22 iops : min= 74, max= 160, avg=111.68, stdev=20.35, samples=22 write: IOPS=119, BW=477KiB/s (488kB/s)(5264KiB/11042msec) slat (usec): min=12, max=4519, avg=99.84, stdev=229.52 clat (msec): min=18, max=115, avg=67.64, stdev=14.28 lat (msec): min=19, max=115, avg=67.74, stdev=14.28 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 87], 95.00th=[ 94], | 99.00th=[ 105], 99.50th=[ 111], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 335, max= 648, per=25.19%, avg=476.00, stdev=79.64, samples=22 iops : min= 83, max= 162, avg=118.77, stdev=20.04, samples=22 lat (msec) : 20=0.04%, 50=12.34%, 100=85.16%, 250=2.46% cpu : usr=0.68%, sys=5.52%, ctx=2530, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17958: Fri Feb 14 02:34:02 2025 read: IOPS=113, BW=453KiB/s (464kB/s)(4984KiB/10998msec) slat (usec): min=21, max=14923, avg=4316.40, stdev=1884.64 clat (msec): min=18, max=116, avg=65.29, stdev=15.50 lat (msec): min=25, max=124, avg=69.61, stdev=16.13 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 42], 10.00th=[ 47], 20.00th=[ 53], | 30.00th=[ 57], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 68], | 70.00th=[ 73], 80.00th=[ 79], 90.00th=[ 87], 95.00th=[ 94], | 99.00th=[ 106], 99.50th=[ 110], 99.90th=[ 116], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 312, max= 664, per=24.75%, avg=449.19, stdev=77.60, samples=21 iops : min= 78, max= 166, avg=112.05, stdev=19.45, samples=21 write: IOPS=119, BW=478KiB/s (489kB/s)(5256KiB/10998msec) slat (usec): min=14, max=4717, avg=103.16, stdev=271.41 clat (msec): min=21, max=117, avg=67.54, stdev=14.44 lat (msec): min=21, max=117, avg=67.64, stdev=14.45 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 46], 10.00th=[ 50], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 80], 90.00th=[ 87], 95.00th=[ 94], | 99.00th=[ 105], 99.50th=[ 109], 99.90th=[ 111], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 295, max= 592, per=24.81%, avg=468.95, stdev=74.54, samples=21 iops : min= 73, max= 148, avg=117.00, stdev=18.69, samples=21 lat (msec) : 20=0.04%, 50=13.32%, 100=84.30%, 250=2.34% cpu : usr=0.00%, sys=6.22%, ctx=2453, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17959: Fri Feb 14 02:34:02 2025 read: IOPS=114, BW=459KiB/s (470kB/s)(5060KiB/11034msec) slat (usec): min=1146, max=14779, avg=4371.61, stdev=1809.67 clat (msec): min=6, max=118, avg=65.38, stdev=15.43 lat (msec): min=10, max=123, avg=69.76, stdev=15.99 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 53], | 30.00th=[ 57], 40.00th=[ 61], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 72], 80.00th=[ 79], 90.00th=[ 87], 95.00th=[ 93], | 99.00th=[ 106], 99.50th=[ 109], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 271, max= 608, per=25.17%, avg=456.82, stdev=79.04, samples=22 iops : min= 67, max= 152, avg=114.05, stdev=19.81, samples=22 write: IOPS=117, BW=469KiB/s (481kB/s)(5180KiB/11034msec) slat (usec): min=13, max=2742, avg=108.17, stdev=216.10 clat (msec): min=33, max=117, avg=67.46, stdev=14.46 lat (msec): min=33, max=118, avg=67.57, stdev=14.46 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 80], 90.00th=[ 88], 95.00th=[ 94], | 99.00th=[ 104], 99.50th=[ 109], 99.90th=[ 117], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 320, max= 616, per=24.77%, avg=468.09, stdev=71.09, samples=22 iops : min= 80, max= 154, avg=116.86, stdev=17.85, samples=22 lat (msec) : 10=0.04%, 20=0.04%, 50=11.91%, 100=85.74%, 250=2.27% cpu : usr=0.98%, sys=5.37%, ctx=2562, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1816KiB/s (1859kB/s), 451KiB/s-459KiB/s (461kB/s-470kB/s), io=19.6MiB (20.5MB), run=10998-11052msec WRITE: bw=1890KiB/s (1936kB/s), 469KiB/s-478KiB/s (481kB/s-489kB/s), io=20.4MiB (21.4MB), run=10998-11052msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=17972: Fri Feb 14 02:34:04 2025 read: IOPS=3, BW=155MiB/s (163MB/s)(40.0MiB/258msec) slat (nsec): min=12098k, max=12098k, avg=12097913.00, stdev= 0.00 clat (nsec): min=232190k, max=232190k, avg=232190243.00, stdev= 0.00 lat (nsec): min=244311k, max=244311k, avg=244310974.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.89%, ctx=7, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=155MiB/s (163MB/s), 155MiB/s-155MiB/s (163MB/s-163MB/s), io=40.0MiB (41.9MB), run=258-258msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout