== sanity test 398c: run fio to test AIO ================= 18:57:54 (1746140274) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.302692 s, 139 MB/s osc.lustre-OST0000-osc-ffff8800a9e5e000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=24351: Thu May 1 18:58:14 2025 write: IOPS=152, BW=611KiB/s (626kB/s)(10.0MiB/16756msec) slat (usec): min=14, max=4703, avg=123.76, stdev=347.31 clat (msec): min=15, max=183, avg=104.25, stdev=21.44 lat (msec): min=16, max=183, avg=104.37, stdev=21.43 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 81], 20.00th=[ 87], | 30.00th=[ 93], 40.00th=[ 97], 50.00th=[ 104], 60.00th=[ 108], | 70.00th=[ 114], 80.00th=[ 122], 90.00th=[ 134], 95.00th=[ 142], | 99.00th=[ 161], 99.50th=[ 165], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 464, max= 768, per=24.99%, avg=609.42, stdev=76.02, samples=33 iops : min= 116, max= 192, avg=152.24, stdev=18.96, samples=33 lat (msec) : 20=0.04%, 50=0.66%, 100=44.65%, 250=54.65% cpu : usr=0.20%, sys=2.45%, ctx=847, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24352: Thu May 1 18:58:14 2025 write: IOPS=152, BW=612KiB/s (626kB/s)(10.0MiB/16740msec) slat (usec): min=14, max=7244, avg=147.46, stdev=412.93 clat (msec): min=11, max=179, avg=104.11, stdev=20.74 lat (msec): min=12, max=179, avg=104.26, stdev=20.74 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 77], 10.00th=[ 82], 20.00th=[ 87], | 30.00th=[ 92], 40.00th=[ 97], 50.00th=[ 102], 60.00th=[ 108], | 70.00th=[ 114], 80.00th=[ 121], 90.00th=[ 131], 95.00th=[ 142], | 99.00th=[ 159], 99.50th=[ 165], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 464, max= 720, per=24.89%, avg=606.97, stdev=71.47, samples=33 iops : min= 116, max= 180, avg=151.64, stdev=17.86, samples=33 lat (msec) : 20=0.20%, 50=0.31%, 100=46.76%, 250=52.73% cpu : usr=0.68%, sys=2.35%, ctx=1181, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24353: Thu May 1 18:58:14 2025 write: IOPS=152, BW=611KiB/s (625kB/s)(10.0MiB/16771msec) slat (usec): min=14, max=4511, avg=87.04, stdev=278.50 clat (msec): min=47, max=182, avg=104.20, stdev=20.33 lat (msec): min=47, max=182, avg=104.29, stdev=20.34 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 75], 10.00th=[ 81], 20.00th=[ 87], | 30.00th=[ 93], 40.00th=[ 99], 50.00th=[ 103], 60.00th=[ 109], | 70.00th=[ 115], 80.00th=[ 122], 90.00th=[ 131], 95.00th=[ 142], | 99.00th=[ 159], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 399, max= 766, per=24.95%, avg=608.52, stdev=86.97, samples=33 iops : min= 99, max= 191, avg=151.94, stdev=21.80, samples=33 lat (msec) : 50=0.27%, 100=46.21%, 250=53.52% cpu : usr=0.42%, sys=1.46%, ctx=576, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24354: Thu May 1 18:58:14 2025 write: IOPS=152, BW=610KiB/s (624kB/s)(10.0MiB/16791msec) slat (usec): min=12, max=7813, avg=104.05, stdev=319.45 clat (msec): min=25, max=175, avg=104.45, stdev=20.05 lat (msec): min=25, max=175, avg=104.56, stdev=20.05 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 78], 10.00th=[ 83], 20.00th=[ 88], | 30.00th=[ 92], 40.00th=[ 97], 50.00th=[ 102], 60.00th=[ 109], | 70.00th=[ 114], 80.00th=[ 122], 90.00th=[ 133], 95.00th=[ 144], | 99.00th=[ 155], 99.50th=[ 171], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 432, max= 758, per=24.86%, avg=606.24, stdev=85.82, samples=33 iops : min= 108, max= 189, avg=151.45, stdev=21.36, samples=33 lat (msec) : 50=0.31%, 100=47.50%, 250=52.19% cpu : usr=0.23%, sys=2.14%, ctx=826, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2439KiB/s (2498kB/s), 610KiB/s-612KiB/s (624kB/s-626kB/s), io=40.0MiB (41.9MB), run=16740-16791msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24378: Thu May 1 18:58:31 2025 read: IOPS=80, BW=323KiB/s (331kB/s)(5048KiB/15640msec) slat (usec): min=1897, max=25377, avg=6309.74, stdev=2777.65 clat (msec): min=9, max=164, avg=91.81, stdev=19.22 lat (msec): min=12, max=170, avg=98.12, stdev=19.71 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 75], | 30.00th=[ 81], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 102], 80.00th=[ 109], 90.00th=[ 117], 95.00th=[ 125], | 99.00th=[ 144], 99.50th=[ 153], 99.90th=[ 159], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 216, max= 408, per=25.06%, avg=321.32, stdev=56.11, samples=31 iops : min= 54, max= 102, avg=80.03, stdev=13.97, samples=31 write: IOPS=82, BW=332KiB/s (340kB/s)(5192KiB/15640msec) slat (usec): min=15, max=4996, avg=127.22, stdev=305.28 clat (msec): min=6, max=157, avg=95.56, stdev=18.43 lat (msec): min=6, max=157, avg=95.68, stdev=18.42 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 80], | 30.00th=[ 86], 40.00th=[ 91], 50.00th=[ 95], 60.00th=[ 100], | 70.00th=[ 106], 80.00th=[ 111], 90.00th=[ 121], 95.00th=[ 127], | 99.00th=[ 144], 99.50th=[ 150], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 231, max= 408, per=24.80%, avg=331.13, stdev=44.98, samples=31 iops : min= 57, max= 102, avg=82.48, stdev=11.28, samples=31 lat (msec) : 10=0.08%, 50=0.08%, 100=64.38%, 250=35.47% cpu : usr=0.67%, sys=4.99%, ctx=2311, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24379: Thu May 1 18:58:31 2025 read: IOPS=80, BW=321KiB/s (329kB/s)(4976KiB/15488msec) slat (usec): min=21, max=24323, avg=6318.48, stdev=2732.91 clat (msec): min=19, max=161, avg=91.02, stdev=20.34 lat (msec): min=19, max=168, avg=97.35, stdev=20.97 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 80], 40.00th=[ 86], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 102], 80.00th=[ 108], 90.00th=[ 118], 95.00th=[ 128], | 99.00th=[ 142], 99.50th=[ 146], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 214, max= 408, per=24.96%, avg=319.97, stdev=50.41, samples=30 iops : min= 53, max= 102, avg=79.73, stdev=12.64, samples=30 write: IOPS=84, BW=340KiB/s (348kB/s)(5264KiB/15488msec) slat (usec): min=15, max=7037, avg=122.90, stdev=306.69 clat (msec): min=44, max=177, avg=95.77, stdev=18.72 lat (msec): min=44, max=177, avg=95.90, stdev=18.73 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 67], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 86], 40.00th=[ 91], 50.00th=[ 95], 60.00th=[ 100], | 70.00th=[ 106], 80.00th=[ 112], 90.00th=[ 120], 95.00th=[ 128], | 99.00th=[ 144], 99.50th=[ 148], 99.90th=[ 174], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 224, max= 432, per=25.49%, avg=340.27, stdev=48.11, samples=30 iops : min= 56, max= 108, avg=84.80, stdev=12.08, samples=30 lat (msec) : 20=0.12%, 50=0.39%, 100=63.87%, 250=35.62% cpu : usr=0.86%, sys=4.77%, ctx=2221, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24380: Thu May 1 18:58:31 2025 read: IOPS=79, BW=319KiB/s (326kB/s)(4984KiB/15645msec) slat (usec): min=1940, max=25741, avg=6389.28, stdev=2725.77 clat (msec): min=18, max=159, avg=92.25, stdev=19.17 lat (msec): min=27, max=167, avg=98.65, stdev=19.76 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 103], 80.00th=[ 108], 90.00th=[ 116], 95.00th=[ 126], | 99.00th=[ 142], 99.50th=[ 150], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 184, max= 424, per=24.80%, avg=318.00, stdev=47.50, samples=31 iops : min= 46, max= 106, avg=79.23, stdev=11.88, samples=31 write: IOPS=83, BW=336KiB/s (344kB/s)(5256KiB/15645msec) slat (usec): min=13, max=6125, avg=127.14, stdev=355.43 clat (msec): min=32, max=160, avg=95.06, stdev=17.43 lat (msec): min=32, max=160, avg=95.19, stdev=17.43 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 69], 10.00th=[ 73], 20.00th=[ 80], | 30.00th=[ 86], 40.00th=[ 90], 50.00th=[ 95], 60.00th=[ 100], | 70.00th=[ 104], 80.00th=[ 109], 90.00th=[ 118], 95.00th=[ 127], | 99.00th=[ 140], 99.50th=[ 146], 99.90th=[ 159], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 223, max= 464, per=24.98%, avg=333.48, stdev=64.85, samples=31 iops : min= 55, max= 116, avg=83.10, stdev=16.37, samples=31 lat (msec) : 20=0.04%, 50=0.04%, 100=64.22%, 250=35.70% cpu : usr=0.00%, sys=5.57%, ctx=2259, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24381: Thu May 1 18:58:31 2025 read: IOPS=81, BW=325KiB/s (332kB/s)(5060KiB/15584msec) slat (usec): min=1731, max=23751, avg=6439.18, stdev=2713.95 clat (msec): min=26, max=166, avg=92.18, stdev=19.88 lat (msec): min=31, max=172, avg=98.63, stdev=20.40 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 63], 10.00th=[ 69], 20.00th=[ 74], | 30.00th=[ 81], 40.00th=[ 86], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 103], 80.00th=[ 110], 90.00th=[ 118], 95.00th=[ 128], | 99.00th=[ 144], 99.50th=[ 150], 99.90th=[ 161], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 240, max= 463, per=25.27%, avg=324.00, stdev=48.08, samples=31 iops : min= 60, max= 115, avg=80.77, stdev=11.92, samples=31 write: IOPS=83, BW=332KiB/s (340kB/s)(5180KiB/15584msec) slat (usec): min=12, max=7504, avg=120.66, stdev=336.61 clat (msec): min=31, max=180, avg=94.83, stdev=18.14 lat (msec): min=31, max=180, avg=94.95, stdev=18.14 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 80], | 30.00th=[ 85], 40.00th=[ 89], 50.00th=[ 94], 60.00th=[ 100], | 70.00th=[ 105], 80.00th=[ 110], 90.00th=[ 118], 95.00th=[ 125], | 99.00th=[ 146], 99.50th=[ 150], 99.90th=[ 167], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 232, max= 455, per=24.87%, avg=332.00, stdev=50.96, samples=31 iops : min= 58, max= 113, avg=82.77, stdev=12.69, samples=31 lat (msec) : 50=0.08%, 100=64.26%, 250=35.66% cpu : usr=1.20%, sys=4.43%, ctx=2285, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1283KiB/s (1313kB/s), 319KiB/s-325KiB/s (326kB/s-332kB/s), io=19.6MiB (20.5MB), run=15488-15645msec WRITE: bw=1335KiB/s (1367kB/s), 332KiB/s-340KiB/s (340kB/s-348kB/s), io=20.4MiB (21.4MB), run=15488-15645msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=24398: Thu May 1 18:58:34 2025 read: IOPS=2, BW=109MiB/s (114MB/s)(40.0MiB/367msec) slat (nsec): min=19087k, max=19087k, avg=19086973.00, stdev= 0.00 clat (nsec): min=327539k, max=327539k, avg=327538545.00, stdev= 0.00 lat (nsec): min=346654k, max=346654k, avg=346653571.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.46%, ctx=14, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=109MiB/s (114MB/s), 109MiB/s-109MiB/s (114MB/s-114MB/s), io=40.0MiB (41.9MB), run=367-367msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout