== sanity test 398c: run fio to test AIO ================= 20:55:22 (1746147322) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.180068 s, 233 MB/s osc.lustre-OST0000-osc-ffff8800aa852800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=22322: Thu May 1 20:55:40 2025 write: IOPS=167, BW=671KiB/s (687kB/s)(10.0MiB/15258msec) slat (usec): min=14, max=3606, avg=105.54, stdev=276.85 clat (msec): min=35, max=163, avg=94.71, stdev=20.01 lat (msec): min=36, max=163, avg=94.82, stdev=20.02 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 66], 10.00th=[ 72], 20.00th=[ 79], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 99], | 70.00th=[ 104], 80.00th=[ 111], 90.00th=[ 121], 95.00th=[ 131], | 99.00th=[ 150], 99.50th=[ 157], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 512, max= 888, per=24.89%, avg=667.93, stdev=100.23, samples=30 iops : min= 128, max= 222, avg=166.87, stdev=25.03, samples=30 lat (msec) : 50=0.74%, 100=62.66%, 250=36.60% cpu : usr=0.62%, sys=2.14%, ctx=952, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22323: Thu May 1 20:55:40 2025 write: IOPS=167, BW=672KiB/s (688kB/s)(10.0MiB/15248msec) slat (usec): min=11, max=9176, avg=127.96, stdev=363.90 clat (msec): min=21, max=157, avg=94.62, stdev=20.62 lat (msec): min=21, max=157, avg=94.75, stdev=20.61 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 64], 10.00th=[ 71], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 97], | 70.00th=[ 105], 80.00th=[ 111], 90.00th=[ 122], 95.00th=[ 133], | 99.00th=[ 150], 99.50th=[ 155], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 159] bw ( KiB/s): min= 510, max= 832, per=24.92%, avg=668.97, stdev=98.43, samples=30 iops : min= 127, max= 208, avg=167.07, stdev=24.63, samples=30 lat (msec) : 50=0.82%, 100=63.01%, 250=36.17% cpu : usr=0.51%, sys=2.62%, ctx=1183, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22324: Thu May 1 20:55:40 2025 write: IOPS=167, BW=671KiB/s (687kB/s)(10.0MiB/15253msec) slat (usec): min=11, max=3643, avg=105.54, stdev=276.87 clat (msec): min=14, max=159, avg=94.60, stdev=20.23 lat (msec): min=14, max=159, avg=94.70, stdev=20.22 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 65], 10.00th=[ 69], 20.00th=[ 79], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 100], | 70.00th=[ 105], 80.00th=[ 111], 90.00th=[ 120], 95.00th=[ 131], | 99.00th=[ 153], 99.50th=[ 159], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 512, max= 896, per=24.94%, avg=669.33, stdev=103.59, samples=30 iops : min= 128, max= 224, avg=167.17, stdev=25.86, samples=30 lat (msec) : 20=0.08%, 50=0.31%, 100=61.72%, 250=37.89% cpu : usr=0.65%, sys=2.15%, ctx=903, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22325: Thu May 1 20:55:40 2025 write: IOPS=167, BW=671KiB/s (687kB/s)(10.0MiB/15255msec) slat (usec): min=10, max=2936, avg=108.89, stdev=253.84 clat (msec): min=11, max=160, avg=94.66, stdev=20.72 lat (msec): min=11, max=160, avg=94.77, stdev=20.72 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 65], 10.00th=[ 69], 20.00th=[ 78], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 94], 60.00th=[ 99], | 70.00th=[ 104], 80.00th=[ 111], 90.00th=[ 123], 95.00th=[ 131], | 99.00th=[ 155], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 161] bw ( KiB/s): min= 512, max= 880, per=24.84%, avg=666.60, stdev=103.03, samples=30 iops : min= 128, max= 220, avg=166.50, stdev=25.78, samples=30 lat (msec) : 20=0.08%, 50=0.78%, 100=61.95%, 250=37.19% cpu : usr=0.43%, sys=2.58%, ctx=1022, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2684KiB/s (2749kB/s), 671KiB/s-672KiB/s (687kB/s-688kB/s), io=40.0MiB (41.9MB), run=15248-15258msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22347: Thu May 1 20:55:57 2025 read: IOPS=83, BW=335KiB/s (343kB/s)(5048KiB/15075msec) slat (usec): min=17, max=18192, avg=6328.24, stdev=2657.62 clat (msec): min=6, max=149, avg=89.33, stdev=20.51 lat (msec): min=10, max=162, avg=95.66, stdev=21.27 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 72], | 30.00th=[ 79], 40.00th=[ 84], 50.00th=[ 88], 60.00th=[ 93], | 70.00th=[ 100], 80.00th=[ 109], 90.00th=[ 117], 95.00th=[ 125], | 99.00th=[ 138], 99.50th=[ 142], 99.90th=[ 148], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 248, max= 448, per=25.21%, avg=333.97, stdev=52.69, samples=30 iops : min= 62, max= 112, avg=83.37, stdev=13.18, samples=30 write: IOPS=86, BW=344KiB/s (353kB/s)(5192KiB/15075msec) slat (usec): min=15, max=3350, avg=125.47, stdev=286.27 clat (msec): min=17, max=147, avg=92.44, stdev=18.56 lat (msec): min=17, max=149, avg=92.57, stdev=18.56 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 78], | 30.00th=[ 84], 40.00th=[ 88], 50.00th=[ 92], 60.00th=[ 96], | 70.00th=[ 102], 80.00th=[ 109], 90.00th=[ 118], 95.00th=[ 125], | 99.00th=[ 134], 99.50th=[ 138], 99.90th=[ 146], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 224, max= 440, per=24.97%, avg=344.37, stdev=59.23, samples=30 iops : min= 56, max= 110, avg=85.97, stdev=14.81, samples=30 lat (msec) : 10=0.04%, 20=0.16%, 50=0.74%, 100=68.05%, 250=31.02% cpu : usr=1.40%, sys=4.59%, ctx=2199, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22348: Thu May 1 20:55:57 2025 read: IOPS=82, BW=330KiB/s (338kB/s)(4976KiB/15075msec) slat (usec): min=1264, max=23727, avg=6304.85, stdev=2692.40 clat (msec): min=16, max=154, avg=88.90, stdev=19.95 lat (msec): min=23, max=161, avg=95.21, stdev=20.58 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 72], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 99], 80.00th=[ 106], 90.00th=[ 117], 95.00th=[ 126], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 150], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 232, max= 432, per=24.84%, avg=329.17, stdev=53.91, samples=30 iops : min= 58, max= 108, avg=82.17, stdev=13.49, samples=30 write: IOPS=87, BW=349KiB/s (358kB/s)(5264KiB/15075msec) slat (usec): min=15, max=6476, avg=137.88, stdev=348.10 clat (msec): min=34, max=151, avg=92.53, stdev=19.22 lat (msec): min=35, max=151, avg=92.67, stdev=19.23 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 63], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 92], 60.00th=[ 97], | 70.00th=[ 103], 80.00th=[ 110], 90.00th=[ 117], 95.00th=[ 127], | 99.00th=[ 140], 99.50th=[ 146], 99.90th=[ 150], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 224, max= 432, per=25.32%, avg=349.17, stdev=55.25, samples=30 iops : min= 56, max= 108, avg=87.17, stdev=13.83, samples=30 lat (msec) : 20=0.04%, 50=0.98%, 100=68.20%, 250=30.78% cpu : usr=1.18%, sys=4.76%, ctx=2236, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22349: Thu May 1 20:55:57 2025 read: IOPS=82, BW=329KiB/s (337kB/s)(4984KiB/15145msec) slat (usec): min=1343, max=24873, avg=6357.84, stdev=2628.24 clat (msec): min=23, max=155, avg=90.06, stdev=21.10 lat (msec): min=31, max=168, avg=96.42, stdev=21.67 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 60], 10.00th=[ 65], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 84], 50.00th=[ 88], 60.00th=[ 94], | 70.00th=[ 101], 80.00th=[ 109], 90.00th=[ 120], 95.00th=[ 129], | 99.00th=[ 144], 99.50th=[ 146], 99.90th=[ 150], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 231, max= 439, per=24.70%, avg=327.30, stdev=52.42, samples=30 iops : min= 57, max= 109, avg=81.53, stdev=13.07, samples=30 write: IOPS=86, BW=347KiB/s (355kB/s)(5256KiB/15145msec) slat (usec): min=15, max=5127, avg=122.84, stdev=271.53 clat (msec): min=37, max=155, avg=91.91, stdev=18.00 lat (msec): min=37, max=155, avg=92.04, stdev=17.98 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 65], 10.00th=[ 70], 20.00th=[ 79], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 95], | 70.00th=[ 101], 80.00th=[ 106], 90.00th=[ 117], 95.00th=[ 126], | 99.00th=[ 138], 99.50th=[ 142], 99.90th=[ 150], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 240, max= 416, per=24.95%, avg=344.10, stdev=53.48, samples=30 iops : min= 60, max= 104, avg=85.73, stdev=13.40, samples=30 lat (msec) : 50=0.74%, 100=69.65%, 250=29.61% cpu : usr=1.06%, sys=4.69%, ctx=2218, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22350: Thu May 1 20:55:57 2025 read: IOPS=83, BW=334KiB/s (342kB/s)(5060KiB/15131msec) slat (usec): min=1609, max=20120, avg=6354.42, stdev=2614.96 clat (msec): min=26, max=157, avg=88.88, stdev=20.29 lat (msec): min=32, max=166, avg=95.24, stdev=20.90 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 71], | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 93], | 70.00th=[ 99], 80.00th=[ 106], 90.00th=[ 115], 95.00th=[ 127], | 99.00th=[ 142], 99.50th=[ 153], 99.90th=[ 155], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 247, max= 439, per=25.13%, avg=332.97, stdev=53.70, samples=30 iops : min= 61, max= 109, avg=82.83, stdev=13.52, samples=30 write: IOPS=85, BW=342KiB/s (351kB/s)(5180KiB/15131msec) slat (usec): min=14, max=4917, avg=129.96, stdev=291.52 clat (msec): min=30, max=150, avg=92.94, stdev=18.96 lat (msec): min=30, max=150, avg=93.08, stdev=18.95 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 68], 20.00th=[ 78], | 30.00th=[ 84], 40.00th=[ 89], 50.00th=[ 93], 60.00th=[ 97], | 70.00th=[ 103], 80.00th=[ 109], 90.00th=[ 118], 95.00th=[ 128], | 99.00th=[ 140], 99.50th=[ 142], 99.90th=[ 146], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 223, max= 456, per=24.68%, avg=340.40, stdev=50.42, samples=30 iops : min= 55, max= 114, avg=84.70, stdev=12.73, samples=30 lat (msec) : 50=0.94%, 100=68.71%, 250=30.35% cpu : usr=0.71%, sys=5.06%, ctx=2274, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1325KiB/s (1357kB/s), 329KiB/s-335KiB/s (337kB/s-343kB/s), io=19.6MiB (20.5MB), run=15075-15145msec WRITE: bw=1379KiB/s (1413kB/s), 342KiB/s-349KiB/s (351kB/s-358kB/s), io=20.4MiB (21.4MB), run=15075-15145msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=22367: Thu May 1 20:56:00 2025 read: IOPS=2, BW=109MiB/s (114MB/s)(40.0MiB/368msec) slat (nsec): min=12778k, max=12778k, avg=12778060.00, stdev= 0.00 clat (nsec): min=341119k, max=341119k, avg=341118762.00, stdev= 0.00 lat (nsec): min=353996k, max=353996k, avg=353995817.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 342], 10.00th=[ 342], 20.00th=[ 342], | 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.82%, ctx=13, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=109MiB/s (114MB/s), 109MiB/s-109MiB/s (114MB/s-114MB/s), io=40.0MiB (41.9MB), run=368-368msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout