== sanity test 398c: run fio to test AIO ================= 23:04:50 (1746155090) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.187254 s, 224 MB/s osc.lustre-OST0000-osc-ffff88012c472000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=23540: Thu May 1 23:05:08 2025 write: IOPS=171, BW=688KiB/s (704kB/s)(10.0MiB/14887msec) slat (usec): min=10, max=5240, avg=111.24, stdev=286.99 clat (msec): min=18, max=157, avg=92.60, stdev=19.81 lat (msec): min=18, max=157, avg=92.71, stdev=19.82 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 75], | 30.00th=[ 81], 40.00th=[ 86], 50.00th=[ 91], 60.00th=[ 97], | 70.00th=[ 103], 80.00th=[ 109], 90.00th=[ 121], 95.00th=[ 129], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 150], 99.95th=[ 155], | 99.99th=[ 157] bw ( KiB/s): min= 520, max= 856, per=24.96%, avg=683.83, stdev=78.69, samples=29 iops : min= 130, max= 214, avg=170.79, stdev=19.65, samples=29 lat (msec) : 20=0.08%, 50=0.35%, 100=66.21%, 250=33.36% cpu : usr=0.50%, sys=2.45%, ctx=1133, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23541: Thu May 1 23:05:08 2025 write: IOPS=171, BW=686KiB/s (702kB/s)(10.0MiB/14928msec) slat (usec): min=10, max=3641, avg=87.75, stdev=232.07 clat (msec): min=21, max=147, avg=92.86, stdev=19.58 lat (msec): min=21, max=147, avg=92.95, stdev=19.59 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 66], 10.00th=[ 69], 20.00th=[ 75], | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 103], 80.00th=[ 110], 90.00th=[ 121], 95.00th=[ 129], | 99.00th=[ 140], 99.50th=[ 146], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 512, max= 848, per=24.83%, avg=680.21, stdev=87.89, samples=29 iops : min= 128, max= 212, avg=170.00, stdev=22.02, samples=29 lat (msec) : 50=0.55%, 100=65.39%, 250=34.06% cpu : usr=0.35%, sys=2.00%, ctx=746, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23542: Thu May 1 23:05:08 2025 write: IOPS=171, BW=685KiB/s (702kB/s)(10.0MiB/14947msec) slat (usec): min=11, max=4023, avg=98.93, stdev=255.38 clat (msec): min=9, max=155, avg=92.76, stdev=19.72 lat (msec): min=9, max=155, avg=92.86, stdev=19.73 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 65], 10.00th=[ 68], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 88], 50.00th=[ 92], 60.00th=[ 97], | 70.00th=[ 103], 80.00th=[ 110], 90.00th=[ 120], 95.00th=[ 129], | 99.00th=[ 144], 99.50th=[ 148], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 157] bw ( KiB/s): min= 518, max= 854, per=24.70%, avg=676.76, stdev=87.46, samples=29 iops : min= 129, max= 213, avg=169.03, stdev=21.89, samples=29 lat (msec) : 10=0.04%, 20=0.04%, 50=0.90%, 100=65.51%, 250=33.52% cpu : usr=0.38%, sys=2.47%, ctx=981, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23543: Thu May 1 23:05:08 2025 write: IOPS=171, BW=686KiB/s (702kB/s)(10.0MiB/14937msec) slat (usec): min=10, max=6207, avg=108.61, stdev=296.16 clat (msec): min=12, max=156, avg=92.82, stdev=18.86 lat (msec): min=13, max=156, avg=92.93, stdev=18.87 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 65], 10.00th=[ 70], 20.00th=[ 75], | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 92], 60.00th=[ 97], | 70.00th=[ 102], 80.00th=[ 110], 90.00th=[ 117], 95.00th=[ 127], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 510, max= 790, per=24.85%, avg=680.93, stdev=82.62, samples=29 iops : min= 127, max= 197, avg=170.10, stdev=20.59, samples=29 lat (msec) : 20=0.08%, 50=0.16%, 100=67.38%, 250=32.38% cpu : usr=0.43%, sys=2.41%, ctx=1027, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2740KiB/s (2806kB/s), 685KiB/s-688KiB/s (702kB/s-704kB/s), io=40.0MiB (41.9MB), run=14887-14947msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23565: Thu May 1 23:05:22 2025 read: IOPS=101, BW=407KiB/s (417kB/s)(5048KiB/12400msec) slat (usec): min=17, max=18739, avg=5059.24, stdev=2245.75 clat (msec): min=7, max=152, avg=73.23, stdev=20.90 lat (msec): min=7, max=161, avg=78.29, stdev=21.71 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 45], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 71], 60.00th=[ 75], | 70.00th=[ 82], 80.00th=[ 89], 90.00th=[ 102], 95.00th=[ 112], | 99.00th=[ 136], 99.50th=[ 146], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 302, max= 568, per=25.34%, avg=407.21, stdev=69.64, samples=24 iops : min= 75, max= 142, avg=101.58, stdev=17.39, samples=24 write: IOPS=104, BW=419KiB/s (429kB/s)(5192KiB/12400msec) slat (usec): min=11, max=2586, avg=109.32, stdev=221.65 clat (msec): min=23, max=155, avg=76.27, stdev=18.54 lat (msec): min=23, max=155, avg=76.38, stdev=18.55 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 60], | 30.00th=[ 66], 40.00th=[ 71], 50.00th=[ 75], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 91], 90.00th=[ 102], 95.00th=[ 110], | 99.00th=[ 128], 99.50th=[ 133], 99.90th=[ 150], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 287, max= 550, per=25.04%, avg=418.92, stdev=69.86, samples=24 iops : min= 71, max= 137, avg=104.50, stdev=17.49, samples=24 lat (msec) : 10=0.12%, 20=0.04%, 50=7.42%, 100=81.52%, 250=10.90% cpu : usr=1.33%, sys=4.88%, ctx=2383, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23566: Thu May 1 23:05:22 2025 read: IOPS=100, BW=401KiB/s (410kB/s)(4976KiB/12422msec) slat (usec): min=1131, max=20835, avg=5096.38, stdev=2382.89 clat (msec): min=10, max=151, avg=73.72, stdev=20.23 lat (msec): min=13, max=158, avg=78.82, stdev=20.90 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 57], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 82], 80.00th=[ 90], 90.00th=[ 101], 95.00th=[ 111], | 99.00th=[ 136], 99.50th=[ 144], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 296, max= 520, per=24.77%, avg=398.04, stdev=70.15, samples=24 iops : min= 74, max= 130, avg=99.29, stdev=17.55, samples=24 write: IOPS=105, BW=424KiB/s (434kB/s)(5264KiB/12422msec) slat (usec): min=12, max=13492, avg=115.23, stdev=498.54 clat (msec): min=6, max=150, avg=75.63, stdev=18.37 lat (msec): min=6, max=150, avg=75.75, stdev=18.36 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 90], 90.00th=[ 99], 95.00th=[ 108], | 99.00th=[ 130], 99.50th=[ 140], 99.90th=[ 148], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 320, max= 598, per=25.32%, avg=423.63, stdev=71.66, samples=24 iops : min= 80, max= 149, avg=105.71, stdev=17.91, samples=24 lat (msec) : 10=0.04%, 20=0.08%, 50=7.54%, 100=82.97%, 250=9.38% cpu : usr=1.89%, sys=4.28%, ctx=2301, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23567: Thu May 1 23:05:22 2025 read: IOPS=99, BW=399KiB/s (409kB/s)(4984KiB/12485msec) slat (usec): min=1388, max=22188, avg=5133.52, stdev=2406.93 clat (msec): min=14, max=152, avg=73.87, stdev=21.17 lat (msec): min=20, max=159, avg=79.01, stdev=22.01 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 82], 80.00th=[ 90], 90.00th=[ 104], 95.00th=[ 112], | 99.00th=[ 142], 99.50th=[ 148], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 272, max= 552, per=24.85%, avg=399.29, stdev=86.16, samples=24 iops : min= 68, max= 138, avg=99.67, stdev=21.48, samples=24 write: IOPS=105, BW=421KiB/s (431kB/s)(5256KiB/12485msec) slat (usec): min=12, max=4874, avg=112.42, stdev=266.22 clat (msec): min=28, max=147, avg=76.02, stdev=18.48 lat (msec): min=28, max=147, avg=76.13, stdev=18.49 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 50], 10.00th=[ 55], 20.00th=[ 60], | 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 102], 95.00th=[ 110], | 99.00th=[ 130], 99.50th=[ 134], 99.90th=[ 146], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 287, max= 640, per=24.91%, avg=416.67, stdev=74.45, samples=24 iops : min= 71, max= 160, avg=104.00, stdev=18.61, samples=24 lat (msec) : 20=0.04%, 50=7.42%, 100=81.41%, 250=11.13% cpu : usr=1.95%, sys=4.33%, ctx=2246, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23568: Thu May 1 23:05:22 2025 read: IOPS=101, BW=405KiB/s (415kB/s)(5060KiB/12483msec) slat (usec): min=950, max=20519, avg=5061.16, stdev=2340.62 clat (msec): min=14, max=159, avg=73.66, stdev=20.53 lat (msec): min=20, max=167, avg=78.72, stdev=21.21 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 57], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 84], 80.00th=[ 90], 90.00th=[ 102], 95.00th=[ 110], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 153], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 248, max= 582, per=24.94%, avg=400.75, stdev=71.87, samples=24 iops : min= 62, max= 145, avg=99.83, stdev=17.94, samples=24 write: IOPS=103, BW=415KiB/s (425kB/s)(5180KiB/12483msec) slat (usec): min=14, max=5623, avg=121.80, stdev=293.19 clat (msec): min=31, max=152, avg=76.09, stdev=19.32 lat (msec): min=31, max=152, avg=76.21, stdev=19.33 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 60], | 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 92], 90.00th=[ 102], 95.00th=[ 110], | 99.00th=[ 134], 99.50th=[ 138], 99.90th=[ 148], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 295, max= 568, per=24.69%, avg=413.08, stdev=76.77, samples=24 iops : min= 73, max= 142, avg=102.92, stdev=19.25, samples=24 lat (msec) : 20=0.04%, 50=8.48%, 100=81.02%, 250=10.47% cpu : usr=1.68%, sys=4.38%, ctx=2337, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1607KiB/s (1646kB/s), 399KiB/s-407KiB/s (409kB/s-417kB/s), io=19.6MiB (20.5MB), run=12400-12485msec WRITE: bw=1673KiB/s (1714kB/s), 415KiB/s-424KiB/s (425kB/s-434kB/s), io=20.4MiB (21.4MB), run=12400-12485msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=23582: Thu May 1 23:05:24 2025 read: IOPS=2, BW=106MiB/s (112MB/s)(40.0MiB/376msec) slat (nsec): min=2267.9k, max=2267.9k, avg=2267922.00, stdev= 0.00 clat (nsec): min=362437k, max=362437k, avg=362436940.00, stdev= 0.00 lat (nsec): min=366384k, max=366384k, avg=366384469.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.53%, ctx=3, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=106MiB/s (112MB/s), 106MiB/s-106MiB/s (112MB/s-112MB/s), io=40.0MiB (41.9MB), run=376-376msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout