== sanity test 398c: run fio to test AIO ================= 20:08:31 (1746144511) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.199807 s, 210 MB/s osc.lustre-OST0000-osc-ffff88013731f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=22732: Thu May 1 20:08:46 2025 write: IOPS=214, BW=860KiB/s (880kB/s)(10.0MiB/11913msec) slat (usec): min=9, max=6351, avg=110.80, stdev=274.49 clat (msec): min=13, max=126, avg=74.08, stdev=14.64 lat (msec): min=14, max=126, avg=74.19, stdev=14.64 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 55], 10.00th=[ 58], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 88], 90.00th=[ 95], 95.00th=[ 101], | 99.00th=[ 111], 99.50th=[ 112], 99.90th=[ 122], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 712, max= 1093, per=24.98%, avg=857.22, stdev=118.73, samples=23 iops : min= 178, max= 273, avg=214.17, stdev=29.60, samples=23 lat (msec) : 20=0.20%, 50=1.88%, 100=93.01%, 250=4.92% cpu : usr=0.00%, sys=3.55%, ctx=1176, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22733: Thu May 1 20:08:46 2025 write: IOPS=214, BW=860KiB/s (881kB/s)(10.0MiB/11907msec) slat (usec): min=9, max=2986, avg=87.06, stdev=187.86 clat (msec): min=10, max=121, avg=74.02, stdev=14.37 lat (msec): min=10, max=121, avg=74.11, stdev=14.38 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 54], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 72], 60.00th=[ 77], | 70.00th=[ 82], 80.00th=[ 88], 90.00th=[ 94], 95.00th=[ 101], | 99.00th=[ 109], 99.50th=[ 114], 99.90th=[ 118], 99.95th=[ 121], | 99.99th=[ 122] bw ( KiB/s): min= 712, max= 1149, per=24.93%, avg=855.48, stdev=118.59, samples=23 iops : min= 178, max= 287, avg=213.74, stdev=29.55, samples=23 lat (msec) : 20=0.08%, 50=2.50%, 100=92.50%, 250=4.92% cpu : usr=0.00%, sys=3.18%, ctx=1031, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22734: Thu May 1 20:08:46 2025 write: IOPS=214, BW=858KiB/s (878kB/s)(10.0MiB/11938msec) slat (usec): min=10, max=2552, avg=88.08, stdev=188.63 clat (msec): min=7, max=123, avg=74.10, stdev=13.96 lat (msec): min=8, max=123, avg=74.19, stdev=13.96 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 55], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 72], 60.00th=[ 77], | 70.00th=[ 82], 80.00th=[ 87], 90.00th=[ 93], 95.00th=[ 100], | 99.00th=[ 108], 99.50th=[ 110], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 124] bw ( KiB/s): min= 678, max= 1152, per=24.81%, avg=851.35, stdev=119.72, samples=23 iops : min= 169, max= 288, avg=212.74, stdev=29.94, samples=23 lat (msec) : 10=0.04%, 20=0.08%, 50=1.13%, 100=93.83%, 250=4.92% cpu : usr=0.49%, sys=2.54%, ctx=970, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22735: Thu May 1 20:08:46 2025 write: IOPS=214, BW=859KiB/s (879kB/s)(10.0MiB/11927msec) slat (usec): min=10, max=2040, avg=93.20, stdev=182.68 clat (msec): min=10, max=117, avg=74.11, stdev=13.75 lat (msec): min=10, max=117, avg=74.21, stdev=13.76 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 55], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 72], 60.00th=[ 77], | 70.00th=[ 82], 80.00th=[ 86], 90.00th=[ 94], 95.00th=[ 101], | 99.00th=[ 108], 99.50th=[ 112], 99.90th=[ 116], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 678, max= 1144, per=24.93%, avg=855.43, stdev=119.36, samples=23 iops : min= 169, max= 286, avg=213.70, stdev=29.90, samples=23 lat (msec) : 20=0.04%, 50=1.76%, 100=93.20%, 250=5.00% cpu : usr=0.42%, sys=2.98%, ctx=1205, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3431KiB/s (3513kB/s), 858KiB/s-860KiB/s (878kB/s-881kB/s), io=40.0MiB (41.9MB), run=11907-11938msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22755: Thu May 1 20:08:57 2025 read: IOPS=119, BW=479KiB/s (490kB/s)(5048KiB/10543msec) slat (usec): min=9, max=11745, avg=4297.84, stdev=1757.65 clat (msec): min=30, max=109, avg=62.87, stdev=14.01 lat (msec): min=31, max=116, avg=67.18, stdev=14.56 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 82], 95.00th=[ 87], | 99.00th=[ 97], 99.50th=[ 104], 99.90th=[ 107], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 359, max= 704, per=25.21%, avg=477.05, stdev=87.27, samples=21 iops : min= 89, max= 176, avg=119.10, stdev=21.95, samples=21 write: IOPS=123, BW=492KiB/s (504kB/s)(5192KiB/10543msec) slat (usec): min=9, max=3269, avg=103.73, stdev=207.01 clat (msec): min=22, max=104, avg=64.37, stdev=13.41 lat (msec): min=22, max=104, avg=64.47, stdev=13.43 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 42], 10.00th=[ 47], 20.00th=[ 53], | 30.00th=[ 58], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 77], 90.00th=[ 82], 95.00th=[ 87], | 99.00th=[ 95], 99.50th=[ 102], 99.90th=[ 104], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 384, max= 624, per=24.97%, avg=491.86, stdev=68.88, samples=21 iops : min= 96, max= 156, avg=122.81, stdev=17.31, samples=21 lat (msec) : 50=17.58%, 100=81.80%, 250=0.62% cpu : usr=0.84%, sys=5.95%, ctx=2361, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22756: Thu May 1 20:08:57 2025 read: IOPS=117, BW=470KiB/s (481kB/s)(4976KiB/10592msec) slat (usec): min=1213, max=16190, avg=4335.99, stdev=1790.39 clat (msec): min=4, max=110, avg=62.72, stdev=14.24 lat (msec): min=10, max=117, avg=67.06, stdev=14.90 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 82], 95.00th=[ 88], | 99.00th=[ 97], 99.50th=[ 101], 99.90th=[ 110], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 320, max= 654, per=24.66%, avg=466.52, stdev=97.09, samples=21 iops : min= 80, max= 163, avg=116.43, stdev=24.19, samples=21 write: IOPS=124, BW=497KiB/s (509kB/s)(5264KiB/10592msec) slat (usec): min=13, max=3045, avg=89.30, stdev=167.84 clat (msec): min=29, max=103, avg=64.58, stdev=12.92 lat (msec): min=29, max=103, avg=64.68, stdev=12.93 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 63], 50.00th=[ 65], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 82], 95.00th=[ 86], | 99.00th=[ 95], 99.50th=[ 96], 99.90th=[ 101], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 392, max= 702, per=25.15%, avg=495.43, stdev=80.67, samples=21 iops : min= 98, max= 175, avg=123.67, stdev=20.13, samples=21 lat (msec) : 10=0.04%, 50=16.91%, 100=82.66%, 250=0.39% cpu : usr=1.48%, sys=4.99%, ctx=2483, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22757: Thu May 1 20:08:57 2025 read: IOPS=117, BW=470KiB/s (481kB/s)(4984KiB/10602msec) slat (usec): min=1171, max=13996, avg=4296.64, stdev=1748.58 clat (msec): min=7, max=109, avg=62.90, stdev=14.37 lat (msec): min=12, max=116, avg=67.20, stdev=14.92 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 56], 40.00th=[ 60], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 82], 95.00th=[ 88], | 99.00th=[ 95], 99.50th=[ 100], 99.90th=[ 109], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 344, max= 704, per=24.72%, avg=467.71, stdev=85.38, samples=21 iops : min= 86, max= 176, avg=116.71, stdev=21.35, samples=21 write: IOPS=123, BW=496KiB/s (508kB/s)(5256KiB/10602msec) slat (usec): min=12, max=3203, avg=104.66, stdev=219.77 clat (msec): min=20, max=111, avg=64.59, stdev=13.29 lat (msec): min=20, max=112, avg=64.69, stdev=13.29 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 82], 95.00th=[ 86], | 99.00th=[ 97], 99.50th=[ 102], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 392, max= 742, per=24.98%, avg=492.14, stdev=91.79, samples=21 iops : min= 98, max= 185, avg=122.81, stdev=23.00, samples=21 lat (msec) : 10=0.04%, 20=0.04%, 50=17.58%, 100=81.88%, 250=0.47% cpu : usr=1.38%, sys=5.53%, ctx=2352, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22758: Thu May 1 20:08:57 2025 read: IOPS=119, BW=477KiB/s (489kB/s)(5060KiB/10602msec) slat (usec): min=1182, max=15046, avg=4321.86, stdev=1665.75 clat (msec): min=14, max=111, avg=62.70, stdev=14.35 lat (msec): min=18, max=116, avg=67.02, stdev=14.88 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 74], 90.00th=[ 82], 95.00th=[ 88], | 99.00th=[ 102], 99.50th=[ 103], 99.90th=[ 106], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 360, max= 664, per=25.10%, avg=474.81, stdev=82.90, samples=21 iops : min= 90, max= 166, avg=118.57, stdev=20.80, samples=21 write: IOPS=122, BW=489KiB/s (500kB/s)(5180KiB/10602msec) slat (usec): min=13, max=4856, avg=107.67, stdev=284.73 clat (msec): min=27, max=105, avg=64.82, stdev=13.56 lat (msec): min=27, max=106, avg=64.93, stdev=13.56 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 53], | 30.00th=[ 57], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 82], 95.00th=[ 88], | 99.00th=[ 102], 99.50th=[ 103], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 304, max= 632, per=24.60%, avg=484.57, stdev=84.50, samples=21 iops : min= 76, max= 158, avg=121.05, stdev=21.10, samples=21 lat (msec) : 20=0.04%, 50=17.50%, 100=81.29%, 250=1.17% cpu : usr=1.59%, sys=4.81%, ctx=2410, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1893KiB/s (1938kB/s), 470KiB/s-479KiB/s (481kB/s-490kB/s), io=19.6MiB (20.5MB), run=10543-10602msec WRITE: bw=1971KiB/s (2018kB/s), 489KiB/s-497KiB/s (500kB/s-509kB/s), io=20.4MiB (21.4MB), run=10543-10602msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=22771: Thu May 1 20:08:59 2025 read: IOPS=4, BW=177MiB/s (186MB/s)(40.0MiB/226msec) slat (nsec): min=3616.9k, max=3616.9k, avg=3616915.00, stdev= 0.00 clat (nsec): min=213941k, max=213941k, avg=213940565.00, stdev= 0.00 lat (nsec): min=217580k, max=217580k, avg=217579807.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=0.89%, ctx=13, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177MiB/s (186MB/s), 177MiB/s-177MiB/s (186MB/s-186MB/s), io=40.0MiB (41.9MB), run=226-226msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout