== sanity test 398c: run fio to test AIO ================= 23:00:59 (1783566059) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.146324 s, 287 MB/s osc.lustre-OST0000-osc-ffff9fbe51fb0000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=184527: Wed Jul 8 23:01:11 2026 write: IOPS=315, BW=1261KiB/s (1291kB/s)(10.0MiB/8121msec); 0 zone resets slat (usec): min=37, max=3659, avg=175.24, stdev=153.04 clat (msec): min=10, max=108, avg=50.08, stdev=12.80 lat (msec): min=10, max=108, avg=50.26, stdev=12.81 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 35], 20.00th=[ 38], | 30.00th=[ 42], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 56], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 72], | 99.00th=[ 88], 99.50th=[ 97], 99.90th=[ 106], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 896, max= 1640, per=25.38%, avg=1280.07, stdev=205.32, samples=15 iops : min= 224, max= 410, avg=319.93, stdev=51.38, samples=15 lat (msec) : 20=0.04%, 50=54.22%, 100=45.43%, 250=0.31% cpu : usr=0.32%, sys=4.16%, ctx=2741, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184528: Wed Jul 8 23:01:11 2026 write: IOPS=315, BW=1264KiB/s (1294kB/s)(10.0MiB/8104msec); 0 zone resets slat (usec): min=38, max=3232, avg=183.64, stdev=142.28 clat (msec): min=5, max=107, avg=50.06, stdev=12.70 lat (msec): min=5, max=108, avg=50.25, stdev=12.71 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 34], 10.00th=[ 35], 20.00th=[ 38], | 30.00th=[ 42], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 62], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 88], 99.50th=[ 101], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 872, max= 1656, per=25.41%, avg=1281.67, stdev=207.42, samples=15 iops : min= 218, max= 414, avg=320.33, stdev=51.92, samples=15 lat (msec) : 10=0.08%, 20=0.04%, 50=54.92%, 100=44.61%, 250=0.35% cpu : usr=0.39%, sys=4.57%, ctx=2667, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184529: Wed Jul 8 23:01:11 2026 write: IOPS=315, BW=1262KiB/s (1292kB/s)(10.0MiB/8113msec); 0 zone resets slat (usec): min=35, max=2802, avg=174.51, stdev=160.21 clat (msec): min=8, max=109, avg=50.07, stdev=12.80 lat (msec): min=8, max=110, avg=50.25, stdev=12.82 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 34], 10.00th=[ 36], 20.00th=[ 38], | 30.00th=[ 42], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 62], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 87], 99.50th=[ 102], 99.90th=[ 108], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 888, max= 1664, per=25.39%, avg=1280.67, stdev=201.16, samples=15 iops : min= 222, max= 416, avg=320.07, stdev=50.36, samples=15 lat (msec) : 10=0.04%, 50=55.43%, 100=43.83%, 250=0.70% cpu : usr=0.43%, sys=3.97%, ctx=2630, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184530: Wed Jul 8 23:01:11 2026 write: IOPS=319, BW=1276KiB/s (1307kB/s)(10.0MiB/8023msec); 0 zone resets slat (usec): min=53, max=2794, avg=191.13, stdev=133.80 clat (msec): min=5, max=118, avg=49.86, stdev=13.32 lat (msec): min=6, max=119, avg=50.05, stdev=13.32 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 35], 20.00th=[ 38], | 30.00th=[ 42], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 52], | 70.00th=[ 56], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 93], 99.50th=[ 105], 99.90th=[ 111], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 848, max= 1672, per=25.63%, avg=1292.33, stdev=208.77, samples=15 iops : min= 212, max= 418, avg=323.00, stdev=52.26, samples=15 lat (msec) : 10=0.27%, 20=0.08%, 50=55.55%, 100=43.28%, 250=0.82% cpu : usr=0.37%, sys=4.62%, ctx=2686, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=5044KiB/s (5165kB/s), 1261KiB/s-1276KiB/s (1291kB/s-1307kB/s), io=40.0MiB (41.9MB), run=8023-8121msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=184541: Wed Jul 8 23:01:23 2026 read: IOPS=143, BW=574KiB/s (587kB/s)(5048KiB/8802msec) slat (usec): min=53, max=15190, avg=3645.67, stdev=1600.52 clat (usec): min=16822, max=96731, avg=51995.81, stdev=11011.32 lat (msec): min=17, max=103, avg=55.64, stdev=11.56 clat percentiles (usec): | 1.00th=[28705], 5.00th=[35914], 10.00th=[39060], 20.00th=[42730], | 30.00th=[45351], 40.00th=[47973], 50.00th=[51119], 60.00th=[54264], | 70.00th=[57934], 80.00th=[61604], 90.00th=[66323], 95.00th=[70779], | 99.00th=[78119], 99.50th=[80217], 99.90th=[94897], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 472, max= 772, per=25.00%, avg=568.47, stdev=85.03, samples=17 iops : min= 118, max= 193, avg=142.00, stdev=21.26, samples=17 write: IOPS=147, BW=590KiB/s (604kB/s)(5192KiB/8802msec); 0 zone resets slat (usec): min=43, max=6184, avg=185.36, stdev=281.96 clat (usec): min=13930, max=88801, avg=53680.65, stdev=10688.65 lat (usec): min=14416, max=88900, avg=53867.09, stdev=10694.14 clat percentiles (usec): | 1.00th=[32900], 5.00th=[38011], 10.00th=[40633], 20.00th=[44303], | 30.00th=[47449], 40.00th=[50070], 50.00th=[52691], 60.00th=[55837], | 70.00th=[58983], 80.00th=[63177], 90.00th=[67634], 95.00th=[71828], | 99.00th=[80217], 99.50th=[82314], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 424, max= 696, per=24.75%, avg=585.76, stdev=82.67, samples=17 iops : min= 106, max= 174, avg=146.29, stdev=20.74, samples=17 lat (msec) : 20=0.20%, 50=42.58%, 100=57.23% cpu : usr=0.52%, sys=4.41%, ctx=4554, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184542: Wed Jul 8 23:01:23 2026 read: IOPS=141, BW=564KiB/s (578kB/s)(4976KiB/8817msec) slat (usec): min=50, max=11646, avg=3673.67, stdev=1568.20 clat (usec): min=28193, max=96159, avg=52341.48, stdev=10804.32 lat (usec): min=30463, max=98660, avg=56018.39, stdev=11225.10 clat percentiles (usec): | 1.00th=[31589], 5.00th=[36439], 10.00th=[39060], 20.00th=[42730], | 30.00th=[45351], 40.00th=[48497], 50.00th=[51119], 60.00th=[54789], | 70.00th=[57410], 80.00th=[61080], 90.00th=[66323], 95.00th=[71828], | 99.00th=[80217], 99.50th=[82314], 99.90th=[89654], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 392, max= 752, per=24.57%, avg=558.76, stdev=92.03, samples=17 iops : min= 98, max= 188, avg=139.53, stdev=23.05, samples=17 write: IOPS=149, BW=597KiB/s (611kB/s)(5264KiB/8817msec); 0 zone resets slat (usec): min=42, max=4773, avg=195.50, stdev=221.21 clat (usec): min=20193, max=92345, avg=53482.26, stdev=10389.56 lat (usec): min=20333, max=92554, avg=53678.91, stdev=10411.25 clat percentiles (usec): | 1.00th=[33817], 5.00th=[38536], 10.00th=[40633], 20.00th=[44303], | 30.00th=[46924], 40.00th=[49546], 50.00th=[52691], 60.00th=[55313], | 70.00th=[58459], 80.00th=[62653], 90.00th=[67634], 95.00th=[71828], | 99.00th=[78119], 99.50th=[81265], 99.90th=[88605], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 432, max= 704, per=25.08%, avg=593.53, stdev=78.54, samples=17 iops : min= 108, max= 176, avg=148.24, stdev=19.67, samples=17 lat (msec) : 50=43.28%, 100=56.72% cpu : usr=0.49%, sys=4.56%, ctx=4475, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184543: Wed Jul 8 23:01:23 2026 read: IOPS=141, BW=567KiB/s (581kB/s)(4984KiB/8783msec) slat (usec): min=51, max=13098, avg=3697.12, stdev=1622.50 clat (usec): min=3543, max=98132, avg=52622.49, stdev=11053.02 lat (msec): min=7, max=103, avg=56.32, stdev=11.53 clat percentiles (usec): | 1.00th=[30278], 5.00th=[36439], 10.00th=[40109], 20.00th=[43254], | 30.00th=[46924], 40.00th=[49021], 50.00th=[51643], 60.00th=[54789], | 70.00th=[57410], 80.00th=[61604], 90.00th=[66847], 95.00th=[71828], | 99.00th=[80217], 99.50th=[84411], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 424, max= 717, per=24.92%, avg=566.59, stdev=73.73, samples=17 iops : min= 106, max= 179, avg=141.53, stdev=18.39, samples=17 write: IOPS=149, BW=598KiB/s (613kB/s)(5256KiB/8783msec); 0 zone resets slat (usec): min=49, max=3800, avg=189.16, stdev=196.04 clat (usec): min=18334, max=92062, avg=53126.62, stdev=10653.40 lat (usec): min=18997, max=92159, avg=53317.00, stdev=10660.65 clat percentiles (usec): | 1.00th=[30540], 5.00th=[37487], 10.00th=[40109], 20.00th=[44303], | 30.00th=[47449], 40.00th=[49546], 50.00th=[52167], 60.00th=[54789], | 70.00th=[57934], 80.00th=[62129], 90.00th=[67634], 95.00th=[72877], | 99.00th=[79168], 99.50th=[82314], 99.90th=[87557], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 432, max= 840, per=25.10%, avg=594.18, stdev=113.87, samples=17 iops : min= 108, max= 210, avg=148.41, stdev=28.52, samples=17 lat (msec) : 4=0.04%, 10=0.08%, 20=0.16%, 50=42.50%, 100=57.23% cpu : usr=0.32%, sys=4.74%, ctx=4459, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184544: Wed Jul 8 23:01:23 2026 read: IOPS=143, BW=574KiB/s (587kB/s)(5060KiB/8823msec) slat (usec): min=1312, max=13484, avg=3657.51, stdev=1600.45 clat (msec): min=2, max=100, avg=52.13, stdev=11.39 lat (msec): min=7, max=103, avg=55.79, stdev=11.80 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 43], | 30.00th=[ 46], 40.00th=[ 49], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 62], 90.00th=[ 67], 95.00th=[ 71], | 99.00th=[ 81], 99.50th=[ 83], 99.90th=[ 93], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 384, max= 720, per=24.93%, avg=566.94, stdev=85.83, samples=17 iops : min= 96, max= 180, avg=141.59, stdev=21.48, samples=17 write: IOPS=146, BW=587KiB/s (601kB/s)(5180KiB/8823msec); 0 zone resets slat (usec): min=59, max=2792, avg=183.24, stdev=175.16 clat (usec): min=7295, max=88283, avg=53456.91, stdev=11585.85 lat (usec): min=7453, max=88824, avg=53641.26, stdev=11596.81 clat percentiles (usec): | 1.00th=[30278], 5.00th=[36439], 10.00th=[40109], 20.00th=[44303], | 30.00th=[46924], 40.00th=[49546], 50.00th=[52167], 60.00th=[55313], | 70.00th=[58983], 80.00th=[63701], 90.00th=[68682], 95.00th=[72877], | 99.00th=[82314], 99.50th=[85459], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 456, max= 741, per=24.43%, avg=578.18, stdev=80.23, samples=17 iops : min= 114, max= 185, avg=144.41, stdev=20.09, samples=17 lat (msec) : 4=0.04%, 10=0.23%, 20=0.27%, 50=42.85%, 100=56.56% lat (msec) : 250=0.04% cpu : usr=0.40%, sys=4.58%, ctx=4514, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2275KiB/s (2329kB/s), 564KiB/s-574KiB/s (578kB/s-587kB/s), io=19.6MiB (20.5MB), run=8783-8823msec WRITE: bw=2368KiB/s (2425kB/s), 587KiB/s-598KiB/s (601kB/s-613kB/s), io=20.4MiB (21.4MB), run=8783-8823msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=184549: Wed Jul 8 23:01:27 2026 read: IOPS=7, BW=308MiB/s (323MB/s)(40.0MiB/130msec) slat (nsec): min=13050k, max=13050k, avg=13049668.00, stdev= 0.00 clat (nsec): min=109762k, max=109762k, avg=109762368.00, stdev= 0.00 lat (nsec): min=122816k, max=122816k, avg=122816266.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.53%, ctx=11, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=308MiB/s (323MB/s), 308MiB/s-308MiB/s (323MB/s-323MB/s), io=40.0MiB (41.9MB), run=130-130msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout