== sanity test 398c: run fio to test AIO ================= 15:42:55 (1788205375) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.569951 s, 73.6 MB/s osc.lustre-OST0000-osc-ffff8bfce9774800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146078: Mon Aug 31 15:45:47 2026 write: IOPS=15, BW=63.7KiB/s (65.2kB/s)(10.0MiB/160744msec); 0 zone resets slat (usec): min=49, max=21230, avg=352.44, stdev=976.29 clat (msec): min=391, max=1807, avg=1003.42, stdev=217.76 lat (msec): min=391, max=1807, avg=1003.77, stdev=217.76 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 617], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1053], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1552], 99.50th=[ 1653], 99.90th=[ 1737], 99.95th=[ 1770], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 128, per=25.62%, avg=65.07, stdev=26.09, samples=310 iops : min= 1, max= 32, avg=15.83, stdev= 6.55, samples=310 lat (msec) : 500=0.43%, 750=10.55%, 1000=38.75%, 2000=50.27% cpu : usr=0.06%, sys=0.28%, ctx=2245, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146079: Mon Aug 31 15:45:47 2026 write: IOPS=15, BW=63.8KiB/s (65.3kB/s)(10.0MiB/160598msec); 0 zone resets slat (usec): min=49, max=16777, avg=360.63, stdev=976.08 clat (msec): min=332, max=1986, avg=1002.67, stdev=227.53 lat (msec): min=332, max=1986, avg=1003.03, stdev=227.54 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 625], 10.00th=[ 751], 20.00th=[ 827], | 30.00th=[ 885], 40.00th=[ 936], 50.00th=[ 995], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1385], | 99.00th=[ 1603], 99.50th=[ 1653], 99.90th=[ 1737], 99.95th=[ 1804], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 128, per=25.68%, avg=65.23, stdev=26.77, samples=309 iops : min= 1, max= 32, avg=15.87, stdev= 6.72, samples=309 lat (msec) : 500=0.86%, 750=9.49%, 1000=41.80%, 2000=47.85% cpu : usr=0.05%, sys=0.32%, ctx=2188, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146080: Mon Aug 31 15:45:47 2026 write: IOPS=15, BW=63.9KiB/s (65.4kB/s)(10.0MiB/160294msec); 0 zone resets slat (usec): min=53, max=16990, avg=355.29, stdev=958.43 clat (msec): min=121, max=1777, avg=1001.12, stdev=224.37 lat (msec): min=121, max=1777, avg=1001.48, stdev=224.40 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 625], 10.00th=[ 726], 20.00th=[ 835], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 995], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1167], 90.00th=[ 1284], 95.00th=[ 1368], | 99.00th=[ 1569], 99.50th=[ 1653], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 128, per=25.44%, avg=64.61, stdev=27.93, samples=312 iops : min= 1, max= 32, avg=15.72, stdev= 7.02, samples=312 lat (msec) : 250=0.59%, 500=0.20%, 750=11.05%, 1000=39.49%, 2000=48.67% cpu : usr=0.04%, sys=0.33%, ctx=2418, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146081: Mon Aug 31 15:45:47 2026 write: IOPS=15, BW=63.7KiB/s (65.2kB/s)(10.0MiB/160751msec); 0 zone resets slat (usec): min=48, max=19510, avg=329.81, stdev=931.32 clat (msec): min=332, max=1758, avg=1003.50, stdev=221.94 lat (msec): min=333, max=1758, avg=1003.83, stdev=221.92 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 634], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1053], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1284], 95.00th=[ 1385], | 99.00th=[ 1636], 99.50th=[ 1687], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 128, per=25.20%, avg=64.00, stdev=24.92, samples=315 iops : min= 1, max= 32, avg=15.57, stdev= 6.27, samples=315 lat (msec) : 500=0.82%, 750=10.23%, 1000=38.95%, 2000=50.00% cpu : usr=0.04%, sys=0.31%, ctx=2138, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=255KiB/s (261kB/s), 63.7KiB/s-63.9KiB/s (65.2kB/s-65.4kB/s), io=40.0MiB (41.9MB), run=160294-160751msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146113: Mon Aug 31 15:47:54 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(5048KiB/120849msec) slat (usec): min=2753, max=56266, avg=15361.16, stdev=8214.33 clat (msec): min=21, max=1015, avg=468.07, stdev=120.14 lat (msec): min=29, max=1036, avg=483.44, stdev=121.07 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 300], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 405], 40.00th=[ 430], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 567], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 785], 99.50th=[ 835], 99.90th=[ 995], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 112, per=25.44%, avg=41.98, stdev=19.73, samples=235 iops : min= 1, max= 28, avg=10.02, stdev= 4.97, samples=235 write: IOPS=10, BW=42.0KiB/s (43.0kB/s)(5192KiB/120849msec); 0 zone resets slat (usec): min=55, max=13757, avg=355.48, stdev=883.29 clat (msec): min=356, max=1949, avg=1016.82, stdev=288.05 lat (msec): min=356, max=1949, avg=1017.18, stdev=288.02 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 844], 40.00th=[ 919], 50.00th=[ 1003], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1720], 99.50th=[ 1821], 99.90th=[ 1938], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 6, max= 104, per=24.95%, avg=42.92, stdev=21.16, samples=237 iops : min= 1, max= 26, avg=10.25, stdev= 5.31, samples=237 lat (msec) : 50=0.12%, 100=0.08%, 250=0.47%, 500=31.80%, 750=26.09% lat (msec) : 1000=15.74%, 2000=25.70% cpu : usr=0.10%, sys=0.62%, ctx=4415, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146114: Mon Aug 31 15:47:54 2026 read: IOPS=10, BW=40.0KiB/s (41.0kB/s)(4976KiB/121386msec) slat (usec): min=3378, max=65333, avg=15330.04, stdev=8205.38 clat (msec): min=20, max=999, avg=468.86, stdev=114.29 lat (msec): min=40, max=1021, avg=484.20, stdev=115.28 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 309], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 558], 90.00th=[ 617], 95.00th=[ 676], | 99.00th=[ 785], 99.50th=[ 827], 99.90th=[ 978], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 127, per=25.10%, avg=41.41, stdev=19.35, samples=235 iops : min= 1, max= 31, avg= 9.86, stdev= 4.86, samples=235 write: IOPS=10, BW=43.4KiB/s (44.4kB/s)(5264KiB/121386msec); 0 zone resets slat (usec): min=55, max=13494, avg=366.09, stdev=910.07 clat (msec): min=378, max=1924, avg=1015.67, stdev=296.36 lat (msec): min=378, max=1924, avg=1016.04, stdev=296.33 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 911], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1267], 90.00th=[ 1401], 95.00th=[ 1502], | 99.00th=[ 1770], 99.50th=[ 1854], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 95, per=24.95%, avg=42.91, stdev=19.46, samples=240 iops : min= 1, max= 23, avg=10.25, stdev= 4.90, samples=240 lat (msec) : 50=0.08%, 100=0.16%, 250=0.23%, 500=32.58%, 750=24.69% lat (msec) : 1000=16.17%, 2000=26.09% cpu : usr=0.08%, sys=0.66%, ctx=4406, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146115: Mon Aug 31 15:47:54 2026 read: IOPS=10, BW=41.1KiB/s (42.0kB/s)(4984KiB/121386msec) slat (usec): min=57, max=55886, avg=15313.00, stdev=8080.01 clat (msec): min=24, max=1076, avg=465.97, stdev=121.22 lat (msec): min=24, max=1090, avg=481.30, stdev=122.25 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 300], 10.00th=[ 330], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 430], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 558], 90.00th=[ 609], 95.00th=[ 684], | 99.00th=[ 793], 99.50th=[ 860], 99.90th=[ 1070], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 119, per=25.00%, avg=41.26, stdev=21.59, samples=237 iops : min= 1, max= 29, avg= 9.84, stdev= 5.42, samples=237 write: IOPS=10, BW=43.3KiB/s (44.3kB/s)(5256KiB/121386msec); 0 zone resets slat (usec): min=59, max=14194, avg=383.82, stdev=883.26 clat (msec): min=118, max=1952, avg=1020.42, stdev=291.81 lat (msec): min=120, max=1953, avg=1020.81, stdev=291.79 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 760], | 30.00th=[ 844], 40.00th=[ 927], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 1737], 99.50th=[ 1804], 99.90th=[ 1938], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 111, per=24.91%, avg=42.85, stdev=19.87, samples=239 iops : min= 1, max= 27, avg=10.23, stdev= 5.02, samples=239 lat (msec) : 50=0.23%, 100=0.31%, 250=0.66%, 500=30.94%, 750=25.70% lat (msec) : 1000=15.35%, 2000=26.80% cpu : usr=0.08%, sys=0.61%, ctx=4428, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146116: Mon Aug 31 15:47:54 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(5060KiB/121186msec) slat (usec): min=3328, max=51718, avg=14996.47, stdev=7906.59 clat (msec): min=6, max=1115, avg=472.40, stdev=115.29 lat (msec): min=28, max=1137, avg=487.41, stdev=116.09 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 305], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 414], 40.00th=[ 439], 50.00th=[ 464], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 558], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 785], 99.50th=[ 827], 99.90th=[ 1099], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 112, per=25.26%, avg=41.68, stdev=19.55, samples=238 iops : min= 1, max= 28, avg= 9.95, stdev= 4.94, samples=238 write: IOPS=10, BW=42.7KiB/s (43.8kB/s)(5180KiB/121186msec); 0 zone resets slat (usec): min=56, max=15954, avg=409.75, stdev=1154.10 clat (msec): min=184, max=1938, avg=1018.61, stdev=291.43 lat (msec): min=184, max=1938, avg=1019.02, stdev=291.43 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 567], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 844], 40.00th=[ 927], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1754], 99.50th=[ 1838], 99.90th=[ 1921], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 103, per=24.69%, avg=42.47, stdev=18.51, samples=238 iops : min= 1, max= 25, avg=10.13, stdev= 4.67, samples=238 lat (msec) : 10=0.04%, 100=0.08%, 250=0.51%, 500=32.03%, 750=26.25% lat (msec) : 1000=14.18%, 2000=26.91% cpu : usr=0.09%, sys=0.65%, ctx=4547, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=165KiB/s (169kB/s), 40.0KiB/s-41.8KiB/s (41.0kB/s-42.8kB/s), io=19.6MiB (20.5MB), run=120849-121386msec WRITE: bw=172KiB/s (176kB/s), 42.7KiB/s-43.4KiB/s (43.8kB/s-44.4kB/s), io=20.4MiB (21.4MB), run=120849-121386msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146140: Mon Aug 31 15:48:04 2026 read: IOPS=1, BW=72.9MiB/s (76.4MB/s)(40.0MiB/549msec) slat (nsec): min=36399k, max=36399k, avg=36399259.00, stdev= 0.00 clat (nsec): min=484067k, max=484067k, avg=484067489.00, stdev= 0.00 lat (nsec): min=520471k, max=520471k, avg=520471480.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 485], 10.00th=[ 485], 20.00th=[ 485], | 30.00th=[ 485], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.37%, ctx=28, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=72.9MiB/s (76.4MB/s), 72.9MiB/s-72.9MiB/s (76.4MB/s-76.4MB/s), io=40.0MiB (41.9MB), run=549-549msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout