== sanity test 398c: run fio to test AIO ================= 11:57:19 (1776182239) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.423021 s, 99.2 MB/s osc.lustre-OST0000-osc-ffff965505c5c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=125235: Tue Apr 14 11:59:49 2026 write: IOPS=17, BW=71.6KiB/s (73.3kB/s)(10.0MiB/142955msec); 0 zone resets slat (usec): min=25, max=17101, avg=289.94, stdev=956.69 clat (msec): min=205, max=2025, avg=892.52, stdev=279.86 lat (msec): min=206, max=2026, avg=892.81, stdev=279.91 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 510], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 860], 60.00th=[ 927], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1234], 95.00th=[ 1368], | 99.00th=[ 1703], 99.50th=[ 1871], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 191, per=24.93%, avg=71.29, stdev=32.42, samples=282 iops : min= 1, max= 47, avg=17.33, stdev= 8.16, samples=282 lat (msec) : 250=0.78%, 500=3.71%, 750=29.34%, 1000=34.34%, 2000=31.80% lat (msec) : >=2000=0.04% cpu : usr=0.06%, sys=0.30%, ctx=1661, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125236: Tue Apr 14 11:59:49 2026 write: IOPS=17, BW=71.7KiB/s (73.4kB/s)(10.0MiB/142834msec); 0 zone resets slat (usec): min=19, max=17078, avg=287.07, stdev=918.56 clat (msec): min=232, max=2023, avg=891.65, stdev=281.28 lat (msec): min=232, max=2024, avg=891.94, stdev=281.29 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 514], 10.00th=[ 567], 20.00th=[ 651], | 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 860], 60.00th=[ 936], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1653], 99.50th=[ 1821], 99.90th=[ 1989], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 219, per=25.12%, avg=71.85, stdev=31.94, samples=280 iops : min= 1, max= 54, avg=17.47, stdev= 8.02, samples=280 lat (msec) : 250=1.25%, 500=3.12%, 750=28.55%, 1000=35.66%, 2000=31.33% lat (msec) : >=2000=0.08% cpu : usr=0.05%, sys=0.30%, ctx=1654, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125237: Tue Apr 14 11:59:49 2026 write: IOPS=17, BW=71.8KiB/s (73.5kB/s)(10.0MiB/142598msec); 0 zone resets slat (usec): min=21, max=16795, avg=289.95, stdev=928.91 clat (msec): min=122, max=2025, avg=890.46, stdev=283.00 lat (msec): min=123, max=2025, avg=890.75, stdev=283.07 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 502], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1670], 99.50th=[ 1804], 99.90th=[ 1972], 99.95th=[ 2005], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 254, per=25.58%, avg=73.17, stdev=34.51, samples=275 iops : min= 1, max= 63, avg=17.80, stdev= 8.66, samples=275 lat (msec) : 250=1.25%, 500=3.71%, 750=28.63%, 1000=33.59%, 2000=32.77% lat (msec) : >=2000=0.04% cpu : usr=0.05%, sys=0.31%, ctx=1677, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125238: Tue Apr 14 11:59:49 2026 write: IOPS=17, BW=71.7KiB/s (73.4kB/s)(10.0MiB/142869msec); 0 zone resets slat (usec): min=25, max=56443, avg=335.55, stdev=1523.71 clat (msec): min=129, max=2005, avg=891.92, stdev=276.75 lat (msec): min=129, max=2005, avg=892.26, stdev=276.78 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 502], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1028], 80.00th=[ 1116], 90.00th=[ 1234], 95.00th=[ 1351], | 99.00th=[ 1670], 99.50th=[ 1871], 99.90th=[ 1989], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 218, per=25.49%, avg=72.89, stdev=31.46, samples=276 iops : min= 1, max= 54, avg=17.73, stdev= 7.89, samples=276 lat (msec) : 250=1.25%, 500=3.48%, 750=28.09%, 1000=34.69%, 2000=32.42% lat (msec) : >=2000=0.08% cpu : usr=0.07%, sys=0.33%, ctx=1832, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=287KiB/s (293kB/s), 71.6KiB/s-71.8KiB/s (73.3kB/s-73.5kB/s), io=40.0MiB (41.9MB), run=142598-142955msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=125269: Tue Apr 14 12:01:35 2026 read: IOPS=12, BW=51.1KiB/s (52.3kB/s)(5048KiB/98760msec) slat (usec): min=2172, max=50920, avg=11821.87, stdev=7017.31 clat (msec): min=5, max=876, avg=399.40, stdev=111.73 lat (msec): min=14, max=901, avg=411.24, stdev=112.67 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 243], 10.00th=[ 271], 20.00th=[ 317], | 30.00th=[ 334], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 418], | 70.00th=[ 451], 80.00th=[ 481], 90.00th=[ 542], 95.00th=[ 617], | 99.00th=[ 693], 99.50th=[ 726], 99.90th=[ 768], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 119, per=24.89%, avg=50.53, stdev=21.79, samples=196 iops : min= 1, max= 29, avg=12.14, stdev= 5.49, samples=196 write: IOPS=13, BW=52.6KiB/s (53.8kB/s)(5192KiB/98760msec); 0 zone resets slat (usec): min=26, max=14640, avg=428.53, stdev=1280.54 clat (msec): min=246, max=1689, avg=814.98, stdev=230.79 lat (msec): min=256, max=1689, avg=815.42, stdev=230.79 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 481], 10.00th=[ 542], 20.00th=[ 625], | 30.00th=[ 684], 40.00th=[ 743], 50.00th=[ 802], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 986], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1435], 99.50th=[ 1469], 99.90th=[ 1653], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 118, per=24.72%, avg=52.15, stdev=21.28, samples=195 iops : min= 1, max= 29, avg=12.55, stdev= 5.37, samples=195 lat (msec) : 10=0.04%, 50=0.12%, 100=0.27%, 250=2.73%, 500=42.03% lat (msec) : 750=24.69%, 1000=20.66%, 2000=9.45% cpu : usr=0.12%, sys=0.88%, ctx=3250, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125270: Tue Apr 14 12:01:36 2026 read: IOPS=12, BW=50.5KiB/s (51.7kB/s)(4976KiB/98504msec) slat (usec): min=2351, max=60195, avg=11826.21, stdev=7191.83 clat (msec): min=10, max=950, avg=396.70, stdev=113.35 lat (msec): min=14, max=957, avg=408.54, stdev=114.43 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 236], 10.00th=[ 271], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 447], 80.00th=[ 481], 90.00th=[ 550], 95.00th=[ 600], | 99.00th=[ 709], 99.50th=[ 776], 99.90th=[ 802], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 136, per=24.63%, avg=49.99, stdev=21.66, samples=195 iops : min= 1, max= 34, avg=12.01, stdev= 5.47, samples=195 write: IOPS=13, BW=53.4KiB/s (54.7kB/s)(5264KiB/98504msec); 0 zone resets slat (usec): min=28, max=15382, avg=392.72, stdev=1144.42 clat (msec): min=248, max=1588, avg=809.79, stdev=227.57 lat (msec): min=248, max=1588, avg=810.19, stdev=227.55 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 460], 10.00th=[ 527], 20.00th=[ 617], | 30.00th=[ 676], 40.00th=[ 751], 50.00th=[ 810], 60.00th=[ 860], | 70.00th=[ 911], 80.00th=[ 995], 90.00th=[ 1099], 95.00th=[ 1217], | 99.00th=[ 1401], 99.50th=[ 1469], 99.90th=[ 1485], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 15, max= 110, per=25.10%, avg=52.96, stdev=19.28, samples=195 iops : min= 3, max= 27, avg=12.75, stdev= 4.88, samples=195 lat (msec) : 20=0.04%, 50=0.08%, 100=0.27%, 250=2.77%, 500=41.68% lat (msec) : 750=24.02%, 1000=21.45%, 2000=9.69% cpu : usr=0.15%, sys=0.85%, ctx=3308, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125271: Tue Apr 14 12:01:36 2026 read: IOPS=12, BW=50.5KiB/s (51.7kB/s)(4984KiB/98699msec) slat (usec): min=2150, max=57932, avg=12347.74, stdev=7581.70 clat (msec): min=27, max=870, avg=398.46, stdev=110.26 lat (msec): min=40, max=881, avg=410.82, stdev=111.53 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 241], 10.00th=[ 271], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 418], | 70.00th=[ 451], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 609], | 99.00th=[ 684], 99.50th=[ 735], 99.90th=[ 802], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 7, max= 119, per=24.75%, avg=50.24, stdev=22.90, samples=195 iops : min= 1, max= 29, avg=12.07, stdev= 5.76, samples=195 write: IOPS=13, BW=53.3KiB/s (54.5kB/s)(5256KiB/98699msec); 0 zone resets slat (usec): min=28, max=15268, avg=361.07, stdev=1089.51 clat (msec): min=154, max=1590, avg=810.28, stdev=229.82 lat (msec): min=155, max=1590, avg=810.64, stdev=229.86 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 468], 10.00th=[ 527], 20.00th=[ 617], | 30.00th=[ 676], 40.00th=[ 743], 50.00th=[ 802], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 978], 90.00th=[ 1116], 95.00th=[ 1234], | 99.00th=[ 1418], 99.50th=[ 1469], 99.90th=[ 1569], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 110, per=25.12%, avg=53.01, stdev=20.48, samples=194 iops : min= 1, max= 27, avg=12.76, stdev= 5.15, samples=194 lat (msec) : 50=0.12%, 100=0.20%, 250=3.05%, 500=42.23%, 750=23.95% lat (msec) : 1000=21.29%, 2000=9.18% cpu : usr=0.12%, sys=0.83%, ctx=3141, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125272: Tue Apr 14 12:01:36 2026 read: IOPS=12, BW=51.8KiB/s (53.1kB/s)(5060KiB/97629msec) slat (usec): min=65, max=51548, avg=11636.60, stdev=7124.24 clat (msec): min=14, max=1217, avg=389.73, stdev=118.41 lat (msec): min=21, max=1227, avg=401.37, stdev=119.54 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 232], 10.00th=[ 262], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 355], 50.00th=[ 380], 60.00th=[ 401], | 70.00th=[ 430], 80.00th=[ 472], 90.00th=[ 535], 95.00th=[ 600], | 99.00th=[ 735], 99.50th=[ 776], 99.90th=[ 1183], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 135, per=25.36%, avg=51.48, stdev=23.82, samples=193 iops : min= 1, max= 33, avg=12.37, stdev= 6.01, samples=193 write: IOPS=13, BW=53.1KiB/s (54.3kB/s)(5180KiB/97629msec); 0 zone resets slat (usec): min=25, max=11687, avg=402.78, stdev=1098.14 clat (msec): min=116, max=1716, avg=812.78, stdev=244.55 lat (msec): min=116, max=1716, avg=813.18, stdev=244.61 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 460], 10.00th=[ 510], 20.00th=[ 609], | 30.00th=[ 684], 40.00th=[ 743], 50.00th=[ 802], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 1003], 90.00th=[ 1150], 95.00th=[ 1267], | 99.00th=[ 1469], 99.50th=[ 1552], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 111, per=24.89%, avg=52.52, stdev=23.12, samples=193 iops : min= 1, max= 27, avg=12.63, stdev= 5.81, samples=193 lat (msec) : 20=0.08%, 50=0.27%, 100=0.20%, 250=3.55%, 500=42.93% lat (msec) : 750=22.62%, 1000=20.23%, 2000=10.12% cpu : usr=0.13%, sys=0.83%, ctx=3177, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=203KiB/s (208kB/s), 50.5KiB/s-51.8KiB/s (51.7kB/s-53.1kB/s), io=19.6MiB (20.5MB), run=97629-98760msec WRITE: bw=212KiB/s (217kB/s), 52.6KiB/s-53.4KiB/s (53.8kB/s-54.7kB/s), io=20.4MiB (21.4MB), run=97629-98760msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=125303: Tue Apr 14 12:01:45 2026 read: IOPS=2, BW=82.8MiB/s (86.8MB/s)(40.0MiB/483msec) slat (nsec): min=31038k, max=31038k, avg=31037662.00, stdev= 0.00 clat (nsec): min=437720k, max=437720k, avg=437720250.00, stdev= 0.00 lat (nsec): min=468764k, max=468764k, avg=468763793.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 439], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.53%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=82.8MiB/s (86.8MB/s), 82.8MiB/s-82.8MiB/s (86.8MB/s-86.8MB/s), io=40.0MiB (41.9MB), run=483-483msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck