== sanity test 398c: run fio to test AIO ================= 01:43:37 (1787377417) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.569144 s, 73.7 MB/s osc.lustre-OST0000-osc-ffff8b6f603c3800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145684: Sat Aug 22 01:46:10 2026 write: IOPS=17, BW=71.6KiB/s (73.3kB/s)(10.0MiB/143004msec); 0 zone resets slat (usec): min=21, max=18757, avg=172.92, stdev=736.68 clat (msec): min=118, max=1877, avg=893.15, stdev=204.96 lat (msec): min=118, max=1877, avg=893.32, stdev=204.98 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 584], 10.00th=[ 634], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 894], 60.00th=[ 936], | 70.00th=[ 986], 80.00th=[ 1036], 90.00th=[ 1133], 95.00th=[ 1200], | 99.00th=[ 1435], 99.50th=[ 1653], 99.90th=[ 1821], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 15, max= 128, per=25.02%, avg=71.31, stdev=29.54, samples=283 iops : min= 3, max= 32, avg=17.39, stdev= 7.39, samples=283 lat (msec) : 250=0.62%, 500=1.88%, 750=18.75%, 1000=51.17%, 2000=27.58% cpu : usr=0.05%, sys=0.21%, ctx=1959, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145685: Sat Aug 22 01:46:10 2026 write: IOPS=17, BW=71.6KiB/s (73.3kB/s)(10.0MiB/143111msec); 0 zone resets slat (usec): min=20, max=10138, avg=135.49, stdev=401.94 clat (msec): min=347, max=1692, avg=893.44, stdev=194.70 lat (msec): min=348, max=1692, avg=893.58, stdev=194.68 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 575], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 902], 60.00th=[ 944], | 70.00th=[ 986], 80.00th=[ 1045], 90.00th=[ 1116], 95.00th=[ 1200], | 99.00th=[ 1368], 99.50th=[ 1603], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1687] bw ( KiB/s): min= 8, max= 231, per=25.20%, avg=71.82, stdev=29.70, samples=281 iops : min= 2, max= 57, avg=17.52, stdev= 7.42, samples=281 lat (msec) : 500=3.36%, 750=17.07%, 1000=52.03%, 2000=27.54% cpu : usr=0.05%, sys=0.20%, ctx=1450, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145686: Sat Aug 22 01:46:10 2026 write: IOPS=17, BW=71.4KiB/s (73.1kB/s)(10.0MiB/143366msec); 0 zone resets slat (usec): min=21, max=14092, avg=178.85, stdev=745.00 clat (msec): min=95, max=1694, avg=895.08, stdev=192.07 lat (msec): min=95, max=1694, avg=895.26, stdev=192.09 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 592], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 902], 60.00th=[ 944], | 70.00th=[ 986], 80.00th=[ 1036], 90.00th=[ 1133], 95.00th=[ 1217], | 99.00th=[ 1385], 99.50th=[ 1536], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 128, per=25.34%, avg=72.23, stdev=31.47, samples=279 iops : min= 1, max= 32, avg=17.62, stdev= 7.84, samples=279 lat (msec) : 100=0.04%, 250=0.08%, 500=1.52%, 750=19.38%, 1000=51.21% lat (msec) : 2000=27.77% cpu : usr=0.05%, sys=0.24%, ctx=1783, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145687: Sat Aug 22 01:46:10 2026 write: IOPS=17, BW=71.5KiB/s (73.2kB/s)(10.0MiB/143285msec); 0 zone resets slat (usec): min=21, max=23408, avg=184.89, stdev=811.21 clat (msec): min=75, max=1741, avg=894.49, stdev=194.81 lat (msec): min=76, max=1741, avg=894.68, stdev=194.80 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 567], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 894], 60.00th=[ 936], | 70.00th=[ 986], 80.00th=[ 1053], 90.00th=[ 1116], 95.00th=[ 1200], | 99.00th=[ 1435], 99.50th=[ 1620], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 160, per=25.12%, avg=71.59, stdev=32.30, samples=282 iops : min= 1, max= 40, avg=17.45, stdev= 8.10, samples=282 lat (msec) : 100=0.04%, 250=0.04%, 500=2.03%, 750=18.40%, 1000=51.99% lat (msec) : 2000=27.50% cpu : usr=0.05%, sys=0.23%, ctx=2107, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=286KiB/s (293kB/s), 71.4KiB/s-71.6KiB/s (73.1kB/s-73.3kB/s), io=40.0MiB (41.9MB), run=143004-143366msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145718: Sat Aug 22 01:47:55 2026 read: IOPS=13, BW=52.4KiB/s (53.7kB/s)(5048KiB/96288msec) slat (usec): min=36, max=49332, avg=13679.44, stdev=7570.12 clat (msec): min=16, max=1088, avg=386.83, stdev=109.99 lat (msec): min=31, max=1102, avg=400.52, stdev=111.45 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 236], 10.00th=[ 262], 20.00th=[ 305], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 376], 60.00th=[ 405], | 70.00th=[ 430], 80.00th=[ 460], 90.00th=[ 514], 95.00th=[ 567], | 99.00th=[ 709], 99.50th=[ 827], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 128, per=25.32%, avg=52.42, stdev=21.85, samples=189 iops : min= 1, max= 32, avg=12.63, stdev= 5.50, samples=189 write: IOPS=13, BW=53.9KiB/s (55.2kB/s)(5192KiB/96288msec); 0 zone resets slat (usec): min=29, max=17222, avg=237.86, stdev=812.66 clat (msec): min=124, max=1544, avg=796.33, stdev=204.17 lat (msec): min=124, max=1545, avg=796.57, stdev=204.21 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 481], 10.00th=[ 550], 20.00th=[ 609], | 30.00th=[ 684], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 852], | 70.00th=[ 902], 80.00th=[ 961], 90.00th=[ 1053], 95.00th=[ 1133], | 99.00th=[ 1301], 99.50th=[ 1435], 99.90th=[ 1519], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 120, per=24.79%, avg=53.54, stdev=20.83, samples=190 iops : min= 1, max= 30, avg=12.91, stdev= 5.24, samples=190 lat (msec) : 20=0.04%, 50=0.23%, 100=0.08%, 250=3.48%, 500=42.42% lat (msec) : 750=24.14%, 1000=21.56%, 2000=8.05% cpu : usr=0.12%, sys=0.63%, ctx=4815, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145719: Sat Aug 22 01:47:55 2026 read: IOPS=12, BW=51.5KiB/s (52.7kB/s)(4976KiB/96659msec) slat (usec): min=2679, max=46850, avg=13393.07, stdev=7094.34 clat (msec): min=27, max=831, avg=385.98, stdev=98.82 lat (msec): min=32, max=840, avg=399.38, stdev=99.41 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 245], 10.00th=[ 275], 20.00th=[ 305], | 30.00th=[ 334], 40.00th=[ 363], 50.00th=[ 380], 60.00th=[ 401], | 70.00th=[ 426], 80.00th=[ 456], 90.00th=[ 510], 95.00th=[ 550], | 99.00th=[ 693], 99.50th=[ 709], 99.90th=[ 810], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7, max= 136, per=24.65%, avg=51.03, stdev=21.45, samples=191 iops : min= 1, max= 34, avg=12.28, stdev= 5.37, samples=191 write: IOPS=13, BW=54.5KiB/s (55.8kB/s)(5264KiB/96659msec); 0 zone resets slat (usec): min=23, max=15462, avg=280.56, stdev=978.90 clat (msec): min=81, max=1489, avg=795.40, stdev=200.55 lat (msec): min=84, max=1489, avg=795.68, stdev=200.55 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 485], 10.00th=[ 550], 20.00th=[ 617], | 30.00th=[ 684], 40.00th=[ 735], 50.00th=[ 802], 60.00th=[ 852], | 70.00th=[ 902], 80.00th=[ 969], 90.00th=[ 1045], 95.00th=[ 1099], | 99.00th=[ 1250], 99.50th=[ 1435], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 7, max= 112, per=25.06%, avg=54.13, stdev=21.35, samples=191 iops : min= 1, max= 28, avg=13.05, stdev= 5.34, samples=191 lat (msec) : 50=0.12%, 100=0.23%, 250=2.97%, 500=42.50%, 750=24.61% lat (msec) : 1000=21.76%, 2000=7.81% cpu : usr=0.10%, sys=0.66%, ctx=4804, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145720: Sat Aug 22 01:47:55 2026 read: IOPS=12, BW=51.6KiB/s (52.8kB/s)(4984KiB/96654msec) slat (usec): min=2355, max=56110, avg=13722.49, stdev=7507.70 clat (msec): min=24, max=824, avg=388.78, stdev=99.81 lat (msec): min=32, max=848, avg=402.51, stdev=100.88 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 249], 10.00th=[ 279], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 380], 60.00th=[ 401], | 70.00th=[ 426], 80.00th=[ 460], 90.00th=[ 514], 95.00th=[ 567], | 99.00th=[ 693], 99.50th=[ 726], 99.90th=[ 818], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 136, per=24.80%, avg=51.35, stdev=22.17, samples=191 iops : min= 1, max= 34, avg=12.36, stdev= 5.52, samples=191 write: IOPS=13, BW=54.4KiB/s (55.7kB/s)(5256KiB/96654msec); 0 zone resets slat (usec): min=23, max=15425, avg=261.44, stdev=881.30 clat (msec): min=173, max=1500, avg=792.58, stdev=192.33 lat (msec): min=174, max=1500, avg=792.84, stdev=192.33 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 481], 10.00th=[ 558], 20.00th=[ 625], | 30.00th=[ 676], 40.00th=[ 735], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 894], 80.00th=[ 953], 90.00th=[ 1036], 95.00th=[ 1083], | 99.00th=[ 1250], 99.50th=[ 1401], 99.90th=[ 1452], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 15, max= 112, per=24.81%, avg=53.58, stdev=19.52, samples=192 iops : min= 3, max= 28, avg=12.92, stdev= 4.92, samples=192 lat (msec) : 50=0.04%, 100=0.27%, 250=2.42%, 500=42.97%, 750=24.02% lat (msec) : 1000=23.28%, 2000=6.99% cpu : usr=0.12%, sys=0.65%, ctx=4713, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145721: Sat Aug 22 01:47:55 2026 read: IOPS=13, BW=52.4KiB/s (53.7kB/s)(5060KiB/96529msec) slat (usec): min=2361, max=53173, avg=13762.19, stdev=7406.50 clat (msec): min=4, max=816, avg=390.94, stdev=100.63 lat (msec): min=9, max=831, avg=404.70, stdev=101.87 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 247], 10.00th=[ 275], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 430], 80.00th=[ 468], 90.00th=[ 518], 95.00th=[ 567], | 99.00th=[ 659], 99.50th=[ 693], 99.90th=[ 760], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 7, max= 112, per=25.32%, avg=52.42, stdev=19.76, samples=190 iops : min= 1, max= 28, avg=12.63, stdev= 5.03, samples=190 write: IOPS=13, BW=53.7KiB/s (54.9kB/s)(5180KiB/96529msec); 0 zone resets slat (usec): min=35, max=12462, avg=235.76, stdev=738.93 clat (msec): min=324, max=1522, avg=794.41, stdev=193.45 lat (msec): min=324, max=1522, avg=794.65, stdev=193.42 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 502], 10.00th=[ 550], 20.00th=[ 617], | 30.00th=[ 676], 40.00th=[ 726], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 894], 80.00th=[ 969], 90.00th=[ 1045], 95.00th=[ 1099], | 99.00th=[ 1250], 99.50th=[ 1351], 99.90th=[ 1485], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 15, max= 104, per=24.57%, avg=53.07, stdev=20.10, samples=191 iops : min= 3, max= 26, avg=12.79, stdev= 5.08, samples=191 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.12%, 250=2.34% lat (msec) : 500=42.27%, 750=26.17%, 1000=21.17%, 2000=7.73% cpu : usr=0.11%, sys=0.72%, ctx=5072, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=208KiB/s (213kB/s), 51.5KiB/s-52.4KiB/s (52.7kB/s-53.7kB/s), io=19.6MiB (20.5MB), run=96288-96659msec WRITE: bw=216KiB/s (221kB/s), 53.7KiB/s-54.5KiB/s (54.9kB/s-55.8kB/s), io=20.4MiB (21.4MB), run=96288-96659msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145741: Sat Aug 22 01:48:06 2026 read: IOPS=2, BW=83.7MiB/s (87.7MB/s)(40.0MiB/478msec) slat (nsec): min=36049k, max=36049k, avg=36048999.00, stdev= 0.00 clat (nsec): min=427227k, max=427227k, avg=427227489.00, stdev= 0.00 lat (nsec): min=463282k, max=463282k, avg=463281864.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.73%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.7MiB/s (87.7MB/s), 83.7MiB/s-83.7MiB/s (87.7MB/s-87.7MB/s), io=40.0MiB (41.9MB), run=478-478msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout