== sanity test 398c: run fio to test AIO ================= 22:22:22 (1787797342) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.492381 s, 85.2 MB/s osc.lustre-OST0000-osc-ffff9054cd62a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145309: Wed Aug 26 22:25:19 2026 write: IOPS=15, BW=61.2KiB/s (62.6kB/s)(10.0MiB/167382msec); 0 zone resets slat (usec): min=28, max=24106, avg=235.54, stdev=1087.28 clat (msec): min=249, max=2784, avg=1044.84, stdev=315.40 lat (msec): min=252, max=2785, avg=1045.07, stdev=315.39 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 609], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1062], | 70.00th=[ 1150], 80.00th=[ 1250], 90.00th=[ 1418], 95.00th=[ 1620], | 99.00th=[ 2072], 99.50th=[ 2232], 99.90th=[ 2702], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 256, per=25.00%, avg=61.00, stdev=30.01, samples=330 iops : min= 1, max= 64, avg=14.79, stdev= 7.54, samples=330 lat (msec) : 250=0.04%, 500=3.12%, 750=8.16%, 1000=37.62%, 2000=49.53% lat (msec) : >=2000=1.52% cpu : usr=0.05%, sys=0.20%, ctx=1704, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145310: Wed Aug 26 22:25:19 2026 write: IOPS=15, BW=61.2KiB/s (62.6kB/s)(10.0MiB/167419msec); 0 zone resets slat (usec): min=28, max=16062, avg=214.10, stdev=839.24 clat (msec): min=264, max=2433, avg=1045.01, stdev=307.55 lat (msec): min=264, max=2433, avg=1045.23, stdev=307.54 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1418], 95.00th=[ 1603], | 99.00th=[ 2005], 99.50th=[ 2198], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 222, per=25.07%, avg=61.18, stdev=31.20, samples=329 iops : min= 1, max= 55, avg=14.84, stdev= 7.82, samples=329 lat (msec) : 500=3.28%, 750=8.09%, 1000=37.58%, 2000=49.96%, >=2000=1.09% cpu : usr=0.04%, sys=0.19%, ctx=1581, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145311: Wed Aug 26 22:25:19 2026 write: IOPS=15, BW=61.2KiB/s (62.7kB/s)(10.0MiB/167192msec); 0 zone resets slat (usec): min=27, max=15042, avg=236.77, stdev=953.48 clat (msec): min=128, max=3122, avg=1044.13, stdev=322.82 lat (msec): min=128, max=3122, avg=1044.37, stdev=322.85 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 609], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1418], 95.00th=[ 1603], | 99.00th=[ 2039], 99.50th=[ 2232], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 200, per=24.99%, avg=60.97, stdev=25.12, samples=330 iops : min= 1, max= 50, avg=14.78, stdev= 6.33, samples=330 lat (msec) : 250=0.08%, 500=3.44%, 750=8.12%, 1000=37.97%, 2000=49.14% lat (msec) : >=2000=1.25% cpu : usr=0.05%, sys=0.23%, ctx=1909, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145312: Wed Aug 26 22:25:19 2026 write: IOPS=15, BW=61.2KiB/s (62.7kB/s)(10.0MiB/167195msec); 0 zone resets slat (usec): min=27, max=21217, avg=205.00, stdev=855.89 clat (msec): min=250, max=3257, avg=1044.00, stdev=315.93 lat (msec): min=251, max=3257, avg=1044.21, stdev=315.93 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 609], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1418], 95.00th=[ 1620], | 99.00th=[ 2039], 99.50th=[ 2232], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 3272] bw ( KiB/s): min= 7, max= 256, per=25.00%, avg=61.01, stdev=29.40, samples=330 iops : min= 1, max= 64, avg=14.80, stdev= 7.39, samples=330 lat (msec) : 500=3.36%, 750=7.50%, 1000=39.53%, 2000=48.32%, >=2000=1.29% cpu : usr=0.05%, sys=0.20%, ctx=1804, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=245KiB/s (251kB/s), 61.2KiB/s-61.2KiB/s (62.6kB/s-62.7kB/s), io=40.0MiB (41.9MB), run=167192-167419msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145346: Wed Aug 26 22:27:28 2026 read: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5048KiB/120391msec) slat (usec): min=44, max=129040, avg=15869.95, stdev=9374.87 clat (msec): min=22, max=1512, avg=478.25, stdev=157.24 lat (msec): min=43, max=1518, avg=494.12, stdev=159.33 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 300], 10.00th=[ 342], 20.00th=[ 376], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 558], 90.00th=[ 642], 95.00th=[ 726], | 99.00th=[ 1083], 99.50th=[ 1301], 99.90th=[ 1469], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 104, per=25.01%, avg=41.51, stdev=20.95, samples=237 iops : min= 1, max= 26, avg= 9.84, stdev= 5.25, samples=237 write: IOPS=10, BW=43.1KiB/s (44.2kB/s)(5192KiB/120391msec); 0 zone resets slat (usec): min=31, max=13276, avg=276.24, stdev=912.36 clat (msec): min=190, max=2291, avg=1001.65, stdev=327.34 lat (msec): min=191, max=2294, avg=1001.93, stdev=327.43 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 802], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1045], | 70.00th=[ 1150], 80.00th=[ 1250], 90.00th=[ 1435], 95.00th=[ 1603], | 99.00th=[ 2072], 99.50th=[ 2165], 99.90th=[ 2198], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 111, per=24.73%, avg=42.78, stdev=18.82, samples=236 iops : min= 1, max= 27, avg=10.16, stdev= 4.77, samples=236 lat (msec) : 50=0.16%, 100=0.39%, 250=0.59%, 500=32.15%, 750=25.31% lat (msec) : 1000=18.28%, 2000=22.42%, >=2000=0.70% cpu : usr=0.11%, sys=0.56%, ctx=4082, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145347: Wed Aug 26 22:27:28 2026 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(4976KiB/120483msec) slat (usec): min=2974, max=65480, avg=15878.45, stdev=8742.21 clat (msec): min=31, max=1366, avg=476.96, stdev=138.57 lat (msec): min=44, max=1396, avg=492.84, stdev=140.31 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 305], 10.00th=[ 338], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 481], | 70.00th=[ 514], 80.00th=[ 575], 90.00th=[ 651], 95.00th=[ 726], | 99.00th=[ 961], 99.50th=[ 1062], 99.90th=[ 1318], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 7, max= 111, per=24.81%, avg=41.18, stdev=20.82, samples=236 iops : min= 1, max= 27, avg= 9.76, stdev= 5.25, samples=236 write: IOPS=10, BW=43.7KiB/s (44.7kB/s)(5264KiB/120483msec); 0 zone resets slat (usec): min=29, max=12012, avg=255.03, stdev=819.83 clat (msec): min=363, max=2316, avg=994.26, stdev=319.55 lat (msec): min=363, max=2317, avg=994.52, stdev=319.59 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 550], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1401], 95.00th=[ 1569], | 99.00th=[ 2056], 99.50th=[ 2198], 99.90th=[ 2265], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 103, per=25.05%, avg=43.34, stdev=20.49, samples=237 iops : min= 1, max= 25, avg=10.30, stdev= 5.17, samples=237 lat (msec) : 50=0.08%, 100=0.08%, 250=0.31%, 500=32.62%, 750=25.47% lat (msec) : 1000=18.75%, 2000=21.99%, >=2000=0.70% cpu : usr=0.10%, sys=0.60%, ctx=4017, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145348: Wed Aug 26 22:27:28 2026 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(4984KiB/120636msec) slat (msec): min=3, max=461, avg=16.03, stdev=16.91 clat (msec): min=14, max=1322, avg=481.04, stdev=150.34 lat (msec): min=28, max=1357, avg=497.08, stdev=151.88 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 313], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 575], 90.00th=[ 659], 95.00th=[ 776], | 99.00th=[ 1003], 99.50th=[ 1062], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 104, per=24.52%, avg=40.70, stdev=20.70, samples=238 iops : min= 1, max= 26, avg= 9.64, stdev= 5.20, samples=238 write: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5256KiB/120636msec); 0 zone resets slat (usec): min=32, max=16898, avg=264.98, stdev=928.91 clat (msec): min=61, max=2261, avg=992.98, stdev=321.00 lat (msec): min=62, max=2261, avg=993.25, stdev=320.99 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 718], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1053], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1569], | 99.00th=[ 1938], 99.50th=[ 2056], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 96, per=25.11%, avg=43.44, stdev=20.90, samples=234 iops : min= 1, max= 24, avg=10.33, stdev= 5.31, samples=234 lat (msec) : 20=0.04%, 50=0.12%, 100=0.35%, 250=1.02%, 500=31.41% lat (msec) : 750=24.84%, 1000=18.36%, 2000=23.44%, >=2000=0.43% cpu : usr=0.09%, sys=0.53%, ctx=3775, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145349: Wed Aug 26 22:27:28 2026 read: IOPS=10, BW=42.0KiB/s (43.0kB/s)(5060KiB/120386msec) slat (usec): min=3252, max=68716, avg=15484.65, stdev=8762.26 clat (msec): min=25, max=1383, avg=481.24, stdev=147.54 lat (msec): min=35, max=1414, avg=496.73, stdev=149.67 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 305], 10.00th=[ 338], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 527], 80.00th=[ 567], 90.00th=[ 659], 95.00th=[ 760], | 99.00th=[ 961], 99.50th=[ 1083], 99.90th=[ 1334], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 5, max= 135, per=25.51%, avg=42.34, stdev=21.94, samples=234 iops : min= 1, max= 33, avg=10.06, stdev= 5.52, samples=234 write: IOPS=10, BW=43.0KiB/s (44.1kB/s)(5180KiB/120386msec); 0 zone resets slat (usec): min=30, max=16979, avg=294.92, stdev=1068.75 clat (msec): min=360, max=2192, avg=996.89, stdev=308.58 lat (msec): min=362, max=2192, avg=997.18, stdev=308.61 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1401], 95.00th=[ 1519], | 99.00th=[ 1972], 99.50th=[ 2089], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 103, per=24.80%, avg=42.90, stdev=18.93, samples=235 iops : min= 1, max= 25, avg=10.20, stdev= 4.79, samples=235 lat (msec) : 50=0.12%, 100=0.08%, 250=0.35%, 500=31.56%, 750=26.13% lat (msec) : 1000=19.10%, 2000=22.23%, >=2000=0.43% cpu : usr=0.10%, sys=0.52%, ctx=3719, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=166KiB/s (170kB/s), 41.3KiB/s-42.0KiB/s (42.3kB/s-43.0kB/s), io=19.6MiB (20.5MB), run=120386-120636msec WRITE: bw=173KiB/s (177kB/s), 43.0KiB/s-43.7KiB/s (44.1kB/s-44.7kB/s), io=20.4MiB (21.4MB), run=120386-120636msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145371: Wed Aug 26 22:27:39 2026 read: IOPS=2, BW=81.5MiB/s (85.4MB/s)(40.0MiB/491msec) slat (nsec): min=59762k, max=59762k, avg=59762006.00, stdev= 0.00 clat (nsec): min=394815k, max=394815k, avg=394815314.00, stdev= 0.00 lat (nsec): min=454583k, max=454583k, avg=454582836.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.06%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=81.5MiB/s (85.4MB/s), 81.5MiB/s-81.5MiB/s (85.4MB/s-85.4MB/s), io=40.0MiB (41.9MB), run=491-491msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout