== sanity test 398c: run fio to test AIO ================= 13:50:10 (1780336210) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.455181 s, 92.1 MB/s osc.lustre-OST0000-osc-ffff9c7ec92e7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146864: Mon Jun 1 13:52:31 2026 write: IOPS=19, BW=77.8KiB/s (79.7kB/s)(10.0MiB/131586msec); 0 zone resets slat (usec): min=37, max=14031, avg=233.24, stdev=611.20 clat (msec): min=89, max=1740, avg=821.10, stdev=212.39 lat (msec): min=89, max=1740, avg=821.33, stdev=212.40 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 527], 10.00th=[ 609], 20.00th=[ 667], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 810], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 978], 90.00th=[ 1070], 95.00th=[ 1183], | 99.00th=[ 1435], 99.50th=[ 1469], 99.90th=[ 1586], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 392, per=24.86%, avg=77.31, stdev=33.91, samples=261 iops : min= 1, max= 98, avg=18.89, stdev= 8.50, samples=261 lat (msec) : 100=0.04%, 250=1.29%, 500=3.09%, 750=32.73%, 1000=46.45% lat (msec) : 2000=16.41% cpu : usr=0.05%, sys=0.35%, ctx=2456, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146865: Mon Jun 1 13:52:31 2026 write: IOPS=19, BW=77.9KiB/s (79.7kB/s)(10.0MiB/131514msec); 0 zone resets slat (usec): min=36, max=20234, avg=254.67, stdev=838.20 clat (msec): min=130, max=1946, avg=820.51, stdev=214.28 lat (msec): min=131, max=1946, avg=820.77, stdev=214.25 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 523], 10.00th=[ 600], 20.00th=[ 667], | 30.00th=[ 709], 40.00th=[ 760], 50.00th=[ 810], 60.00th=[ 860], | 70.00th=[ 919], 80.00th=[ 978], 90.00th=[ 1062], 95.00th=[ 1183], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1586], 99.95th=[ 1787], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 384, per=25.25%, avg=78.53, stdev=33.49, samples=257 iops : min= 1, max= 96, avg=19.19, stdev= 8.39, samples=257 lat (msec) : 250=1.25%, 500=3.01%, 750=33.87%, 1000=45.62%, 2000=16.25% cpu : usr=0.04%, sys=0.32%, ctx=2376, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146866: Mon Jun 1 13:52:31 2026 write: IOPS=19, BW=77.8KiB/s (79.7kB/s)(10.0MiB/131574msec); 0 zone resets slat (usec): min=37, max=14258, avg=251.39, stdev=757.73 clat (msec): min=75, max=1657, avg=821.11, stdev=208.57 lat (msec): min=75, max=1658, avg=821.37, stdev=208.57 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 535], 10.00th=[ 609], 20.00th=[ 676], | 30.00th=[ 726], 40.00th=[ 768], 50.00th=[ 810], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 978], 90.00th=[ 1070], 95.00th=[ 1167], | 99.00th=[ 1418], 99.50th=[ 1452], 99.90th=[ 1569], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 8, max= 384, per=24.86%, avg=77.32, stdev=32.64, samples=261 iops : min= 2, max= 96, avg=18.89, stdev= 8.22, samples=261 lat (msec) : 100=0.08%, 250=1.17%, 500=2.97%, 750=31.80%, 1000=47.97% lat (msec) : 2000=16.02% cpu : usr=0.04%, sys=0.31%, ctx=2234, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146867: Mon Jun 1 13:52:31 2026 write: IOPS=19, BW=77.0KiB/s (79.9kB/s)(10.0MiB/131292msec); 0 zone resets slat (usec): min=35, max=14929, avg=267.53, stdev=778.43 clat (msec): min=89, max=1657, avg=819.80, stdev=211.05 lat (msec): min=90, max=1657, avg=820.07, stdev=211.04 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 535], 10.00th=[ 609], 20.00th=[ 676], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 810], 60.00th=[ 860], | 70.00th=[ 911], 80.00th=[ 961], 90.00th=[ 1083], 95.00th=[ 1150], | 99.00th=[ 1418], 99.50th=[ 1485], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 15, max= 328, per=25.03%, avg=77.86, stdev=32.01, samples=259 iops : min= 3, max= 82, avg=19.03, stdev= 8.05, samples=259 lat (msec) : 100=0.08%, 250=1.60%, 500=2.85%, 750=31.60%, 1000=47.85% lat (msec) : 2000=16.02% cpu : usr=0.05%, sys=0.32%, ctx=2398, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=311KiB/s (319kB/s), 77.8KiB/s-77.0KiB/s (79.7kB/s-79.9kB/s), io=40.0MiB (41.9MB), run=131292-131586msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146896: Mon Jun 1 13:54:14 2026 read: IOPS=13, BW=54.1KiB/s (55.4kB/s)(5048KiB/93344msec) slat (usec): min=2381, max=53454, avg=13525.63, stdev=7944.98 clat (msec): min=7, max=974, avg=369.61, stdev=105.40 lat (msec): min=18, max=992, avg=383.15, stdev=106.49 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 232], 10.00th=[ 262], 20.00th=[ 288], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 409], 80.00th=[ 447], 90.00th=[ 493], 95.00th=[ 558], | 99.00th=[ 693], 99.50th=[ 726], 99.90th=[ 919], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 15, max= 151, per=24.96%, avg=53.42, stdev=23.76, samples=185 iops : min= 3, max= 37, avg=12.84, stdev= 5.96, samples=185 write: IOPS=13, BW=55.6KiB/s (56.0kB/s)(5192KiB/93344msec); 0 zone resets slat (usec): min=38, max=24942, avg=322.88, stdev=1081.32 clat (msec): min=112, max=1513, avg=775.34, stdev=198.81 lat (msec): min=112, max=1513, avg=775.66, stdev=198.84 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 472], 10.00th=[ 542], 20.00th=[ 617], | 30.00th=[ 667], 40.00th=[ 709], 50.00th=[ 760], 60.00th=[ 818], | 70.00th=[ 877], 80.00th=[ 944], 90.00th=[ 1028], 95.00th=[ 1116], | 99.00th=[ 1318], 99.50th=[ 1368], 99.90th=[ 1485], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 8, max= 119, per=24.71%, avg=55.11, stdev=20.08, samples=185 iops : min= 2, max= 29, avg=13.26, stdev= 5.03, samples=185 lat (msec) : 10=0.04%, 20=0.04%, 50=0.20%, 100=0.12%, 250=3.71% lat (msec) : 500=43.91%, 750=25.78%, 1000=19.92%, 2000=6.29% cpu : usr=0.11%, sys=0.84%, ctx=5051, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146897: Mon Jun 1 13:54:14 2026 read: IOPS=13, BW=53.2KiB/s (54.5kB/s)(4976KiB/93448msec) slat (usec): min=2494, max=73726, avg=13556.88, stdev=7754.57 clat (msec): min=24, max=829, avg=363.20, stdev=101.67 lat (msec): min=36, max=857, avg=376.77, stdev=102.69 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 222], 10.00th=[ 255], 20.00th=[ 288], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 351], 60.00th=[ 372], | 70.00th=[ 401], 80.00th=[ 435], 90.00th=[ 485], 95.00th=[ 550], | 99.00th=[ 684], 99.50th=[ 735], 99.90th=[ 810], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 119, per=24.72%, avg=52.90, stdev=23.77, samples=184 iops : min= 1, max= 29, avg=12.71, stdev= 5.96, samples=184 write: IOPS=14, BW=56.3KiB/s (57.7kB/s)(5264KiB/93448msec); 0 zone resets slat (usec): min=42, max=27031, avg=279.67, stdev=926.57 clat (msec): min=195, max=1497, avg=777.83, stdev=200.88 lat (msec): min=195, max=1497, avg=778.11, stdev=200.88 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 477], 10.00th=[ 542], 20.00th=[ 609], | 30.00th=[ 667], 40.00th=[ 709], 50.00th=[ 768], 60.00th=[ 827], | 70.00th=[ 894], 80.00th=[ 944], 90.00th=[ 1036], 95.00th=[ 1099], | 99.00th=[ 1301], 99.50th=[ 1351], 99.90th=[ 1385], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 119, per=24.99%, avg=55.74, stdev=22.04, samples=185 iops : min= 1, max= 29, avg=13.42, stdev= 5.50, samples=185 lat (msec) : 50=0.20%, 100=0.23%, 250=4.02%, 500=43.87%, 750=24.61% lat (msec) : 1000=19.92%, 2000=7.15% cpu : usr=0.11%, sys=0.72%, ctx=4877, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146898: Mon Jun 1 13:54:14 2026 read: IOPS=13, BW=53.4KiB/s (54.7kB/s)(4984KiB/93265msec) slat (usec): min=2520, max=57183, avg=13719.79, stdev=7775.99 clat (msec): min=27, max=894, avg=365.17, stdev=102.07 lat (msec): min=35, max=913, avg=378.90, stdev=103.43 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 236], 10.00th=[ 259], 20.00th=[ 288], | 30.00th=[ 309], 40.00th=[ 330], 50.00th=[ 351], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 435], 90.00th=[ 489], 95.00th=[ 558], | 99.00th=[ 701], 99.50th=[ 718], 99.90th=[ 835], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 128, per=24.88%, avg=53.24, stdev=23.66, samples=184 iops : min= 1, max= 32, avg=12.79, stdev= 5.90, samples=184 write: IOPS=14, BW=56.4KiB/s (57.7kB/s)(5256KiB/93265msec); 0 zone resets slat (usec): min=51, max=10283, avg=325.46, stdev=835.61 clat (msec): min=179, max=1479, avg=773.42, stdev=194.42 lat (msec): min=180, max=1480, avg=773.74, stdev=194.44 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 481], 10.00th=[ 542], 20.00th=[ 609], | 30.00th=[ 667], 40.00th=[ 709], 50.00th=[ 760], 60.00th=[ 818], | 70.00th=[ 877], 80.00th=[ 936], 90.00th=[ 1020], 95.00th=[ 1099], | 99.00th=[ 1284], 99.50th=[ 1318], 99.90th=[ 1452], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 15, max= 112, per=24.92%, avg=55.57, stdev=20.78, samples=185 iops : min= 3, max= 28, avg=13.38, stdev= 5.21, samples=185 lat (msec) : 50=0.12%, 100=0.16%, 250=3.67%, 500=43.67%, 750=25.90% lat (msec) : 1000=20.43%, 2000=6.05% cpu : usr=0.11%, sys=0.75%, ctx=4821, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146899: Mon Jun 1 13:54:14 2026 read: IOPS=13, BW=54.0KiB/s (56.3kB/s)(5060KiB/92030msec) slat (usec): min=91, max=60113, avg=13608.87, stdev=7815.68 clat (msec): min=7, max=811, avg=364.75, stdev=107.00 lat (msec): min=15, max=827, avg=378.36, stdev=108.41 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 220], 10.00th=[ 253], 20.00th=[ 288], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 355], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 439], 90.00th=[ 498], 95.00th=[ 558], | 99.00th=[ 684], 99.50th=[ 709], 99.90th=[ 793], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 8, max= 119, per=25.81%, avg=55.23, stdev=21.40, samples=180 iops : min= 2, max= 29, avg=13.30, stdev= 5.34, samples=180 write: IOPS=14, BW=56.3KiB/s (57.6kB/s)(5180KiB/92030msec); 0 zone resets slat (usec): min=45, max=14764, avg=325.94, stdev=820.32 clat (msec): min=122, max=1504, avg=766.22, stdev=203.58 lat (msec): min=122, max=1505, avg=766.54, stdev=203.60 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 460], 10.00th=[ 514], 20.00th=[ 609], | 30.00th=[ 667], 40.00th=[ 701], 50.00th=[ 751], 60.00th=[ 802], | 70.00th=[ 869], 80.00th=[ 936], 90.00th=[ 1028], 95.00th=[ 1099], | 99.00th=[ 1301], 99.50th=[ 1385], 99.90th=[ 1485], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 130, per=24.94%, avg=55.62, stdev=20.93, samples=182 iops : min= 1, max= 32, avg=13.39, stdev= 5.26, samples=182 lat (msec) : 10=0.04%, 50=0.43%, 100=0.23%, 250=4.38%, 500=43.98% lat (msec) : 750=25.55%, 1000=19.22%, 2000=6.17% cpu : usr=0.10%, sys=0.76%, ctx=4991, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=215KiB/s (220kB/s), 53.2KiB/s-54.0KiB/s (54.5kB/s-56.3kB/s), io=19.6MiB (20.5MB), run=92030-93448msec WRITE: bw=224KiB/s (229kB/s), 55.6KiB/s-56.4KiB/s (56.0kB/s-57.7kB/s), io=20.4MiB (21.4MB), run=92030-93448msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146916: Mon Jun 1 13:54:23 2026 read: IOPS=1, BW=78.6MiB/s (82.4MB/s)(40.0MiB/509msec) slat (nsec): min=22150k, max=22150k, avg=22150088.00, stdev= 0.00 clat (nsec): min=474673k, max=474673k, avg=474673415.00, stdev= 0.00 lat (nsec): min=496827k, max=496827k, avg=496827022.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 477], 10.00th=[ 477], 20.00th=[ 477], | 30.00th=[ 477], 40.00th=[ 477], 50.00th=[ 477], 60.00th=[ 477], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.56%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=78.6MiB/s (82.4MB/s), 78.6MiB/s-78.6MiB/s (82.4MB/s-82.4MB/s), io=40.0MiB (41.9MB), run=509-509msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout