== sanity test 398c: run fio to test AIO ================= 20:15:42 (1787789742) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.412977 s, 102 MB/s osc.lustre-OST0000-osc-ffff99c2096b7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144827: Wed Aug 26 20:18:07 2026 write: IOPS=18, BW=75.3KiB/s (77.1kB/s)(10.0MiB/136015msec); 0 zone resets slat (usec): min=22, max=14050, avg=200.80, stdev=650.73 clat (msec): min=59, max=1365, avg=849.20, stdev=200.51 lat (msec): min=59, max=1365, avg=849.40, stdev=200.50 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 493], 10.00th=[ 600], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 802], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 944], 80.00th=[ 1011], 90.00th=[ 1099], 95.00th=[ 1183], | 99.00th=[ 1301], 99.50th=[ 1318], 99.90th=[ 1351], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 15, max= 239, per=24.87%, avg=74.86, stdev=32.21, samples=269 iops : min= 3, max= 59, avg=18.28, stdev= 8.03, samples=269 lat (msec) : 100=0.04%, 250=0.59%, 500=4.65%, 750=22.89%, 1000=49.92% lat (msec) : 2000=21.91% cpu : usr=0.04%, sys=0.26%, ctx=2131, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144828: Wed Aug 26 20:18:07 2026 write: IOPS=18, BW=75.4KiB/s (77.2kB/s)(10.0MiB/135884msec); 0 zone resets slat (usec): min=21, max=15823, avg=220.21, stdev=774.98 clat (msec): min=71, max=1463, avg=848.80, stdev=201.89 lat (msec): min=71, max=1463, avg=849.02, stdev=201.89 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 493], 10.00th=[ 609], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 802], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 944], 80.00th=[ 1028], 90.00th=[ 1116], 95.00th=[ 1183], | 99.00th=[ 1301], 99.50th=[ 1334], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 207, per=24.82%, avg=74.70, stdev=30.94, samples=270 iops : min= 1, max= 51, avg=18.23, stdev= 7.72, samples=270 lat (msec) : 100=0.04%, 250=0.66%, 500=4.38%, 750=22.97%, 1000=49.41% lat (msec) : 2000=22.54% cpu : usr=0.06%, sys=0.23%, ctx=1933, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144829: Wed Aug 26 20:18:07 2026 write: IOPS=18, BW=75.3KiB/s (77.1kB/s)(10.0MiB/135965msec); 0 zone resets slat (usec): min=22, max=18303, avg=204.54, stdev=821.46 clat (msec): min=229, max=1434, avg=848.75, stdev=197.66 lat (msec): min=229, max=1442, avg=848.96, stdev=197.69 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 481], 10.00th=[ 609], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 944], 80.00th=[ 1011], 90.00th=[ 1099], 95.00th=[ 1167], | 99.00th=[ 1301], 99.50th=[ 1334], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 254, per=24.99%, avg=75.21, stdev=32.88, samples=268 iops : min= 1, max= 63, avg=18.36, stdev= 8.23, samples=268 lat (msec) : 250=0.62%, 500=4.92%, 750=22.38%, 1000=50.70%, 2000=21.37% cpu : usr=0.05%, sys=0.25%, ctx=1997, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144830: Wed Aug 26 20:18:07 2026 write: IOPS=18, BW=75.3KiB/s (77.1kB/s)(10.0MiB/136019msec); 0 zone resets slat (usec): min=21, max=11262, avg=175.99, stdev=591.58 clat (msec): min=80, max=1440, avg=849.13, stdev=205.35 lat (msec): min=80, max=1441, avg=849.31, stdev=205.35 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 472], 10.00th=[ 600], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1020], 90.00th=[ 1099], 95.00th=[ 1167], | 99.00th=[ 1318], 99.50th=[ 1334], 99.90th=[ 1385], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 159, per=25.48%, avg=76.70, stdev=33.00, samples=261 iops : min= 1, max= 39, avg=18.73, stdev= 8.22, samples=261 lat (msec) : 100=0.12%, 250=0.20%, 500=5.31%, 750=23.36%, 1000=47.85% lat (msec) : 2000=23.16% cpu : usr=0.05%, sys=0.23%, ctx=1666, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=301KiB/s (308kB/s), 75.3KiB/s-75.4KiB/s (77.1kB/s-77.2kB/s), io=40.0MiB (41.9MB), run=135884-136019msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144859: Wed Aug 26 20:19:38 2026 read: IOPS=14, BW=59.0KiB/s (60.4kB/s)(5048KiB/85512msec) slat (usec): min=2168, max=53910, avg=11845.37, stdev=6853.79 clat (msec): min=13, max=823, avg=329.39, stdev=103.05 lat (msec): min=20, max=839, avg=341.24, stdev=104.51 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 192], 10.00th=[ 215], 20.00th=[ 245], | 30.00th=[ 271], 40.00th=[ 296], 50.00th=[ 321], 60.00th=[ 347], | 70.00th=[ 372], 80.00th=[ 405], 90.00th=[ 464], 95.00th=[ 514], | 99.00th=[ 609], 99.50th=[ 726], 99.90th=[ 810], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 15, max= 135, per=25.12%, avg=58.78, stdev=24.66, samples=169 iops : min= 3, max= 33, avg=14.28, stdev= 6.23, samples=169 write: IOPS=15, BW=60.7KiB/s (62.2kB/s)(5192KiB/85512msec); 0 zone resets slat (usec): min=27, max=20217, avg=229.69, stdev=808.47 clat (msec): min=166, max=1740, avg=719.27, stdev=198.38 lat (msec): min=166, max=1741, avg=719.50, stdev=198.43 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 443], 10.00th=[ 489], 20.00th=[ 558], | 30.00th=[ 600], 40.00th=[ 642], 50.00th=[ 684], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 877], 90.00th=[ 995], 95.00th=[ 1083], | 99.00th=[ 1234], 99.50th=[ 1318], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 6, max= 112, per=24.84%, avg=60.36, stdev=22.76, samples=169 iops : min= 1, max= 28, avg=14.67, stdev= 5.70, samples=169 lat (msec) : 20=0.04%, 50=0.31%, 100=0.27%, 250=10.43%, 500=41.37% lat (msec) : 750=27.38%, 1000=15.27%, 2000=4.92% cpu : usr=0.10%, sys=0.75%, ctx=5299, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144860: Wed Aug 26 20:19:38 2026 read: IOPS=14, BW=58.1KiB/s (59.5kB/s)(4976KiB/85666msec) slat (usec): min=1957, max=63755, avg=11887.90, stdev=6830.19 clat (msec): min=9, max=711, avg=327.48, stdev=98.14 lat (msec): min=13, max=726, avg=339.37, stdev=99.49 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 192], 10.00th=[ 218], 20.00th=[ 249], | 30.00th=[ 271], 40.00th=[ 300], 50.00th=[ 317], 60.00th=[ 342], | 70.00th=[ 368], 80.00th=[ 401], 90.00th=[ 460], 95.00th=[ 510], | 99.00th=[ 592], 99.50th=[ 634], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 7, max= 127, per=24.78%, avg=57.98, stdev=25.27, samples=169 iops : min= 1, max= 31, avg=14.08, stdev= 6.37, samples=169 write: IOPS=15, BW=61.4KiB/s (62.9kB/s)(5264KiB/85666msec); 0 zone resets slat (usec): min=25, max=14375, avg=236.03, stdev=754.66 clat (msec): min=208, max=1313, avg=718.68, stdev=196.31 lat (msec): min=208, max=1314, avg=718.91, stdev=196.30 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 435], 10.00th=[ 485], 20.00th=[ 558], | 30.00th=[ 600], 40.00th=[ 642], 50.00th=[ 693], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 877], 90.00th=[ 1003], 95.00th=[ 1099], | 99.00th=[ 1217], 99.50th=[ 1284], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 112, per=25.00%, avg=60.75, stdev=23.39, samples=170 iops : min= 1, max= 28, avg=14.77, stdev= 5.93, samples=170 lat (msec) : 10=0.04%, 20=0.04%, 50=0.27%, 100=0.23%, 250=9.73% lat (msec) : 500=41.72%, 750=27.38%, 1000=15.27%, 2000=5.31% cpu : usr=0.11%, sys=0.67%, ctx=4905, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144861: Wed Aug 26 20:19:38 2026 read: IOPS=14, BW=58.1KiB/s (59.5kB/s)(4984KiB/85712msec) slat (usec): min=2335, max=46417, avg=11563.21, stdev=7118.53 clat (msec): min=9, max=846, avg=323.79, stdev=100.32 lat (msec): min=16, max=862, avg=335.36, stdev=101.98 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 180], 10.00th=[ 211], 20.00th=[ 243], | 30.00th=[ 266], 40.00th=[ 288], 50.00th=[ 313], 60.00th=[ 338], | 70.00th=[ 368], 80.00th=[ 401], 90.00th=[ 460], 95.00th=[ 506], | 99.00th=[ 609], 99.50th=[ 634], 99.90th=[ 693], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 7, max= 167, per=24.69%, avg=57.77, stdev=29.05, samples=170 iops : min= 1, max= 41, avg=14.02, stdev= 7.33, samples=170 write: IOPS=15, BW=61.3KiB/s (62.8kB/s)(5256KiB/85712msec); 0 zone resets slat (usec): min=25, max=11058, avg=232.78, stdev=661.41 clat (msec): min=89, max=1345, avg=723.32, stdev=201.60 lat (msec): min=89, max=1345, avg=723.56, stdev=201.62 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 435], 10.00th=[ 481], 20.00th=[ 558], | 30.00th=[ 609], 40.00th=[ 651], 50.00th=[ 693], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 885], 90.00th=[ 1028], 95.00th=[ 1099], | 99.00th=[ 1217], 99.50th=[ 1234], 99.90th=[ 1267], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 15, max= 112, per=24.86%, avg=60.41, stdev=21.38, samples=170 iops : min= 3, max= 28, avg=14.68, stdev= 5.38, samples=170 lat (msec) : 10=0.04%, 20=0.08%, 50=0.12%, 100=0.31%, 250=10.86% lat (msec) : 500=40.90%, 750=27.23%, 1000=14.34%, 2000=6.13% cpu : usr=0.10%, sys=0.72%, ctx=5178, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144862: Wed Aug 26 20:19:38 2026 read: IOPS=14, BW=59.1KiB/s (60.5kB/s)(5060KiB/85613msec) slat (usec): min=53, max=43113, avg=11587.50, stdev=6924.75 clat (msec): min=21, max=836, avg=330.00, stdev=101.09 lat (msec): min=24, max=840, avg=341.60, stdev=102.90 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 197], 10.00th=[ 220], 20.00th=[ 247], | 30.00th=[ 271], 40.00th=[ 296], 50.00th=[ 321], 60.00th=[ 342], | 70.00th=[ 372], 80.00th=[ 405], 90.00th=[ 456], 95.00th=[ 506], | 99.00th=[ 609], 99.50th=[ 768], 99.90th=[ 827], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7, max= 135, per=25.06%, avg=58.64, stdev=25.01, samples=170 iops : min= 1, max= 33, avg=14.24, stdev= 6.31, samples=170 write: IOPS=15, BW=60.5KiB/s (61.0kB/s)(5180KiB/85613msec); 0 zone resets slat (usec): min=23, max=22229, avg=253.88, stdev=1040.47 clat (msec): min=146, max=1730, avg=722.94, stdev=207.15 lat (msec): min=146, max=1730, avg=723.20, stdev=207.17 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 435], 10.00th=[ 481], 20.00th=[ 558], | 30.00th=[ 609], 40.00th=[ 651], 50.00th=[ 693], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 885], 90.00th=[ 1003], 95.00th=[ 1099], | 99.00th=[ 1267], 99.50th=[ 1552], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 152, per=24.70%, avg=60.02, stdev=22.92, samples=169 iops : min= 1, max= 38, avg=14.58, stdev= 5.78, samples=169 lat (msec) : 50=0.31%, 100=0.16%, 250=10.31%, 500=42.19%, 750=26.60% lat (msec) : 1000=15.39%, 2000=5.04% cpu : usr=0.11%, sys=0.72%, ctx=4923, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=234KiB/s (240kB/s), 58.1KiB/s-59.1KiB/s (59.5kB/s-60.5kB/s), io=19.6MiB (20.5MB), run=85512-85712msec WRITE: bw=244KiB/s (250kB/s), 60.5KiB/s-61.4KiB/s (61.0kB/s-62.9kB/s), io=20.4MiB (21.4MB), run=85512-85712msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144878: Wed Aug 26 20:19:47 2026 read: IOPS=2, BW=95.0MiB/s (99.6MB/s)(40.0MiB/421msec) slat (nsec): min=37600k, max=37600k, avg=37600062.00, stdev= 0.00 clat (nsec): min=354072k, max=354072k, avg=354072106.00, stdev= 0.00 lat (nsec): min=391676k, max=391676k, avg=391676041.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.10%, ctx=25, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=95.0MiB/s (99.6MB/s), 95.0MiB/s-95.0MiB/s (99.6MB/s-99.6MB/s), io=40.0MiB (41.9MB), run=421-421msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout