== sanity test 398c: run fio to test AIO ================= 20:27:40 (1785457660) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.3794 s, 111 MB/s osc.lustre-OST0000-osc-ffff979d10912800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145963: Thu Jul 30 20:31:01 2026 write: IOPS=13, BW=53.9KiB/s (55.2kB/s)(10.0MiB/190108msec); 0 zone resets slat (usec): min=48, max=32561, avg=319.92, stdev=1098.44 clat (msec): min=99, max=2598, avg=1187.35, stdev=411.09 lat (msec): min=99, max=2598, avg=1187.67, stdev=411.10 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 659], 10.00th=[ 751], 20.00th=[ 869], | 30.00th=[ 953], 40.00th=[ 1053], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1452], 90.00th=[ 1804], 95.00th=[ 2005], | 99.00th=[ 2366], 99.50th=[ 2433], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 215, per=25.85%, avg=55.57, stdev=28.82, samples=361 iops : min= 1, max= 53, avg=13.41, stdev= 7.24, samples=361 lat (msec) : 100=0.04%, 250=0.08%, 500=3.16%, 750=6.45%, 1000=24.45% lat (msec) : 2000=60.90%, >=2000=4.92% cpu : usr=0.05%, sys=0.25%, ctx=2111, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145964: Thu Jul 30 20:31:01 2026 write: IOPS=13, BW=53.8KiB/s (55.1kB/s)(10.0MiB/190268msec); 0 zone resets slat (usec): min=49, max=28699, avg=300.16, stdev=1256.13 clat (msec): min=306, max=2449, avg=1187.85, stdev=403.72 lat (msec): min=306, max=2449, avg=1188.15, stdev=403.75 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 667], 10.00th=[ 760], 20.00th=[ 860], | 30.00th=[ 953], 40.00th=[ 1053], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1318], 80.00th=[ 1469], 90.00th=[ 1754], 95.00th=[ 2005], | 99.00th=[ 2333], 99.50th=[ 2400], 99.90th=[ 2433], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 255, per=26.15%, avg=56.23, stdev=31.71, samples=357 iops : min= 1, max= 63, avg=13.57, stdev= 7.97, samples=357 lat (msec) : 500=3.20%, 750=6.41%, 1000=24.73%, 2000=60.70%, >=2000=4.96% cpu : usr=0.03%, sys=0.24%, ctx=1798, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145965: Thu Jul 30 20:31:01 2026 write: IOPS=13, BW=53.9KiB/s (55.2kB/s)(10.0MiB/189862msec); 0 zone resets slat (usec): min=49, max=13548, avg=298.21, stdev=935.27 clat (msec): min=243, max=2569, avg=1185.88, stdev=404.83 lat (msec): min=243, max=2569, avg=1186.19, stdev=404.86 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 676], 10.00th=[ 776], 20.00th=[ 852], | 30.00th=[ 944], 40.00th=[ 1045], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1318], 80.00th=[ 1469], 90.00th=[ 1804], 95.00th=[ 1972], | 99.00th=[ 2232], 99.50th=[ 2299], 99.90th=[ 2433], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 252, per=26.79%, avg=57.59, stdev=33.48, samples=348 iops : min= 1, max= 63, avg=13.92, stdev= 8.43, samples=348 lat (msec) : 250=0.04%, 500=3.75%, 750=5.00%, 1000=25.82%, 2000=60.98% lat (msec) : >=2000=4.41% cpu : usr=0.03%, sys=0.23%, ctx=1844, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145966: Thu Jul 30 20:31:01 2026 write: IOPS=13, BW=53.8KiB/s (55.1kB/s)(10.0MiB/190278msec); 0 zone resets slat (usec): min=47, max=25646, avg=271.92, stdev=915.85 clat (msec): min=314, max=2516, avg=1188.01, stdev=399.73 lat (msec): min=314, max=2516, avg=1188.28, stdev=399.74 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 676], 10.00th=[ 743], 20.00th=[ 869], | 30.00th=[ 969], 40.00th=[ 1053], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1318], 80.00th=[ 1469], 90.00th=[ 1787], 95.00th=[ 1972], | 99.00th=[ 2299], 99.50th=[ 2366], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 255, per=26.31%, avg=56.56, stdev=31.79, samples=355 iops : min= 1, max= 63, avg=13.65, stdev= 7.97, samples=355 lat (msec) : 500=3.12%, 750=7.11%, 1000=22.70%, 2000=62.97%, >=2000=4.10% cpu : usr=0.03%, sys=0.22%, ctx=1931, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=215KiB/s (220kB/s), 53.8KiB/s-53.9KiB/s (55.1kB/s-55.2kB/s), io=40.0MiB (41.9MB), run=189862-190278msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146005: Thu Jul 30 20:33:10 2026 read: IOPS=10, BW=42.2KiB/s (43.2kB/s)(5048KiB/119748msec) slat (usec): min=3017, max=60706, avg=15498.68, stdev=8463.05 clat (msec): min=31, max=2118, avg=473.27, stdev=160.87 lat (msec): min=40, max=2125, avg=488.78, stdev=162.26 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 489], | 70.00th=[ 531], 80.00th=[ 592], 90.00th=[ 651], 95.00th=[ 718], | 99.00th=[ 835], 99.50th=[ 877], 99.90th=[ 1955], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 119, per=25.30%, avg=42.24, stdev=20.21, samples=234 iops : min= 1, max= 29, avg=10.05, stdev= 5.07, samples=234 write: IOPS=10, BW=43.4KiB/s (44.4kB/s)(5192KiB/119748msec); 0 zone resets slat (usec): min=58, max=19937, avg=358.15, stdev=1096.49 clat (msec): min=314, max=2866, avg=996.24, stdev=295.52 lat (msec): min=314, max=2867, avg=996.60, stdev=295.49 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 584], 10.00th=[ 642], 20.00th=[ 768], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1955], 99.50th=[ 2140], 99.90th=[ 2769], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 6, max= 119, per=24.79%, avg=43.13, stdev=20.26, samples=235 iops : min= 1, max= 29, avg=10.27, stdev= 5.09, samples=235 lat (msec) : 50=0.12%, 100=0.08%, 250=0.86%, 500=30.86%, 750=25.20% lat (msec) : 1000=20.47%, 2000=21.91%, >=2000=0.51% cpu : usr=0.09%, sys=0.63%, ctx=4683, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146006: Thu Jul 30 20:33:10 2026 read: IOPS=10, BW=41.6KiB/s (42.5kB/s)(4976KiB/119754msec) slat (usec): min=83, max=65099, avg=15178.53, stdev=8770.40 clat (msec): min=30, max=1958, avg=473.07, stdev=162.64 lat (msec): min=32, max=1968, avg=488.26, stdev=164.27 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 368], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 575], 90.00th=[ 651], 95.00th=[ 718], | 99.00th=[ 835], 99.50th=[ 1435], 99.90th=[ 1921], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 118, per=24.76%, avg=41.36, stdev=21.04, samples=235 iops : min= 1, max= 29, avg= 9.83, stdev= 5.28, samples=235 write: IOPS=10, BW=43.0KiB/s (45.0kB/s)(5264KiB/119754msec); 0 zone resets slat (usec): min=54, max=28072, avg=414.19, stdev=1450.84 clat (msec): min=189, max=2538, avg=992.28, stdev=292.26 lat (msec): min=190, max=2538, avg=992.69, stdev=292.25 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 760], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1871], 99.50th=[ 2072], 99.90th=[ 2400], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 103, per=25.03%, avg=43.56, stdev=19.27, samples=236 iops : min= 1, max= 25, avg=10.38, stdev= 4.82, samples=236 lat (msec) : 50=0.08%, 100=0.27%, 250=1.21%, 500=30.90%, 750=24.34% lat (msec) : 1000=20.04%, 2000=22.81%, >=2000=0.35% cpu : usr=0.10%, sys=0.66%, ctx=4930, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146007: Thu Jul 30 20:33:10 2026 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(4984KiB/119915msec) slat (usec): min=2863, max=65238, avg=15568.20, stdev=8692.23 clat (msec): min=19, max=1018, avg=462.54, stdev=131.84 lat (msec): min=26, max=1074, avg=478.11, stdev=133.75 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 447], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 567], 90.00th=[ 642], 95.00th=[ 693], | 99.00th=[ 810], 99.50th=[ 844], 99.90th=[ 1011], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 119, per=24.89%, avg=41.56, stdev=21.92, samples=235 iops : min= 1, max= 29, avg= 9.88, stdev= 5.50, samples=235 write: IOPS=10, BW=43.8KiB/s (44.9kB/s)(5256KiB/119915msec); 0 zone resets slat (usec): min=60, max=16284, avg=366.24, stdev=1039.52 clat (msec): min=130, max=2157, avg=1002.96, stdev=283.03 lat (msec): min=130, max=2157, avg=1003.33, stdev=283.03 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 575], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 852], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1368], 95.00th=[ 1502], | 99.00th=[ 1787], 99.50th=[ 1905], 99.90th=[ 2123], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 120, per=24.66%, avg=42.91, stdev=20.30, samples=238 iops : min= 1, max= 30, avg=10.21, stdev= 5.08, samples=238 lat (msec) : 20=0.04%, 100=0.20%, 250=1.13%, 500=30.94%, 750=23.79% lat (msec) : 1000=19.73%, 2000=24.10%, >=2000=0.08% cpu : usr=0.09%, sys=0.66%, ctx=4567, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146008: Thu Jul 30 20:33:10 2026 read: IOPS=10, BW=42.4KiB/s (43.4kB/s)(5060KiB/119378msec) slat (usec): min=2863, max=59531, avg=15647.35, stdev=8834.27 clat (msec): min=29, max=2355, avg=473.39, stdev=169.20 lat (msec): min=40, max=2372, avg=489.04, stdev=170.80 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 363], | 30.00th=[ 393], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 489], | 70.00th=[ 523], 80.00th=[ 575], 90.00th=[ 634], 95.00th=[ 735], | 99.00th=[ 835], 99.50th=[ 894], 99.90th=[ 2299], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 103, per=25.45%, avg=42.49, stdev=19.22, samples=233 iops : min= 1, max= 25, avg=10.12, stdev= 4.80, samples=233 write: IOPS=10, BW=43.4KiB/s (44.4kB/s)(5180KiB/119378msec); 0 zone resets slat (usec): min=56, max=17086, avg=341.94, stdev=920.89 clat (msec): min=336, max=2792, avg=992.63, stdev=292.66 lat (msec): min=337, max=2792, avg=992.97, stdev=292.65 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 575], 10.00th=[ 659], 20.00th=[ 776], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1871], 99.50th=[ 2056], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 126, per=24.94%, avg=43.40, stdev=19.56, samples=233 iops : min= 1, max= 31, avg=10.34, stdev= 4.90, samples=233 lat (msec) : 50=0.08%, 100=0.31%, 250=0.86%, 500=31.33%, 750=23.95% lat (msec) : 1000=21.29%, 2000=21.76%, >=2000=0.43% cpu : usr=0.09%, sys=0.65%, ctx=4481, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=167KiB/s (171kB/s), 41.6KiB/s-42.4KiB/s (42.5kB/s-43.4kB/s), io=19.6MiB (20.5MB), run=119378-119915msec WRITE: bw=174KiB/s (178kB/s), 43.4KiB/s-43.0KiB/s (44.4kB/s-45.0kB/s), io=20.4MiB (21.4MB), run=119378-119915msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146029: Thu Jul 30 20:33:20 2026 read: IOPS=2, BW=96.9MiB/s (102MB/s)(40.0MiB/413msec) slat (nsec): min=63361k, max=63361k, avg=63360540.00, stdev= 0.00 clat (nsec): min=329902k, max=329902k, avg=329901670.00, stdev= 0.00 lat (nsec): min=393267k, max=393267k, avg=393267293.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.18%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=96.9MiB/s (102MB/s), 96.9MiB/s-96.9MiB/s (102MB/s-102MB/s), io=40.0MiB (41.9MB), run=413-413msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout