== sanity test 398c: run fio to test AIO ================= 11:47:17 (1787845637) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.473489 s, 88.6 MB/s osc.lustre-OST0000-osc-ffff9efda6a3a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147509: Thu Aug 27 11:50:24 2026 write: IOPS=14, BW=57.9KiB/s (59.2kB/s)(10.0MiB/177005msec); 0 zone resets slat (usec): min=28, max=19959, avg=223.78, stdev=888.32 clat (msec): min=209, max=2425, avg=1105.21, stdev=365.25 lat (msec): min=209, max=2431, avg=1105.43, stdev=365.26 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 818], | 30.00th=[ 902], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1133], | 70.00th=[ 1234], 80.00th=[ 1401], 90.00th=[ 1620], 95.00th=[ 1787], | 99.00th=[ 2106], 99.50th=[ 2232], 99.90th=[ 2400], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 256, per=25.17%, avg=58.14, stdev=30.14, samples=346 iops : min= 1, max= 64, avg=14.08, stdev= 7.58, samples=346 lat (msec) : 250=0.62%, 500=2.66%, 750=9.84%, 1000=30.12%, 2000=54.61% lat (msec) : >=2000=2.15% cpu : usr=0.04%, sys=0.21%, ctx=1691, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147510: Thu Aug 27 11:50:24 2026 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(10.0MiB/177179msec); 0 zone resets slat (usec): min=28, max=14579, avg=235.22, stdev=850.71 clat (msec): min=195, max=2503, avg=1106.34, stdev=366.17 lat (msec): min=195, max=2503, avg=1106.58, stdev=366.17 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 969], 50.00th=[ 1045], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1385], 90.00th=[ 1620], 95.00th=[ 1804], | 99.00th=[ 2106], 99.50th=[ 2265], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 216, per=24.81%, avg=57.30, stdev=27.32, samples=351 iops : min= 1, max= 54, avg=13.87, stdev= 6.89, samples=351 lat (msec) : 250=0.66%, 500=2.62%, 750=9.88%, 1000=31.25%, 2000=53.63% lat (msec) : >=2000=1.95% cpu : usr=0.05%, sys=0.22%, ctx=1900, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147511: Thu Aug 27 11:50:24 2026 write: IOPS=14, BW=57.9KiB/s (59.3kB/s)(10.0MiB/176913msec); 0 zone resets slat (usec): min=29, max=17460, avg=210.35, stdev=725.05 clat (msec): min=106, max=2498, avg=1105.28, stdev=367.95 lat (msec): min=106, max=2498, avg=1105.50, stdev=367.97 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 592], 10.00th=[ 701], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1036], 60.00th=[ 1133], | 70.00th=[ 1234], 80.00th=[ 1385], 90.00th=[ 1620], 95.00th=[ 1787], | 99.00th=[ 2165], 99.50th=[ 2265], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 232, per=24.86%, avg=57.43, stdev=28.15, samples=350 iops : min= 1, max= 58, avg=13.90, stdev= 7.07, samples=350 lat (msec) : 250=0.55%, 500=2.58%, 750=10.43%, 1000=30.70%, 2000=53.44% lat (msec) : >=2000=2.30% cpu : usr=0.04%, sys=0.21%, ctx=1984, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147512: Thu Aug 27 11:50:24 2026 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(10.0MiB/177138msec); 0 zone resets slat (usec): min=29, max=19633, avg=207.96, stdev=793.32 clat (msec): min=220, max=2501, avg=1106.25, stdev=365.00 lat (msec): min=220, max=2502, avg=1106.46, stdev=365.00 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 969], 50.00th=[ 1045], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1385], 90.00th=[ 1620], 95.00th=[ 1804], | 99.00th=[ 2140], 99.50th=[ 2232], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 176, per=24.95%, avg=57.63, stdev=28.53, samples=349 iops : min= 1, max= 44, avg=13.95, stdev= 7.18, samples=349 lat (msec) : 250=0.62%, 500=2.30%, 750=10.78%, 1000=30.04%, 2000=53.71% lat (msec) : >=2000=2.54% cpu : usr=0.04%, sys=0.19%, ctx=1684, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=231KiB/s (237kB/s), 57.8KiB/s-57.9KiB/s (59.2kB/s-59.3kB/s), io=40.0MiB (41.9MB), run=176913-177179msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147549: Thu Aug 27 11:52:37 2026 read: IOPS=10, BW=40.3KiB/s (41.3kB/s)(5048KiB/125172msec) slat (usec): min=26, max=60024, avg=15194.97, stdev=8332.22 clat (msec): min=17, max=1022, avg=475.82, stdev=131.09 lat (msec): min=30, max=1043, avg=491.02, stdev=132.67 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 292], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 405], 40.00th=[ 439], 50.00th=[ 468], 60.00th=[ 502], | 70.00th=[ 535], 80.00th=[ 584], 90.00th=[ 642], 95.00th=[ 709], | 99.00th=[ 835], 99.50th=[ 877], 99.90th=[ 919], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 95, per=25.56%, avg=40.63, stdev=19.87, samples=243 iops : min= 1, max= 23, avg= 9.64, stdev= 4.99, samples=243 write: IOPS=10, BW=41.5KiB/s (42.5kB/s)(5192KiB/125172msec); 0 zone resets slat (usec): min=29, max=9231, avg=245.99, stdev=604.04 clat (msec): min=132, max=2417, avg=1064.63, stdev=307.92 lat (msec): min=133, max=2417, avg=1064.88, stdev=307.91 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 642], 10.00th=[ 693], 20.00th=[ 818], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1620], | 99.00th=[ 2022], 99.50th=[ 2265], 99.90th=[ 2366], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 111, per=24.89%, avg=41.32, stdev=18.33, samples=245 iops : min= 1, max= 27, avg= 9.81, stdev= 4.61, samples=245 lat (msec) : 20=0.08%, 50=0.23%, 100=0.20%, 250=1.02%, 500=28.67% lat (msec) : 750=24.92%, 1000=16.25%, 2000=28.12%, >=2000=0.51% cpu : usr=0.09%, sys=0.56%, ctx=4182, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147550: Thu Aug 27 11:52:37 2026 read: IOPS=9, BW=39.6KiB/s (40.6kB/s)(4976KiB/125561msec) slat (usec): min=2568, max=88381, avg=15295.54, stdev=8245.10 clat (msec): min=21, max=885, avg=479.13, stdev=133.04 lat (msec): min=27, max=904, avg=494.44, stdev=134.39 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 292], 10.00th=[ 334], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 498], | 70.00th=[ 535], 80.00th=[ 592], 90.00th=[ 659], 95.00th=[ 726], | 99.00th=[ 844], 99.50th=[ 860], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 96, per=24.55%, avg=39.04, stdev=20.22, samples=248 iops : min= 1, max= 24, avg= 9.23, stdev= 5.10, samples=248 write: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5264KiB/125561msec); 0 zone resets slat (usec): min=30, max=23247, avg=285.60, stdev=905.68 clat (msec): min=270, max=2357, avg=1057.04, stdev=300.80 lat (msec): min=270, max=2357, avg=1057.32, stdev=300.79 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 642], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1418], 95.00th=[ 1603], | 99.00th=[ 2022], 99.50th=[ 2089], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 104, per=24.88%, avg=41.29, stdev=19.17, samples=249 iops : min= 1, max= 26, avg= 9.80, stdev= 4.82, samples=249 lat (msec) : 50=0.23%, 250=0.86%, 500=29.30%, 750=23.67%, 1000=16.60% lat (msec) : 2000=28.67%, >=2000=0.66% cpu : usr=0.10%, sys=0.53%, ctx=4283, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147551: Thu Aug 27 11:52:37 2026 read: IOPS=9, BW=39.7KiB/s (40.7kB/s)(4984KiB/125509msec) slat (usec): min=3090, max=50080, avg=14942.58, stdev=7939.11 clat (msec): min=23, max=1034, avg=467.57, stdev=130.31 lat (msec): min=37, max=1039, avg=482.52, stdev=132.01 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 284], 10.00th=[ 321], 20.00th=[ 363], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 481], | 70.00th=[ 527], 80.00th=[ 567], 90.00th=[ 634], 95.00th=[ 701], | 99.00th=[ 827], 99.50th=[ 902], 99.90th=[ 936], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 104, per=25.46%, avg=40.49, stdev=21.14, samples=241 iops : min= 1, max= 26, avg= 9.60, stdev= 5.33, samples=241 write: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5256KiB/125509msec); 0 zone resets slat (usec): min=35, max=21980, avg=337.41, stdev=1220.63 clat (msec): min=241, max=2415, avg=1068.65, stdev=307.42 lat (msec): min=241, max=2415, avg=1068.99, stdev=307.43 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 642], 10.00th=[ 718], 20.00th=[ 818], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1053], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1418], 95.00th=[ 1636], | 99.00th=[ 1989], 99.50th=[ 2265], 99.90th=[ 2366], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 96, per=24.77%, avg=41.12, stdev=17.61, samples=249 iops : min= 1, max= 24, avg= 9.76, stdev= 4.46, samples=249 lat (msec) : 50=0.20%, 100=0.20%, 250=0.82%, 500=30.98%, 750=22.03% lat (msec) : 1000=16.29%, 2000=29.06%, >=2000=0.43% cpu : usr=0.11%, sys=0.60%, ctx=4514, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147552: Thu Aug 27 11:52:37 2026 read: IOPS=10, BW=40.5KiB/s (41.5kB/s)(5060KiB/124931msec) slat (usec): min=2958, max=65651, avg=15281.06, stdev=8425.05 clat (msec): min=20, max=1601, avg=479.27, stdev=148.38 lat (msec): min=32, max=1646, avg=494.56, stdev=149.52 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 284], 10.00th=[ 326], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 464], 60.00th=[ 493], | 70.00th=[ 531], 80.00th=[ 584], 90.00th=[ 659], 95.00th=[ 726], | 99.00th=[ 894], 99.50th=[ 969], 99.90th=[ 1485], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 102, per=25.86%, avg=41.12, stdev=18.14, samples=241 iops : min= 1, max= 25, avg= 9.76, stdev= 4.63, samples=241 write: IOPS=10, BW=41.5KiB/s (42.5kB/s)(5180KiB/124931msec); 0 zone resets slat (usec): min=33, max=9014, avg=261.50, stdev=681.70 clat (msec): min=168, max=2438, avg=1057.57, stdev=310.66 lat (msec): min=168, max=2438, avg=1057.83, stdev=310.63 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 642], 10.00th=[ 693], 20.00th=[ 802], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1620], | 99.00th=[ 2072], 99.50th=[ 2232], 99.90th=[ 2366], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 103, per=24.60%, avg=40.83, stdev=19.87, samples=247 iops : min= 1, max= 25, avg= 9.69, stdev= 4.98, samples=247 lat (msec) : 50=0.12%, 100=0.31%, 250=0.94%, 500=29.65%, 750=24.06% lat (msec) : 1000=16.91%, 2000=27.46%, >=2000=0.55% cpu : usr=0.09%, sys=0.59%, ctx=4609, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=160KiB/s (164kB/s), 39.6KiB/s-40.5KiB/s (40.6kB/s-41.5kB/s), io=19.6MiB (20.5MB), run=124931-125561msec WRITE: bw=166KiB/s (170kB/s), 41.5KiB/s-41.9KiB/s (42.5kB/s-42.9kB/s), io=20.4MiB (21.4MB), run=124931-125561msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147576: Thu Aug 27 11:52:47 2026 read: IOPS=1, BW=66.3MiB/s (69.6MB/s)(40.0MiB/603msec) slat (nsec): min=44157k, max=44157k, avg=44156586.00, stdev= 0.00 clat (nsec): min=533153k, max=533153k, avg=533152644.00, stdev= 0.00 lat (nsec): min=577314k, max=577314k, avg=577313760.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 535], 10.00th=[ 535], 20.00th=[ 535], | 30.00th=[ 535], 40.00th=[ 535], 50.00th=[ 535], 60.00th=[ 535], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.16%, ctx=44, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=66.3MiB/s (69.6MB/s), 66.3MiB/s-66.3MiB/s (69.6MB/s-69.6MB/s), io=40.0MiB (41.9MB), run=603-603msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout