== sanity test 398c: run fio to test AIO ================= 16:31:04 (1783974664) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.536401 s, 78.2 MB/s osc.lustre-OST0000-osc-ffff8d4c51cb7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144346: Mon Jul 13 16:34:01 2026 write: IOPS=15, BW=60.0KiB/s (62.4kB/s)(10.0MiB/168002msec); 0 zone resets slat (usec): min=37, max=24459, avg=392.63, stdev=1318.25 clat (msec): min=192, max=2921, avg=1048.57, stdev=369.53 lat (msec): min=192, max=2921, avg=1048.96, stdev=369.57 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 575], 10.00th=[ 684], 20.00th=[ 776], | 30.00th=[ 860], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1485], 95.00th=[ 1703], | 99.00th=[ 2433], 99.50th=[ 2735], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 255, per=25.40%, avg=61.72, stdev=31.44, samples=326 iops : min= 1, max= 63, avg=14.91, stdev= 7.89, samples=326 lat (msec) : 250=0.62%, 500=3.20%, 750=12.11%, 1000=33.44%, 2000=48.20% lat (msec) : >=2000=2.42% cpu : usr=0.05%, sys=0.35%, ctx=1950, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144347: Mon Jul 13 16:34:01 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168162msec); 0 zone resets slat (usec): min=39, max=17113, avg=348.31, stdev=1052.05 clat (msec): min=271, max=2926, avg=1049.24, stdev=347.69 lat (msec): min=273, max=2926, avg=1049.59, stdev=347.68 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1737], | 99.00th=[ 2140], 99.50th=[ 2366], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 135, per=25.48%, avg=61.91, stdev=31.06, samples=325 iops : min= 1, max= 33, avg=14.97, stdev= 7.77, samples=325 lat (msec) : 500=3.28%, 750=11.56%, 1000=34.18%, 2000=48.91%, >=2000=2.07% cpu : usr=0.06%, sys=0.30%, ctx=2028, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144348: Mon Jul 13 16:34:01 2026 write: IOPS=15, BW=60.0KiB/s (62.4kB/s)(10.0MiB/168006msec); 0 zone resets slat (usec): min=37, max=23092, avg=373.87, stdev=1205.46 clat (msec): min=181, max=2865, avg=1049.07, stdev=360.25 lat (msec): min=181, max=2866, avg=1049.45, stdev=360.25 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 785], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1284], 90.00th=[ 1502], 95.00th=[ 1703], | 99.00th=[ 2333], 99.50th=[ 2500], 99.90th=[ 2668], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 255, per=25.40%, avg=61.73, stdev=31.09, samples=326 iops : min= 1, max= 63, avg=14.91, stdev= 7.82, samples=326 lat (msec) : 250=0.62%, 500=3.09%, 750=11.56%, 1000=34.18%, 2000=47.85% lat (msec) : >=2000=2.70% cpu : usr=0.05%, sys=0.31%, ctx=2054, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144349: Mon Jul 13 16:34:01 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168163msec); 0 zone resets slat (usec): min=38, max=18036, avg=379.03, stdev=1112.89 clat (msec): min=96, max=2863, avg=1049.36, stdev=357.89 lat (msec): min=97, max=2865, avg=1049.74, stdev=357.89 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1485], 95.00th=[ 1687], | 99.00th=[ 2366], 99.50th=[ 2668], 99.90th=[ 2869], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 143, per=25.88%, avg=62.88, stdev=32.41, samples=320 iops : min= 1, max= 35, avg=15.21, stdev= 8.14, samples=320 lat (msec) : 100=0.04%, 250=0.27%, 500=3.05%, 750=11.80%, 1000=33.87% lat (msec) : 2000=48.79%, >=2000=2.19% cpu : usr=0.05%, sys=0.32%, ctx=2190, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=244KiB/s (249kB/s), 60.9KiB/s-60.0KiB/s (62.4kB/s-62.4kB/s), io=40.0MiB (41.9MB), run=168002-168163msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144383: Mon Jul 13 16:36:25 2026 read: IOPS=9, BW=37.8KiB/s (38.7kB/s)(5048KiB/133681msec) slat (usec): min=3766, max=78738, avg=17345.63, stdev=9210.46 clat (msec): min=55, max=1332, avg=528.18, stdev=154.56 lat (msec): min=68, max=1360, avg=545.53, stdev=155.89 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 330], 10.00th=[ 363], 20.00th=[ 405], | 30.00th=[ 451], 40.00th=[ 477], 50.00th=[ 506], 60.00th=[ 542], | 70.00th=[ 575], 80.00th=[ 642], 90.00th=[ 726], 95.00th=[ 818], | 99.00th=[ 961], 99.50th=[ 1083], 99.90th=[ 1267], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 95, per=25.16%, avg=37.74, stdev=18.69, samples=261 iops : min= 1, max= 23, avg= 8.97, stdev= 4.72, samples=261 write: IOPS=9, BW=38.8KiB/s (39.8kB/s)(5192KiB/133681msec); 0 zone resets slat (usec): min=49, max=19300, avg=403.95, stdev=1267.85 clat (msec): min=155, max=2716, avg=1113.24, stdev=357.12 lat (msec): min=155, max=2716, avg=1113.64, stdev=357.07 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 625], 10.00th=[ 701], 20.00th=[ 802], | 30.00th=[ 902], 40.00th=[ 986], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1250], 80.00th=[ 1385], 90.00th=[ 1552], 95.00th=[ 1737], | 99.00th=[ 2366], 99.50th=[ 2567], 99.90th=[ 2702], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 103, per=25.00%, avg=38.99, stdev=19.48, samples=260 iops : min= 1, max= 25, avg= 9.28, stdev= 4.90, samples=260 lat (msec) : 100=0.27%, 250=0.51%, 500=23.20%, 750=28.55%, 1000=17.07% lat (msec) : 2000=29.41%, >=2000=0.98% cpu : usr=0.09%, sys=0.65%, ctx=4771, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144384: Mon Jul 13 16:36:25 2026 read: IOPS=9, BW=37.3KiB/s (38.2kB/s)(4976KiB/133535msec) slat (usec): min=3000, max=64637, avg=17290.23, stdev=9969.64 clat (msec): min=25, max=1412, avg=515.67, stdev=156.16 lat (msec): min=33, max=1428, avg=532.97, stdev=158.41 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 317], 10.00th=[ 351], 20.00th=[ 397], | 30.00th=[ 435], 40.00th=[ 468], 50.00th=[ 493], 60.00th=[ 523], | 70.00th=[ 558], 80.00th=[ 617], 90.00th=[ 718], 95.00th=[ 818], | 99.00th=[ 961], 99.50th=[ 1053], 99.90th=[ 1334], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 95, per=24.83%, avg=37.25, stdev=18.83, samples=261 iops : min= 1, max= 23, avg= 8.85, stdev= 4.73, samples=261 write: IOPS=9, BW=39.4KiB/s (40.4kB/s)(5264KiB/133535msec); 0 zone resets slat (usec): min=52, max=15344, avg=380.07, stdev=1049.19 clat (msec): min=369, max=2805, avg=1116.51, stdev=365.11 lat (msec): min=369, max=2805, avg=1116.89, stdev=365.09 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 642], 10.00th=[ 701], 20.00th=[ 793], | 30.00th=[ 894], 40.00th=[ 1003], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1267], 80.00th=[ 1385], 90.00th=[ 1569], 95.00th=[ 1770], | 99.00th=[ 2467], 99.50th=[ 2601], 99.90th=[ 2769], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 103, per=25.21%, avg=39.33, stdev=19.90, samples=262 iops : min= 1, max= 25, avg= 9.37, stdev= 5.01, samples=262 lat (msec) : 50=0.23%, 100=0.16%, 250=0.35%, 500=25.16%, 750=26.60% lat (msec) : 1000=16.21%, 2000=30.43%, >=2000=0.86% cpu : usr=0.10%, sys=0.61%, ctx=4334, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144385: Mon Jul 13 16:36:25 2026 read: IOPS=9, BW=37.3KiB/s (38.2kB/s)(4984KiB/133543msec) slat (usec): min=86, max=68157, avg=17391.55, stdev=9844.23 clat (msec): min=22, max=1412, avg=517.15, stdev=153.36 lat (msec): min=38, max=1427, avg=534.55, stdev=155.49 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 326], 10.00th=[ 355], 20.00th=[ 401], | 30.00th=[ 435], 40.00th=[ 468], 50.00th=[ 498], 60.00th=[ 527], | 70.00th=[ 558], 80.00th=[ 625], 90.00th=[ 718], 95.00th=[ 793], | 99.00th=[ 953], 99.50th=[ 1036], 99.90th=[ 1334], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 104, per=24.97%, avg=37.45, stdev=19.56, samples=261 iops : min= 1, max= 26, avg= 8.90, stdev= 4.96, samples=261 write: IOPS=9, BW=39.4KiB/s (40.3kB/s)(5256KiB/133543msec); 0 zone resets slat (usec): min=54, max=19944, avg=395.48, stdev=1127.46 clat (msec): min=235, max=2688, avg=1118.03, stdev=369.23 lat (msec): min=235, max=2689, avg=1118.43, stdev=369.24 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 902], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1267], 80.00th=[ 1385], 90.00th=[ 1552], 95.00th=[ 1770], | 99.00th=[ 2467], 99.50th=[ 2534], 99.90th=[ 2635], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 95, per=25.16%, avg=39.25, stdev=18.98, samples=261 iops : min= 1, max= 23, avg= 9.35, stdev= 4.77, samples=261 lat (msec) : 50=0.12%, 100=0.23%, 250=0.39%, 500=24.77%, 750=27.19% lat (msec) : 1000=15.62%, 2000=30.55%, >=2000=1.13% cpu : usr=0.09%, sys=0.66%, ctx=4700, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144386: Mon Jul 13 16:36:25 2026 read: IOPS=9, BW=37.9KiB/s (38.8kB/s)(5060KiB/133544msec) slat (usec): min=3371, max=91475, avg=17499.49, stdev=9547.82 clat (msec): min=52, max=1332, avg=524.73, stdev=155.27 lat (msec): min=65, max=1355, avg=542.23, stdev=156.37 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 330], 10.00th=[ 363], 20.00th=[ 405], | 30.00th=[ 439], 40.00th=[ 468], 50.00th=[ 498], 60.00th=[ 531], | 70.00th=[ 575], 80.00th=[ 651], 90.00th=[ 726], 95.00th=[ 827], | 99.00th=[ 978], 99.50th=[ 1062], 99.90th=[ 1301], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 103, per=25.60%, avg=38.40, stdev=18.57, samples=258 iops : min= 1, max= 25, avg= 9.14, stdev= 4.67, samples=258 write: IOPS=9, BW=38.8KiB/s (39.7kB/s)(5180KiB/133544msec); 0 zone resets slat (usec): min=40, max=13954, avg=348.43, stdev=867.56 clat (msec): min=420, max=2788, avg=1116.91, stdev=362.27 lat (msec): min=420, max=2788, avg=1117.26, stdev=362.30 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 894], 40.00th=[ 995], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1368], 90.00th=[ 1569], 95.00th=[ 1754], | 99.00th=[ 2534], 99.50th=[ 2601], 99.90th=[ 2735], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 96, per=24.85%, avg=38.77, stdev=19.03, samples=261 iops : min= 1, max= 24, avg= 9.23, stdev= 4.79, samples=261 lat (msec) : 100=0.23%, 250=0.43%, 500=24.80%, 750=26.88%, 1000=17.15% lat (msec) : 2000=29.49%, >=2000=1.02% cpu : usr=0.07%, sys=0.65%, ctx=4591, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=150KiB/s (154kB/s), 37.3KiB/s-37.9KiB/s (38.2kB/s-38.8kB/s), io=19.6MiB (20.5MB), run=133535-133681msec WRITE: bw=156KiB/s (160kB/s), 38.8KiB/s-39.4KiB/s (39.7kB/s-40.4kB/s), io=20.4MiB (21.4MB), run=133535-133681msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144410: Mon Jul 13 16:36:36 2026 read: IOPS=1, BW=43.6MiB/s (45.7MB/s)(40.0MiB/917msec) slat (nsec): min=53642k, max=53642k, avg=53642490.00, stdev= 0.00 clat (nsec): min=834339k, max=834339k, avg=834339240.00, stdev= 0.00 lat (nsec): min=887987k, max=887987k, avg=887987033.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 835], 5.00th=[ 835], 10.00th=[ 835], 20.00th=[ 835], | 30.00th=[ 835], 40.00th=[ 835], 50.00th=[ 835], 60.00th=[ 835], | 70.00th=[ 835], 80.00th=[ 835], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=1.20%, ctx=21, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=43.6MiB/s (45.7MB/s), 43.6MiB/s-43.6MiB/s (45.7MB/s-45.7MB/s), io=40.0MiB (41.9MB), run=917-917msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout