== sanity test 398c: run fio to test AIO ================= 03:57:06 (1788940626) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.552853 s, 75.9 MB/s osc.lustre-OST0000-osc-ffff8895e0637800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145384: Wed Sep 9 03:59:52 2026 write: IOPS=16, BW=65.7KiB/s (67.2kB/s)(10.0MiB/155938msec); 0 zone resets slat (usec): min=21, max=18334, avg=179.89, stdev=723.59 clat (msec): min=117, max=3243, avg=974.10, stdev=385.82 lat (msec): min=117, max=3243, avg=974.28, stdev=385.83 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 531], 10.00th=[ 642], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 885], 60.00th=[ 953], | 70.00th=[ 1045], 80.00th=[ 1183], 90.00th=[ 1401], 95.00th=[ 1670], | 99.00th=[ 2467], 99.50th=[ 2735], 99.90th=[ 3239], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 7, max= 212, per=24.92%, avg=65.28, stdev=32.28, samples=308 iops : min= 1, max= 53, avg=15.84, stdev= 8.14, samples=308 lat (msec) : 250=0.62%, 500=3.44%, 750=20.78%, 1000=40.98%, 2000=31.09% lat (msec) : >=2000=3.09% cpu : usr=0.05%, sys=0.20%, ctx=1572, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145385: Wed Sep 9 03:59:52 2026 write: IOPS=16, BW=65.6KiB/s (67.2kB/s)(10.0MiB/155998msec); 0 zone resets slat (usec): min=21, max=19832, avg=184.54, stdev=811.92 clat (msec): min=199, max=3231, avg=974.18, stdev=384.35 lat (msec): min=200, max=3231, avg=974.37, stdev=384.36 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 567], 10.00th=[ 651], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 953], | 70.00th=[ 1036], 80.00th=[ 1167], 90.00th=[ 1385], 95.00th=[ 1687], | 99.00th=[ 2467], 99.50th=[ 2735], 99.90th=[ 3171], 99.95th=[ 3171], | 99.99th=[ 3239] bw ( KiB/s): min= 7, max= 212, per=25.33%, avg=66.36, stdev=30.39, samples=303 iops : min= 1, max= 53, avg=16.11, stdev= 7.65, samples=303 lat (msec) : 250=0.62%, 500=3.28%, 750=21.52%, 1000=41.02%, 2000=30.35% lat (msec) : >=2000=3.20% cpu : usr=0.05%, sys=0.21%, ctx=1610, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145386: Wed Sep 9 03:59:52 2026 write: IOPS=16, BW=65.6KiB/s (67.2kB/s)(10.0MiB/156026msec); 0 zone resets slat (usec): min=22, max=16063, avg=195.96, stdev=758.08 clat (msec): min=98, max=3234, avg=974.33, stdev=384.60 lat (msec): min=98, max=3234, avg=974.53, stdev=384.58 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 567], 10.00th=[ 659], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 885], 60.00th=[ 944], | 70.00th=[ 1028], 80.00th=[ 1183], 90.00th=[ 1385], 95.00th=[ 1670], | 99.00th=[ 2534], 99.50th=[ 2735], 99.90th=[ 3239], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 7, max= 199, per=25.78%, avg=67.55, stdev=31.76, samples=298 iops : min= 1, max= 49, avg=16.41, stdev= 7.98, samples=298 lat (msec) : 100=0.04%, 250=0.20%, 500=4.06%, 750=19.96%, 1000=42.54% lat (msec) : 2000=30.16%, >=2000=3.05% cpu : usr=0.05%, sys=0.20%, ctx=1739, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145387: Wed Sep 9 03:59:52 2026 write: IOPS=16, BW=65.6KiB/s (67.2kB/s)(10.0MiB/156100msec); 0 zone resets slat (usec): min=22, max=21146, avg=192.95, stdev=830.73 clat (msec): min=127, max=3229, avg=974.69, stdev=379.59 lat (msec): min=127, max=3229, avg=974.88, stdev=379.59 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 542], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 953], | 70.00th=[ 1036], 80.00th=[ 1183], 90.00th=[ 1435], 95.00th=[ 1720], | 99.00th=[ 2400], 99.50th=[ 2567], 99.90th=[ 2937], 99.95th=[ 3205], | 99.99th=[ 3239] bw ( KiB/s): min= 7, max= 254, per=26.19%, avg=68.62, stdev=38.03, samples=293 iops : min= 1, max= 63, avg=16.68, stdev= 9.58, samples=293 lat (msec) : 250=0.08%, 500=4.22%, 750=20.70%, 1000=40.66%, 2000=31.45% lat (msec) : >=2000=2.89% cpu : usr=0.04%, sys=0.19%, ctx=1489, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=262KiB/s (269kB/s), 65.6KiB/s-65.7KiB/s (67.2kB/s-67.2kB/s), io=40.0MiB (41.9MB), run=155938-156100msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145419: Wed Sep 9 04:01:49 2026 read: IOPS=11, BW=47.3KiB/s (48.5kB/s)(5048KiB/106616msec) slat (usec): min=2415, max=72121, avg=14893.33, stdev=8629.85 clat (msec): min=24, max=1052, avg=416.77, stdev=114.14 lat (msec): min=34, max=1077, avg=431.67, stdev=115.40 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 268], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 380], 50.00th=[ 401], 60.00th=[ 422], | 70.00th=[ 447], 80.00th=[ 489], 90.00th=[ 558], 95.00th=[ 617], | 99.00th=[ 810], 99.50th=[ 844], 99.90th=[ 1036], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 7, max= 120, per=25.18%, avg=47.33, stdev=21.20, samples=209 iops : min= 1, max= 30, avg=11.36, stdev= 5.34, samples=209 write: IOPS=12, BW=48.7KiB/s (49.9kB/s)(5192KiB/106616msec); 0 zone resets slat (usec): min=33, max=11114, avg=249.13, stdev=765.93 clat (msec): min=384, max=2018, avg=890.34, stdev=260.67 lat (msec): min=384, max=2018, avg=890.59, stdev=260.68 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 567], 10.00th=[ 617], 20.00th=[ 676], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 978], 80.00th=[ 1070], 90.00th=[ 1217], 95.00th=[ 1401], | 99.00th=[ 1737], 99.50th=[ 1888], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 103, per=24.88%, avg=48.52, stdev=18.87, samples=210 iops : min= 1, max= 25, avg=11.66, stdev= 4.76, samples=210 lat (msec) : 50=0.12%, 100=0.20%, 250=1.17%, 500=40.23%, 750=24.26% lat (msec) : 1000=20.08%, 2000=13.83%, >=2000=0.12% cpu : usr=0.12%, sys=0.59%, ctx=4261, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145420: Wed Sep 9 04:01:49 2026 read: IOPS=11, BW=46.7KiB/s (47.8kB/s)(4976KiB/106664msec) slat (msec): min=3, max=462, avg=15.50, stdev=15.49 clat (msec): min=10, max=1030, avg=420.86, stdev=128.23 lat (msec): min=17, max=1059, avg=436.37, stdev=129.56 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 259], 10.00th=[ 300], 20.00th=[ 342], | 30.00th=[ 363], 40.00th=[ 384], 50.00th=[ 405], 60.00th=[ 422], | 70.00th=[ 456], 80.00th=[ 498], 90.00th=[ 567], 95.00th=[ 642], | 99.00th=[ 902], 99.50th=[ 969], 99.90th=[ 1020], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 111, per=24.55%, avg=46.16, stdev=21.04, samples=211 iops : min= 1, max= 27, avg=11.07, stdev= 5.28, samples=211 write: IOPS=12, BW=49.4KiB/s (50.5kB/s)(5264KiB/106664msec); 0 zone resets slat (usec): min=33, max=15388, avg=195.01, stdev=657.39 clat (msec): min=306, max=2078, avg=881.84, stdev=262.86 lat (msec): min=306, max=2078, avg=882.04, stdev=262.86 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 514], 10.00th=[ 592], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 902], | 70.00th=[ 978], 80.00th=[ 1062], 90.00th=[ 1217], 95.00th=[ 1418], | 99.00th=[ 1703], 99.50th=[ 1921], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 127, per=25.13%, avg=49.00, stdev=21.49, samples=211 iops : min= 1, max= 31, avg=11.78, stdev= 5.39, samples=211 lat (msec) : 20=0.08%, 50=0.27%, 100=0.35%, 250=1.29%, 500=39.45% lat (msec) : 750=22.97%, 1000=21.64%, 2000=13.79%, >=2000=0.16% cpu : usr=0.10%, sys=0.53%, ctx=4074, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145421: Wed Sep 9 04:01:49 2026 read: IOPS=11, BW=46.7KiB/s (47.9kB/s)(4984KiB/106618msec) slat (usec): min=2693, max=69955, avg=14607.06, stdev=8581.73 clat (msec): min=20, max=946, avg=414.36, stdev=111.12 lat (msec): min=30, max=966, avg=428.98, stdev=112.61 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 268], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 456], 80.00th=[ 485], 90.00th=[ 550], 95.00th=[ 609], | 99.00th=[ 785], 99.50th=[ 818], 99.90th=[ 919], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 103, per=24.94%, avg=46.88, stdev=21.44, samples=209 iops : min= 1, max= 25, avg=11.25, stdev= 5.38, samples=209 write: IOPS=12, BW=49.3KiB/s (50.5kB/s)(5256KiB/106618msec); 0 zone resets slat (usec): min=28, max=31749, avg=244.27, stdev=1228.95 clat (msec): min=350, max=2053, avg=888.04, stdev=252.66 lat (msec): min=350, max=2053, avg=888.28, stdev=252.69 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 567], 10.00th=[ 609], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 978], 80.00th=[ 1070], 90.00th=[ 1234], 95.00th=[ 1385], | 99.00th=[ 1620], 99.50th=[ 1770], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 110, per=24.87%, avg=48.50, stdev=20.94, samples=212 iops : min= 1, max= 27, avg=11.66, stdev= 5.28, samples=212 lat (msec) : 50=0.04%, 100=0.16%, 250=1.21%, 500=40.08%, 750=23.32% lat (msec) : 1000=21.29%, 2000=13.83%, >=2000=0.08% cpu : usr=0.11%, sys=0.61%, ctx=4203, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145422: Wed Sep 9 04:01:49 2026 read: IOPS=11, BW=47.7KiB/s (48.9kB/s)(5060KiB/105999msec) slat (usec): min=52, max=72368, avg=14808.48, stdev=8694.61 clat (msec): min=8, max=1119, avg=416.00, stdev=122.60 lat (msec): min=8, max=1139, avg=430.82, stdev=124.02 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 334], | 30.00th=[ 355], 40.00th=[ 376], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 451], 80.00th=[ 489], 90.00th=[ 558], 95.00th=[ 634], | 99.00th=[ 852], 99.50th=[ 986], 99.90th=[ 1036], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 104, per=25.18%, avg=47.33, stdev=22.18, samples=210 iops : min= 1, max= 26, avg=11.36, stdev= 5.56, samples=210 write: IOPS=12, BW=48.9KiB/s (50.0kB/s)(5180KiB/105999msec); 0 zone resets slat (usec): min=28, max=15897, avg=245.37, stdev=942.28 clat (msec): min=190, max=2087, avg=887.30, stdev=269.88 lat (msec): min=190, max=2087, avg=887.55, stdev=269.87 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 902], | 70.00th=[ 969], 80.00th=[ 1070], 90.00th=[ 1217], 95.00th=[ 1418], | 99.00th=[ 1804], 99.50th=[ 1938], 99.90th=[ 2056], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 104, per=24.83%, avg=48.43, stdev=20.01, samples=209 iops : min= 1, max= 26, avg=11.63, stdev= 5.01, samples=209 lat (msec) : 10=0.08%, 20=0.12%, 50=0.08%, 100=0.20%, 250=1.17% lat (msec) : 500=40.20%, 750=23.20%, 1000=21.72%, 2000=13.01%, >=2000=0.23% cpu : usr=0.12%, sys=0.56%, ctx=4065, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=188KiB/s (193kB/s), 46.7KiB/s-47.7KiB/s (47.8kB/s-48.9kB/s), io=19.6MiB (20.5MB), run=105999-106664msec WRITE: bw=196KiB/s (201kB/s), 48.7KiB/s-49.4KiB/s (49.9kB/s-50.5kB/s), io=20.4MiB (21.4MB), run=105999-106664msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145453: Wed Sep 9 04:02:04 2026 read: IOPS=1, BW=43.2MiB/s (45.3MB/s)(40.0MiB/925msec) slat (nsec): min=38346k, max=38346k, avg=38346115.00, stdev= 0.00 clat (nsec): min=831023k, max=831023k, avg=831022799.00, stdev= 0.00 lat (nsec): min=869375k, max=869375k, avg=869374729.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 835], 5.00th=[ 835], 10.00th=[ 835], 20.00th=[ 835], | 30.00th=[ 835], 40.00th=[ 835], 50.00th=[ 835], 60.00th=[ 835], | 70.00th=[ 835], 80.00th=[ 835], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=1.62%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=43.2MiB/s (45.3MB/s), 43.2MiB/s-43.2MiB/s (45.3MB/s-45.3MB/s), io=40.0MiB (41.9MB), run=925-925msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout