== sanity test 398c: run fio to test AIO ================= 09:33:28 (1787146408) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.607186 s, 69.1 MB/s osc.lustre-OST0000-osc-ffff915e4310e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145541: Wed Aug 19 09:36:28 2026 write: IOPS=15, BW=60.7KiB/s (62.2kB/s)(10.0MiB/168714msec); 0 zone resets slat (usec): min=25, max=17193, avg=190.21, stdev=831.16 clat (msec): min=163, max=2098, avg=1053.64, stdev=316.15 lat (msec): min=163, max=2098, avg=1053.84, stdev=316.18 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 625], 10.00th=[ 726], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1502], 95.00th=[ 1620], | 99.00th=[ 1905], 99.50th=[ 1972], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2106] bw ( KiB/s): min= 8, max= 238, per=25.28%, avg=61.17, stdev=29.42, samples=329 iops : min= 2, max= 59, avg=14.81, stdev= 7.41, samples=329 lat (msec) : 250=1.25%, 500=1.37%, 750=10.90%, 1000=36.21%, 2000=49.92% lat (msec) : >=2000=0.35% cpu : usr=0.04%, sys=0.19%, ctx=1538, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145542: Wed Aug 19 09:36:28 2026 write: IOPS=15, BW=60.6KiB/s (62.0kB/s)(10.0MiB/169018msec); 0 zone resets slat (usec): min=22, max=18965, avg=184.60, stdev=809.72 clat (msec): min=193, max=2019, avg=1055.35, stdev=309.64 lat (msec): min=193, max=2019, avg=1055.54, stdev=309.64 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1301], 90.00th=[ 1502], 95.00th=[ 1620], | 99.00th=[ 1888], 99.50th=[ 1972], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 152, per=25.14%, avg=60.84, stdev=29.47, samples=331 iops : min= 1, max= 38, avg=14.74, stdev= 7.42, samples=331 lat (msec) : 250=0.62%, 500=2.11%, 750=9.18%, 1000=37.11%, 2000=50.94% lat (msec) : >=2000=0.04% cpu : usr=0.04%, sys=0.19%, ctx=1657, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145543: Wed Aug 19 09:36:28 2026 write: IOPS=15, BW=60.6KiB/s (62.1kB/s)(10.0MiB/168878msec); 0 zone resets slat (usec): min=22, max=12808, avg=189.68, stdev=702.93 clat (msec): min=189, max=2024, avg=1053.95, stdev=308.91 lat (msec): min=189, max=2024, avg=1054.15, stdev=308.93 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 642], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1167], 80.00th=[ 1301], 90.00th=[ 1485], 95.00th=[ 1620], | 99.00th=[ 1888], 99.50th=[ 1972], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 128, per=25.44%, avg=61.57, stdev=30.39, samples=327 iops : min= 1, max= 32, avg=14.91, stdev= 7.64, samples=327 lat (msec) : 250=0.62%, 500=1.52%, 750=11.48%, 1000=34.73%, 2000=51.45% lat (msec) : >=2000=0.20% cpu : usr=0.04%, sys=0.18%, ctx=1666, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145544: Wed Aug 19 09:36:28 2026 write: IOPS=15, BW=60.7KiB/s (62.2kB/s)(10.0MiB/168713msec); 0 zone resets slat (usec): min=23, max=18641, avg=198.57, stdev=947.12 clat (msec): min=151, max=2001, avg=1053.49, stdev=315.62 lat (msec): min=151, max=2001, avg=1053.70, stdev=315.65 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 634], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1318], 90.00th=[ 1502], 95.00th=[ 1636], | 99.00th=[ 1854], 99.50th=[ 1938], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 239, per=25.44%, avg=61.57, stdev=29.62, samples=327 iops : min= 1, max= 59, avg=14.92, stdev= 7.45, samples=327 lat (msec) : 250=1.25%, 500=1.56%, 750=10.31%, 1000=36.84%, 2000=50.00% lat (msec) : >=2000=0.04% cpu : usr=0.05%, sys=0.20%, ctx=1735, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=242KiB/s (248kB/s), 60.6KiB/s-60.7KiB/s (62.0kB/s-62.2kB/s), io=40.0MiB (41.9MB), run=168713-169018msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145578: Wed Aug 19 09:38:31 2026 read: IOPS=11, BW=44.7KiB/s (45.8kB/s)(5048KiB/112939msec) slat (usec): min=2631, max=57580, avg=14524.24, stdev=8491.01 clat (msec): min=9, max=1032, avg=440.26, stdev=126.91 lat (msec): min=15, max=1058, avg=454.79, stdev=128.34 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 485], 80.00th=[ 527], 90.00th=[ 600], 95.00th=[ 693], | 99.00th=[ 844], 99.50th=[ 869], 99.90th=[ 894], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 120, per=25.04%, avg=44.32, stdev=23.10, samples=223 iops : min= 1, max= 30, avg=10.63, stdev= 5.83, samples=223 write: IOPS=11, BW=45.0KiB/s (47.1kB/s)(5192KiB/112939msec); 0 zone resets slat (usec): min=26, max=18188, avg=278.57, stdev=1113.27 clat (msec): min=382, max=2001, avg=946.83, stdev=270.59 lat (msec): min=383, max=2001, avg=947.11, stdev=270.65 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 518], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1435], | 99.00th=[ 1720], 99.50th=[ 1770], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 108, per=24.78%, avg=45.60, stdev=20.40, samples=223 iops : min= 1, max= 27, avg=10.95, stdev= 5.12, samples=223 lat (msec) : 10=0.04%, 50=0.16%, 100=0.16%, 250=1.02%, 500=37.19% lat (msec) : 750=21.88%, 1000=19.34%, 2000=20.20%, >=2000=0.04% cpu : usr=0.09%, sys=0.60%, ctx=4555, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145579: Wed Aug 19 09:38:31 2026 read: IOPS=10, BW=43.0KiB/s (45.0kB/s)(4976KiB/113183msec) slat (usec): min=2802, max=56017, avg=14720.82, stdev=8328.91 clat (msec): min=14, max=1031, avg=440.47, stdev=127.17 lat (msec): min=23, max=1059, avg=455.20, stdev=128.81 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 372], 40.00th=[ 393], 50.00th=[ 414], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 527], 90.00th=[ 609], 95.00th=[ 709], | 99.00th=[ 835], 99.50th=[ 869], 99.90th=[ 1011], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 104, per=24.62%, avg=43.57, stdev=21.22, samples=223 iops : min= 1, max= 26, avg=10.43, stdev= 5.31, samples=223 write: IOPS=11, BW=46.5KiB/s (47.6kB/s)(5264KiB/113183msec); 0 zone resets slat (usec): min=24, max=21979, avg=260.99, stdev=1112.82 clat (msec): min=242, max=2086, avg=943.69, stdev=281.78 lat (msec): min=242, max=2086, avg=943.95, stdev=281.81 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 510], 10.00th=[ 600], 20.00th=[ 701], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1284], 95.00th=[ 1452], | 99.00th=[ 1787], 99.50th=[ 1938], 99.90th=[ 2072], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 111, per=25.08%, avg=46.14, stdev=19.38, samples=224 iops : min= 1, max= 27, avg=11.08, stdev= 4.88, samples=224 lat (msec) : 20=0.04%, 50=0.16%, 100=0.08%, 250=0.94%, 500=37.62% lat (msec) : 750=21.76%, 1000=19.69%, 2000=19.53%, >=2000=0.20% cpu : usr=0.10%, sys=0.59%, ctx=4691, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145580: Wed Aug 19 09:38:31 2026 read: IOPS=11, BW=44.0KiB/s (45.1kB/s)(4984KiB/113183msec) slat (usec): min=2896, max=63202, avg=14944.19, stdev=8621.88 clat (msec): min=12, max=1032, avg=441.86, stdev=124.00 lat (msec): min=23, max=1050, avg=456.81, stdev=125.45 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 451], | 70.00th=[ 481], 80.00th=[ 531], 90.00th=[ 600], 95.00th=[ 693], | 99.00th=[ 818], 99.50th=[ 852], 99.90th=[ 1028], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 6, max= 136, per=25.00%, avg=44.25, stdev=22.65, samples=221 iops : min= 1, max= 34, avg=10.61, stdev= 5.71, samples=221 write: IOPS=11, BW=46.4KiB/s (47.6kB/s)(5256KiB/113183msec); 0 zone resets slat (usec): min=27, max=14026, avg=265.85, stdev=927.08 clat (msec): min=351, max=2083, avg=942.00, stdev=278.96 lat (msec): min=351, max=2083, avg=942.27, stdev=279.01 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 502], 10.00th=[ 600], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1435], | 99.00th=[ 1804], 99.50th=[ 1854], 99.90th=[ 2039], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 111, per=24.83%, avg=45.68, stdev=21.33, samples=225 iops : min= 1, max= 27, avg=10.96, stdev= 5.35, samples=225 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=1.09%, 500=37.15% lat (msec) : 750=21.45%, 1000=19.49%, 2000=20.47%, >=2000=0.08% cpu : usr=0.10%, sys=0.68%, ctx=4679, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145581: Wed Aug 19 09:38:31 2026 read: IOPS=11, BW=45.0KiB/s (46.1kB/s)(5060KiB/112341msec) slat (usec): min=62, max=68944, avg=14570.05, stdev=8294.60 clat (msec): min=14, max=1369, avg=442.57, stdev=137.62 lat (msec): min=23, max=1394, avg=457.15, stdev=139.11 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 451], | 70.00th=[ 489], 80.00th=[ 535], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 844], 99.50th=[ 860], 99.90th=[ 1318], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 7, max= 119, per=25.60%, avg=45.32, stdev=20.90, samples=219 iops : min= 1, max= 29, avg=10.87, stdev= 5.23, samples=219 write: IOPS=11, BW=46.1KiB/s (47.2kB/s)(5180KiB/112341msec); 0 zone resets slat (usec): min=34, max=15245, avg=270.69, stdev=917.34 clat (msec): min=142, max=2086, avg=939.96, stdev=290.34 lat (msec): min=142, max=2086, avg=940.23, stdev=290.36 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 510], 10.00th=[ 592], 20.00th=[ 693], | 30.00th=[ 776], 40.00th=[ 844], 50.00th=[ 919], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1452], | 99.00th=[ 1754], 99.50th=[ 1787], 99.90th=[ 2072], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 104, per=24.87%, avg=45.76, stdev=22.44, samples=221 iops : min= 1, max= 26, avg=10.98, stdev= 5.64, samples=221 lat (msec) : 20=0.04%, 50=0.31%, 100=0.31%, 250=1.29%, 500=36.80% lat (msec) : 750=22.77%, 1000=18.36%, 2000=20.04%, >=2000=0.08% cpu : usr=0.09%, sys=0.60%, ctx=4575, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177KiB/s (182kB/s), 43.0KiB/s-45.0KiB/s (45.0kB/s-46.1kB/s), io=19.6MiB (20.5MB), run=112341-113183msec WRITE: bw=185KiB/s (189kB/s), 45.0KiB/s-46.5KiB/s (47.1kB/s-47.6kB/s), io=20.4MiB (21.4MB), run=112341-113183msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145612: Wed Aug 19 09:38:42 2026 read: IOPS=1, BW=64.7MiB/s (67.9MB/s)(40.0MiB/618msec) slat (nsec): min=26379k, max=26379k, avg=26379235.00, stdev= 0.00 clat (nsec): min=567366k, max=567366k, avg=567366011.00, stdev= 0.00 lat (nsec): min=593783k, max=593783k, avg=593783462.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.11%, ctx=4, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=64.7MiB/s (67.9MB/s), 64.7MiB/s-64.7MiB/s (67.9MB/s-67.9MB/s), io=40.0MiB (41.9MB), run=618-618msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout