== sanity test 398c: run fio to test AIO ================= 23:21:31 (1789269691) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.559061 s, 75.0 MB/s osc.lustre-OST0000-osc-ffff90ba82d04800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145110: Sat Sep 12 23:24:28 2026 write: IOPS=15, BW=60.0KiB/s (62.4kB/s)(10.0MiB/167938msec); 0 zone resets slat (usec): min=23, max=18846, avg=188.82, stdev=809.02 clat (msec): min=157, max=2512, avg=1048.62, stdev=314.86 lat (msec): min=160, max=2512, avg=1048.81, stdev=314.88 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1418], 95.00th=[ 1552], | 99.00th=[ 2072], 99.50th=[ 2299], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 255, per=25.73%, avg=62.52, stdev=34.00, samples=322 iops : min= 1, max= 63, avg=15.14, stdev= 8.50, samples=322 lat (msec) : 250=0.08%, 500=2.50%, 750=9.65%, 1000=34.57%, 2000=51.84% lat (msec) : >=2000=1.37% cpu : usr=0.03%, sys=0.19%, ctx=1563, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145111: Sat Sep 12 23:24:28 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168025msec); 0 zone resets slat (usec): min=22, max=13543, avg=161.09, stdev=606.88 clat (msec): min=267, max=2512, avg=1049.11, stdev=306.84 lat (msec): min=267, max=2512, avg=1049.27, stdev=306.82 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1586], | 99.00th=[ 1972], 99.50th=[ 2072], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 128, per=26.39%, avg=64.12, stdev=31.62, samples=314 iops : min= 1, max= 32, avg=15.55, stdev= 7.95, samples=314 lat (msec) : 500=2.66%, 750=9.38%, 1000=36.95%, 2000=50.16%, >=2000=0.86% cpu : usr=0.04%, sys=0.18%, ctx=1622, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145112: Sat Sep 12 23:24:28 2026 write: IOPS=15, BW=61.1KiB/s (62.5kB/s)(10.0MiB/167682msec); 0 zone resets slat (usec): min=21, max=12109, avg=180.48, stdev=713.46 clat (msec): min=219, max=2334, avg=1046.91, stdev=315.59 lat (msec): min=219, max=2334, avg=1047.10, stdev=315.59 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 625], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1620], | 99.00th=[ 2072], 99.50th=[ 2198], 99.90th=[ 2299], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 255, per=25.41%, avg=61.74, stdev=33.71, samples=326 iops : min= 1, max= 63, avg=14.95, stdev= 8.46, samples=326 lat (msec) : 250=1.25%, 500=1.68%, 750=9.80%, 1000=35.16%, 2000=51.02% lat (msec) : >=2000=1.09% cpu : usr=0.04%, sys=0.18%, ctx=1439, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145113: Sat Sep 12 23:24:28 2026 write: IOPS=15, BW=61.0KiB/s (62.5kB/s)(10.0MiB/167834msec); 0 zone resets slat (usec): min=22, max=15819, avg=184.57, stdev=782.23 clat (msec): min=86, max=2412, avg=1048.29, stdev=315.22 lat (msec): min=86, max=2412, avg=1048.48, stdev=315.22 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 818], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1418], 95.00th=[ 1586], | 99.00th=[ 2072], 99.50th=[ 2265], 99.90th=[ 2366], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 239, per=25.09%, avg=60.97, stdev=28.56, samples=330 iops : min= 1, max= 59, avg=14.76, stdev= 7.20, samples=330 lat (msec) : 100=0.12%, 250=0.51%, 500=2.30%, 750=8.79%, 1000=36.17% lat (msec) : 2000=50.55%, >=2000=1.56% cpu : usr=0.04%, sys=0.19%, ctx=1711, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=244KiB/s (250kB/s), 60.9KiB/s-61.1KiB/s (62.4kB/s-62.5kB/s), io=40.0MiB (41.9MB), run=167682-168025msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145147: Sat Sep 12 23:26:18 2026 read: IOPS=12, BW=49.0KiB/s (51.2kB/s)(5048KiB/100998msec) slat (usec): min=2406, max=71637, avg=13760.32, stdev=7935.01 clat (msec): min=13, max=967, avg=396.44, stdev=114.98 lat (msec): min=21, max=1008, avg=410.21, stdev=115.92 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 243], 10.00th=[ 275], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 468], 90.00th=[ 550], 95.00th=[ 617], | 99.00th=[ 709], 99.50th=[ 877], 99.90th=[ 961], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 128, per=25.15%, avg=49.55, stdev=22.76, samples=200 iops : min= 1, max= 32, avg=11.89, stdev= 5.73, samples=200 write: IOPS=12, BW=51.4KiB/s (52.6kB/s)(5192KiB/100998msec); 0 zone resets slat (usec): min=28, max=9820, avg=193.09, stdev=452.95 clat (msec): min=165, max=1674, avg=843.13, stdev=242.52 lat (msec): min=165, max=1674, avg=843.32, stdev=242.52 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 481], 10.00th=[ 567], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 760], 50.00th=[ 810], 60.00th=[ 877], | 70.00th=[ 944], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1334], | 99.00th=[ 1519], 99.50th=[ 1586], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 112, per=25.06%, avg=51.38, stdev=21.84, samples=198 iops : min= 1, max= 28, avg=12.36, stdev= 5.51, samples=198 lat (msec) : 20=0.12%, 50=0.16%, 100=0.16%, 250=2.38%, 500=41.64% lat (msec) : 750=24.06%, 1000=19.69%, 2000=11.80% cpu : usr=0.12%, sys=0.60%, ctx=4359, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145148: Sat Sep 12 23:26:18 2026 read: IOPS=12, BW=49.1KiB/s (50.2kB/s)(4976KiB/101441msec) slat (usec): min=49, max=52873, avg=13766.24, stdev=7706.28 clat (msec): min=6, max=916, avg=399.65, stdev=111.04 lat (msec): min=12, max=920, avg=413.42, stdev=112.43 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 245], 10.00th=[ 275], 20.00th=[ 313], | 30.00th=[ 342], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 443], 80.00th=[ 481], 90.00th=[ 542], 95.00th=[ 609], | 99.00th=[ 709], 99.50th=[ 726], 99.90th=[ 911], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 7, max= 143, per=24.68%, avg=48.62, stdev=23.44, samples=200 iops : min= 1, max= 35, avg=11.67, stdev= 5.88, samples=200 write: IOPS=12, BW=51.9KiB/s (53.1kB/s)(5264KiB/101441msec); 0 zone resets slat (usec): min=25, max=20511, avg=242.82, stdev=884.23 clat (msec): min=162, max=1684, avg=841.72, stdev=244.09 lat (msec): min=162, max=1685, avg=841.96, stdev=244.05 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 477], 10.00th=[ 550], 20.00th=[ 634], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 818], 60.00th=[ 877], | 70.00th=[ 944], 80.00th=[ 1020], 90.00th=[ 1167], 95.00th=[ 1318], | 99.00th=[ 1519], 99.50th=[ 1586], 99.90th=[ 1653], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 160, per=25.05%, avg=51.35, stdev=22.27, samples=201 iops : min= 1, max= 40, avg=12.34, stdev= 5.63, samples=201 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=0.08%, 250=2.70% lat (msec) : 500=40.59%, 750=23.87%, 1000=21.05%, 2000=11.56% cpu : usr=0.12%, sys=0.62%, ctx=4136, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145149: Sat Sep 12 23:26:18 2026 read: IOPS=12, BW=49.2KiB/s (50.3kB/s)(4984KiB/101391msec) slat (usec): min=2868, max=56778, avg=13937.86, stdev=7901.02 clat (msec): min=36, max=927, avg=398.07, stdev=105.39 lat (msec): min=50, max=935, avg=412.02, stdev=106.73 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 253], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 468], 90.00th=[ 535], 95.00th=[ 600], | 99.00th=[ 709], 99.50th=[ 735], 99.90th=[ 919], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 7, max= 167, per=24.86%, avg=48.98, stdev=23.65, samples=200 iops : min= 1, max= 41, avg=11.75, stdev= 5.97, samples=200 write: IOPS=12, BW=51.8KiB/s (53.1kB/s)(5256KiB/101391msec); 0 zone resets slat (usec): min=25, max=29011, avg=257.26, stdev=1074.24 clat (msec): min=287, max=1657, avg=841.00, stdev=235.08 lat (msec): min=287, max=1657, avg=841.26, stdev=235.09 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 489], 10.00th=[ 567], 20.00th=[ 642], | 30.00th=[ 709], 40.00th=[ 768], 50.00th=[ 818], 60.00th=[ 869], | 70.00th=[ 927], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1301], | 99.00th=[ 1469], 99.50th=[ 1586], 99.90th=[ 1636], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 128, per=25.21%, avg=51.68, stdev=20.47, samples=199 iops : min= 1, max= 32, avg=12.43, stdev= 5.16, samples=199 lat (msec) : 50=0.04%, 250=2.27%, 500=41.80%, 750=23.48%, 1000=21.02% lat (msec) : 2000=11.41% cpu : usr=0.12%, sys=0.62%, ctx=4011, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145150: Sat Sep 12 23:26:18 2026 read: IOPS=12, BW=50.1KiB/s (51.3kB/s)(5060KiB/100996msec) slat (usec): min=2441, max=54737, avg=14145.26, stdev=7853.79 clat (msec): min=20, max=992, avg=397.80, stdev=113.13 lat (msec): min=32, max=1007, avg=411.95, stdev=114.40 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 251], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 380], 60.00th=[ 405], | 70.00th=[ 435], 80.00th=[ 472], 90.00th=[ 550], 95.00th=[ 625], | 99.00th=[ 718], 99.50th=[ 785], 99.90th=[ 969], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 135, per=25.64%, avg=50.51, stdev=21.22, samples=197 iops : min= 1, max= 33, avg=12.14, stdev= 5.31, samples=197 write: IOPS=12, BW=51.3KiB/s (52.5kB/s)(5180KiB/100996msec); 0 zone resets slat (usec): min=26, max=22095, avg=186.64, stdev=705.00 clat (msec): min=293, max=1699, avg=841.88, stdev=240.60 lat (msec): min=293, max=1699, avg=842.07, stdev=240.59 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 489], 10.00th=[ 567], 20.00th=[ 634], | 30.00th=[ 701], 40.00th=[ 760], 50.00th=[ 818], 60.00th=[ 869], | 70.00th=[ 944], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1301], | 99.00th=[ 1586], 99.50th=[ 1620], 99.90th=[ 1687], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 112, per=24.83%, avg=50.91, stdev=20.43, samples=199 iops : min= 1, max= 28, avg=12.24, stdev= 5.15, samples=199 lat (msec) : 50=0.16%, 100=0.04%, 250=2.11%, 500=42.07%, 750=23.48% lat (msec) : 1000=20.70%, 2000=11.45% cpu : usr=0.10%, sys=0.63%, ctx=4239, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=198KiB/s (203kB/s), 49.1KiB/s-50.1KiB/s (50.2kB/s-51.3kB/s), io=19.6MiB (20.5MB), run=100996-101441msec WRITE: bw=206KiB/s (211kB/s), 51.3KiB/s-51.9KiB/s (52.5kB/s-53.1kB/s), io=20.4MiB (21.4MB), run=100996-101441msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145168: Sat Sep 12 23:26:29 2026 read: IOPS=1, BW=65.7MiB/s (68.9MB/s)(40.0MiB/609msec) slat (nsec): min=30248k, max=30248k, avg=30248406.00, stdev= 0.00 clat (nsec): min=555983k, max=555983k, avg=555982862.00, stdev= 0.00 lat (nsec): min=586237k, max=586237k, avg=586236540.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 558], 10.00th=[ 558], 20.00th=[ 558], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.81%, ctx=6, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.7MiB/s (68.9MB/s), 65.7MiB/s-65.7MiB/s (68.9MB/s-68.9MB/s), io=40.0MiB (41.9MB), run=609-609msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout