== sanity test 398c: run fio to test AIO ================= 16:52:16 (1775767936) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.655019 s, 64.0 MB/s osc.lustre-OST0000-osc-ffff953a1f444000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147911: Thu Apr 9 16:55:30 2026 write: IOPS=14, BW=56.1KiB/s (57.4kB/s)(10.0MiB/182592msec); 0 zone resets slat (usec): min=36, max=32232, avg=284.26, stdev=1229.56 clat (msec): min=115, max=2829, avg=1140.35, stdev=418.00 lat (msec): min=115, max=2829, avg=1140.63, stdev=418.04 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 558], 10.00th=[ 667], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 995], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1267], 80.00th=[ 1435], 90.00th=[ 1703], 95.00th=[ 1921], | 99.00th=[ 2534], 99.50th=[ 2668], 99.90th=[ 2802], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 192, per=25.53%, avg=56.94, stdev=29.69, samples=354 iops : min= 1, max= 48, avg=13.82, stdev= 7.46, samples=354 lat (msec) : 250=0.43%, 500=3.32%, 750=11.05%, 1000=26.21%, 2000=55.12% lat (msec) : >=2000=3.87% cpu : usr=0.04%, sys=0.23%, ctx=1964, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147912: Thu Apr 9 16:55:30 2026 write: IOPS=13, BW=55.0KiB/s (57.3kB/s)(10.0MiB/182964msec); 0 zone resets slat (usec): min=35, max=21527, avg=300.50, stdev=1060.51 clat (msec): min=233, max=2921, avg=1142.17, stdev=418.38 lat (msec): min=234, max=2921, avg=1142.47, stdev=418.37 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 584], 10.00th=[ 684], 20.00th=[ 827], | 30.00th=[ 919], 40.00th=[ 995], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1267], 80.00th=[ 1435], 90.00th=[ 1703], 95.00th=[ 1938], | 99.00th=[ 2567], 99.50th=[ 2668], 99.90th=[ 2836], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 160, per=24.96%, avg=55.66, stdev=28.80, samples=362 iops : min= 1, max= 40, avg=13.50, stdev= 7.25, samples=362 lat (msec) : 250=0.62%, 500=3.36%, 750=10.43%, 1000=26.17%, 2000=55.08% lat (msec) : >=2000=4.34% cpu : usr=0.04%, sys=0.23%, ctx=2013, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147913: Thu Apr 9 16:55:30 2026 write: IOPS=13, BW=55.0KiB/s (57.3kB/s)(10.0MiB/182904msec); 0 zone resets slat (usec): min=36, max=26173, avg=286.36, stdev=1108.41 clat (msec): min=217, max=2879, avg=1141.75, stdev=414.17 lat (msec): min=218, max=2892, avg=1142.03, stdev=414.20 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 835], | 30.00th=[ 911], 40.00th=[ 995], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1267], 80.00th=[ 1435], 90.00th=[ 1703], 95.00th=[ 1921], | 99.00th=[ 2601], 99.50th=[ 2735], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 144, per=25.17%, avg=56.13, stdev=31.42, samples=359 iops : min= 1, max= 36, avg=13.62, stdev= 7.91, samples=359 lat (msec) : 250=0.62%, 500=3.16%, 750=8.98%, 1000=27.70%, 2000=55.74% lat (msec) : >=2000=3.79% cpu : usr=0.03%, sys=0.23%, ctx=1862, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147914: Thu Apr 9 16:55:30 2026 write: IOPS=13, BW=55.0KiB/s (57.3kB/s)(10.0MiB/182961msec); 0 zone resets slat (usec): min=36, max=31605, avg=291.68, stdev=1153.40 clat (msec): min=92, max=2846, avg=1141.94, stdev=423.17 lat (msec): min=93, max=2846, avg=1142.23, stdev=423.16 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 584], 10.00th=[ 684], 20.00th=[ 827], | 30.00th=[ 919], 40.00th=[ 995], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1267], 80.00th=[ 1435], 90.00th=[ 1737], 95.00th=[ 1972], | 99.00th=[ 2567], 99.50th=[ 2668], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 240, per=25.68%, avg=57.27, stdev=31.91, samples=352 iops : min= 1, max= 60, avg=13.90, stdev= 8.02, samples=352 lat (msec) : 100=0.04%, 250=0.66%, 500=3.16%, 750=10.27%, 1000=26.60% lat (msec) : 2000=54.96%, >=2000=4.30% cpu : usr=0.04%, sys=0.23%, ctx=1891, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=224KiB/s (229kB/s), 55.0KiB/s-56.1KiB/s (57.3kB/s-57.4kB/s), io=40.0MiB (41.9MB), run=182592-182964msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147951: Thu Apr 9 16:57:37 2026 read: IOPS=10, BW=43.1KiB/s (44.1kB/s)(5048KiB/117126msec) slat (usec): min=2389, max=54384, avg=14961.16, stdev=8273.52 clat (msec): min=25, max=965, avg=455.61, stdev=128.72 lat (msec): min=30, max=994, avg=470.58, stdev=129.97 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 418], 50.00th=[ 439], 60.00th=[ 472], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 625], 95.00th=[ 676], | 99.00th=[ 827], 99.50th=[ 927], 99.90th=[ 961], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 135, per=25.08%, avg=42.89, stdev=21.75, samples=230 iops : min= 1, max= 33, avg=10.20, stdev= 5.46, samples=230 write: IOPS=11, BW=44.3KiB/s (45.4kB/s)(5192KiB/117126msec); 0 zone resets slat (usec): min=42, max=35116, avg=346.46, stdev=1417.73 clat (msec): min=347, max=1824, avg=982.89, stdev=280.26 lat (msec): min=348, max=1824, avg=983.24, stdev=280.26 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1502], | 99.00th=[ 1720], 99.50th=[ 1737], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 96, per=24.86%, avg=44.25, stdev=19.23, samples=229 iops : min= 1, max= 24, avg=10.54, stdev= 4.79, samples=229 lat (msec) : 50=0.23%, 100=0.20%, 250=0.70%, 500=33.71%, 750=23.87% lat (msec) : 1000=18.83%, 2000=22.46% cpu : usr=0.09%, sys=0.61%, ctx=4460, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147952: Thu Apr 9 16:57:37 2026 read: IOPS=10, BW=42.5KiB/s (43.5kB/s)(4976KiB/117124msec) slat (usec): min=3559, max=78227, avg=14803.81, stdev=8070.73 clat (msec): min=36, max=1015, avg=456.72, stdev=135.95 lat (msec): min=41, max=1030, avg=471.53, stdev=137.30 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 477], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 634], 95.00th=[ 693], | 99.00th=[ 885], 99.50th=[ 944], 99.90th=[ 986], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 144, per=24.84%, avg=42.48, stdev=21.09, samples=229 iops : min= 1, max= 36, avg=10.09, stdev= 5.29, samples=229 write: IOPS=11, BW=44.9KiB/s (46.0kB/s)(5264KiB/117124msec); 0 zone resets slat (usec): min=44, max=17542, avg=384.19, stdev=1247.30 clat (msec): min=313, max=1825, avg=975.74, stdev=279.32 lat (msec): min=314, max=1825, avg=976.12, stdev=279.41 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1485], | 99.00th=[ 1703], 99.50th=[ 1754], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 127, per=24.91%, avg=44.33, stdev=19.63, samples=232 iops : min= 1, max= 31, avg=10.56, stdev= 4.94, samples=232 lat (msec) : 50=0.08%, 100=0.23%, 250=1.17%, 500=32.77%, 750=24.26% lat (msec) : 1000=18.36%, 2000=23.12% cpu : usr=0.10%, sys=0.69%, ctx=4850, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147953: Thu Apr 9 16:57:37 2026 read: IOPS=10, BW=42.6KiB/s (43.6kB/s)(4984KiB/117010msec) slat (usec): min=71, max=72569, avg=15100.24, stdev=8756.43 clat (msec): min=51, max=965, avg=457.99, stdev=133.64 lat (msec): min=59, max=1017, avg=473.10, stdev=135.15 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 634], 95.00th=[ 718], | 99.00th=[ 827], 99.50th=[ 911], 99.90th=[ 953], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 119, per=24.94%, avg=42.65, stdev=21.65, samples=229 iops : min= 1, max= 29, avg=10.14, stdev= 5.45, samples=229 write: IOPS=11, BW=44.9KiB/s (45.0kB/s)(5256KiB/117010msec); 0 zone resets slat (usec): min=38, max=21961, avg=356.31, stdev=1081.53 clat (msec): min=183, max=1823, avg=974.74, stdev=289.95 lat (msec): min=184, max=1823, avg=975.10, stdev=289.94 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 527], 10.00th=[ 609], 20.00th=[ 726], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1653], 99.50th=[ 1737], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 104, per=24.89%, avg=44.31, stdev=21.11, samples=231 iops : min= 1, max= 26, avg=10.55, stdev= 5.28, samples=231 lat (msec) : 100=0.43%, 250=0.90%, 500=33.79%, 750=23.91%, 1000=18.28% lat (msec) : 2000=22.70% cpu : usr=0.09%, sys=0.68%, ctx=4731, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147954: Thu Apr 9 16:57:37 2026 read: IOPS=10, BW=43.2KiB/s (44.3kB/s)(5060KiB/117069msec) slat (usec): min=2622, max=69068, avg=14987.64, stdev=8389.33 clat (msec): min=50, max=1015, avg=462.29, stdev=128.54 lat (msec): min=60, max=1032, avg=477.28, stdev=129.87 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 292], 10.00th=[ 321], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 447], 60.00th=[ 481], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 818], 99.50th=[ 936], 99.90th=[ 1003], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 95, per=25.13%, avg=42.97, stdev=19.59, samples=230 iops : min= 1, max= 23, avg=10.21, stdev= 4.91, samples=230 write: IOPS=11, BW=44.2KiB/s (45.3kB/s)(5180KiB/117069msec); 0 zone resets slat (usec): min=40, max=14512, avg=351.24, stdev=1069.50 clat (msec): min=374, max=1861, avg=976.73, stdev=283.16 lat (msec): min=374, max=1861, avg=977.08, stdev=283.15 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 550], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 793], 40.00th=[ 869], 50.00th=[ 953], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1485], | 99.00th=[ 1687], 99.50th=[ 1754], 99.90th=[ 1838], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 112, per=24.72%, avg=44.00, stdev=20.25, samples=230 iops : min= 1, max= 28, avg=10.47, stdev= 5.09, samples=230 lat (msec) : 100=0.27%, 250=0.70%, 500=33.32%, 750=25.35%, 1000=17.62% lat (msec) : 2000=22.73% cpu : usr=0.10%, sys=0.62%, ctx=4731, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=171KiB/s (175kB/s), 42.5KiB/s-43.2KiB/s (43.5kB/s-44.3kB/s), io=19.6MiB (20.5MB), run=117010-117126msec WRITE: bw=178KiB/s (183kB/s), 44.2KiB/s-44.9KiB/s (45.3kB/s-46.0kB/s), io=20.4MiB (21.4MB), run=117010-117126msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147975: Thu Apr 9 16:57:50 2026 read: IOPS=1, BW=59.9MiB/s (62.8MB/s)(40.0MiB/668msec) slat (nsec): min=62652k, max=62652k, avg=62652377.00, stdev= 0.00 clat (nsec): min=555934k, max=555934k, avg=555933617.00, stdev= 0.00 lat (nsec): min=618591k, max=618591k, avg=618591310.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 558], 10.00th=[ 558], 20.00th=[ 558], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.35%, ctx=4, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=59.9MiB/s (62.8MB/s), 59.9MiB/s-59.9MiB/s (62.8MB/s-62.8MB/s), io=40.0MiB (41.9MB), run=668-668msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout