== sanity test 398c: run fio to test AIO ================= 23:29:30 (1789442970) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.502662 s, 83.4 MB/s osc.lustre-OST0000-osc-ffff9507df073800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145901: Mon Sep 14 23:32:46 2026 write: IOPS=13, BW=55.3KiB/s (56.6kB/s)(10.0MiB/185302msec); 0 zone resets slat (usec): min=28, max=17840, avg=333.59, stdev=1206.00 clat (msec): min=341, max=2668, avg=1156.21, stdev=313.24 lat (msec): min=341, max=2669, avg=1156.54, stdev=313.23 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 676], 10.00th=[ 802], 20.00th=[ 919], | 30.00th=[ 995], 40.00th=[ 1062], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1536], 95.00th=[ 1687], | 99.00th=[ 2022], 99.50th=[ 2232], 99.90th=[ 2601], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 127, per=25.57%, avg=56.25, stdev=25.08, samples=357 iops : min= 1, max= 31, avg=13.54, stdev= 6.27, samples=357 lat (msec) : 500=2.54%, 750=4.57%, 1000=23.59%, 2000=68.20%, >=2000=1.09% cpu : usr=0.05%, sys=0.28%, ctx=1951, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145902: Mon Sep 14 23:32:46 2026 write: IOPS=13, BW=55.2KiB/s (56.6kB/s)(10.0MiB/185406msec); 0 zone resets slat (usec): min=29, max=14671, avg=303.52, stdev=1026.73 clat (msec): min=269, max=2446, avg=1157.31, stdev=299.14 lat (msec): min=281, max=2446, avg=1157.61, stdev=299.09 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 709], 10.00th=[ 810], 20.00th=[ 927], | 30.00th=[ 1003], 40.00th=[ 1070], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1536], 95.00th=[ 1687], | 99.00th=[ 1972], 99.50th=[ 2140], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 127, per=25.00%, avg=55.00, stdev=28.39, samples=365 iops : min= 1, max= 31, avg=13.23, stdev= 7.09, samples=365 lat (msec) : 500=1.80%, 750=5.47%, 1000=22.77%, 2000=69.22%, >=2000=0.74% cpu : usr=0.05%, sys=0.24%, ctx=1946, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145903: Mon Sep 14 23:32:46 2026 write: IOPS=13, BW=55.4KiB/s (56.7kB/s)(10.0MiB/184838msec); 0 zone resets slat (usec): min=31, max=23771, avg=296.02, stdev=1070.68 clat (msec): min=148, max=2380, avg=1154.36, stdev=310.10 lat (msec): min=148, max=2380, avg=1154.66, stdev=310.07 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 676], 10.00th=[ 793], 20.00th=[ 919], | 30.00th=[ 995], 40.00th=[ 1070], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1552], 95.00th=[ 1703], | 99.00th=[ 2005], 99.50th=[ 2123], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 143, per=25.00%, avg=55.00, stdev=24.35, samples=365 iops : min= 1, max= 35, avg=13.23, stdev= 6.10, samples=365 lat (msec) : 250=0.35%, 500=1.64%, 750=5.47%, 1000=23.05%, 2000=68.40% lat (msec) : >=2000=1.09% cpu : usr=0.05%, sys=0.25%, ctx=2079, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145904: Mon Sep 14 23:32:46 2026 write: IOPS=13, BW=55.2KiB/s (56.6kB/s)(10.0MiB/185398msec); 0 zone resets slat (usec): min=23, max=25163, avg=320.39, stdev=1200.34 clat (msec): min=300, max=2394, avg=1157.43, stdev=302.55 lat (msec): min=300, max=2394, avg=1157.76, stdev=302.52 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 676], 10.00th=[ 810], 20.00th=[ 919], | 30.00th=[ 995], 40.00th=[ 1062], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1385], 90.00th=[ 1536], 95.00th=[ 1687], | 99.00th=[ 1972], 99.50th=[ 2140], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 128, per=25.56%, avg=56.24, stdev=26.77, samples=357 iops : min= 1, max= 32, avg=13.54, stdev= 6.69, samples=357 lat (msec) : 500=1.91%, 750=5.55%, 1000=22.93%, 2000=68.79%, >=2000=0.82% cpu : usr=0.04%, sys=0.22%, ctx=1715, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=221KiB/s (226kB/s), 55.2KiB/s-55.4KiB/s (56.6kB/s-56.7kB/s), io=40.0MiB (41.9MB), run=184838-185406msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145940: Mon Sep 14 23:34:57 2026 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(5048KiB/121267msec) slat (msec): min=3, max=275, avg=15.58, stdev=11.13 clat (msec): min=37, max=1032, avg=470.93, stdev=121.94 lat (msec): min=55, max=1045, avg=486.52, stdev=122.34 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 305], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 558], 90.00th=[ 617], 95.00th=[ 676], | 99.00th=[ 835], 99.50th=[ 860], 99.90th=[ 1028], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 104, per=25.08%, avg=41.38, stdev=19.04, samples=239 iops : min= 1, max= 26, avg= 9.86, stdev= 4.78, samples=239 write: IOPS=10, BW=42.8KiB/s (43.8kB/s)(5192KiB/121267msec); 0 zone resets slat (usec): min=35, max=22105, avg=320.34, stdev=1159.06 clat (msec): min=252, max=2023, avg=1018.22, stdev=263.73 lat (msec): min=252, max=2023, avg=1018.55, stdev=263.66 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 625], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 919], 50.00th=[ 995], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 1703], 99.50th=[ 1821], 99.90th=[ 1921], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 96, per=24.49%, avg=42.13, stdev=19.30, samples=241 iops : min= 1, max= 24, avg=10.05, stdev= 4.85, samples=241 lat (msec) : 50=0.08%, 100=0.31%, 250=0.35%, 500=30.94%, 750=23.63% lat (msec) : 1000=20.04%, 2000=24.61%, >=2000=0.04% cpu : usr=0.09%, sys=0.61%, ctx=4308, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145941: Mon Sep 14 23:34:57 2026 read: IOPS=10, BW=41.0KiB/s (42.0kB/s)(4976KiB/121264msec) slat (usec): min=2877, max=67234, avg=15621.05, stdev=8474.37 clat (msec): min=61, max=881, avg=468.37, stdev=114.42 lat (msec): min=83, max=927, avg=484.00, stdev=115.34 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 309], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 558], 90.00th=[ 617], 95.00th=[ 667], | 99.00th=[ 793], 99.50th=[ 844], 99.90th=[ 877], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 95, per=24.50%, avg=40.43, stdev=19.03, samples=241 iops : min= 1, max= 23, avg= 9.63, stdev= 4.80, samples=241 write: IOPS=10, BW=43.4KiB/s (44.5kB/s)(5264KiB/121264msec); 0 zone resets slat (usec): min=33, max=25791, avg=346.07, stdev=1268.09 clat (msec): min=254, max=2024, avg=1014.11, stdev=260.01 lat (msec): min=254, max=2024, avg=1014.46, stdev=260.03 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 634], 10.00th=[ 709], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1368], 95.00th=[ 1452], | 99.00th=[ 1703], 99.50th=[ 1770], 99.90th=[ 2005], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 95, per=24.84%, avg=42.72, stdev=18.96, samples=241 iops : min= 1, max= 23, avg=10.20, stdev= 4.72, samples=241 lat (msec) : 100=0.20%, 250=0.43%, 500=31.45%, 750=23.36%, 1000=18.83% lat (msec) : 2000=25.66%, >=2000=0.08% cpu : usr=0.10%, sys=0.58%, ctx=4533, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145942: Mon Sep 14 23:34:57 2026 read: IOPS=10, BW=41.1KiB/s (42.1kB/s)(4984KiB/121264msec) slat (usec): min=35, max=69121, avg=15117.53, stdev=8044.83 clat (msec): min=21, max=912, avg=470.35, stdev=123.45 lat (msec): min=30, max=926, avg=485.48, stdev=124.14 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 305], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 464], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 684], | 99.00th=[ 835], 99.50th=[ 860], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 7, max= 96, per=24.96%, avg=41.18, stdev=19.02, samples=238 iops : min= 1, max= 24, avg= 9.82, stdev= 4.79, samples=238 write: IOPS=10, BW=43.3KiB/s (44.4kB/s)(5256KiB/121264msec); 0 zone resets slat (usec): min=34, max=13785, avg=324.46, stdev=878.66 clat (msec): min=115, max=2029, avg=1014.99, stdev=265.28 lat (msec): min=118, max=2029, avg=1015.32, stdev=265.25 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 625], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 986], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1720], 99.50th=[ 1787], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 103, per=24.70%, avg=42.49, stdev=19.45, samples=241 iops : min= 1, max= 25, avg=10.14, stdev= 4.86, samples=241 lat (msec) : 50=0.27%, 100=0.16%, 250=0.78%, 500=31.05%, 750=22.07% lat (msec) : 1000=20.90%, 2000=24.69%, >=2000=0.08% cpu : usr=0.09%, sys=0.54%, ctx=4226, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145943: Mon Sep 14 23:34:57 2026 read: IOPS=10, BW=41.7KiB/s (42.7kB/s)(5060KiB/121261msec) slat (usec): min=3097, max=78630, avg=15457.01, stdev=8221.35 clat (msec): min=10, max=890, avg=467.97, stdev=114.73 lat (msec): min=21, max=926, avg=483.44, stdev=116.15 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 300], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 667], | 99.00th=[ 776], 99.50th=[ 835], 99.90th=[ 885], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 104, per=25.41%, avg=41.92, stdev=19.66, samples=237 iops : min= 1, max= 26, avg=10.00, stdev= 4.95, samples=237 write: IOPS=10, BW=42.7KiB/s (43.7kB/s)(5180KiB/121261msec); 0 zone resets slat (usec): min=32, max=18792, avg=319.78, stdev=1011.41 clat (msec): min=192, max=2090, avg=1023.02, stdev=267.25 lat (msec): min=192, max=2090, avg=1023.34, stdev=267.26 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 625], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 1737], 99.50th=[ 1787], 99.90th=[ 1921], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 95, per=24.38%, avg=41.93, stdev=18.00, samples=241 iops : min= 1, max= 23, avg=10.00, stdev= 4.52, samples=241 lat (msec) : 20=0.08%, 50=0.08%, 100=0.16%, 250=0.39%, 500=32.07% lat (msec) : 750=22.66%, 1000=19.06%, 2000=25.47%, >=2000=0.04% cpu : usr=0.08%, sys=0.56%, ctx=4518, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=165KiB/s (169kB/s), 41.0KiB/s-41.7KiB/s (42.0kB/s-42.7kB/s), io=19.6MiB (20.5MB), run=121261-121267msec WRITE: bw=172KiB/s (176kB/s), 42.7KiB/s-43.4KiB/s (43.7kB/s-44.5kB/s), io=20.4MiB (21.4MB), run=121261-121267msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145966: Mon Sep 14 23:35:07 2026 read: IOPS=1, BW=63.3MiB/s (66.4MB/s)(40.0MiB/632msec) slat (nsec): min=31391k, max=31391k, avg=31390638.00, stdev= 0.00 clat (nsec): min=572087k, max=572087k, avg=572086548.00, stdev= 0.00 lat (nsec): min=603483k, max=603483k, avg=603482765.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.74%, ctx=4, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.3MiB/s (66.4MB/s), 63.3MiB/s-63.3MiB/s (66.4MB/s-66.4MB/s), io=40.0MiB (41.9MB), run=632-632msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout