== sanity test 398c: run fio to test AIO ================= 07:53:06 (1787745186) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.590962 s, 71.0 MB/s osc.lustre-OST0000-osc-ffff9493d8872000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144865: Wed Aug 26 07:57:01 2026 write: IOPS=11, BW=46.4KiB/s (47.5kB/s)(10.0MiB/220844msec); 0 zone resets slat (usec): min=23, max=62721, avg=287.74, stdev=1659.88 clat (msec): min=64, max=3484, avg=1378.67, stdev=514.20 lat (msec): min=65, max=3485, avg=1378.96, stdev=514.19 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 514], 10.00th=[ 827], 20.00th=[ 1020], | 30.00th=[ 1116], 40.00th=[ 1217], 50.00th=[ 1318], 60.00th=[ 1435], | 70.00th=[ 1552], 80.00th=[ 1737], 90.00th=[ 2056], 95.00th=[ 2366], | 99.00th=[ 2970], 99.50th=[ 3071], 99.90th=[ 3406], 99.95th=[ 3473], | 99.99th=[ 3473] bw ( KiB/s): min= 7, max= 270, per=25.25%, avg=46.71, stdev=27.35, samples=429 iops : min= 1, max= 67, avg=11.17, stdev= 6.85, samples=429 lat (msec) : 100=0.08%, 250=0.78%, 500=3.95%, 750=2.46%, 1000=10.55% lat (msec) : 2000=70.12%, >=2000=12.07% cpu : usr=0.03%, sys=0.19%, ctx=1852, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144866: Wed Aug 26 07:57:01 2026 write: IOPS=11, BW=46.4KiB/s (47.5kB/s)(10.0MiB/220635msec); 0 zone resets slat (usec): min=22, max=39027, avg=262.76, stdev=1457.14 clat (msec): min=151, max=3359, avg=1378.13, stdev=510.99 lat (msec): min=151, max=3359, avg=1378.39, stdev=511.05 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 485], 10.00th=[ 852], 20.00th=[ 1020], | 30.00th=[ 1133], 40.00th=[ 1217], 50.00th=[ 1301], 60.00th=[ 1418], | 70.00th=[ 1536], 80.00th=[ 1754], 90.00th=[ 2056], 95.00th=[ 2366], | 99.00th=[ 2903], 99.50th=[ 3071], 99.90th=[ 3339], 99.95th=[ 3373], | 99.99th=[ 3373] bw ( KiB/s): min= 6, max= 254, per=26.64%, avg=49.28, stdev=30.92, samples=407 iops : min= 1, max= 63, avg=11.81, stdev= 7.75, samples=407 lat (msec) : 250=1.25%, 500=3.83%, 750=2.30%, 1000=10.74%, 2000=71.29% lat (msec) : >=2000=10.59% cpu : usr=0.05%, sys=0.18%, ctx=1485, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144867: Wed Aug 26 07:57:01 2026 write: IOPS=11, BW=46.4KiB/s (47.5kB/s)(10.0MiB/220843msec); 0 zone resets slat (usec): min=23, max=37986, avg=278.21, stdev=1397.23 clat (msec): min=217, max=3347, avg=1378.65, stdev=507.24 lat (msec): min=217, max=3348, avg=1378.93, stdev=507.28 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 531], 10.00th=[ 827], 20.00th=[ 1028], | 30.00th=[ 1133], 40.00th=[ 1217], 50.00th=[ 1301], 60.00th=[ 1418], | 70.00th=[ 1569], 80.00th=[ 1737], 90.00th=[ 2039], 95.00th=[ 2299], | 99.00th=[ 2869], 99.50th=[ 3071], 99.90th=[ 3306], 99.95th=[ 3306], | 99.99th=[ 3339] bw ( KiB/s): min= 7, max= 254, per=25.87%, avg=47.87, stdev=29.83, samples=419 iops : min= 1, max= 63, avg=11.46, stdev= 7.49, samples=419 lat (msec) : 250=0.62%, 500=3.95%, 750=2.89%, 1000=10.51%, 2000=71.02% lat (msec) : >=2000=11.02% cpu : usr=0.04%, sys=0.17%, ctx=1978, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144868: Wed Aug 26 07:57:01 2026 write: IOPS=11, BW=46.4KiB/s (47.5kB/s)(10.0MiB/220839msec); 0 zone resets slat (usec): min=23, max=23633, avg=224.65, stdev=1043.47 clat (msec): min=211, max=3492, avg=1378.84, stdev=506.56 lat (msec): min=211, max=3492, avg=1379.07, stdev=506.59 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 485], 10.00th=[ 835], 20.00th=[ 1036], | 30.00th=[ 1133], 40.00th=[ 1217], 50.00th=[ 1318], 60.00th=[ 1418], | 70.00th=[ 1536], 80.00th=[ 1737], 90.00th=[ 2056], 95.00th=[ 2299], | 99.00th=[ 2802], 99.50th=[ 3037], 99.90th=[ 3406], 99.95th=[ 3406], | 99.99th=[ 3507] bw ( KiB/s): min= 7, max= 254, per=25.63%, avg=47.41, stdev=29.44, samples=423 iops : min= 1, max= 63, avg=11.34, stdev= 7.39, samples=423 lat (msec) : 250=0.62%, 500=4.45%, 750=2.46%, 1000=9.96%, 2000=71.09% lat (msec) : >=2000=11.41% cpu : usr=0.04%, sys=0.19%, ctx=1753, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=185KiB/s (190kB/s), 46.4KiB/s-46.4KiB/s (47.5kB/s-47.5kB/s), io=40.0MiB (41.9MB), run=220635-220844msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144909: Wed Aug 26 07:59:29 2026 read: IOPS=9, BW=36.0KiB/s (37.9kB/s)(5048KiB/136470msec) slat (usec): min=48, max=126291, avg=18340.75, stdev=10849.32 clat (msec): min=16, max=1838, avg=549.63, stdev=182.55 lat (msec): min=16, max=1909, avg=567.97, stdev=184.91 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 330], 10.00th=[ 368], 20.00th=[ 418], | 30.00th=[ 451], 40.00th=[ 481], 50.00th=[ 518], 60.00th=[ 558], | 70.00th=[ 609], 80.00th=[ 667], 90.00th=[ 776], 95.00th=[ 852], | 99.00th=[ 1200], 99.50th=[ 1301], 99.90th=[ 1821], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 96, per=25.80%, avg=37.41, stdev=19.33, samples=263 iops : min= 1, max= 24, avg= 8.79, stdev= 4.87, samples=263 write: IOPS=9, BW=38.0KiB/s (38.0kB/s)(5192KiB/136470msec); 0 zone resets slat (usec): min=35, max=60141, avg=326.99, stdev=1875.32 clat (msec): min=148, max=2636, avg=1128.40, stdev=356.30 lat (msec): min=148, max=2636, avg=1128.73, stdev=356.27 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 835], | 30.00th=[ 919], 40.00th=[ 1003], 50.00th=[ 1083], 60.00th=[ 1183], | 70.00th=[ 1301], 80.00th=[ 1401], 90.00th=[ 1586], 95.00th=[ 1787], | 99.00th=[ 2165], 99.50th=[ 2333], 99.90th=[ 2500], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 108, per=25.11%, avg=37.91, stdev=18.54, samples=266 iops : min= 1, max= 27, avg= 8.92, stdev= 4.68, samples=266 lat (msec) : 20=0.04%, 50=0.16%, 100=0.16%, 250=0.35%, 500=22.73% lat (msec) : 750=26.05%, 1000=19.18%, 2000=30.43%, >=2000=0.90% cpu : usr=0.10%, sys=0.56%, ctx=4190, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144910: Wed Aug 26 07:59:29 2026 read: IOPS=9, BW=36.2KiB/s (37.1kB/s)(4976KiB/137500msec) slat (usec): min=3413, max=94295, avg=17640.52, stdev=10136.42 clat (msec): min=28, max=1883, avg=539.33, stdev=185.39 lat (msec): min=48, max=1900, avg=556.98, stdev=186.53 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 321], 10.00th=[ 359], 20.00th=[ 405], | 30.00th=[ 439], 40.00th=[ 477], 50.00th=[ 510], 60.00th=[ 550], | 70.00th=[ 584], 80.00th=[ 642], 90.00th=[ 751], 95.00th=[ 844], | 99.00th=[ 1099], 99.50th=[ 1401], 99.90th=[ 1838], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 112, per=25.17%, avg=36.50, stdev=19.81, samples=265 iops : min= 1, max= 28, avg= 8.56, stdev= 5.01, samples=265 write: IOPS=9, BW=38.3KiB/s (39.2kB/s)(5264KiB/137500msec); 0 zone resets slat (usec): min=31, max=18756, avg=308.78, stdev=1091.21 clat (msec): min=285, max=2679, avg=1142.36, stdev=345.13 lat (msec): min=285, max=2679, avg=1142.67, stdev=345.11 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 642], 10.00th=[ 718], 20.00th=[ 852], | 30.00th=[ 936], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1401], 90.00th=[ 1586], 95.00th=[ 1770], | 99.00th=[ 2106], 99.50th=[ 2198], 99.90th=[ 2601], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 120, per=25.76%, avg=38.90, stdev=19.07, samples=264 iops : min= 1, max= 30, avg= 9.17, stdev= 4.81, samples=264 lat (msec) : 50=0.04%, 100=0.12%, 250=0.31%, 500=23.12%, 750=25.78% lat (msec) : 1000=16.72%, 2000=32.89%, >=2000=1.02% cpu : usr=0.10%, sys=0.53%, ctx=4278, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144911: Wed Aug 26 07:59:29 2026 read: IOPS=9, BW=36.2KiB/s (37.1kB/s)(4984KiB/137499msec) slat (msec): min=3, max=167, avg=18.56, stdev=11.34 clat (msec): min=10, max=1916, avg=546.26, stdev=197.54 lat (msec): min=24, max=1932, avg=564.84, stdev=199.90 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 330], 10.00th=[ 368], 20.00th=[ 409], | 30.00th=[ 439], 40.00th=[ 477], 50.00th=[ 514], 60.00th=[ 550], | 70.00th=[ 600], 80.00th=[ 659], 90.00th=[ 751], 95.00th=[ 860], | 99.00th=[ 1418], 99.50th=[ 1720], 99.90th=[ 1888], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 112, per=25.34%, avg=36.74, stdev=20.02, samples=265 iops : min= 1, max= 28, avg= 8.62, stdev= 5.05, samples=265 write: IOPS=9, BW=38.2KiB/s (39.1kB/s)(5256KiB/137499msec); 0 zone resets slat (usec): min=33, max=17195, avg=302.96, stdev=1048.91 clat (msec): min=314, max=2384, avg=1135.30, stdev=348.53 lat (msec): min=314, max=2384, avg=1135.61, stdev=348.51 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 852], | 30.00th=[ 936], 40.00th=[ 1011], 50.00th=[ 1099], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1569], 95.00th=[ 1770], | 99.00th=[ 2039], 99.50th=[ 2333], 99.90th=[ 2366], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 104, per=25.12%, avg=37.93, stdev=18.79, samples=269 iops : min= 1, max= 26, avg= 8.92, stdev= 4.74, samples=269 lat (msec) : 20=0.04%, 50=0.16%, 100=0.16%, 250=0.20%, 500=23.05% lat (msec) : 750=26.29%, 1000=16.99%, 2000=32.30%, >=2000=0.82% cpu : usr=0.08%, sys=0.58%, ctx=4416, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144912: Wed Aug 26 07:59:29 2026 read: IOPS=9, BW=36.9KiB/s (37.8kB/s)(5060KiB/136967msec) slat (usec): min=3342, max=82641, avg=18145.95, stdev=10866.89 clat (msec): min=13, max=1869, avg=544.89, stdev=186.90 lat (msec): min=37, max=1894, avg=563.05, stdev=189.67 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 338], 10.00th=[ 368], 20.00th=[ 409], | 30.00th=[ 443], 40.00th=[ 477], 50.00th=[ 510], 60.00th=[ 550], | 70.00th=[ 592], 80.00th=[ 659], 90.00th=[ 768], 95.00th=[ 852], | 99.00th=[ 1167], 99.50th=[ 1552], 99.90th=[ 1854], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 103, per=26.19%, avg=37.98, stdev=19.40, samples=260 iops : min= 1, max= 25, avg= 8.93, stdev= 4.89, samples=260 write: IOPS=9, BW=37.8KiB/s (38.7kB/s)(5180KiB/136967msec); 0 zone resets slat (usec): min=28, max=24949, avg=281.72, stdev=1065.64 clat (msec): min=284, max=2346, avg=1138.58, stdev=342.31 lat (msec): min=285, max=2346, avg=1138.87, stdev=342.23 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 852], | 30.00th=[ 944], 40.00th=[ 1020], 50.00th=[ 1099], 60.00th=[ 1200], | 70.00th=[ 1318], 80.00th=[ 1418], 90.00th=[ 1569], 95.00th=[ 1770], | 99.00th=[ 2039], 99.50th=[ 2072], 99.90th=[ 2198], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 144, per=25.56%, avg=38.60, stdev=17.98, samples=261 iops : min= 1, max= 36, avg= 9.09, stdev= 4.53, samples=261 lat (msec) : 20=0.04%, 50=0.12%, 100=0.08%, 250=0.12%, 500=23.63% lat (msec) : 750=26.13%, 1000=17.19%, 2000=31.95%, >=2000=0.74% cpu : usr=0.07%, sys=0.56%, ctx=4358, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=146KiB/s (149kB/s), 36.2KiB/s-36.0KiB/s (37.1kB/s-37.9kB/s), io=19.6MiB (20.5MB), run=136470-137500msec WRITE: bw=152KiB/s (156kB/s), 37.8KiB/s-38.3KiB/s (38.7kB/s-39.2kB/s), io=20.4MiB (21.4MB), run=136470-137500msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144936: Wed Aug 26 07:59:41 2026 read: IOPS=1, BW=64.2MiB/s (67.3MB/s)(40.0MiB/623msec) slat (nsec): min=33402k, max=33402k, avg=33401634.00, stdev= 0.00 clat (nsec): min=570805k, max=570805k, avg=570804913.00, stdev= 0.00 lat (nsec): min=604214k, max=604214k, avg=604213566.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.09%, ctx=6, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=64.2MiB/s (67.3MB/s), 64.2MiB/s-64.2MiB/s (67.3MB/s-67.3MB/s), io=40.0MiB (41.9MB), run=623-623msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout