== sanity test 398c: run fio to test AIO ================= 13:01:01 (1784048461) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.529144 s, 79.3 MB/s osc.lustre-OST0000-osc-ffff8d7e44f78000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145989: Tue Jul 14 13:04:17 2026 write: IOPS=13, BW=55.9KiB/s (57.2kB/s)(10.0MiB/183318msec); 0 zone resets slat (usec): min=37, max=19619, avg=287.63, stdev=959.70 clat (msec): min=106, max=2438, avg=1144.58, stdev=331.63 lat (msec): min=106, max=2438, avg=1144.87, stdev=331.65 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 667], 10.00th=[ 760], 20.00th=[ 877], | 30.00th=[ 953], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1569], 95.00th=[ 1737], | 99.00th=[ 2140], 99.50th=[ 2265], 99.90th=[ 2333], 99.95th=[ 2400], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=25.14%, avg=55.81, stdev=27.13, samples=361 iops : min= 1, max= 32, avg=13.55, stdev= 6.80, samples=361 lat (msec) : 250=0.39%, 500=0.86%, 750=7.54%, 1000=27.89%, 2000=62.11% lat (msec) : >=2000=1.21% cpu : usr=0.04%, sys=0.24%, ctx=2245, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145990: Tue Jul 14 13:04:17 2026 write: IOPS=13, BW=55.8KiB/s (57.1kB/s)(10.0MiB/183658msec); 0 zone resets slat (usec): min=45, max=12760, avg=247.76, stdev=733.50 clat (msec): min=305, max=2470, avg=1146.49, stdev=336.99 lat (msec): min=305, max=2471, avg=1146.74, stdev=337.05 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 642], 10.00th=[ 760], 20.00th=[ 877], | 30.00th=[ 953], 40.00th=[ 1036], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1267], 80.00th=[ 1401], 90.00th=[ 1586], 95.00th=[ 1770], | 99.00th=[ 2106], 99.50th=[ 2265], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 128, per=26.24%, avg=58.25, stdev=31.29, samples=346 iops : min= 1, max= 32, avg=14.15, stdev= 7.83, samples=346 lat (msec) : 500=1.37%, 750=7.23%, 1000=27.66%, 2000=62.27%, >=2000=1.48% cpu : usr=0.04%, sys=0.21%, ctx=1883, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145991: Tue Jul 14 13:04:17 2026 write: IOPS=13, BW=55.8KiB/s (57.1kB/s)(10.0MiB/183662msec); 0 zone resets slat (usec): min=53, max=17545, avg=251.70, stdev=749.53 clat (msec): min=170, max=2438, avg=1147.09, stdev=338.75 lat (msec): min=170, max=2438, avg=1147.34, stdev=338.76 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 659], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 961], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1418], 90.00th=[ 1586], 95.00th=[ 1754], | 99.00th=[ 2165], 99.50th=[ 2299], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=26.08%, avg=57.91, stdev=31.72, samples=348 iops : min= 1, max= 32, avg=14.06, stdev= 7.92, samples=348 lat (msec) : 250=0.62%, 500=0.62%, 750=7.85%, 1000=27.50%, 2000=62.11% lat (msec) : >=2000=1.29% cpu : usr=0.04%, sys=0.22%, ctx=1908, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145992: Tue Jul 14 13:04:17 2026 write: IOPS=13, BW=55.7KiB/s (57.1kB/s)(10.0MiB/183735msec); 0 zone resets slat (usec): min=51, max=20069, avg=269.76, stdev=884.65 clat (msec): min=287, max=2493, avg=1147.05, stdev=331.67 lat (msec): min=287, max=2493, avg=1147.32, stdev=331.64 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 684], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 953], 40.00th=[ 1020], 50.00th=[ 1099], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1418], 90.00th=[ 1586], 95.00th=[ 1754], | 99.00th=[ 2106], 99.50th=[ 2265], 99.90th=[ 2400], 99.95th=[ 2467], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 128, per=25.71%, avg=57.08, stdev=30.46, samples=353 iops : min= 1, max= 32, avg=13.86, stdev= 7.61, samples=353 lat (msec) : 500=0.98%, 750=6.99%, 1000=30.31%, 2000=60.20%, >=2000=1.52% cpu : usr=0.04%, sys=0.24%, ctx=2055, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=223KiB/s (228kB/s), 55.7KiB/s-55.9KiB/s (57.1kB/s-57.2kB/s), io=40.0MiB (41.9MB), run=183318-183735msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146029: Tue Jul 14 13:06:07 2026 read: IOPS=12, BW=49.0KiB/s (50.2kB/s)(5048KiB/102916msec) slat (usec): min=56, max=58315, avg=13782.53, stdev=7739.55 clat (msec): min=11, max=903, avg=404.10, stdev=110.91 lat (msec): min=12, max=908, avg=417.89, stdev=112.51 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 259], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 447], 80.00th=[ 481], 90.00th=[ 550], 95.00th=[ 600], | 99.00th=[ 718], 99.50th=[ 751], 99.90th=[ 835], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 103, per=25.15%, avg=48.54, stdev=21.06, samples=204 iops : min= 1, max= 25, avg=11.62, stdev= 5.27, samples=204 write: IOPS=12, BW=50.4KiB/s (51.7kB/s)(5192KiB/102916msec); 0 zone resets slat (usec): min=57, max=12717, avg=282.78, stdev=642.36 clat (msec): min=213, max=1486, avg=861.17, stdev=217.28 lat (msec): min=213, max=1486, avg=861.45, stdev=217.27 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 510], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1028], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1418], 99.50th=[ 1435], 99.90th=[ 1452], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 7, max= 127, per=24.89%, avg=50.02, stdev=20.32, samples=203 iops : min= 1, max= 31, avg=12.00, stdev= 5.08, samples=203 lat (msec) : 20=0.04%, 50=0.16%, 100=0.31%, 250=1.76%, 500=41.05% lat (msec) : 750=20.70%, 1000=24.49%, 2000=11.48% cpu : usr=0.10%, sys=0.63%, ctx=5022, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146030: Tue Jul 14 13:06:07 2026 read: IOPS=12, BW=48.2KiB/s (49.3kB/s)(4976KiB/103300msec) slat (usec): min=2671, max=58735, avg=13691.54, stdev=8104.66 clat (msec): min=15, max=829, avg=406.18, stdev=108.87 lat (msec): min=23, max=841, avg=419.87, stdev=110.57 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 257], 10.00th=[ 284], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 489], 90.00th=[ 558], 95.00th=[ 609], | 99.00th=[ 701], 99.50th=[ 709], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 104, per=24.89%, avg=48.04, stdev=19.26, samples=203 iops : min= 1, max= 26, avg=11.50, stdev= 4.85, samples=203 write: IOPS=12, BW=50.0KiB/s (52.2kB/s)(5264KiB/103300msec); 0 zone resets slat (usec): min=59, max=26864, avg=333.23, stdev=1190.55 clat (msec): min=315, max=1464, avg=856.01, stdev=212.70 lat (msec): min=315, max=1467, avg=856.34, stdev=212.73 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 502], 10.00th=[ 584], 20.00th=[ 676], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 902], | 70.00th=[ 953], 80.00th=[ 1020], 90.00th=[ 1133], 95.00th=[ 1217], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 135, per=25.16%, avg=50.57, stdev=20.71, samples=204 iops : min= 1, max= 33, avg=12.13, stdev= 5.19, samples=204 lat (msec) : 20=0.04%, 50=0.12%, 100=0.31%, 250=1.29%, 500=40.82% lat (msec) : 750=21.02%, 1000=25.35%, 2000=11.05% cpu : usr=0.09%, sys=0.66%, ctx=4794, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146031: Tue Jul 14 13:06:07 2026 read: IOPS=12, BW=48.1KiB/s (49.3kB/s)(4984KiB/103548msec) slat (usec): min=2975, max=57589, avg=13920.30, stdev=7712.18 clat (msec): min=8, max=848, avg=402.81, stdev=110.96 lat (msec): min=15, max=867, avg=416.74, stdev=112.51 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 255], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 368], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 489], 90.00th=[ 550], 95.00th=[ 600], | 99.00th=[ 701], 99.50th=[ 768], 99.90th=[ 835], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 7, max= 135, per=24.75%, avg=47.76, stdev=23.08, samples=205 iops : min= 1, max= 33, avg=11.43, stdev= 5.79, samples=205 write: IOPS=12, BW=50.8KiB/s (51.0kB/s)(5256KiB/103548msec); 0 zone resets slat (usec): min=50, max=15065, avg=295.23, stdev=689.30 clat (msec): min=277, max=1509, avg=862.53, stdev=212.33 lat (msec): min=278, max=1509, avg=862.82, stdev=212.31 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 523], 10.00th=[ 584], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 818], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1167], 95.00th=[ 1234], | 99.00th=[ 1351], 99.50th=[ 1401], 99.90th=[ 1452], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 111, per=24.82%, avg=49.88, stdev=20.33, samples=206 iops : min= 1, max= 27, avg=11.97, stdev= 5.10, samples=206 lat (msec) : 10=0.04%, 20=0.08%, 50=0.16%, 100=0.12%, 250=1.72% lat (msec) : 500=40.27%, 750=21.56%, 1000=23.83%, 2000=12.23% cpu : usr=0.10%, sys=0.63%, ctx=4824, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146032: Tue Jul 14 13:06:07 2026 read: IOPS=12, BW=48.9KiB/s (50.1kB/s)(5060KiB/103496msec) slat (usec): min=2466, max=52959, avg=13564.87, stdev=7681.49 clat (msec): min=25, max=852, avg=404.28, stdev=107.59 lat (msec): min=44, max=859, avg=417.85, stdev=108.93 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 259], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 447], 80.00th=[ 481], 90.00th=[ 542], 95.00th=[ 600], | 99.00th=[ 709], 99.50th=[ 785], 99.90th=[ 835], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 8, max= 119, per=25.85%, avg=49.90, stdev=20.36, samples=199 iops : min= 2, max= 29, avg=11.96, stdev= 5.11, samples=199 write: IOPS=12, BW=50.0KiB/s (51.3kB/s)(5180KiB/103496msec); 0 zone resets slat (usec): min=46, max=10508, avg=307.66, stdev=721.13 clat (msec): min=331, max=1462, avg=866.83, stdev=212.68 lat (msec): min=331, max=1462, avg=867.14, stdev=212.66 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 518], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1385], 99.50th=[ 1418], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 119, per=24.74%, avg=49.73, stdev=20.62, samples=204 iops : min= 1, max= 29, avg=11.92, stdev= 5.17, samples=204 lat (msec) : 50=0.16%, 100=0.31%, 250=1.76%, 500=41.02%, 750=20.59% lat (msec) : 1000=23.95%, 2000=12.23% cpu : usr=0.08%, sys=0.66%, ctx=4669, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=194KiB/s (198kB/s), 48.1KiB/s-49.0KiB/s (49.3kB/s-50.2kB/s), io=19.6MiB (20.5MB), run=102916-103548msec WRITE: bw=202KiB/s (207kB/s), 50.0KiB/s-50.0KiB/s (51.3kB/s-52.2kB/s), io=20.4MiB (21.4MB), run=102916-103548msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146052: Tue Jul 14 13:06:22 2026 read: IOPS=1, BW=56.0MiB/s (58.7MB/s)(40.0MiB/714msec) slat (nsec): min=39363k, max=39363k, avg=39363033.00, stdev= 0.00 clat (nsec): min=634340k, max=634340k, avg=634339700.00, stdev= 0.00 lat (nsec): min=673707k, max=673707k, avg=673707088.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 634], 5.00th=[ 634], 10.00th=[ 634], 20.00th=[ 634], | 30.00th=[ 634], 40.00th=[ 634], 50.00th=[ 634], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.82%, ctx=8, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=56.0MiB/s (58.7MB/s), 56.0MiB/s-56.0MiB/s (58.7MB/s-58.7MB/s), io=40.0MiB (41.9MB), run=714-714msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout