== sanity test 398c: run fio to test AIO ================= 14:15:23 (1787854523) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.546859 s, 76.7 MB/s osc.lustre-OST0000-osc-ffff8ee85305d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145904: Thu Aug 27 14:18:07 2026 write: IOPS=16, BW=67.3KiB/s (68.9kB/s)(10.0MiB/152181msec); 0 zone resets slat (usec): min=29, max=11757, avg=213.80, stdev=720.56 clat (msec): min=90, max=2077, avg=950.20, stdev=250.35 lat (msec): min=90, max=2077, avg=950.42, stdev=250.33 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 584], 10.00th=[ 651], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 927], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1653], 99.50th=[ 1938], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 175, per=25.50%, avg=68.60, stdev=31.23, samples=294 iops : min= 1, max= 43, avg=16.70, stdev= 7.78, samples=294 lat (msec) : 100=0.04%, 250=0.43%, 500=2.15%, 750=15.35%, 1000=44.38% lat (msec) : 2000=37.30%, >=2000=0.35% cpu : usr=0.04%, sys=0.24%, ctx=1992, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145905: Thu Aug 27 14:18:07 2026 write: IOPS=16, BW=67.3KiB/s (68.9kB/s)(10.0MiB/152247msec); 0 zone resets slat (usec): min=28, max=18720, avg=225.93, stdev=972.79 clat (msec): min=290, max=2086, avg=950.44, stdev=249.23 lat (msec): min=290, max=2087, avg=950.66, stdev=249.19 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 567], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 936], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1687], 99.50th=[ 1905], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 128, per=24.82%, avg=66.76, stdev=25.76, samples=302 iops : min= 1, max= 32, avg=16.24, stdev= 6.43, samples=302 lat (msec) : 500=2.66%, 750=16.41%, 1000=42.89%, 2000=37.77%, >=2000=0.27% cpu : usr=0.04%, sys=0.24%, ctx=1737, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145906: Thu Aug 27 14:18:07 2026 write: IOPS=16, BW=67.6KiB/s (69.2kB/s)(10.0MiB/151551msec); 0 zone resets slat (usec): min=28, max=19005, avg=211.75, stdev=804.74 clat (msec): min=146, max=2086, avg=946.63, stdev=254.23 lat (msec): min=146, max=2086, avg=946.85, stdev=254.24 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 575], 10.00th=[ 651], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 936], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1385], | 99.00th=[ 1636], 99.50th=[ 1989], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 252, per=25.32%, avg=68.12, stdev=34.72, samples=296 iops : min= 1, max= 63, avg=16.58, stdev= 8.70, samples=296 lat (msec) : 250=0.70%, 500=2.62%, 750=14.84%, 1000=43.79%, 2000=37.58% lat (msec) : >=2000=0.47% cpu : usr=0.04%, sys=0.22%, ctx=1609, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145907: Thu Aug 27 14:18:07 2026 write: IOPS=16, BW=67.3KiB/s (68.9kB/s)(10.0MiB/152247msec); 0 zone resets slat (usec): min=28, max=17346, avg=197.57, stdev=768.68 clat (msec): min=81, max=2077, avg=950.35, stdev=246.33 lat (msec): min=81, max=2077, avg=950.55, stdev=246.31 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 584], 10.00th=[ 651], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 936], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1653], 99.50th=[ 1804], 99.90th=[ 2022], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 135, per=24.90%, avg=66.99, stdev=32.07, samples=301 iops : min= 1, max= 33, avg=16.29, stdev= 8.02, samples=301 lat (msec) : 100=0.04%, 500=2.54%, 750=14.77%, 1000=45.04%, 2000=37.46% lat (msec) : >=2000=0.16% cpu : usr=0.04%, sys=0.20%, ctx=1473, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=269KiB/s (275kB/s), 67.3KiB/s-67.6KiB/s (68.9kB/s-69.2kB/s), io=40.0MiB (41.9MB), run=151551-152247msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145940: Thu Aug 27 14:20:00 2026 read: IOPS=12, BW=48.3KiB/s (49.5kB/s)(5048KiB/104495msec) slat (usec): min=2955, max=64303, avg=13708.49, stdev=7793.52 clat (msec): min=33, max=1080, avg=400.85, stdev=113.81 lat (msec): min=50, max=1098, avg=414.57, stdev=115.73 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 255], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 443], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 592], | 99.00th=[ 793], 99.50th=[ 810], 99.90th=[ 1028], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 128, per=25.07%, avg=47.89, stdev=22.73, samples=206 iops : min= 1, max= 32, avg=11.48, stdev= 5.69, samples=206 write: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5192KiB/104495msec); 0 zone resets slat (usec): min=33, max=14040, avg=295.50, stdev=950.64 clat (msec): min=155, max=1749, avg=882.36, stdev=240.15 lat (msec): min=155, max=1749, avg=882.66, stdev=240.15 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 531], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 927], | 70.00th=[ 978], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1301], | 99.00th=[ 1620], 99.50th=[ 1703], 99.90th=[ 1737], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 104, per=24.77%, avg=49.30, stdev=18.81, samples=206 iops : min= 1, max= 26, avg=11.83, stdev= 4.76, samples=206 lat (msec) : 50=0.04%, 100=0.16%, 250=2.23%, 500=40.20%, 750=21.02% lat (msec) : 1000=22.85%, 2000=13.52% cpu : usr=0.11%, sys=0.61%, ctx=4269, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145941: Thu Aug 27 14:20:00 2026 read: IOPS=11, BW=47.5KiB/s (48.7kB/s)(4976KiB/104689msec) slat (usec): min=2550, max=91135, avg=14047.52, stdev=8247.13 clat (msec): min=40, max=1031, avg=405.99, stdev=106.53 lat (msec): min=53, max=1048, avg=420.04, stdev=108.52 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 259], 10.00th=[ 292], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 489], 90.00th=[ 542], 95.00th=[ 592], | 99.00th=[ 684], 99.50th=[ 768], 99.90th=[ 969], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 128, per=24.59%, avg=46.97, stdev=22.29, samples=207 iops : min= 1, max= 32, avg=11.24, stdev= 5.58, samples=207 write: IOPS=12, BW=50.3KiB/s (51.5kB/s)(5264KiB/104689msec); 0 zone resets slat (usec): min=31, max=14058, avg=247.46, stdev=694.01 clat (msec): min=104, max=1849, avg=873.48, stdev=253.52 lat (msec): min=105, max=1849, avg=873.73, stdev=253.52 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 502], 10.00th=[ 567], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1062], 90.00th=[ 1200], 95.00th=[ 1301], | 99.00th=[ 1603], 99.50th=[ 1737], 99.90th=[ 1821], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 112, per=24.83%, avg=49.41, stdev=21.19, samples=208 iops : min= 1, max= 28, avg=11.86, stdev= 5.34, samples=208 lat (msec) : 50=0.04%, 100=0.16%, 250=2.07%, 500=40.23%, 750=22.27% lat (msec) : 1000=21.48%, 2000=13.75% cpu : usr=0.09%, sys=0.60%, ctx=4443, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145942: Thu Aug 27 14:20:00 2026 read: IOPS=11, BW=47.8KiB/s (48.0kB/s)(4984KiB/104218msec) slat (usec): min=36, max=66842, avg=13885.89, stdev=8038.66 clat (msec): min=18, max=943, avg=398.70, stdev=115.27 lat (msec): min=23, max=970, avg=412.59, stdev=117.21 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 255], 10.00th=[ 279], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 439], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 600], | 99.00th=[ 751], 99.50th=[ 793], 99.90th=[ 936], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 120, per=25.02%, avg=47.78, stdev=22.13, samples=205 iops : min= 1, max= 30, avg=11.45, stdev= 5.58, samples=205 write: IOPS=12, BW=50.4KiB/s (51.6kB/s)(5256KiB/104218msec); 0 zone resets slat (usec): min=31, max=8753, avg=260.80, stdev=725.46 clat (msec): min=158, max=1802, avg=876.57, stdev=245.06 lat (msec): min=158, max=1802, avg=876.84, stdev=245.07 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 514], 10.00th=[ 592], 20.00th=[ 684], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 911], | 70.00th=[ 969], 80.00th=[ 1062], 90.00th=[ 1183], 95.00th=[ 1284], | 99.00th=[ 1620], 99.50th=[ 1703], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 109, per=24.95%, avg=49.65, stdev=19.39, samples=207 iops : min= 1, max= 27, avg=11.92, stdev= 4.88, samples=207 lat (msec) : 20=0.04%, 50=0.20%, 100=0.47%, 250=1.99%, 500=39.84% lat (msec) : 750=20.59%, 1000=23.32%, 2000=13.55% cpu : usr=0.09%, sys=0.57%, ctx=4095, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145943: Thu Aug 27 14:20:00 2026 read: IOPS=12, BW=48.4KiB/s (49.5kB/s)(5060KiB/104581msec) slat (usec): min=3189, max=67691, avg=13937.73, stdev=7956.36 clat (msec): min=54, max=1046, avg=407.47, stdev=108.63 lat (msec): min=62, max=1061, avg=421.41, stdev=110.64 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 257], 10.00th=[ 284], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 422], | 70.00th=[ 451], 80.00th=[ 493], 90.00th=[ 542], 95.00th=[ 609], | 99.00th=[ 718], 99.50th=[ 768], 99.90th=[ 810], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 7, max= 117, per=25.26%, avg=48.24, stdev=21.08, samples=206 iops : min= 1, max= 29, avg=11.56, stdev= 5.30, samples=206 write: IOPS=12, BW=49.5KiB/s (50.7kB/s)(5180KiB/104581msec); 0 zone resets slat (usec): min=33, max=16810, avg=293.80, stdev=916.18 clat (msec): min=136, max=1832, avg=877.98, stdev=238.65 lat (msec): min=136, max=1833, avg=878.28, stdev=238.64 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 518], 10.00th=[ 584], 20.00th=[ 684], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1062], 90.00th=[ 1183], 95.00th=[ 1301], | 99.00th=[ 1586], 99.50th=[ 1703], 99.90th=[ 1754], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 104, per=24.49%, avg=48.73, stdev=19.30, samples=208 iops : min= 1, max= 26, avg=11.68, stdev= 4.85, samples=208 lat (msec) : 100=0.16%, 250=2.15%, 500=40.16%, 750=20.82%, 1000=23.20% lat (msec) : 2000=13.52% cpu : usr=0.10%, sys=0.64%, ctx=4319, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=192KiB/s (196kB/s), 47.5KiB/s-48.4KiB/s (48.7kB/s-49.5kB/s), io=19.6MiB (20.5MB), run=104218-104689msec WRITE: bw=200KiB/s (204kB/s), 49.5KiB/s-50.4KiB/s (50.7kB/s-51.6kB/s), io=20.4MiB (21.4MB), run=104218-104689msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145961: Thu Aug 27 14:20:11 2026 read: IOPS=1, BW=58.3MiB/s (61.1MB/s)(40.0MiB/686msec) slat (nsec): min=39924k, max=39924k, avg=39924005.00, stdev= 0.00 clat (nsec): min=606258k, max=606258k, avg=606257794.00, stdev= 0.00 lat (nsec): min=646186k, max=646186k, avg=646186475.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 609], 10.00th=[ 609], 20.00th=[ 609], | 30.00th=[ 609], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 609], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.04%, ctx=38, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=58.3MiB/s (61.1MB/s), 58.3MiB/s-58.3MiB/s (61.1MB/s-61.1MB/s), io=40.0MiB (41.9MB), run=686-686msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout