== sanity test 398c: run fio to test AIO ================= 04:51:24 (1785401484) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.688547 s, 60.9 MB/s osc.lustre-OST0000-osc-ffffa0f3a577e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145362: Thu Jul 30 04:56:12 2026 write: IOPS=9, BW=37.5KiB/s (38.4kB/s)(10.0MiB/273246msec); 0 zone resets slat (usec): min=39, max=27487, avg=247.45, stdev=1132.62 clat (msec): min=252, max=5107, avg=1706.34, stdev=733.73 lat (msec): min=252, max=5107, avg=1706.58, stdev=733.72 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 617], 10.00th=[ 818], 20.00th=[ 1070], | 30.00th=[ 1334], 40.00th=[ 1502], 50.00th=[ 1653], 60.00th=[ 1821], | 70.00th=[ 2039], 80.00th=[ 2265], 90.00th=[ 2534], 95.00th=[ 2970], | 99.00th=[ 4044], 99.50th=[ 4665], 99.90th=[ 4866], 99.95th=[ 4866], | 99.99th=[ 5134] bw ( KiB/s): min= 7, max= 255, per=27.89%, avg=41.56, stdev=29.87, samples=482 iops : min= 1, max= 63, avg= 9.90, stdev= 7.47, samples=482 lat (msec) : 500=3.16%, 750=4.88%, 1000=9.14%, 2000=51.45%, >=2000=31.37% cpu : usr=0.03%, sys=0.16%, ctx=1769, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145363: Thu Jul 30 04:56:12 2026 write: IOPS=9, BW=37.5KiB/s (38.4kB/s)(10.0MiB/273244msec); 0 zone resets slat (usec): min=44, max=32609, avg=251.85, stdev=1224.74 clat (msec): min=358, max=5294, avg=1706.06, stdev=727.48 lat (msec): min=359, max=5294, avg=1706.32, stdev=727.56 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 634], 10.00th=[ 835], 20.00th=[ 1099], | 30.00th=[ 1334], 40.00th=[ 1519], 50.00th=[ 1670], 60.00th=[ 1804], | 70.00th=[ 2005], 80.00th=[ 2198], 90.00th=[ 2534], 95.00th=[ 2970], | 99.00th=[ 4077], 99.50th=[ 4732], 99.90th=[ 5201], 99.95th=[ 5269], | 99.99th=[ 5269] bw ( KiB/s): min= 7, max= 255, per=27.54%, avg=41.04, stdev=28.04, samples=488 iops : min= 1, max= 63, avg= 9.76, stdev= 7.06, samples=488 lat (msec) : 500=3.09%, 750=4.73%, 1000=8.48%, 2000=53.05%, >=2000=30.66% cpu : usr=0.03%, sys=0.15%, ctx=1605, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145364: Thu Jul 30 04:56:12 2026 write: IOPS=9, BW=37.5KiB/s (38.4kB/s)(10.0MiB/272747msec); 0 zone resets slat (usec): min=39, max=47641, avg=234.23, stdev=1235.17 clat (msec): min=78, max=5107, avg=1703.06, stdev=723.39 lat (msec): min=79, max=5108, avg=1703.30, stdev=723.41 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 659], 10.00th=[ 844], 20.00th=[ 1116], | 30.00th=[ 1318], 40.00th=[ 1519], 50.00th=[ 1670], 60.00th=[ 1804], | 70.00th=[ 2005], 80.00th=[ 2198], 90.00th=[ 2500], 95.00th=[ 2937], | 99.00th=[ 4245], 99.50th=[ 4665], 99.90th=[ 5067], 99.95th=[ 5067], | 99.99th=[ 5134] bw ( KiB/s): min= 7, max= 223, per=27.16%, avg=40.47, stdev=28.46, samples=495 iops : min= 1, max= 55, avg= 9.63, stdev= 7.15, samples=495 lat (msec) : 100=0.04%, 250=0.04%, 500=2.97%, 750=5.00%, 1000=8.59% lat (msec) : 2000=53.09%, >=2000=30.27% cpu : usr=0.03%, sys=0.16%, ctx=1822, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145365: Thu Jul 30 04:56:12 2026 write: IOPS=9, BW=37.5KiB/s (38.4kB/s)(10.0MiB/273245msec); 0 zone resets slat (usec): min=36, max=31448, avg=252.31, stdev=1114.45 clat (msec): min=195, max=5082, avg=1706.79, stdev=736.16 lat (msec): min=195, max=5082, avg=1707.04, stdev=736.20 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 584], 10.00th=[ 827], 20.00th=[ 1099], | 30.00th=[ 1351], 40.00th=[ 1519], 50.00th=[ 1670], 60.00th=[ 1821], | 70.00th=[ 1989], 80.00th=[ 2198], 90.00th=[ 2534], 95.00th=[ 2937], | 99.00th=[ 4396], 99.50th=[ 4665], 99.90th=[ 5067], 99.95th=[ 5067], | 99.99th=[ 5067] bw ( KiB/s): min= 7, max= 247, per=27.90%, avg=41.57, stdev=29.28, samples=482 iops : min= 1, max= 61, avg= 9.91, stdev= 7.36, samples=482 lat (msec) : 250=0.55%, 500=2.62%, 750=4.80%, 1000=7.77%, 2000=55.12% lat (msec) : >=2000=29.14% cpu : usr=0.02%, sys=0.17%, ctx=1936, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=150KiB/s (153kB/s), 37.5KiB/s-37.5KiB/s (38.4kB/s-38.4kB/s), io=40.0MiB (41.9MB), run=272747-273246msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145414: Thu Jul 30 04:59:55 2026 read: IOPS=6, BW=24.3KiB/s (24.9kB/s)(5048KiB/207550msec) slat (msec): min=2, max=137, avg=22.53, stdev=16.06 clat (msec): min=18, max=2393, avg=794.52, stdev=302.77 lat (msec): min=24, max=2407, avg=817.05, stdev=306.05 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 305], 10.00th=[ 380], 20.00th=[ 523], | 30.00th=[ 642], 40.00th=[ 718], 50.00th=[ 810], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1636], 99.50th=[ 1687], 99.90th=[ 2165], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 96, per=27.35%, avg=26.26, stdev=16.66, samples=372 iops : min= 1, max= 24, avg= 6.04, stdev= 4.21, samples=372 write: IOPS=6, BW=25.0KiB/s (25.6kB/s)(5192KiB/207550msec); 0 zone resets slat (usec): min=54, max=19968, avg=330.40, stdev=1238.24 clat (msec): min=377, max=3683, avg=1758.43, stdev=596.54 lat (msec): min=377, max=3683, avg=1758.76, stdev=596.57 clat percentiles (msec): | 1.00th=[ 659], 5.00th=[ 776], 10.00th=[ 919], 20.00th=[ 1150], | 30.00th=[ 1452], 40.00th=[ 1653], 50.00th=[ 1787], 60.00th=[ 1905], | 70.00th=[ 2039], 80.00th=[ 2265], 90.00th=[ 2534], 95.00th=[ 2735], | 99.00th=[ 3205], 99.50th=[ 3373], 99.90th=[ 3641], 99.95th=[ 3675], | 99.99th=[ 3675] bw ( KiB/s): min= 7, max= 96, per=26.52%, avg=26.52, stdev=16.62, samples=379 iops : min= 1, max= 24, avg= 6.11, stdev= 4.22, samples=379 lat (msec) : 20=0.04%, 50=0.16%, 100=0.23%, 250=0.82%, 500=7.97% lat (msec) : 750=14.34%, 1000=20.70%, 2000=39.22%, >=2000=16.52% cpu : usr=0.04%, sys=0.36%, ctx=4324, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145415: Thu Jul 30 04:59:55 2026 read: IOPS=5, BW=23.9KiB/s (24.5kB/s)(4976KiB/208194msec) slat (msec): min=2, max=142, avg=21.94, stdev=15.90 clat (msec): min=42, max=2307, avg=790.85, stdev=296.93 lat (msec): min=68, max=2316, avg=812.80, stdev=300.72 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 309], 10.00th=[ 388], 20.00th=[ 523], | 30.00th=[ 634], 40.00th=[ 718], 50.00th=[ 793], 60.00th=[ 877], | 70.00th=[ 953], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1519], 99.50th=[ 1636], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 104, per=27.67%, avg=26.56, stdev=17.28, samples=362 iops : min= 1, max= 26, avg= 6.12, stdev= 4.35, samples=362 write: IOPS=6, BW=25.3KiB/s (25.9kB/s)(5264KiB/208194msec); 0 zone resets slat (usec): min=44, max=29360, avg=435.79, stdev=1936.37 clat (msec): min=320, max=3629, avg=1758.91, stdev=606.37 lat (msec): min=320, max=3630, avg=1759.35, stdev=606.38 clat percentiles (msec): | 1.00th=[ 617], 5.00th=[ 793], 10.00th=[ 936], 20.00th=[ 1133], | 30.00th=[ 1452], 40.00th=[ 1620], 50.00th=[ 1787], 60.00th=[ 1905], | 70.00th=[ 2056], 80.00th=[ 2299], 90.00th=[ 2534], 95.00th=[ 2769], | 99.00th=[ 3239], 99.50th=[ 3373], 99.90th=[ 3540], 99.95th=[ 3641], | 99.99th=[ 3641] bw ( KiB/s): min= 7, max= 96, per=26.85%, avg=26.85, stdev=16.68, samples=380 iops : min= 1, max= 24, avg= 6.18, stdev= 4.25, samples=380 lat (msec) : 50=0.08%, 100=0.04%, 250=0.43%, 500=8.55%, 750=14.22% lat (msec) : 1000=20.70%, 2000=38.98%, >=2000=16.99% cpu : usr=0.06%, sys=0.40%, ctx=4260, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145416: Thu Jul 30 04:59:55 2026 read: IOPS=5, BW=23.0KiB/s (24.6kB/s)(4984KiB/207865msec) slat (msec): min=2, max=129, avg=22.21, stdev=15.33 clat (msec): min=37, max=2338, avg=792.66, stdev=299.98 lat (msec): min=51, max=2386, avg=814.89, stdev=304.41 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 326], 10.00th=[ 376], 20.00th=[ 506], | 30.00th=[ 634], 40.00th=[ 709], 50.00th=[ 810], 60.00th=[ 885], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1519], 99.50th=[ 1636], 99.90th=[ 2198], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 104, per=27.45%, avg=26.35, stdev=17.77, samples=367 iops : min= 1, max= 26, avg= 6.07, stdev= 4.51, samples=367 write: IOPS=6, BW=25.3KiB/s (25.9kB/s)(5256KiB/207865msec); 0 zone resets slat (usec): min=52, max=32301, avg=352.68, stdev=1524.42 clat (msec): min=360, max=3715, avg=1753.11, stdev=590.72 lat (msec): min=361, max=3715, avg=1753.47, stdev=590.78 clat percentiles (msec): | 1.00th=[ 659], 5.00th=[ 793], 10.00th=[ 953], 20.00th=[ 1167], | 30.00th=[ 1435], 40.00th=[ 1620], 50.00th=[ 1770], 60.00th=[ 1905], | 70.00th=[ 2039], 80.00th=[ 2232], 90.00th=[ 2534], 95.00th=[ 2735], | 99.00th=[ 3205], 99.50th=[ 3440], 99.90th=[ 3708], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 7, max= 88, per=27.30%, avg=27.30, stdev=16.35, samples=372 iops : min= 1, max= 22, avg= 6.30, stdev= 4.15, samples=372 lat (msec) : 50=0.04%, 100=0.16%, 250=0.59%, 500=8.95%, 750=13.09% lat (msec) : 1000=19.53%, 2000=40.94%, >=2000=16.72% cpu : usr=0.05%, sys=0.34%, ctx=4295, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145417: Thu Jul 30 04:59:55 2026 read: IOPS=6, BW=24.7KiB/s (25.3kB/s)(5060KiB/205037msec) slat (usec): min=62, max=107530, avg=21320.97, stdev=14641.84 clat (msec): min=29, max=3720, avg=780.04, stdev=319.71 lat (msec): min=29, max=3759, avg=801.37, stdev=323.21 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 309], 10.00th=[ 372], 20.00th=[ 498], | 30.00th=[ 617], 40.00th=[ 701], 50.00th=[ 776], 60.00th=[ 860], | 70.00th=[ 953], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1485], 99.50th=[ 1569], 99.90th=[ 3507], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 7, max= 128, per=28.54%, avg=27.40, stdev=17.20, samples=359 iops : min= 1, max= 32, avg= 6.32, stdev= 4.36, samples=359 write: IOPS=6, BW=25.3KiB/s (25.9kB/s)(5180KiB/205037msec); 0 zone resets slat (usec): min=52, max=25368, avg=334.31, stdev=1212.17 clat (msec): min=184, max=5513, avg=1747.47, stdev=658.25 lat (msec): min=191, max=5513, avg=1747.81, stdev=658.24 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 760], 10.00th=[ 911], 20.00th=[ 1150], | 30.00th=[ 1452], 40.00th=[ 1620], 50.00th=[ 1754], 60.00th=[ 1871], | 70.00th=[ 2005], 80.00th=[ 2198], 90.00th=[ 2534], 95.00th=[ 2769], | 99.00th=[ 3406], 99.50th=[ 5201], 99.90th=[ 5537], 99.95th=[ 5537], | 99.99th=[ 5537] bw ( KiB/s): min= 7, max= 88, per=26.50%, avg=26.50, stdev=15.60, samples=377 iops : min= 1, max= 22, avg= 6.09, stdev= 3.97, samples=377 lat (msec) : 50=0.16%, 100=0.12%, 250=0.78%, 500=9.57%, 750=14.84% lat (msec) : 1000=19.69%, 2000=39.45%, >=2000=15.39% cpu : usr=0.05%, sys=0.35%, ctx=4139, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=96.4KiB/s (98.7kB/s), 23.9KiB/s-24.7KiB/s (24.5kB/s-25.3kB/s), io=19.6MiB (20.5MB), run=205037-208194msec WRITE: bw=100KiB/s (103kB/s), 25.0KiB/s-25.3KiB/s (25.6kB/s-25.9kB/s), io=20.4MiB (21.4MB), run=205037-208194msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145454: Thu Jul 30 05:00:17 2026 read: IOPS=1, BW=43.5MiB/s (45.6MB/s)(40.0MiB/919msec) slat (nsec): min=72594k, max=72594k, avg=72593806.00, stdev= 0.00 clat (nsec): min=801647k, max=801647k, avg=801647476.00, stdev= 0.00 lat (nsec): min=874246k, max=874246k, avg=874246322.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 802], 5.00th=[ 802], 10.00th=[ 802], 20.00th=[ 802], | 30.00th=[ 802], 40.00th=[ 802], 50.00th=[ 802], 60.00th=[ 802], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=1.53%, ctx=6, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=43.5MiB/s (45.6MB/s), 43.5MiB/s-43.5MiB/s (45.6MB/s-45.6MB/s), io=40.0MiB (41.9MB), run=919-919msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout