== sanity test 398c: run fio to test AIO ================= 03:08:11 (1785395291) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.661443 s, 63.4 MB/s osc.lustre-OST0000-osc-ffff99e2371f8800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144327: Thu Jul 30 03:13:01 2026 write: IOPS=9, BW=36.9KiB/s (37.8kB/s)(10.0MiB/277259msec); 0 zone resets slat (usec): min=45, max=32471, avg=240.30, stdev=1124.43 clat (msec): min=142, max=3533, avg=1732.31, stdev=680.36 lat (msec): min=142, max=3533, avg=1732.55, stdev=680.37 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 558], 10.00th=[ 684], 20.00th=[ 1099], | 30.00th=[ 1401], 40.00th=[ 1620], 50.00th=[ 1804], 60.00th=[ 1972], | 70.00th=[ 2140], 80.00th=[ 2333], 90.00th=[ 2567], 95.00th=[ 2769], | 99.00th=[ 3104], 99.50th=[ 3239], 99.90th=[ 3507], 99.95th=[ 3540], | 99.99th=[ 3540] bw ( KiB/s): min= 7, max= 224, per=27.26%, avg=40.07, stdev=28.81, samples=500 iops : min= 1, max= 56, avg= 9.58, stdev= 7.27, samples=500 lat (msec) : 250=0.62%, 500=3.24%, 750=8.36%, 1000=5.00%, 2000=43.98% lat (msec) : >=2000=38.79% cpu : usr=0.02%, sys=0.14%, ctx=1541, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144328: Thu Jul 30 03:13:01 2026 write: IOPS=9, BW=36.9KiB/s (37.8kB/s)(10.0MiB/277382msec); 0 zone resets slat (usec): min=42, max=43816, avg=286.57, stdev=1601.86 clat (msec): min=279, max=3499, avg=1732.61, stdev=681.26 lat (msec): min=279, max=3499, avg=1732.90, stdev=681.27 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 514], 10.00th=[ 684], 20.00th=[ 1070], | 30.00th=[ 1368], 40.00th=[ 1620], 50.00th=[ 1821], 60.00th=[ 1972], | 70.00th=[ 2123], 80.00th=[ 2299], 90.00th=[ 2567], 95.00th=[ 2769], | 99.00th=[ 3272], 99.50th=[ 3440], 99.90th=[ 3507], 99.95th=[ 3507], | 99.99th=[ 3507] bw ( KiB/s): min= 7, max= 128, per=28.11%, avg=41.33, stdev=29.74, samples=485 iops : min= 1, max= 32, avg= 9.89, stdev= 7.48, samples=485 lat (msec) : 500=4.57%, 750=7.34%, 1000=6.13%, 2000=43.98%, >=2000=37.97% cpu : usr=0.02%, sys=0.14%, ctx=1670, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144329: Thu Jul 30 03:13:01 2026 write: IOPS=9, BW=36.9KiB/s (37.8kB/s)(10.0MiB/277301msec); 0 zone resets slat (usec): min=47, max=38731, avg=221.93, stdev=1060.90 clat (msec): min=211, max=3411, avg=1732.43, stdev=677.24 lat (msec): min=212, max=3411, avg=1732.66, stdev=677.25 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 558], 10.00th=[ 676], 20.00th=[ 1099], | 30.00th=[ 1385], 40.00th=[ 1636], 50.00th=[ 1804], 60.00th=[ 1955], | 70.00th=[ 2106], 80.00th=[ 2333], 90.00th=[ 2601], 95.00th=[ 2735], | 99.00th=[ 3104], 99.50th=[ 3239], 99.90th=[ 3373], 99.95th=[ 3406], | 99.99th=[ 3406] bw ( KiB/s): min= 7, max= 256, per=27.67%, avg=40.67, stdev=30.02, samples=492 iops : min= 1, max= 64, avg= 9.73, stdev= 7.55, samples=492 lat (msec) : 250=0.62%, 500=3.79%, 750=7.46%, 1000=5.51%, 2000=45.27% lat (msec) : >=2000=37.34% cpu : usr=0.02%, sys=0.14%, ctx=1456, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144330: Thu Jul 30 03:13:01 2026 write: IOPS=9, BW=36.9KiB/s (37.8kB/s)(10.0MiB/277216msec); 0 zone resets slat (usec): min=46, max=52592, avg=277.61, stdev=1546.71 clat (msec): min=63, max=3628, avg=1731.69, stdev=678.04 lat (msec): min=73, max=3629, avg=1731.97, stdev=677.99 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 535], 10.00th=[ 693], 20.00th=[ 1083], | 30.00th=[ 1401], 40.00th=[ 1620], 50.00th=[ 1804], 60.00th=[ 1972], | 70.00th=[ 2123], 80.00th=[ 2333], 90.00th=[ 2567], 95.00th=[ 2735], | 99.00th=[ 3171], 99.50th=[ 3339], 99.90th=[ 3507], 99.95th=[ 3641], | 99.99th=[ 3641] bw ( KiB/s): min= 7, max= 208, per=26.39%, avg=38.80, stdev=26.99, samples=516 iops : min= 1, max= 52, avg= 9.26, stdev= 6.79, samples=516 lat (msec) : 100=0.12%, 250=0.23%, 500=3.75%, 750=7.15%, 1000=5.78% lat (msec) : 2000=44.77%, >=2000=38.20% cpu : usr=0.03%, sys=0.14%, ctx=1835, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=148KiB/s (151kB/s), 36.9KiB/s-36.9KiB/s (37.8kB/s-37.8kB/s), io=40.0MiB (41.9MB), run=277216-277382msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144382: Thu Jul 30 03:16:08 2026 read: IOPS=7, BW=28.9KiB/s (29.6kB/s)(5048KiB/174781msec) slat (usec): min=58, max=262893, avg=19334.41, stdev=16784.61 clat (msec): min=36, max=1945, avg=675.04, stdev=269.88 lat (msec): min=50, max=1962, avg=694.38, stdev=274.03 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 271], 10.00th=[ 330], 20.00th=[ 426], | 30.00th=[ 510], 40.00th=[ 600], 50.00th=[ 676], 60.00th=[ 735], | 70.00th=[ 810], 80.00th=[ 894], 90.00th=[ 1020], 95.00th=[ 1116], | 99.00th=[ 1401], 99.50th=[ 1485], 99.90th=[ 1552], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 87, per=26.81%, avg=30.29, stdev=18.27, samples=324 iops : min= 1, max= 21, avg= 7.10, stdev= 4.60, samples=324 write: IOPS=7, BW=29.7KiB/s (30.4kB/s)(5192KiB/174781msec); 0 zone resets slat (usec): min=50, max=16741, avg=309.20, stdev=1077.03 clat (msec): min=214, max=3224, avg=1477.43, stdev=550.56 lat (msec): min=214, max=3224, avg=1477.74, stdev=550.57 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 709], 10.00th=[ 818], 20.00th=[ 969], | 30.00th=[ 1133], 40.00th=[ 1284], 50.00th=[ 1435], 60.00th=[ 1586], | 70.00th=[ 1737], 80.00th=[ 1905], 90.00th=[ 2198], 95.00th=[ 2500], | 99.00th=[ 2970], 99.50th=[ 3071], 99.90th=[ 3205], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 7, max= 87, per=26.50%, avg=31.27, stdev=17.90, samples=322 iops : min= 1, max= 21, avg= 7.34, stdev= 4.51, samples=322 lat (msec) : 50=0.04%, 100=0.35%, 250=1.56%, 500=12.81%, 750=19.14% lat (msec) : 1000=21.60%, 2000=36.37%, >=2000=8.12% cpu : usr=0.06%, sys=0.37%, ctx=3913, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144383: Thu Jul 30 03:16:08 2026 read: IOPS=7, BW=28.2KiB/s (28.9kB/s)(4976KiB/176148msec) slat (msec): min=2, max=276, avg=19.03, stdev=15.44 clat (msec): min=12, max=1537, avg=675.92, stdev=268.39 lat (msec): min=21, max=1548, avg=694.95, stdev=272.24 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 279], 10.00th=[ 330], 20.00th=[ 418], | 30.00th=[ 498], 40.00th=[ 584], 50.00th=[ 659], 60.00th=[ 751], | 70.00th=[ 844], 80.00th=[ 927], 90.00th=[ 1020], 95.00th=[ 1099], | 99.00th=[ 1318], 99.50th=[ 1385], 99.90th=[ 1502], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 112, per=26.51%, avg=29.95, stdev=19.48, samples=323 iops : min= 1, max= 28, avg= 7.02, stdev= 4.94, samples=323 write: IOPS=7, BW=29.9KiB/s (30.6kB/s)(5264KiB/176148msec); 0 zone resets slat (usec): min=51, max=44497, avg=309.46, stdev=1597.05 clat (msec): min=480, max=3341, avg=1481.00, stdev=535.05 lat (msec): min=480, max=3341, avg=1481.31, stdev=535.09 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 718], 10.00th=[ 827], 20.00th=[ 978], | 30.00th=[ 1133], 40.00th=[ 1301], 50.00th=[ 1452], 60.00th=[ 1586], | 70.00th=[ 1737], 80.00th=[ 1905], 90.00th=[ 2198], 95.00th=[ 2467], | 99.00th=[ 2903], 99.50th=[ 3004], 99.90th=[ 3171], 99.95th=[ 3339], | 99.99th=[ 3339] bw ( KiB/s): min= 7, max= 95, per=25.98%, avg=30.66, stdev=18.23, samples=334 iops : min= 1, max= 23, avg= 7.18, stdev= 4.58, samples=334 lat (msec) : 20=0.04%, 100=0.08%, 250=1.25%, 500=13.63%, 750=17.50% lat (msec) : 1000=21.17%, 2000=38.16%, >=2000=8.16% cpu : usr=0.05%, sys=0.40%, ctx=3948, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144384: Thu Jul 30 03:16:08 2026 read: IOPS=7, BW=28.3KiB/s (28.0kB/s)(4984KiB/176208msec) slat (msec): min=2, max=264, avg=19.63, stdev=15.69 clat (msec): min=26, max=1539, avg=685.99, stdev=275.70 lat (msec): min=45, max=1572, avg=705.63, stdev=279.49 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 300], 10.00th=[ 342], 20.00th=[ 430], | 30.00th=[ 502], 40.00th=[ 600], 50.00th=[ 667], 60.00th=[ 743], | 70.00th=[ 818], 80.00th=[ 927], 90.00th=[ 1062], 95.00th=[ 1150], | 99.00th=[ 1385], 99.50th=[ 1452], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 108, per=26.15%, avg=29.55, stdev=18.53, samples=328 iops : min= 1, max= 27, avg= 6.92, stdev= 4.67, samples=328 write: IOPS=7, BW=29.8KiB/s (30.5kB/s)(5256KiB/176208msec); 0 zone resets slat (usec): min=50, max=27037, avg=302.99, stdev=1212.49 clat (msec): min=391, max=3340, avg=1472.61, stdev=531.70 lat (msec): min=391, max=3341, avg=1472.91, stdev=531.74 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 718], 10.00th=[ 835], 20.00th=[ 995], | 30.00th=[ 1133], 40.00th=[ 1250], 50.00th=[ 1418], 60.00th=[ 1552], | 70.00th=[ 1720], 80.00th=[ 1921], 90.00th=[ 2198], 95.00th=[ 2433], | 99.00th=[ 2903], 99.50th=[ 3071], 99.90th=[ 3205], 99.95th=[ 3339], | 99.99th=[ 3339] bw ( KiB/s): min= 7, max= 87, per=25.46%, avg=30.04, stdev=18.03, samples=339 iops : min= 1, max= 21, avg= 7.04, stdev= 4.50, samples=339 lat (msec) : 50=0.20%, 100=0.16%, 250=0.82%, 500=13.55%, 750=18.05% lat (msec) : 1000=19.45%, 2000=39.49%, >=2000=8.28% cpu : usr=0.06%, sys=0.39%, ctx=3999, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144385: Thu Jul 30 03:16:08 2026 read: IOPS=7, BW=28.8KiB/s (29.5kB/s)(5060KiB/175800msec) slat (msec): min=2, max=259, avg=19.20, stdev=14.98 clat (msec): min=7, max=1462, avg=686.01, stdev=272.45 lat (msec): min=28, max=1488, avg=705.22, stdev=275.94 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 279], 10.00th=[ 342], 20.00th=[ 430], | 30.00th=[ 510], 40.00th=[ 600], 50.00th=[ 676], 60.00th=[ 743], | 70.00th=[ 827], 80.00th=[ 936], 90.00th=[ 1062], 95.00th=[ 1167], | 99.00th=[ 1351], 99.50th=[ 1368], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 102, per=27.15%, avg=30.68, stdev=18.21, samples=322 iops : min= 1, max= 25, avg= 7.19, stdev= 4.58, samples=322 write: IOPS=7, BW=29.5KiB/s (30.2kB/s)(5180KiB/175800msec); 0 zone resets slat (usec): min=52, max=27693, avg=325.50, stdev=1301.42 clat (msec): min=386, max=3303, avg=1479.29, stdev=533.11 lat (msec): min=387, max=3303, avg=1479.61, stdev=533.07 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 735], 10.00th=[ 827], 20.00th=[ 978], | 30.00th=[ 1116], 40.00th=[ 1284], 50.00th=[ 1435], 60.00th=[ 1586], | 70.00th=[ 1737], 80.00th=[ 1921], 90.00th=[ 2198], 95.00th=[ 2433], | 99.00th=[ 2836], 99.50th=[ 2970], 99.90th=[ 3272], 99.95th=[ 3306], | 99.99th=[ 3306] bw ( KiB/s): min= 7, max= 96, per=26.00%, avg=30.68, stdev=17.21, samples=327 iops : min= 1, max= 24, avg= 7.19, stdev= 4.32, samples=327 lat (msec) : 10=0.04%, 50=0.04%, 100=0.16%, 250=1.13%, 500=13.36% lat (msec) : 750=18.12%, 1000=20.39%, 2000=38.28%, >=2000=8.48% cpu : usr=0.06%, sys=0.44%, ctx=4100, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=114KiB/s (117kB/s), 28.2KiB/s-28.9KiB/s (28.9kB/s-29.6kB/s), io=19.6MiB (20.5MB), run=174781-176208msec WRITE: bw=119KiB/s (121kB/s), 29.5KiB/s-29.9KiB/s (30.2kB/s-30.6kB/s), io=20.4MiB (21.4MB), run=174781-176208msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144416: Thu Jul 30 03:16:21 2026 read: IOPS=1, BW=67.5MiB/s (70.7MB/s)(40.0MiB/593msec) slat (nsec): min=23961k, max=23961k, avg=23961475.00, stdev= 0.00 clat (nsec): min=538327k, max=538327k, avg=538326834.00, stdev= 0.00 lat (nsec): min=562292k, max=562292k, avg=562292322.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.03%, ctx=6, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=67.5MiB/s (70.7MB/s), 67.5MiB/s-67.5MiB/s (70.7MB/s-70.7MB/s), io=40.0MiB (41.9MB), run=593-593msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout