== sanity test 398c: run fio to test AIO ================= 16:39:40 (1788467980) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.369781 s, 113 MB/s osc.lustre-OST0000-osc-ffff892746320000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145695: Thu Sep 3 16:42:52 2026 write: IOPS=13, BW=55.7KiB/s (57.0kB/s)(10.0MiB/183853msec); 0 zone resets slat (usec): min=22, max=37242, avg=256.72, stdev=1158.12 clat (msec): min=141, max=2596, avg=1148.36, stdev=382.74 lat (msec): min=144, max=2596, avg=1148.62, stdev=382.74 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 600], 10.00th=[ 735], 20.00th=[ 835], | 30.00th=[ 927], 40.00th=[ 1011], 50.00th=[ 1116], 60.00th=[ 1217], | 70.00th=[ 1318], 80.00th=[ 1469], 90.00th=[ 1636], 95.00th=[ 1838], | 99.00th=[ 2106], 99.50th=[ 2467], 99.90th=[ 2567], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 237, per=25.57%, avg=56.76, stdev=31.16, samples=354 iops : min= 1, max= 59, avg=13.70, stdev= 7.83, samples=354 lat (msec) : 250=0.66%, 500=3.24%, 750=7.27%, 1000=27.34%, 2000=59.45% lat (msec) : >=2000=2.03% cpu : usr=0.05%, sys=0.23%, ctx=1725, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145696: Thu Sep 3 16:42:52 2026 write: IOPS=13, BW=55.6KiB/s (56.9kB/s)(10.0MiB/184169msec); 0 zone resets slat (usec): min=21, max=22393, avg=284.57, stdev=1165.89 clat (msec): min=274, max=2505, avg=1150.04, stdev=386.12 lat (msec): min=274, max=2505, avg=1150.33, stdev=386.13 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 592], 10.00th=[ 709], 20.00th=[ 835], | 30.00th=[ 936], 40.00th=[ 1011], 50.00th=[ 1099], 60.00th=[ 1217], | 70.00th=[ 1334], 80.00th=[ 1452], 90.00th=[ 1670], 95.00th=[ 1888], | 99.00th=[ 2106], 99.50th=[ 2165], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 256, per=25.49%, avg=56.59, stdev=31.65, samples=355 iops : min= 1, max= 64, avg=13.65, stdev= 7.94, samples=355 lat (msec) : 500=4.10%, 750=8.16%, 1000=26.84%, 2000=58.05%, >=2000=2.85% cpu : usr=0.05%, sys=0.22%, ctx=1741, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145697: Thu Sep 3 16:42:52 2026 write: IOPS=13, BW=55.6KiB/s (56.0kB/s)(10.0MiB/184112msec); 0 zone resets slat (usec): min=22, max=36727, avg=278.35, stdev=1290.04 clat (msec): min=123, max=2591, avg=1149.53, stdev=378.89 lat (msec): min=123, max=2591, avg=1149.82, stdev=378.88 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 592], 10.00th=[ 743], 20.00th=[ 844], | 30.00th=[ 936], 40.00th=[ 1020], 50.00th=[ 1099], 60.00th=[ 1217], | 70.00th=[ 1334], 80.00th=[ 1452], 90.00th=[ 1636], 95.00th=[ 1821], | 99.00th=[ 2165], 99.50th=[ 2265], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 256, per=25.57%, avg=56.77, stdev=35.36, samples=354 iops : min= 1, max= 64, avg=13.70, stdev= 8.87, samples=354 lat (msec) : 250=0.23%, 500=4.10%, 750=5.70%, 1000=28.20%, 2000=59.14% lat (msec) : >=2000=2.62% cpu : usr=0.04%, sys=0.25%, ctx=1860, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145698: Thu Sep 3 16:42:52 2026 write: IOPS=13, BW=55.7KiB/s (57.0kB/s)(10.0MiB/183960msec); 0 zone resets slat (usec): min=22, max=26882, avg=314.86, stdev=1188.19 clat (msec): min=59, max=2590, avg=1148.70, stdev=384.66 lat (msec): min=60, max=2591, avg=1149.02, stdev=384.71 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 584], 10.00th=[ 735], 20.00th=[ 835], | 30.00th=[ 927], 40.00th=[ 1011], 50.00th=[ 1116], 60.00th=[ 1217], | 70.00th=[ 1334], 80.00th=[ 1452], 90.00th=[ 1653], 95.00th=[ 1821], | 99.00th=[ 2123], 99.50th=[ 2299], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 208, per=25.22%, avg=56.00, stdev=29.01, samples=359 iops : min= 1, max= 52, avg=13.51, stdev= 7.28, samples=359 lat (msec) : 100=0.04%, 250=0.35%, 500=4.14%, 750=6.09%, 1000=28.63% lat (msec) : 2000=58.48%, >=2000=2.27% cpu : usr=0.05%, sys=0.24%, ctx=1980, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=222KiB/s (228kB/s), 55.6KiB/s-55.7KiB/s (56.9kB/s-57.0kB/s), io=40.0MiB (41.9MB), run=183853-184169msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145735: Thu Sep 3 16:45:05 2026 read: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5048KiB/123627msec) slat (msec): min=2, max=121, avg=16.79, stdev= 9.80 clat (msec): min=32, max=1346, avg=471.70, stdev=167.84 lat (msec): min=48, max=1371, avg=488.50, stdev=169.82 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 338], | 30.00th=[ 380], 40.00th=[ 414], 50.00th=[ 447], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 575], 90.00th=[ 667], 95.00th=[ 793], | 99.00th=[ 1083], 99.50th=[ 1116], 99.90th=[ 1318], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 127, per=25.61%, avg=41.48, stdev=21.98, samples=238 iops : min= 1, max= 31, avg= 9.83, stdev= 5.55, samples=238 write: IOPS=10, BW=41.0KiB/s (43.0kB/s)(5192KiB/123627msec); 0 zone resets slat (usec): min=27, max=38371, avg=371.53, stdev=1571.18 clat (msec): min=403, max=2614, avg=1044.68, stdev=368.26 lat (msec): min=404, max=2615, avg=1045.05, stdev=368.36 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 592], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 844], 40.00th=[ 911], 50.00th=[ 978], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1485], 95.00th=[ 1804], | 99.00th=[ 2366], 99.50th=[ 2534], 99.90th=[ 2567], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 95, per=24.87%, avg=41.79, stdev=19.99, samples=242 iops : min= 1, max= 23, avg= 9.91, stdev= 5.03, samples=242 lat (msec) : 50=0.20%, 100=0.12%, 250=1.25%, 500=30.35%, 750=23.71% lat (msec) : 1000=19.49%, 2000=23.12%, >=2000=1.76% cpu : usr=0.12%, sys=0.72%, ctx=5075, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145736: Thu Sep 3 16:45:05 2026 read: IOPS=10, BW=40.3KiB/s (41.2kB/s)(4976KiB/123624msec) slat (usec): min=3043, max=82659, avg=16667.43, stdev=9252.60 clat (msec): min=17, max=1345, avg=475.75, stdev=166.07 lat (msec): min=24, max=1367, avg=492.42, stdev=168.30 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 384], 40.00th=[ 418], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 575], 90.00th=[ 684], 95.00th=[ 802], | 99.00th=[ 1062], 99.50th=[ 1133], 99.90th=[ 1301], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 135, per=24.65%, avg=39.94, stdev=22.84, samples=243 iops : min= 1, max= 33, avg= 9.44, stdev= 5.72, samples=243 write: IOPS=10, BW=42.6KiB/s (43.6kB/s)(5264KiB/123624msec); 0 zone resets slat (usec): min=35, max=24932, avg=433.68, stdev=1620.36 clat (msec): min=390, max=2838, avg=1034.72, stdev=363.61 lat (msec): min=390, max=2838, avg=1035.16, stdev=363.65 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 575], 10.00th=[ 659], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1502], 95.00th=[ 1754], | 99.00th=[ 2333], 99.50th=[ 2534], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 103, per=25.59%, avg=42.99, stdev=21.95, samples=239 iops : min= 1, max= 25, avg=10.21, stdev= 5.55, samples=239 lat (msec) : 20=0.08%, 50=0.12%, 100=0.12%, 250=0.86%, 500=31.37% lat (msec) : 750=22.70%, 1000=20.86%, 2000=22.54%, >=2000=1.37% cpu : usr=0.12%, sys=0.74%, ctx=4877, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145737: Thu Sep 3 16:45:05 2026 read: IOPS=10, BW=40.4KiB/s (41.3kB/s)(4984KiB/123454msec) slat (usec): min=53, max=222345, avg=16382.36, stdev=11385.89 clat (msec): min=20, max=1248, avg=468.81, stdev=164.67 lat (msec): min=22, max=1298, avg=485.20, stdev=167.35 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 342], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 443], 60.00th=[ 477], | 70.00th=[ 518], 80.00th=[ 567], 90.00th=[ 667], 95.00th=[ 785], | 99.00th=[ 1036], 99.50th=[ 1099], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 119, per=25.42%, avg=41.18, stdev=22.01, samples=237 iops : min= 1, max= 29, avg= 9.76, stdev= 5.55, samples=237 write: IOPS=10, BW=42.6KiB/s (43.6kB/s)(5256KiB/123454msec); 0 zone resets slat (usec): min=30, max=39825, avg=476.07, stdev=1856.47 clat (msec): min=187, max=2668, avg=1041.58, stdev=382.41 lat (msec): min=187, max=2668, avg=1042.06, stdev=382.53 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 558], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 902], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1502], 95.00th=[ 1821], | 99.00th=[ 2333], 99.50th=[ 2601], 99.90th=[ 2668], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 103, per=25.22%, avg=42.37, stdev=20.25, samples=241 iops : min= 1, max= 25, avg=10.05, stdev= 5.09, samples=241 lat (msec) : 50=0.23%, 100=0.27%, 250=0.90%, 500=31.88%, 750=22.93% lat (msec) : 1000=18.36%, 2000=23.67%, >=2000=1.76% cpu : usr=0.11%, sys=0.73%, ctx=4802, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145738: Thu Sep 3 16:45:05 2026 read: IOPS=10, BW=40.0KiB/s (41.9kB/s)(5060KiB/123512msec) slat (msec): min=3, max=213, avg=16.43, stdev=11.18 clat (msec): min=37, max=1416, avg=483.68, stdev=172.26 lat (msec): min=40, max=1423, avg=500.12, stdev=174.99 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 575], 90.00th=[ 693], 95.00th=[ 818], | 99.00th=[ 1099], 99.50th=[ 1133], 99.90th=[ 1351], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 111, per=25.42%, avg=41.18, stdev=20.77, samples=240 iops : min= 1, max= 27, avg= 9.76, stdev= 5.22, samples=240 write: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5180KiB/123512msec); 0 zone resets slat (usec): min=28, max=24565, avg=356.26, stdev=1181.06 clat (msec): min=376, max=2849, avg=1034.27, stdev=369.17 lat (msec): min=377, max=2849, avg=1034.63, stdev=369.20 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 584], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1485], 95.00th=[ 1770], | 99.00th=[ 2366], 99.50th=[ 2567], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 112, per=25.04%, avg=42.08, stdev=20.11, samples=240 iops : min= 1, max= 28, avg= 9.98, stdev= 5.06, samples=240 lat (msec) : 50=0.04%, 100=0.08%, 250=0.98%, 500=31.41%, 750=22.70% lat (msec) : 1000=21.33%, 2000=21.95%, >=2000=1.52% cpu : usr=0.11%, sys=0.70%, ctx=4594, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=162KiB/s (166kB/s), 40.3KiB/s-40.0KiB/s (41.2kB/s-41.9kB/s), io=19.6MiB (20.5MB), run=123454-123627msec WRITE: bw=169KiB/s (173kB/s), 41.9KiB/s-42.6KiB/s (42.9kB/s-43.6kB/s), io=20.4MiB (21.4MB), run=123454-123627msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145768: Thu Sep 3 16:45:16 2026 read: IOPS=1, BW=49.4MiB/s (51.8MB/s)(40.0MiB/810msec) slat (nsec): min=103190k, max=103190k, avg=103189956.00, stdev= 0.00 clat (nsec): min=679980k, max=679980k, avg=679980098.00, stdev= 0.00 lat (nsec): min=783176k, max=783176k, avg=783176276.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 684], 5.00th=[ 684], 10.00th=[ 684], 20.00th=[ 684], | 30.00th=[ 684], 40.00th=[ 684], 50.00th=[ 684], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] lat (msec) : 750=100.00% cpu : usr=0.25%, sys=2.84%, ctx=45, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=49.4MiB/s (51.8MB/s), 49.4MiB/s-49.4MiB/s (51.8MB/s-51.8MB/s), io=40.0MiB (41.9MB), run=810-810msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout