== sanity test 398c: run fio to test AIO ================= 12:48:23 (1788194903) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.587668 s, 71.4 MB/s osc.lustre-OST0000-osc-ffff8c7f4add5000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145316: Mon Aug 31 12:51:45 2026 write: IOPS=13, BW=53.7KiB/s (54.9kB/s)(10.0MiB/190821msec); 0 zone resets slat (usec): min=39, max=13884, avg=279.61, stdev=869.77 clat (msec): min=252, max=2078, avg=1191.20, stdev=319.13 lat (msec): min=252, max=2078, avg=1191.48, stdev=319.16 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 676], 10.00th=[ 818], 20.00th=[ 944], | 30.00th=[ 1020], 40.00th=[ 1099], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1334], 80.00th=[ 1452], 90.00th=[ 1620], 95.00th=[ 1720], | 99.00th=[ 1955], 99.50th=[ 2005], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 128, per=25.32%, avg=54.18, stdev=26.11, samples=371 iops : min= 1, max= 32, avg=13.08, stdev= 6.54, samples=371 lat (msec) : 500=1.80%, 750=4.69%, 1000=20.70%, 2000=72.23%, >=2000=0.59% cpu : usr=0.04%, sys=0.22%, ctx=1811, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145317: Mon Aug 31 12:51:45 2026 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(10.0MiB/190720msec); 0 zone resets slat (usec): min=35, max=30305, avg=277.42, stdev=962.00 clat (msec): min=84, max=2154, avg=1190.74, stdev=316.47 lat (msec): min=84, max=2154, avg=1191.02, stdev=316.48 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 651], 10.00th=[ 818], 20.00th=[ 944], | 30.00th=[ 1028], 40.00th=[ 1099], 50.00th=[ 1183], 60.00th=[ 1267], | 70.00th=[ 1351], 80.00th=[ 1452], 90.00th=[ 1603], 95.00th=[ 1720], | 99.00th=[ 1921], 99.50th=[ 1955], 99.90th=[ 2106], 99.95th=[ 2123], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 128, per=24.98%, avg=53.46, stdev=26.56, samples=376 iops : min= 1, max= 32, avg=12.91, stdev= 6.69, samples=376 lat (msec) : 100=0.04%, 250=0.23%, 500=1.95%, 750=4.34%, 1000=19.45% lat (msec) : 2000=73.75%, >=2000=0.23% cpu : usr=0.04%, sys=0.23%, ctx=2052, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145318: Mon Aug 31 12:51:45 2026 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(10.0MiB/190723msec); 0 zone resets slat (usec): min=40, max=20145, avg=291.52, stdev=952.82 clat (msec): min=271, max=2156, avg=1190.65, stdev=319.34 lat (msec): min=271, max=2156, avg=1190.94, stdev=319.36 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 667], 10.00th=[ 818], 20.00th=[ 936], | 30.00th=[ 1020], 40.00th=[ 1099], 50.00th=[ 1167], 60.00th=[ 1267], | 70.00th=[ 1351], 80.00th=[ 1452], 90.00th=[ 1620], 95.00th=[ 1737], | 99.00th=[ 1921], 99.50th=[ 2005], 99.90th=[ 2106], 99.95th=[ 2123], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 128, per=25.18%, avg=53.89, stdev=24.09, samples=373 iops : min= 1, max= 32, avg=13.02, stdev= 6.02, samples=373 lat (msec) : 500=2.50%, 750=4.18%, 1000=20.74%, 2000=72.07%, >=2000=0.51% cpu : usr=0.04%, sys=0.24%, ctx=2152, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145319: Mon Aug 31 12:51:45 2026 write: IOPS=13, BW=53.7KiB/s (55.0kB/s)(10.0MiB/190529msec); 0 zone resets slat (usec): min=49, max=26225, avg=257.30, stdev=910.39 clat (msec): min=144, max=2072, avg=1190.04, stdev=326.56 lat (msec): min=145, max=2073, avg=1190.30, stdev=326.58 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 651], 10.00th=[ 835], 20.00th=[ 936], | 30.00th=[ 1003], 40.00th=[ 1099], 50.00th=[ 1183], 60.00th=[ 1267], | 70.00th=[ 1351], 80.00th=[ 1452], 90.00th=[ 1620], 95.00th=[ 1737], | 99.00th=[ 1972], 99.50th=[ 2005], 99.90th=[ 2056], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 222, per=26.44%, avg=56.58, stdev=35.11, samples=355 iops : min= 1, max= 55, avg=13.69, stdev= 8.84, samples=355 lat (msec) : 250=0.70%, 500=1.48%, 750=4.84%, 1000=22.19%, 2000=70.27% lat (msec) : >=2000=0.51% cpu : usr=0.05%, sys=0.20%, ctx=1670, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=215KiB/s (220kB/s), 53.7KiB/s-53.7KiB/s (54.9kB/s-55.0kB/s), io=40.0MiB (41.9MB), run=190529-190821msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145356: Mon Aug 31 12:54:02 2026 read: IOPS=9, BW=39.6KiB/s (40.6kB/s)(5048KiB/127413msec) slat (usec): min=3339, max=79515, avg=16362.19, stdev=9421.07 clat (msec): min=19, max=1132, avg=505.80, stdev=141.14 lat (msec): min=34, max=1141, avg=522.17, stdev=142.80 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 313], 10.00th=[ 342], 20.00th=[ 384], | 30.00th=[ 422], 40.00th=[ 460], 50.00th=[ 493], 60.00th=[ 527], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 684], 95.00th=[ 760], | 99.00th=[ 894], 99.50th=[ 944], 99.90th=[ 1116], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 104, per=25.08%, avg=39.38, stdev=20.40, samples=250 iops : min= 1, max= 26, avg= 9.33, stdev= 5.15, samples=250 write: IOPS=10, BW=40.7KiB/s (41.7kB/s)(5192KiB/127413msec); 0 zone resets slat (usec): min=44, max=24016, avg=426.41, stdev=1513.55 clat (msec): min=174, max=2332, avg=1058.66, stdev=327.64 lat (msec): min=174, max=2332, avg=1059.08, stdev=327.65 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 575], 10.00th=[ 667], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 944], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1502], 95.00th=[ 1653], | 99.00th=[ 2056], 99.50th=[ 2140], 99.90th=[ 2265], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 112, per=24.72%, avg=40.29, stdev=19.65, samples=252 iops : min= 1, max= 28, avg= 9.56, stdev= 4.94, samples=252 lat (msec) : 20=0.04%, 50=0.08%, 100=0.12%, 250=0.23%, 500=26.05% lat (msec) : 750=28.67%, 1000=17.54%, 2000=26.60%, >=2000=0.66% cpu : usr=0.09%, sys=0.60%, ctx=4210, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145357: Mon Aug 31 12:54:02 2026 read: IOPS=9, BW=39.1KiB/s (40.0kB/s)(4976KiB/127309msec) slat (usec): min=78, max=83283, avg=16473.12, stdev=9451.73 clat (msec): min=18, max=1085, avg=491.65, stdev=145.63 lat (msec): min=27, max=1102, avg=508.14, stdev=147.62 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 288], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 414], 40.00th=[ 451], 50.00th=[ 485], 60.00th=[ 518], | 70.00th=[ 550], 80.00th=[ 609], 90.00th=[ 684], 95.00th=[ 735], | 99.00th=[ 877], 99.50th=[ 919], 99.90th=[ 1053], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 104, per=24.94%, avg=39.15, stdev=20.43, samples=248 iops : min= 1, max= 26, avg= 9.27, stdev= 5.16, samples=248 write: IOPS=10, BW=41.3KiB/s (42.3kB/s)(5264KiB/127309msec); 0 zone resets slat (usec): min=56, max=13015, avg=326.98, stdev=877.38 clat (msec): min=209, max=2278, avg=1066.25, stdev=342.68 lat (msec): min=213, max=2279, avg=1066.59, stdev=342.68 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 558], 10.00th=[ 651], 20.00th=[ 776], | 30.00th=[ 869], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1351], 90.00th=[ 1536], 95.00th=[ 1636], | 99.00th=[ 2106], 99.50th=[ 2198], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 103, per=25.04%, avg=40.81, stdev=19.67, samples=252 iops : min= 1, max= 25, avg= 9.68, stdev= 4.93, samples=252 lat (msec) : 20=0.04%, 50=0.27%, 100=0.23%, 250=0.86%, 500=26.60% lat (msec) : 750=27.62%, 1000=15.62%, 2000=28.05%, >=2000=0.70% cpu : usr=0.08%, sys=0.60%, ctx=4155, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145358: Mon Aug 31 12:54:02 2026 read: IOPS=9, BW=39.1KiB/s (40.1kB/s)(4984KiB/127402msec) slat (usec): min=3187, max=67814, avg=16468.78, stdev=9681.67 clat (msec): min=66, max=1020, avg=502.67, stdev=139.60 lat (msec): min=75, max=1038, avg=519.15, stdev=141.54 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 309], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 422], 40.00th=[ 451], 50.00th=[ 489], 60.00th=[ 518], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 709], 95.00th=[ 760], | 99.00th=[ 869], 99.50th=[ 894], 99.90th=[ 995], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 104, per=24.86%, avg=39.03, stdev=19.99, samples=250 iops : min= 1, max= 26, avg= 9.24, stdev= 5.05, samples=250 write: IOPS=10, BW=41.3KiB/s (42.2kB/s)(5256KiB/127402msec); 0 zone resets slat (usec): min=56, max=13295, avg=363.05, stdev=971.56 clat (msec): min=395, max=2276, avg=1055.54, stdev=330.14 lat (msec): min=395, max=2276, avg=1055.91, stdev=330.24 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 575], 10.00th=[ 659], 20.00th=[ 785], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1318], 90.00th=[ 1519], 95.00th=[ 1670], | 99.00th=[ 2056], 99.50th=[ 2106], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 96, per=25.45%, avg=41.49, stdev=20.90, samples=247 iops : min= 1, max= 24, avg= 9.85, stdev= 5.29, samples=247 lat (msec) : 100=0.08%, 250=0.35%, 500=26.68%, 750=27.77%, 1000=18.48% lat (msec) : 2000=26.02%, >=2000=0.62% cpu : usr=0.07%, sys=0.60%, ctx=4369, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145359: Mon Aug 31 12:54:02 2026 read: IOPS=9, BW=39.8KiB/s (40.7kB/s)(5060KiB/127222msec) slat (usec): min=3057, max=70510, avg=16355.08, stdev=9441.59 clat (msec): min=6, max=1146, avg=503.62, stdev=144.58 lat (msec): min=15, max=1175, avg=519.98, stdev=146.48 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 305], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 422], 40.00th=[ 460], 50.00th=[ 489], 60.00th=[ 527], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 693], 95.00th=[ 768], | 99.00th=[ 894], 99.50th=[ 936], 99.90th=[ 995], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 104, per=25.70%, avg=40.36, stdev=19.44, samples=245 iops : min= 1, max= 26, avg= 9.57, stdev= 4.91, samples=245 write: IOPS=10, BW=40.7KiB/s (41.7kB/s)(5180KiB/127222msec); 0 zone resets slat (usec): min=56, max=26131, avg=363.24, stdev=1280.75 clat (msec): min=352, max=2332, avg=1060.85, stdev=335.13 lat (msec): min=353, max=2332, avg=1061.21, stdev=335.17 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 567], 10.00th=[ 659], 20.00th=[ 785], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1200], 80.00th=[ 1334], 90.00th=[ 1502], 95.00th=[ 1670], | 99.00th=[ 2072], 99.50th=[ 2165], 99.90th=[ 2232], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 96, per=24.83%, avg=40.47, stdev=20.01, samples=250 iops : min= 1, max= 24, avg= 9.60, stdev= 5.04, samples=250 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=0.23%, 250=0.39% lat (msec) : 500=26.60%, 750=28.16%, 1000=17.46%, 2000=26.25%, >=2000=0.74% cpu : usr=0.08%, sys=0.58%, ctx=4252, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=158KiB/s (161kB/s), 39.1KiB/s-39.8KiB/s (40.0kB/s-40.7kB/s), io=19.6MiB (20.5MB), run=127222-127413msec WRITE: bw=164KiB/s (168kB/s), 40.7KiB/s-41.3KiB/s (41.7kB/s-42.3kB/s), io=20.4MiB (21.4MB), run=127222-127413msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145383: Mon Aug 31 12:54:13 2026 read: IOPS=2, BW=93.0MiB/s (97.5MB/s)(40.0MiB/430msec) slat (nsec): min=21482k, max=21482k, avg=21482467.00, stdev= 0.00 clat (nsec): min=386974k, max=386974k, avg=386973667.00, stdev= 0.00 lat (nsec): min=408460k, max=408460k, avg=408459916.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] lat (msec) : 500=100.00% cpu : usr=0.23%, sys=3.03%, ctx=5, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=93.0MiB/s (97.5MB/s), 93.0MiB/s-93.0MiB/s (97.5MB/s-97.5MB/s), io=40.0MiB (41.9MB), run=430-430msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout