== sanity test 398c: run fio to test AIO ================= 09:58:40 (1787579920) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.555786 s, 75.5 MB/s osc.lustre-OST0000-osc-ffff940f609fe000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180516: Mon Aug 24 09:59:24 2026 write: IOPS=81, BW=326KiB/s (334kB/s)(10.0MiB/31428msec); 0 zone resets slat (usec): min=28, max=25561, avg=225.33, stdev=794.46 clat (msec): min=25, max=414, avg=195.85, stdev=46.03 lat (msec): min=26, max=414, avg=196.08, stdev=46.02 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 207], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 257], 95.00th=[ 275], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 224, max= 461, per=25.06%, avg=326.03, stdev=57.44, samples=62 iops : min= 56, max= 115, avg=81.15, stdev=14.33, samples=62 lat (msec) : 50=0.16%, 100=0.20%, 250=88.01%, 500=11.64% cpu : usr=0.21%, sys=0.98%, ctx=2594, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180517: Mon Aug 24 09:59:24 2026 write: IOPS=82, BW=328KiB/s (336kB/s)(10.0MiB/31209msec); 0 zone resets slat (usec): min=28, max=19321, avg=272.63, stdev=1009.93 clat (msec): min=17, max=338, avg=194.61, stdev=45.91 lat (msec): min=20, max=338, avg=194.88, stdev=45.90 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 127], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 236], 90.00th=[ 257], 95.00th=[ 279], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 338] bw ( KiB/s): min= 216, max= 502, per=25.08%, avg=326.34, stdev=66.61, samples=62 iops : min= 54, max= 125, avg=81.21, stdev=16.61, samples=62 lat (msec) : 20=0.04%, 50=0.04%, 100=0.31%, 250=86.33%, 500=13.28% cpu : usr=0.16%, sys=0.96%, ctx=2206, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180518: Mon Aug 24 09:59:24 2026 write: IOPS=81, BW=325KiB/s (333kB/s)(10.0MiB/31466msec); 0 zone resets slat (usec): min=30, max=20621, avg=229.61, stdev=922.65 clat (msec): min=25, max=336, avg=195.57, stdev=45.19 lat (msec): min=25, max=337, avg=195.80, stdev=45.17 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 207], | 70.00th=[ 215], 80.00th=[ 234], 90.00th=[ 257], 95.00th=[ 275], | 99.00th=[ 326], 99.50th=[ 330], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 184, max= 477, per=24.96%, avg=324.69, stdev=65.84, samples=62 iops : min= 46, max= 119, avg=80.81, stdev=16.40, samples=62 lat (msec) : 50=0.08%, 100=0.23%, 250=86.95%, 500=12.73% cpu : usr=0.17%, sys=0.95%, ctx=2155, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180519: Mon Aug 24 09:59:24 2026 write: IOPS=81, BW=325KiB/s (333kB/s)(10.0MiB/31478msec); 0 zone resets slat (usec): min=29, max=14566, avg=253.99, stdev=812.20 clat (msec): min=23, max=329, avg=195.36, stdev=44.84 lat (msec): min=23, max=329, avg=195.62, stdev=44.82 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 239, max= 493, per=24.92%, avg=324.23, stdev=63.09, samples=62 iops : min= 59, max= 123, avg=80.68, stdev=15.76, samples=62 lat (msec) : 50=0.20%, 100=0.20%, 250=88.28%, 500=11.33% cpu : usr=0.18%, sys=0.91%, ctx=2615, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1301KiB/s (1332kB/s), 325KiB/s-328KiB/s (333kB/s-336kB/s), io=40.0MiB (41.9MB), run=31209-31478msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180533: Mon Aug 24 10:00:07 2026 read: IOPS=34, BW=138KiB/s (142kB/s)(5048KiB/36486msec) slat (usec): min=3148, max=76750, avg=15933.66, stdev=9049.01 clat (msec): min=63, max=438, avg=214.97, stdev=58.88 lat (msec): min=76, max=478, avg=230.92, stdev=61.01 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 180], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 262], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 384], 99.50th=[ 397], 99.90th=[ 414], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 68, max= 215, per=25.01%, avg=137.53, stdev=35.90, samples=72 iops : min= 17, max= 53, avg=33.96, stdev= 9.00, samples=72 write: IOPS=35, BW=142KiB/s (146kB/s)(5192KiB/36486msec); 0 zone resets slat (usec): min=37, max=14021, avg=287.11, stdev=917.14 clat (msec): min=63, max=465, avg=222.41, stdev=56.77 lat (msec): min=63, max=465, avg=222.70, stdev=56.73 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 178], | 30.00th=[ 190], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 228], | 70.00th=[ 245], 80.00th=[ 266], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 380], 99.50th=[ 405], 99.90th=[ 439], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 48, max= 227, per=24.79%, avg=141.81, stdev=37.28, samples=72 iops : min= 12, max= 56, avg=35.03, stdev= 9.34, samples=72 lat (msec) : 100=0.27%, 250=73.75%, 500=25.98% cpu : usr=0.27%, sys=1.65%, ctx=4689, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180534: Mon Aug 24 10:00:07 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(4976KiB/36311msec) slat (usec): min=2901, max=69217, avg=15655.91, stdev=8428.67 clat (msec): min=46, max=466, avg=213.91, stdev=60.51 lat (msec): min=63, max=489, avg=229.58, stdev=63.11 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 239], 80.00th=[ 262], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 401], 99.50th=[ 418], 99.90th=[ 447], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 47, max= 229, per=24.70%, avg=135.88, stdev=39.69, samples=72 iops : min= 11, max= 57, avg=33.54, stdev= 9.96, samples=72 write: IOPS=36, BW=145KiB/s (148kB/s)(5264KiB/36311msec); 0 zone resets slat (usec): min=28, max=14041, avg=262.91, stdev=827.67 clat (msec): min=66, max=455, avg=222.53, stdev=56.09 lat (msec): min=66, max=455, avg=222.80, stdev=56.11 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 232], | 70.00th=[ 245], 80.00th=[ 268], 90.00th=[ 300], 95.00th=[ 326], | 99.00th=[ 376], 99.50th=[ 405], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 46, max= 280, per=25.21%, avg=144.21, stdev=40.53, samples=72 iops : min= 11, max= 70, avg=35.64, stdev=10.23, samples=72 lat (msec) : 50=0.04%, 100=0.47%, 250=73.67%, 500=25.82% cpu : usr=0.24%, sys=1.62%, ctx=4337, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180535: Mon Aug 24 10:00:07 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(4984KiB/36465msec) slat (msec): min=2, max=214, avg=15.90, stdev=10.26 clat (msec): min=45, max=430, avg=216.80, stdev=59.76 lat (msec): min=75, max=452, avg=232.70, stdev=61.72 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 243], 80.00th=[ 266], 90.00th=[ 300], 95.00th=[ 334], | 99.00th=[ 380], 99.50th=[ 393], 99.90th=[ 405], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 27, max= 229, per=24.82%, avg=136.51, stdev=37.76, samples=72 iops : min= 6, max= 57, avg=33.71, stdev= 9.52, samples=72 write: IOPS=36, BW=144KiB/s (148kB/s)(5256KiB/36465msec); 0 zone resets slat (usec): min=33, max=31971, avg=327.70, stdev=1327.90 clat (msec): min=91, max=429, avg=220.45, stdev=58.71 lat (msec): min=91, max=429, avg=220.78, stdev=58.65 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 215], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 268], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 368], 99.50th=[ 397], 99.90th=[ 426], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 48, max= 232, per=25.04%, avg=143.25, stdev=42.42, samples=72 iops : min= 12, max= 58, avg=35.39, stdev=10.64, samples=72 lat (msec) : 50=0.04%, 100=0.31%, 250=73.36%, 500=26.29% cpu : usr=0.26%, sys=1.71%, ctx=4577, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180536: Mon Aug 24 10:00:07 2026 read: IOPS=34, BW=139KiB/s (143kB/s)(5060KiB/36313msec) slat (usec): min=37, max=247609, avg=16111.91, stdev=11630.15 clat (msec): min=20, max=591, avg=216.27, stdev=64.54 lat (msec): min=21, max=604, avg=232.39, stdev=67.77 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 259], 90.00th=[ 292], 95.00th=[ 334], | 99.00th=[ 418], 99.50th=[ 456], 99.90th=[ 584], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 47, max= 216, per=25.15%, avg=138.31, stdev=38.24, samples=72 iops : min= 11, max= 54, avg=34.14, stdev= 9.63, samples=72 write: IOPS=35, BW=143KiB/s (146kB/s)(5180KiB/36313msec); 0 zone resets slat (usec): min=31, max=15385, avg=257.77, stdev=838.86 clat (msec): min=50, max=649, avg=220.78, stdev=63.94 lat (msec): min=51, max=650, avg=221.04, stdev=63.95 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 266], 90.00th=[ 300], 95.00th=[ 334], | 99.00th=[ 409], 99.50th=[ 592], 99.90th=[ 617], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 71, max= 259, per=24.76%, avg=141.63, stdev=38.96, samples=72 iops : min= 17, max= 64, avg=34.99, stdev= 9.77, samples=72 lat (msec) : 50=0.16%, 100=0.35%, 250=74.61%, 500=24.34%, 750=0.55% cpu : usr=0.25%, sys=1.60%, ctx=4375, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=550KiB/s (563kB/s), 137KiB/s-139KiB/s (140kB/s-143kB/s), io=19.6MiB (20.5MB), run=36311-36486msec WRITE: bw=573KiB/s (586kB/s), 142KiB/s-145KiB/s (146kB/s-148kB/s), io=20.4MiB (21.4MB), run=36311-36486msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180546: Mon Aug 24 10:00:19 2026 read: IOPS=1, BW=65.6MiB/s (68.8MB/s)(40.0MiB/610msec) slat (nsec): min=53857k, max=53857k, avg=53857326.00, stdev= 0.00 clat (nsec): min=521162k, max=521162k, avg=521162167.00, stdev= 0.00 lat (nsec): min=575024k, max=575024k, avg=575023608.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 523], | 30.00th=[ 523], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 523], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.81%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.6MiB/s (68.8MB/s), 65.6MiB/s-65.6MiB/s (68.8MB/s-68.8MB/s), io=40.0MiB (41.9MB), run=610-610msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout