== sanity test 398c: run fio to test AIO ================= 21:55:58 (1767840958) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.13903 s, 302 MB/s osc.lustre-OST0000-osc-ffff9c9b8a9f3800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=121853: Wed Jan 7 21:56:56 2026 write: IOPS=45, BW=184KiB/s (188kB/s)(10.0MiB/55686msec); 0 zone resets slat (usec): min=20, max=12189, avg=176.72, stdev=436.47 clat (msec): min=71, max=767, avg=347.71, stdev=159.41 lat (msec): min=71, max=767, avg=347.89, stdev=159.45 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 125], 10.00th=[ 153], 20.00th=[ 184], | 30.00th=[ 220], 40.00th=[ 264], 50.00th=[ 355], 60.00th=[ 405], | 70.00th=[ 460], 80.00th=[ 498], 90.00th=[ 550], 95.00th=[ 609], | 99.00th=[ 693], 99.50th=[ 735], 99.90th=[ 743], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 64, max= 521, per=24.86%, avg=182.69, stdev=96.01, samples=110 iops : min= 16, max= 130, avg=45.45, stdev=24.07, samples=110 lat (msec) : 100=1.68%, 250=35.94%, 500=42.97%, 750=19.34%, 1000=0.08% cpu : usr=0.13%, sys=0.60%, ctx=1635, majf=1, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=121854: Wed Jan 7 21:56:56 2026 write: IOPS=45, BW=184KiB/s (188kB/s)(10.0MiB/55666msec); 0 zone resets slat (usec): min=19, max=7001, avg=172.69, stdev=344.00 clat (msec): min=42, max=829, avg=347.61, stdev=160.08 lat (msec): min=42, max=829, avg=347.78, stdev=160.11 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 126], 10.00th=[ 150], 20.00th=[ 184], | 30.00th=[ 218], 40.00th=[ 266], 50.00th=[ 351], 60.00th=[ 405], | 70.00th=[ 451], 80.00th=[ 502], 90.00th=[ 558], 95.00th=[ 609], | 99.00th=[ 709], 99.50th=[ 735], 99.90th=[ 802], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 80, max= 566, per=24.88%, avg=182.87, stdev=96.06, samples=110 iops : min= 20, max= 141, avg=45.48, stdev=24.09, samples=110 lat (msec) : 50=0.12%, 100=1.72%, 250=35.39%, 500=42.89%, 750=19.73% lat (msec) : 1000=0.16% cpu : usr=0.11%, sys=0.65%, ctx=1785, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=121855: Wed Jan 7 21:56:56 2026 write: IOPS=46, BW=185KiB/s (189kB/s)(10.0MiB/55484msec); 0 zone resets slat (usec): min=20, max=4279, avg=151.10, stdev=220.30 clat (msec): min=15, max=812, avg=346.52, stdev=159.88 lat (msec): min=16, max=812, avg=346.67, stdev=159.90 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 125], 10.00th=[ 153], 20.00th=[ 182], | 30.00th=[ 220], 40.00th=[ 264], 50.00th=[ 351], 60.00th=[ 405], | 70.00th=[ 456], 80.00th=[ 498], 90.00th=[ 558], 95.00th=[ 609], | 99.00th=[ 693], 99.50th=[ 726], 99.90th=[ 751], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 72, max= 631, per=24.89%, avg=182.97, stdev=97.09, samples=110 iops : min= 18, max= 157, avg=45.51, stdev=24.32, samples=110 lat (msec) : 20=0.16%, 50=0.51%, 100=1.25%, 250=35.35%, 500=43.32% lat (msec) : 750=19.34%, 1000=0.08% cpu : usr=0.09%, sys=0.60%, ctx=1492, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=121856: Wed Jan 7 21:56:56 2026 write: IOPS=45, BW=184KiB/s (188kB/s)(10.0MiB/55685msec); 0 zone resets slat (usec): min=21, max=8492, avg=164.00, stdev=369.26 clat (msec): min=70, max=818, avg=347.74, stdev=159.66 lat (msec): min=71, max=819, avg=347.90, stdev=159.70 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 125], 10.00th=[ 153], 20.00th=[ 184], | 30.00th=[ 218], 40.00th=[ 266], 50.00th=[ 347], 60.00th=[ 405], | 70.00th=[ 456], 80.00th=[ 502], 90.00th=[ 550], 95.00th=[ 600], | 99.00th=[ 726], 99.50th=[ 735], 99.90th=[ 802], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 72, max= 522, per=24.83%, avg=182.49, stdev=94.45, samples=110 iops : min= 18, max= 130, avg=45.39, stdev=23.67, samples=110 lat (msec) : 100=1.45%, 250=35.31%, 500=43.44%, 750=19.49%, 1000=0.31% cpu : usr=0.10%, sys=0.61%, ctx=1596, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=736KiB/s (753kB/s), 184KiB/s-185KiB/s (188kB/s-189kB/s), io=40.0MiB (41.9MB), run=55484-55686msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=121873: Wed Jan 7 21:57:26 2026 read: IOPS=48, BW=196KiB/s (201kB/s)(5048KiB/25767msec) slat (usec): min=51, max=11558, avg=2688.08, stdev=1663.22 clat (msec): min=2, max=264, avg=106.98, stdev=32.10 lat (msec): min=2, max=267, avg=109.67, stdev=32.57 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 65], 10.00th=[ 73], 20.00th=[ 82], | 30.00th=[ 89], 40.00th=[ 97], 50.00th=[ 103], 60.00th=[ 110], | 70.00th=[ 121], 80.00th=[ 133], 90.00th=[ 150], 95.00th=[ 165], | 99.00th=[ 197], 99.50th=[ 205], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 95, max= 311, per=25.43%, avg=195.02, stdev=51.34, samples=51 iops : min= 23, max= 77, avg=48.53, stdev=12.86, samples=51 write: IOPS=50, BW=201KiB/s (206kB/s)(5192KiB/25767msec); 0 zone resets slat (usec): min=22, max=3673, avg=141.72, stdev=242.06 clat (msec): min=28, max=460, avg=210.60, stdev=69.48 lat (msec): min=28, max=460, avg=210.75, stdev=69.50 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 113], 10.00th=[ 132], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 184], 50.00th=[ 201], 60.00th=[ 222], | 70.00th=[ 243], 80.00th=[ 264], 90.00th=[ 305], 95.00th=[ 338], | 99.00th=[ 418], 99.50th=[ 426], 99.90th=[ 447], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 104, max= 376, per=25.17%, avg=201.10, stdev=55.71, samples=51 iops : min= 26, max= 94, avg=50.06, stdev=13.92, samples=51 lat (msec) : 4=0.04%, 10=0.23%, 20=0.16%, 50=0.59%, 100=22.46% lat (msec) : 250=63.09%, 500=13.44% cpu : usr=0.20%, sys=1.72%, ctx=2687, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121874: Wed Jan 7 21:57:26 2026 read: IOPS=47, BW=190KiB/s (195kB/s)(4976KiB/26147msec) slat (usec): min=814, max=12041, avg=2677.65, stdev=1564.20 clat (usec): min=1810, max=266916, avg=108479.16, stdev=32132.93 lat (msec): min=3, max=270, avg=111.16, stdev=32.58 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 67], 10.00th=[ 72], 20.00th=[ 83], | 30.00th=[ 90], 40.00th=[ 96], 50.00th=[ 105], 60.00th=[ 113], | 70.00th=[ 122], 80.00th=[ 134], 90.00th=[ 153], 95.00th=[ 167], | 99.00th=[ 194], 99.50th=[ 209], 99.90th=[ 264], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 96, max= 344, per=24.73%, avg=189.65, stdev=52.52, samples=51 iops : min= 24, max= 86, avg=47.20, stdev=13.15, samples=51 write: IOPS=50, BW=201KiB/s (206kB/s)(5264KiB/26147msec); 0 zone resets slat (usec): min=23, max=3864, avg=140.57, stdev=198.06 clat (msec): min=74, max=439, avg=212.24, stdev=67.42 lat (msec): min=74, max=440, avg=212.38, stdev=67.43 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 118], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 184], 50.00th=[ 203], 60.00th=[ 222], | 70.00th=[ 247], 80.00th=[ 266], 90.00th=[ 300], 95.00th=[ 338], | 99.00th=[ 405], 99.50th=[ 418], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 112, max= 312, per=25.20%, avg=201.33, stdev=53.73, samples=51 iops : min= 28, max= 78, avg=50.12, stdev=13.45, samples=51 lat (msec) : 2=0.04%, 4=0.04%, 20=0.20%, 50=0.12%, 100=22.50% lat (msec) : 250=62.58%, 500=14.53% cpu : usr=0.25%, sys=1.63%, ctx=2640, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121875: Wed Jan 7 21:57:26 2026 read: IOPS=47, BW=191KiB/s (195kB/s)(4984KiB/26135msec) slat (usec): min=903, max=12550, avg=2681.19, stdev=1640.02 clat (msec): min=4, max=276, avg=106.66, stdev=30.33 lat (msec): min=6, max=277, avg=109.35, stdev=30.79 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 66], 10.00th=[ 74], 20.00th=[ 83], | 30.00th=[ 89], 40.00th=[ 96], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 118], 80.00th=[ 131], 90.00th=[ 150], 95.00th=[ 161], | 99.00th=[ 188], 99.50th=[ 197], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 72, max= 312, per=24.95%, avg=191.35, stdev=62.75, samples=51 iops : min= 18, max= 78, avg=47.61, stdev=15.74, samples=51 write: IOPS=50, BW=201KiB/s (206kB/s)(5256KiB/26135msec); 0 zone resets slat (usec): min=25, max=3784, avg=146.29, stdev=219.20 clat (msec): min=72, max=438, avg=213.85, stdev=68.72 lat (msec): min=72, max=438, avg=214.00, stdev=68.73 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 117], 10.00th=[ 134], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 186], 50.00th=[ 207], 60.00th=[ 228], | 70.00th=[ 247], 80.00th=[ 271], 90.00th=[ 309], 95.00th=[ 338], | 99.00th=[ 401], 99.50th=[ 422], 99.90th=[ 435], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 96, max= 360, per=25.14%, avg=200.90, stdev=50.31, samples=51 iops : min= 24, max= 90, avg=50.00, stdev=12.57, samples=51 lat (msec) : 10=0.08%, 20=0.16%, 50=0.23%, 100=22.70%, 250=61.99% lat (msec) : 500=14.84% cpu : usr=0.22%, sys=1.63%, ctx=2719, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121876: Wed Jan 7 21:57:26 2026 read: IOPS=48, BW=194KiB/s (198kB/s)(5060KiB/26119msec) slat (usec): min=843, max=13040, avg=2766.10, stdev=1687.72 clat (msec): min=9, max=400, avg=108.74, stdev=34.93 lat (msec): min=14, max=405, avg=111.51, stdev=35.34 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 67], 10.00th=[ 74], 20.00th=[ 83], | 30.00th=[ 91], 40.00th=[ 99], 50.00th=[ 105], 60.00th=[ 112], | 70.00th=[ 121], 80.00th=[ 131], 90.00th=[ 153], 95.00th=[ 165], | 99.00th=[ 197], 99.50th=[ 264], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 87, max= 304, per=25.40%, avg=194.78, stdev=50.47, samples=51 iops : min= 21, max= 76, avg=48.49, stdev=12.59, samples=51 write: IOPS=49, BW=198KiB/s (203kB/s)(5180KiB/26119msec); 0 zone resets slat (usec): min=20, max=2749, avg=133.30, stdev=191.05 clat (msec): min=67, max=451, avg=213.05, stdev=69.57 lat (msec): min=67, max=451, avg=213.19, stdev=69.58 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 116], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 184], 50.00th=[ 201], 60.00th=[ 222], | 70.00th=[ 245], 80.00th=[ 275], 90.00th=[ 309], 95.00th=[ 342], | 99.00th=[ 409], 99.50th=[ 426], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 112, max= 328, per=24.89%, avg=198.86, stdev=51.45, samples=51 iops : min= 28, max= 82, avg=49.49, stdev=12.92, samples=51 lat (msec) : 10=0.04%, 20=0.16%, 50=0.31%, 100=21.56%, 250=63.36% lat (msec) : 500=14.57% cpu : usr=0.24%, sys=1.64%, ctx=2707, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=768KiB/s (786kB/s), 190KiB/s-196KiB/s (195kB/s-201kB/s), io=19.6MiB (20.5MB), run=25767-26147msec WRITE: bw=799KiB/s (818kB/s), 198KiB/s-201KiB/s (203kB/s-206kB/s), io=20.4MiB (21.4MB), run=25767-26147msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=121884: Wed Jan 7 21:57:31 2026 read: IOPS=7, BW=294MiB/s (308MB/s)(40.0MiB/136msec) slat (nsec): min=13600k, max=13600k, avg=13600058.00, stdev= 0.00 clat (nsec): min=116835k, max=116835k, avg=116834879.00, stdev= 0.00 lat (nsec): min=130440k, max=130440k, avg=130440219.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=10.37%, ctx=3, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=294MiB/s (308MB/s), 294MiB/s-294MiB/s (308MB/s-308MB/s), io=40.0MiB (41.9MB), run=136-136msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck