== sanity test 398c: run fio to test AIO ================= 22:47:16 (1775875636) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.321399 s, 131 MB/s osc.lustre-OST0000-osc-ffff953f481f2800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148049: Fri Apr 10 22:48:47 2026 write: IOPS=29, BW=119KiB/s (122kB/s)(10.0MiB/86266msec); 0 zone resets slat (usec): min=36, max=14855, avg=302.69, stdev=631.66 clat (msec): min=134, max=961, avg=538.11, stdev=113.81 lat (msec): min=135, max=961, avg=538.42, stdev=113.83 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 355], 10.00th=[ 401], 20.00th=[ 443], | 30.00th=[ 485], 40.00th=[ 510], 50.00th=[ 535], 60.00th=[ 567], | 70.00th=[ 592], 80.00th=[ 625], 90.00th=[ 676], 95.00th=[ 718], | 99.00th=[ 852], 99.50th=[ 885], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 961] bw ( KiB/s): min= 23, max= 255, per=24.95%, avg=118.27, stdev=30.35, samples=171 iops : min= 5, max= 63, avg=29.20, stdev= 7.62, samples=171 lat (msec) : 250=0.66%, 500=35.35%, 750=60.78%, 1000=3.20% cpu : usr=0.07%, sys=0.53%, ctx=2529, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148050: Fri Apr 10 22:48:47 2026 write: IOPS=29, BW=119KiB/s (122kB/s)(10.0MiB/86079msec); 0 zone resets slat (usec): min=37, max=9173, avg=290.33, stdev=562.37 clat (msec): min=174, max=936, avg=537.46, stdev=114.29 lat (msec): min=175, max=936, avg=537.75, stdev=114.30 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 355], 10.00th=[ 397], 20.00th=[ 447], | 30.00th=[ 481], 40.00th=[ 506], 50.00th=[ 535], 60.00th=[ 567], | 70.00th=[ 592], 80.00th=[ 625], 90.00th=[ 676], 95.00th=[ 718], | 99.00th=[ 852], 99.50th=[ 885], 99.90th=[ 919], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 31, max= 255, per=24.95%, avg=118.26, stdev=30.68, samples=171 iops : min= 7, max= 63, avg=29.20, stdev= 7.73, samples=171 lat (msec) : 250=0.74%, 500=37.27%, 750=58.55%, 1000=3.44% cpu : usr=0.06%, sys=0.56%, ctx=2745, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148051: Fri Apr 10 22:48:47 2026 write: IOPS=29, BW=119KiB/s (122kB/s)(10.0MiB/86261msec); 0 zone resets slat (usec): min=37, max=9788, avg=285.98, stdev=519.86 clat (msec): min=177, max=971, avg=538.08, stdev=113.43 lat (msec): min=177, max=971, avg=538.37, stdev=113.45 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 359], 10.00th=[ 401], 20.00th=[ 439], | 30.00th=[ 472], 40.00th=[ 502], 50.00th=[ 535], 60.00th=[ 567], | 70.00th=[ 592], 80.00th=[ 634], 90.00th=[ 667], 95.00th=[ 726], | 99.00th=[ 860], 99.50th=[ 927], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 48, max= 256, per=25.12%, avg=119.08, stdev=31.39, samples=170 iops : min= 12, max= 64, avg=29.41, stdev= 7.88, samples=170 lat (msec) : 250=0.16%, 500=38.48%, 750=57.62%, 1000=3.75% cpu : usr=0.06%, sys=0.54%, ctx=2955, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148052: Fri Apr 10 22:48:47 2026 write: IOPS=29, BW=119KiB/s (121kB/s)(10.0MiB/86306msec); 0 zone resets slat (usec): min=37, max=8243, avg=306.43, stdev=566.24 clat (msec): min=59, max=926, avg=538.36, stdev=110.62 lat (msec): min=59, max=934, avg=538.67, stdev=110.64 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 372], 10.00th=[ 405], 20.00th=[ 443], | 30.00th=[ 477], 40.00th=[ 510], 50.00th=[ 535], 60.00th=[ 567], | 70.00th=[ 592], 80.00th=[ 625], 90.00th=[ 676], 95.00th=[ 718], | 99.00th=[ 852], 99.50th=[ 894], 99.90th=[ 919], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 24, max= 255, per=24.90%, avg=118.01, stdev=27.38, samples=171 iops : min= 6, max= 63, avg=29.14, stdev= 6.89, samples=171 lat (msec) : 100=0.04%, 250=0.16%, 500=37.42%, 750=59.18%, 1000=3.20% cpu : usr=0.08%, sys=0.55%, ctx=2965, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=475KiB/s (486kB/s), 119KiB/s-119KiB/s (121kB/s-122kB/s), io=40.0MiB (41.9MB), run=86079-86306msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148074: Fri Apr 10 22:49:50 2026 read: IOPS=21, BW=87.7KiB/s (89.8kB/s)(5048KiB/57545msec) slat (usec): min=1744, max=25571, avg=7610.33, stdev=3459.42 clat (msec): min=8, max=370, avg=224.54, stdev=47.08 lat (msec): min=11, max=382, avg=232.16, stdev=47.62 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 188], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 222], 60.00th=[ 234], | 70.00th=[ 249], 80.00th=[ 262], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 342], 99.50th=[ 347], 99.90th=[ 368], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 23, max= 152, per=24.98%, avg=86.92, stdev=28.97, samples=114 iops : min= 5, max= 38, avg=21.38, stdev= 7.27, samples=114 write: IOPS=22, BW=90.2KiB/s (92.4kB/s)(5192KiB/57545msec); 0 zone resets slat (usec): min=43, max=9368, avg=368.28, stdev=637.33 clat (msec): min=121, max=847, avg=481.98, stdev=116.13 lat (msec): min=122, max=847, avg=482.35, stdev=116.16 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 305], 10.00th=[ 342], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 634], 95.00th=[ 684], | 99.00th=[ 785], 99.50th=[ 802], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 40, max= 143, per=24.67%, avg=89.54, stdev=23.50, samples=114 iops : min= 10, max= 35, avg=22.04, stdev= 5.91, samples=114 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=0.12%, 250=35.78% lat (msec) : 500=43.20%, 750=19.92%, 1000=0.82% cpu : usr=0.17%, sys=1.32%, ctx=5323, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148075: Fri Apr 10 22:49:50 2026 read: IOPS=21, BW=86.5KiB/s (88.6kB/s)(4976KiB/57496msec) slat (usec): min=2126, max=27056, avg=7876.73, stdev=3635.26 clat (msec): min=11, max=398, avg=225.81, stdev=49.66 lat (msec): min=18, max=403, avg=233.69, stdev=50.27 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 159], 10.00th=[ 171], 20.00th=[ 188], | 30.00th=[ 199], 40.00th=[ 213], 50.00th=[ 224], 60.00th=[ 234], | 70.00th=[ 249], 80.00th=[ 264], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 380], 99.90th=[ 393], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 23, max= 152, per=24.68%, avg=85.89, stdev=29.74, samples=114 iops : min= 5, max= 38, avg=21.12, stdev= 7.49, samples=114 write: IOPS=22, BW=91.6KiB/s (93.8kB/s)(5264KiB/57496msec); 0 zone resets slat (usec): min=48, max=8638, avg=362.32, stdev=532.72 clat (msec): min=153, max=832, avg=476.42, stdev=117.14 lat (msec): min=153, max=833, avg=476.78, stdev=117.17 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 305], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 464], 60.00th=[ 498], | 70.00th=[ 531], 80.00th=[ 575], 90.00th=[ 634], 95.00th=[ 693], | 99.00th=[ 785], 99.50th=[ 802], 99.90th=[ 827], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 39, max= 143, per=25.12%, avg=91.20, stdev=22.60, samples=114 iops : min= 9, max= 35, avg=22.45, stdev= 5.66, samples=114 lat (msec) : 20=0.04%, 50=0.31%, 100=0.16%, 250=34.38%, 500=44.73% lat (msec) : 750=19.49%, 1000=0.90% cpu : usr=0.14%, sys=1.29%, ctx=5352, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148076: Fri Apr 10 22:49:50 2026 read: IOPS=21, BW=86.7KiB/s (88.8kB/s)(4984KiB/57494msec) slat (usec): min=2267, max=33915, avg=7878.31, stdev=3598.56 clat (msec): min=9, max=388, avg=226.25, stdev=49.31 lat (msec): min=20, max=404, avg=234.14, stdev=49.84 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 150], 10.00th=[ 169], 20.00th=[ 188], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 251], 80.00th=[ 266], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 338], 99.50th=[ 368], 99.90th=[ 384], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 31, max= 200, per=24.78%, avg=86.23, stdev=29.65, samples=114 iops : min= 7, max= 50, avg=21.20, stdev= 7.48, samples=114 write: IOPS=22, BW=91.4KiB/s (93.6kB/s)(5256KiB/57494msec); 0 zone resets slat (usec): min=42, max=7996, avg=359.40, stdev=550.36 clat (msec): min=156, max=853, avg=476.47, stdev=115.93 lat (msec): min=156, max=853, avg=476.83, stdev=115.95 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 309], 10.00th=[ 338], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 464], 60.00th=[ 493], | 70.00th=[ 542], 80.00th=[ 575], 90.00th=[ 634], 95.00th=[ 676], | 99.00th=[ 768], 99.50th=[ 802], 99.90th=[ 844], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 23, max= 159, per=24.95%, avg=90.58, stdev=24.71, samples=114 iops : min= 5, max= 39, avg=22.30, stdev= 6.21, samples=114 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.20%, 250=33.98% lat (msec) : 500=45.86%, 750=19.06%, 1000=0.70% cpu : usr=0.13%, sys=1.27%, ctx=5243, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148077: Fri Apr 10 22:49:50 2026 read: IOPS=22, BW=88.6KiB/s (90.7kB/s)(5060KiB/57097msec) slat (usec): min=88, max=21454, avg=7698.47, stdev=3523.44 clat (msec): min=9, max=391, avg=225.96, stdev=50.77 lat (msec): min=9, max=398, avg=233.66, stdev=51.52 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 153], 10.00th=[ 169], 20.00th=[ 188], | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 236], | 70.00th=[ 251], 80.00th=[ 266], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 338], 99.50th=[ 355], 99.90th=[ 380], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 7, max= 167, per=25.41%, avg=88.43, stdev=30.28, samples=113 iops : min= 1, max= 41, avg=21.76, stdev= 7.60, samples=113 write: IOPS=22, BW=90.7KiB/s (92.9kB/s)(5180KiB/57097msec); 0 zone resets slat (usec): min=51, max=7740, avg=396.77, stdev=683.78 clat (msec): min=68, max=813, avg=476.34, stdev=116.79 lat (msec): min=68, max=814, avg=476.74, stdev=116.81 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 443], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 535], 80.00th=[ 575], 90.00th=[ 625], 95.00th=[ 676], | 99.00th=[ 776], 99.50th=[ 802], 99.90th=[ 810], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 32, max= 146, per=24.77%, avg=89.92, stdev=24.47, samples=113 iops : min= 8, max= 36, avg=22.13, stdev= 6.14, samples=113 lat (msec) : 10=0.04%, 20=0.16%, 50=0.35%, 100=0.31%, 250=34.57% lat (msec) : 500=44.22%, 750=19.61%, 1000=0.74% cpu : usr=0.14%, sys=1.34%, ctx=5292, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=349KiB/s (357kB/s), 86.5KiB/s-88.6KiB/s (88.6kB/s-90.7kB/s), io=19.6MiB (20.5MB), run=57097-57545msec WRITE: bw=363KiB/s (372kB/s), 90.2KiB/s-91.6KiB/s (92.4kB/s-93.8kB/s), io=20.4MiB (21.4MB), run=57097-57545msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148089: Fri Apr 10 22:49:55 2026 read: IOPS=4, BW=188MiB/s (197MB/s)(40.0MiB/213msec) slat (nsec): min=15081k, max=15081k, avg=15080809.00, stdev= 0.00 clat (nsec): min=190462k, max=190462k, avg=190461824.00, stdev= 0.00 lat (nsec): min=205548k, max=205548k, avg=205547883.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.13%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=188MiB/s (197MB/s), 188MiB/s-188MiB/s (197MB/s-197MB/s), io=40.0MiB (41.9MB), run=213-213msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout