== sanity test 398c: run fio to test AIO ================= 22:39:51 (1775875191) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.191109 s, 219 MB/s osc.lustre-OST0000-osc-ffff94b9c166a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149509: Fri Apr 10 22:41:03 2026 write: IOPS=37, BW=152KiB/s (155kB/s)(10.0MiB/67511msec); 0 zone resets slat (usec): min=35, max=4444, avg=242.59, stdev=303.43 clat (msec): min=75, max=891, avg=421.44, stdev=88.61 lat (msec): min=75, max=892, avg=421.69, stdev=88.63 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 284], 10.00th=[ 321], 20.00th=[ 355], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 443], | 70.00th=[ 468], 80.00th=[ 493], 90.00th=[ 527], 95.00th=[ 558], | 99.00th=[ 609], 99.50th=[ 667], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 894] bw ( KiB/s): min= 72, max= 312, per=24.91%, avg=150.96, stdev=35.77, samples=134 iops : min= 18, max= 78, avg=37.43, stdev= 8.95, samples=134 lat (msec) : 100=0.62%, 250=2.70%, 500=78.79%, 750=17.62%, 1000=0.27% cpu : usr=0.08%, sys=0.59%, ctx=2735, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149510: Fri Apr 10 22:41:03 2026 write: IOPS=37, BW=152KiB/s (155kB/s)(10.0MiB/67458msec); 0 zone resets slat (usec): min=35, max=7414, avg=255.64, stdev=350.62 clat (msec): min=84, max=757, avg=420.94, stdev=86.35 lat (msec): min=84, max=757, avg=421.20, stdev=86.37 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 292], 10.00th=[ 326], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 443], | 70.00th=[ 464], 80.00th=[ 493], 90.00th=[ 527], 95.00th=[ 558], | 99.00th=[ 609], 99.50th=[ 651], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 760] bw ( KiB/s): min= 79, max= 384, per=24.94%, avg=151.15, stdev=39.17, samples=134 iops : min= 19, max= 96, avg=37.48, stdev= 9.82, samples=134 lat (msec) : 100=0.62%, 250=2.54%, 500=80.00%, 750=16.80%, 1000=0.04% cpu : usr=0.08%, sys=0.62%, ctx=2768, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149511: Fri Apr 10 22:41:03 2026 write: IOPS=37, BW=152KiB/s (155kB/s)(10.0MiB/67555msec); 0 zone resets slat (usec): min=37, max=4596, avg=275.14, stdev=365.04 clat (msec): min=127, max=756, avg=421.50, stdev=86.74 lat (msec): min=127, max=756, avg=421.78, stdev=86.75 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 443], | 70.00th=[ 468], 80.00th=[ 493], 90.00th=[ 527], 95.00th=[ 567], | 99.00th=[ 625], 99.50th=[ 667], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 72, max= 384, per=24.94%, avg=151.14, stdev=35.66, samples=134 iops : min= 18, max= 96, avg=37.48, stdev= 8.92, samples=134 lat (msec) : 250=3.87%, 500=79.34%, 750=16.68%, 1000=0.12% cpu : usr=0.08%, sys=0.63%, ctx=3000, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149512: Fri Apr 10 22:41:03 2026 write: IOPS=37, BW=152KiB/s (155kB/s)(10.0MiB/67588msec); 0 zone resets slat (usec): min=41, max=7839, avg=268.51, stdev=339.46 clat (msec): min=42, max=708, avg=421.71, stdev=83.29 lat (msec): min=42, max=708, avg=421.98, stdev=83.30 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 292], 10.00th=[ 326], 20.00th=[ 359], | 30.00th=[ 380], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 443], | 70.00th=[ 468], 80.00th=[ 493], 90.00th=[ 527], 95.00th=[ 550], | 99.00th=[ 609], 99.50th=[ 634], 99.90th=[ 684], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 79, max= 288, per=24.91%, avg=150.96, stdev=30.77, samples=134 iops : min= 19, max= 72, avg=37.43, stdev= 7.71, samples=134 lat (msec) : 50=0.04%, 250=3.24%, 500=79.22%, 750=17.50% cpu : usr=0.08%, sys=0.65%, ctx=3013, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=606KiB/s (621kB/s), 152KiB/s-152KiB/s (155kB/s-155kB/s), io=40.0MiB (41.9MB), run=67458-67588msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149532: Fri Apr 10 22:41:45 2026 read: IOPS=33, BW=133KiB/s (136kB/s)(5048KiB/37881msec) slat (usec): min=51, max=21197, avg=4918.43, stdev=2068.28 clat (msec): min=6, max=428, avg=152.93, stdev=43.68 lat (msec): min=11, max=432, avg=157.85, stdev=44.20 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 100], 10.00th=[ 109], 20.00th=[ 123], | 30.00th=[ 132], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 182], 90.00th=[ 199], 95.00th=[ 224], | 99.00th=[ 288], 99.50th=[ 359], 99.90th=[ 426], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 71, max= 223, per=25.29%, avg=133.03, stdev=37.76, samples=75 iops : min= 17, max= 55, avg=33.01, stdev= 9.45, samples=75 write: IOPS=34, BW=137KiB/s (140kB/s)(5192KiB/37881msec); 0 zone resets slat (usec): min=53, max=7929, avg=272.38, stdev=337.26 clat (msec): min=50, max=612, avg=312.76, stdev=84.35 lat (msec): min=50, max=612, avg=313.04, stdev=84.35 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 186], 10.00th=[ 211], 20.00th=[ 247], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 326], | 70.00th=[ 351], 80.00th=[ 376], 90.00th=[ 426], 95.00th=[ 472], | 99.00th=[ 535], 99.50th=[ 550], 99.90th=[ 600], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 64, max= 197, per=24.95%, avg=136.71, stdev=29.16, samples=75 iops : min= 16, max= 49, avg=33.93, stdev= 7.34, samples=75 lat (msec) : 10=0.08%, 20=0.16%, 50=0.59%, 100=2.11%, 250=56.33% lat (msec) : 500=39.26%, 750=1.48% cpu : usr=0.20%, sys=1.55%, ctx=5155, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149533: Fri Apr 10 22:41:45 2026 read: IOPS=32, BW=131KiB/s (134kB/s)(4976KiB/38084msec) slat (usec): min=2051, max=17125, avg=4887.13, stdev=1912.83 clat (msec): min=6, max=310, avg=152.81, stdev=36.16 lat (msec): min=12, max=316, avg=157.70, stdev=36.62 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 104], 10.00th=[ 113], 20.00th=[ 123], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 197], 95.00th=[ 211], | 99.00th=[ 255], 99.50th=[ 292], 99.90th=[ 305], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 48, max= 208, per=24.69%, avg=129.88, stdev=41.99, samples=75 iops : min= 12, max= 52, avg=32.23, stdev=10.52, samples=75 write: IOPS=34, BW=138KiB/s (142kB/s)(5264KiB/38084msec); 0 zone resets slat (usec): min=42, max=4155, avg=260.39, stdev=269.60 clat (msec): min=76, max=590, avg=312.82, stdev=83.73 lat (msec): min=77, max=591, avg=313.08, stdev=83.74 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 182], 10.00th=[ 209], 20.00th=[ 249], | 30.00th=[ 264], 40.00th=[ 288], 50.00th=[ 305], 60.00th=[ 326], | 70.00th=[ 347], 80.00th=[ 384], 90.00th=[ 426], 95.00th=[ 472], | 99.00th=[ 531], 99.50th=[ 550], 99.90th=[ 558], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 71, max= 208, per=25.14%, avg=137.75, stdev=30.92, samples=75 iops : min= 17, max= 52, avg=34.19, stdev= 7.76, samples=75 lat (msec) : 10=0.08%, 50=0.23%, 100=1.60%, 250=56.88%, 500=39.84% lat (msec) : 750=1.37% cpu : usr=0.20%, sys=1.57%, ctx=5146, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149534: Fri Apr 10 22:41:45 2026 read: IOPS=32, BW=131KiB/s (134kB/s)(4984KiB/38102msec) slat (usec): min=1962, max=20127, avg=4904.98, stdev=2023.08 clat (msec): min=13, max=335, avg=153.91, stdev=39.36 lat (msec): min=20, max=344, avg=158.82, stdev=39.82 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 101], 10.00th=[ 109], 20.00th=[ 124], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 171], 80.00th=[ 184], 90.00th=[ 199], 95.00th=[ 222], | 99.00th=[ 275], 99.50th=[ 313], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 47, max= 215, per=24.75%, avg=130.17, stdev=40.77, samples=75 iops : min= 11, max= 53, avg=32.29, stdev=10.20, samples=75 write: IOPS=34, BW=138KiB/s (141kB/s)(5256KiB/38102msec); 0 zone resets slat (usec): min=49, max=2978, avg=258.47, stdev=232.06 clat (msec): min=36, max=603, avg=312.33, stdev=86.72 lat (msec): min=36, max=603, avg=312.59, stdev=86.73 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 176], 10.00th=[ 209], 20.00th=[ 249], | 30.00th=[ 268], 40.00th=[ 288], 50.00th=[ 309], 60.00th=[ 326], | 70.00th=[ 347], 80.00th=[ 380], 90.00th=[ 426], 95.00th=[ 472], | 99.00th=[ 535], 99.50th=[ 575], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 47, max= 224, per=24.74%, avg=135.59, stdev=34.22, samples=75 iops : min= 11, max= 56, avg=33.65, stdev= 8.59, samples=75 lat (msec) : 20=0.08%, 50=0.39%, 100=2.19%, 250=55.70%, 500=40.04% lat (msec) : 750=1.60% cpu : usr=0.18%, sys=1.55%, ctx=5160, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149535: Fri Apr 10 22:41:45 2026 read: IOPS=33, BW=133KiB/s (136kB/s)(5060KiB/37999msec) slat (usec): min=1499, max=17411, avg=4920.92, stdev=2000.27 clat (msec): min=4, max=371, avg=153.10, stdev=39.31 lat (msec): min=7, max=375, avg=158.03, stdev=39.88 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 104], 10.00th=[ 112], 20.00th=[ 124], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 182], 90.00th=[ 199], 95.00th=[ 220], | 99.00th=[ 275], 99.50th=[ 292], 99.90th=[ 338], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 55, max= 232, per=25.37%, avg=133.47, stdev=37.20, samples=75 iops : min= 13, max= 58, avg=33.12, stdev= 9.31, samples=75 write: IOPS=34, BW=136KiB/s (140kB/s)(5180KiB/37999msec); 0 zone resets slat (usec): min=59, max=2952, avg=271.50, stdev=265.50 clat (msec): min=23, max=588, avg=313.72, stdev=82.07 lat (msec): min=23, max=588, avg=313.99, stdev=82.09 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 192], 10.00th=[ 213], 20.00th=[ 245], | 30.00th=[ 266], 40.00th=[ 288], 50.00th=[ 305], 60.00th=[ 326], | 70.00th=[ 351], 80.00th=[ 384], 90.00th=[ 426], 95.00th=[ 464], | 99.00th=[ 542], 99.50th=[ 550], 99.90th=[ 575], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 71, max= 200, per=24.77%, avg=135.73, stdev=29.22, samples=75 iops : min= 17, max= 50, avg=33.69, stdev= 7.35, samples=75 lat (msec) : 10=0.16%, 20=0.12%, 50=0.31%, 100=1.45%, 250=57.62% lat (msec) : 500=39.18%, 750=1.17% cpu : usr=0.16%, sys=1.58%, ctx=5284, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=527KiB/s (539kB/s), 131KiB/s-133KiB/s (134kB/s-136kB/s), io=19.6MiB (20.5MB), run=37881-38102msec WRITE: bw=548KiB/s (561kB/s), 136KiB/s-138KiB/s (140kB/s-142kB/s), io=20.4MiB (21.4MB), run=37881-38102msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149544: Fri Apr 10 22:41:49 2026 read: IOPS=5, BW=227MiB/s (238MB/s)(40.0MiB/176msec) slat (nsec): min=14266k, max=14266k, avg=14265557.00, stdev= 0.00 clat (nsec): min=150799k, max=150799k, avg=150799191.00, stdev= 0.00 lat (nsec): min=165069k, max=165069k, avg=165069234.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.29%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=40.0MiB (41.9MB), run=176-176msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout