== sanity test 398c: run fio to test AIO ================= 03:52:12 (1785397932) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.403301 s, 104 MB/s osc.lustre-OST0000-osc-ffff91711f81c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=178816: Thu Jul 30 03:53:12 2026 write: IOPS=54, BW=217KiB/s (223kB/s)(10.0MiB/47126msec); 0 zone resets slat (usec): min=47, max=29143, avg=310.39, stdev=1436.36 clat (msec): min=26, max=612, avg=294.00, stdev=99.57 lat (msec): min=26, max=612, avg=294.31, stdev=99.61 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 142], 10.00th=[ 167], 20.00th=[ 201], | 30.00th=[ 239], 40.00th=[ 266], 50.00th=[ 292], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 376], 90.00th=[ 418], 95.00th=[ 477], | 99.00th=[ 558], 99.50th=[ 584], 99.90th=[ 592], 99.95th=[ 600], | 99.99th=[ 617] bw ( KiB/s): min= 126, max= 485, per=24.91%, avg=214.99, stdev=78.33, samples=93 iops : min= 31, max= 121, avg=53.34, stdev=19.64, samples=93 lat (msec) : 50=0.08%, 100=0.86%, 250=33.44%, 500=62.62%, 750=3.01% cpu : usr=0.08%, sys=0.64%, ctx=2345, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178817: Thu Jul 30 03:53:12 2026 write: IOPS=53, BW=216KiB/s (221kB/s)(10.0MiB/47432msec); 0 zone resets slat (usec): min=36, max=30613, avg=314.28, stdev=1335.05 clat (msec): min=110, max=636, avg=295.24, stdev=100.31 lat (msec): min=110, max=636, avg=295.56, stdev=100.32 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 146], 10.00th=[ 165], 20.00th=[ 201], | 30.00th=[ 234], 40.00th=[ 271], 50.00th=[ 296], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 376], 90.00th=[ 430], 95.00th=[ 481], | 99.00th=[ 550], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 634] bw ( KiB/s): min= 79, max= 508, per=24.87%, avg=214.60, stdev=80.51, samples=93 iops : min= 19, max= 127, avg=53.25, stdev=20.16, samples=93 lat (msec) : 250=33.75%, 500=63.48%, 750=2.77% cpu : usr=0.08%, sys=0.67%, ctx=2185, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178818: Thu Jul 30 03:53:12 2026 write: IOPS=54, BW=217KiB/s (222kB/s)(10.0MiB/47296msec); 0 zone resets slat (usec): min=49, max=36235, avg=327.28, stdev=1493.25 clat (msec): min=25, max=635, avg=294.96, stdev=98.98 lat (msec): min=25, max=636, avg=295.29, stdev=98.96 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 199], | 30.00th=[ 234], 40.00th=[ 268], 50.00th=[ 292], 60.00th=[ 317], | 70.00th=[ 347], 80.00th=[ 376], 90.00th=[ 418], 95.00th=[ 472], | 99.00th=[ 550], 99.50th=[ 584], 99.90th=[ 592], 99.95th=[ 600], | 99.99th=[ 634] bw ( KiB/s): min= 126, max= 469, per=24.93%, avg=215.17, stdev=75.35, samples=93 iops : min= 31, max= 117, avg=53.39, stdev=18.87, samples=93 lat (msec) : 50=0.20%, 100=0.35%, 250=34.53%, 500=61.72%, 750=3.20% cpu : usr=0.10%, sys=0.75%, ctx=2607, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178819: Thu Jul 30 03:53:12 2026 write: IOPS=53, BW=216KiB/s (221kB/s)(10.0MiB/47440msec); 0 zone resets slat (usec): min=36, max=17257, avg=252.42, stdev=901.42 clat (msec): min=46, max=637, avg=295.28, stdev=98.82 lat (msec): min=46, max=637, avg=295.53, stdev=98.81 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 153], 10.00th=[ 171], 20.00th=[ 201], | 30.00th=[ 234], 40.00th=[ 268], 50.00th=[ 296], 60.00th=[ 321], | 70.00th=[ 347], 80.00th=[ 372], 90.00th=[ 418], 95.00th=[ 477], | 99.00th=[ 558], 99.50th=[ 584], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 124, max= 508, per=24.79%, avg=213.98, stdev=77.33, samples=93 iops : min= 31, max= 127, avg=53.08, stdev=19.38, samples=93 lat (msec) : 50=0.04%, 100=0.82%, 250=32.38%, 500=63.52%, 750=3.24% cpu : usr=0.09%, sys=0.59%, ctx=2094, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=863KiB/s (884kB/s), 216KiB/s-217KiB/s (221kB/s-223kB/s), io=40.0MiB (41.9MB), run=47126-47440msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=178837: Thu Jul 30 03:54:08 2026 read: IOPS=33, BW=133KiB/s (136kB/s)(5048KiB/38074msec) slat (usec): min=67, max=82060, avg=17101.78, stdev=11188.11 clat (msec): min=86, max=482, avg=227.77, stdev=72.96 lat (msec): min=95, max=517, avg=244.88, stdev=76.62 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 126], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 178], 40.00th=[ 197], 50.00th=[ 222], 60.00th=[ 247], | 70.00th=[ 271], 80.00th=[ 296], 90.00th=[ 326], 95.00th=[ 355], | 99.00th=[ 401], 99.50th=[ 447], 99.90th=[ 472], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 47, max= 214, per=25.28%, avg=132.21, stdev=38.41, samples=75 iops : min= 11, max= 53, avg=32.67, stdev= 9.64, samples=75 write: IOPS=34, BW=136KiB/s (140kB/s)(5192KiB/38074msec); 0 zone resets slat (usec): min=56, max=14328, avg=246.80, stdev=711.57 clat (msec): min=96, max=468, avg=230.07, stdev=70.84 lat (msec): min=97, max=468, avg=230.31, stdev=70.82 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 163], | 30.00th=[ 182], 40.00th=[ 201], 50.00th=[ 226], 60.00th=[ 249], | 70.00th=[ 271], 80.00th=[ 292], 90.00th=[ 326], 95.00th=[ 351], | 99.00th=[ 414], 99.50th=[ 435], 99.90th=[ 456], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 55, max= 264, per=25.06%, avg=136.59, stdev=47.50, samples=75 iops : min= 13, max= 66, avg=33.76, stdev=11.90, samples=75 lat (msec) : 100=0.31%, 250=60.70%, 500=38.98% cpu : usr=0.18%, sys=1.47%, ctx=4404, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178838: Thu Jul 30 03:54:08 2026 read: IOPS=32, BW=130KiB/s (133kB/s)(4976KiB/38309msec) slat (usec): min=91, max=100544, avg=17641.80, stdev=11279.44 clat (msec): min=30, max=485, avg=226.98, stdev=70.75 lat (msec): min=31, max=499, avg=244.63, stdev=74.14 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 127], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 180], 40.00th=[ 203], 50.00th=[ 226], 60.00th=[ 247], | 70.00th=[ 268], 80.00th=[ 292], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 393], 99.50th=[ 414], 99.90th=[ 468], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 14, max= 233, per=24.63%, avg=128.82, stdev=39.05, samples=76 iops : min= 3, max= 58, avg=31.82, stdev= 9.80, samples=76 write: IOPS=34, BW=137KiB/s (141kB/s)(5264KiB/38309msec); 0 zone resets slat (usec): min=54, max=11234, avg=235.09, stdev=576.80 clat (msec): min=30, max=428, avg=230.89, stdev=68.59 lat (msec): min=30, max=429, avg=231.12, stdev=68.58 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 134], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 186], 40.00th=[ 207], 50.00th=[ 226], 60.00th=[ 249], | 70.00th=[ 271], 80.00th=[ 296], 90.00th=[ 321], 95.00th=[ 347], | 99.00th=[ 393], 99.50th=[ 405], 99.90th=[ 426], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 36, max= 280, per=25.12%, avg=136.91, stdev=47.64, samples=76 iops : min= 9, max= 70, avg=33.86, stdev=11.92, samples=76 lat (msec) : 50=0.51%, 100=0.59%, 250=60.35%, 500=38.55% cpu : usr=0.22%, sys=1.71%, ctx=4404, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178839: Thu Jul 30 03:54:08 2026 read: IOPS=32, BW=130KiB/s (134kB/s)(4984KiB/38220msec) slat (usec): min=65, max=95081, avg=17030.58, stdev=10673.31 clat (msec): min=64, max=425, avg=224.01, stdev=67.93 lat (msec): min=84, max=462, avg=241.04, stdev=71.37 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 178], 40.00th=[ 199], 50.00th=[ 215], 60.00th=[ 239], | 70.00th=[ 259], 80.00th=[ 284], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 388], 99.50th=[ 401], 99.90th=[ 418], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 29, max= 240, per=24.99%, avg=130.69, stdev=43.12, samples=75 iops : min= 7, max= 60, avg=32.29, stdev=10.84, samples=75 write: IOPS=34, BW=138KiB/s (141kB/s)(5256KiB/38220msec); 0 zone resets slat (usec): min=55, max=23730, avg=282.06, stdev=1077.56 clat (msec): min=90, max=459, avg=233.24, stdev=71.19 lat (msec): min=91, max=461, avg=233.53, stdev=71.20 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 131], 10.00th=[ 142], 20.00th=[ 165], | 30.00th=[ 186], 40.00th=[ 207], 50.00th=[ 228], 60.00th=[ 251], | 70.00th=[ 275], 80.00th=[ 296], 90.00th=[ 334], 95.00th=[ 359], | 99.00th=[ 401], 99.50th=[ 430], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 29, max= 256, per=25.20%, avg=137.36, stdev=43.71, samples=75 iops : min= 7, max= 64, avg=33.97, stdev=10.92, samples=75 lat (msec) : 100=0.43%, 250=62.23%, 500=37.34% cpu : usr=0.19%, sys=1.76%, ctx=4389, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178840: Thu Jul 30 03:54:08 2026 read: IOPS=33, BW=132KiB/s (136kB/s)(5060KiB/38191msec) slat (usec): min=2392, max=74894, avg=17282.13, stdev=10921.30 clat (msec): min=14, max=461, avg=222.80, stdev=67.19 lat (msec): min=20, max=470, avg=240.09, stdev=70.91 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 127], 10.00th=[ 142], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 197], 50.00th=[ 220], 60.00th=[ 239], | 70.00th=[ 262], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 338], | 99.00th=[ 376], 99.50th=[ 388], 99.90th=[ 443], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 21, max= 240, per=25.34%, avg=132.53, stdev=43.70, samples=75 iops : min= 5, max= 60, avg=32.75, stdev=10.97, samples=75 write: IOPS=33, BW=136KiB/s (139kB/s)(5180KiB/38191msec); 0 zone resets slat (usec): min=52, max=12831, avg=243.38, stdev=714.77 clat (msec): min=99, max=426, avg=233.24, stdev=67.90 lat (msec): min=99, max=426, avg=233.49, stdev=67.89 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 205], 50.00th=[ 228], 60.00th=[ 253], | 70.00th=[ 275], 80.00th=[ 292], 90.00th=[ 330], 95.00th=[ 351], | 99.00th=[ 393], 99.50th=[ 401], 99.90th=[ 418], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 29, max= 263, per=24.83%, avg=135.32, stdev=43.02, samples=75 iops : min= 7, max= 65, avg=33.47, stdev=10.77, samples=75 lat (msec) : 20=0.08%, 50=0.04%, 100=0.51%, 250=60.86%, 500=38.52% cpu : usr=0.17%, sys=1.52%, ctx=4468, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=524KiB/s (536kB/s), 130KiB/s-133KiB/s (133kB/s-136kB/s), io=19.6MiB (20.5MB), run=38074-38309msec WRITE: bw=545KiB/s (558kB/s), 136KiB/s-138KiB/s (139kB/s-141kB/s), io=20.4MiB (21.4MB), run=38074-38309msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=178851: Thu Jul 30 03:54:21 2026 read: IOPS=1, BW=69.8MiB/s (73.2MB/s)(40.0MiB/573msec) slat (nsec): min=49881k, max=49881k, avg=49880717.00, stdev= 0.00 clat (nsec): min=500584k, max=500584k, avg=500583619.00, stdev= 0.00 lat (nsec): min=550468k, max=550468k, avg=550468261.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.27%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=69.8MiB/s (73.2MB/s), 69.8MiB/s-69.8MiB/s (73.2MB/s-73.2MB/s), io=40.0MiB (41.9MB), run=573-573msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout