== sanity test 398c: run fio to test AIO ================= 10:01:15 (1747404075) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.152533 s, 275 MB/s osc.lustre-OST0000-osc-ffff88008e783000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=24021: Fri May 16 10:01:42 2025 write: IOPS=101, BW=405KiB/s (414kB/s)(10.0MiB/25307msec) slat (usec): min=13, max=395, avg=70.18, stdev=44.50 clat (msec): min=22, max=284, avg=157.97, stdev=28.36 lat (msec): min=22, max=284, avg=158.04, stdev=28.36 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 113], 10.00th=[ 124], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 171], 80.00th=[ 178], 90.00th=[ 192], 95.00th=[ 209], | 99.00th=[ 228], 99.50th=[ 241], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 288, max= 582, per=24.98%, avg=404.24, stdev=50.82, samples=50 iops : min= 72, max= 145, avg=101.02, stdev=12.67, samples=50 lat (msec) : 50=0.04%, 100=2.15%, 250=97.50%, 500=0.31% cpu : usr=0.15%, sys=0.83%, ctx=2145, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24022: Fri May 16 10:01:42 2025 write: IOPS=101, BW=405KiB/s (414kB/s)(10.0MiB/25307msec) slat (usec): min=12, max=568, avg=70.95, stdev=44.40 clat (msec): min=66, max=247, avg=158.02, stdev=27.51 lat (msec): min=66, max=247, avg=158.09, stdev=27.52 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 112], 10.00th=[ 124], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 153], 50.00th=[ 159], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 180], 90.00th=[ 192], 95.00th=[ 205], | 99.00th=[ 226], 99.50th=[ 236], 99.90th=[ 243], 99.95th=[ 245], | 99.99th=[ 249] bw ( KiB/s): min= 304, max= 566, per=24.89%, avg=402.80, stdev=48.25, samples=50 iops : min= 76, max= 141, avg=100.66, stdev=12.04, samples=50 lat (msec) : 100=2.23%, 250=97.77% cpu : usr=0.14%, sys=0.84%, ctx=2192, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24023: Fri May 16 10:01:42 2025 write: IOPS=101, BW=405KiB/s (415kB/s)(10.0MiB/25292msec) slat (usec): min=13, max=444, avg=72.05, stdev=46.12 clat (msec): min=83, max=268, avg=157.88, stdev=27.67 lat (msec): min=83, max=269, avg=157.96, stdev=27.68 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 112], 10.00th=[ 124], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 171], 80.00th=[ 180], 90.00th=[ 192], 95.00th=[ 207], | 99.00th=[ 226], 99.50th=[ 236], 99.90th=[ 253], 99.95th=[ 262], | 99.99th=[ 271] bw ( KiB/s): min= 320, max= 584, per=24.96%, avg=403.78, stdev=44.24, samples=50 iops : min= 80, max= 146, avg=100.90, stdev=11.08, samples=50 lat (msec) : 100=2.19%, 250=97.70%, 500=0.12% cpu : usr=0.21%, sys=0.81%, ctx=2116, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24024: Fri May 16 10:01:42 2025 write: IOPS=101, BW=406KiB/s (415kB/s)(10.0MiB/25242msec) slat (usec): min=13, max=491, avg=71.38, stdev=44.05 clat (msec): min=56, max=259, avg=157.60, stdev=28.46 lat (msec): min=56, max=259, avg=157.67, stdev=28.47 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 111], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 180], 90.00th=[ 192], 95.00th=[ 209], | 99.00th=[ 228], 99.50th=[ 234], 99.90th=[ 251], 99.95th=[ 257], | 99.99th=[ 259] bw ( KiB/s): min= 319, max= 608, per=25.01%, avg=404.72, stdev=51.44, samples=50 iops : min= 79, max= 152, avg=101.00, stdev=12.91, samples=50 lat (msec) : 100=2.97%, 250=96.91%, 500=0.12% cpu : usr=0.19%, sys=0.82%, ctx=2136, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1619KiB/s (1657kB/s), 405KiB/s-406KiB/s (414kB/s-415kB/s), io=40.0MiB (41.9MB), run=25242-25307msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24054: Fri May 16 10:01:57 2025 read: IOPS=86, BW=346KiB/s (354kB/s)(5048KiB/14591msec) slat (usec): min=494, max=3659, avg=838.60, stdev=228.30 clat (usec): min=1496, max=98815, avg=61682.55, stdev=10943.37 lat (usec): min=2256, max=99407, avg=62522.94, stdev=10948.48 clat percentiles (usec): | 1.00th=[36439], 5.00th=[45351], 10.00th=[49546], 20.00th=[53740], | 30.00th=[55837], 40.00th=[58459], 50.00th=[61080], 60.00th=[63701], | 70.00th=[67634], 80.00th=[69731], 90.00th=[74974], 95.00th=[80217], | 99.00th=[86508], 99.50th=[88605], 99.90th=[98042], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 224, max= 440, per=25.05%, avg=344.14, stdev=60.31, samples=29 iops : min= 56, max= 110, avg=85.93, stdev=15.09, samples=29 write: IOPS=88, BW=356KiB/s (364kB/s)(5192KiB/14591msec) slat (usec): min=13, max=1054, avg=57.53, stdev=43.65 clat (msec): min=54, max=196, avg=118.77, stdev=26.37 lat (msec): min=54, max=196, avg=118.83, stdev=26.38 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 79], 10.00th=[ 86], 20.00th=[ 94], | 30.00th=[ 103], 40.00th=[ 110], 50.00th=[ 118], 60.00th=[ 128], | 70.00th=[ 136], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 163], | 99.00th=[ 178], 99.50th=[ 182], 99.90th=[ 188], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 256, max= 424, per=24.74%, avg=353.79, stdev=33.20, samples=29 iops : min= 64, max= 106, avg=88.34, stdev= 8.28, samples=29 lat (msec) : 2=0.04%, 4=0.04%, 10=0.04%, 20=0.08%, 50=5.16% lat (msec) : 100=57.85%, 250=36.80% cpu : usr=0.21%, sys=1.75%, ctx=3000, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24055: Fri May 16 10:01:57 2025 read: IOPS=85, BW=341KiB/s (349kB/s)(4976KiB/14591msec) slat (usec): min=467, max=2744, avg=829.37, stdev=202.30 clat (usec): min=902, max=118980, avg=61946.22, stdev=11384.66 lat (usec): min=1692, max=119720, avg=62777.60, stdev=11382.23 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 46], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 70], 90.00th=[ 77], 95.00th=[ 82], | 99.00th=[ 90], 99.50th=[ 93], 99.90th=[ 99], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 207, max= 464, per=24.73%, avg=339.72, stdev=67.04, samples=29 iops : min= 51, max= 116, avg=84.83, stdev=16.75, samples=29 write: IOPS=90, BW=361KiB/s (369kB/s)(5264KiB/14591msec) slat (usec): min=12, max=272, avg=54.71, stdev=30.53 clat (msec): min=57, max=189, avg=117.79, stdev=26.33 lat (msec): min=57, max=189, avg=117.85, stdev=26.33 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 77], 10.00th=[ 85], 20.00th=[ 94], | 30.00th=[ 102], 40.00th=[ 110], 50.00th=[ 117], 60.00th=[ 126], | 70.00th=[ 133], 80.00th=[ 142], 90.00th=[ 153], 95.00th=[ 161], | 99.00th=[ 180], 99.50th=[ 182], 99.90th=[ 188], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 288, max= 456, per=25.11%, avg=359.03, stdev=43.15, samples=29 iops : min= 72, max= 114, avg=89.66, stdev=10.79, samples=29 lat (usec) : 1000=0.04% lat (msec) : 4=0.04%, 20=0.27%, 50=5.00%, 100=57.97%, 250=36.68% cpu : usr=0.30%, sys=1.62%, ctx=2944, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24056: Fri May 16 10:01:57 2025 read: IOPS=85, BW=341KiB/s (350kB/s)(4984KiB/14601msec) slat (usec): min=493, max=4561, avg=834.70, stdev=213.57 clat (usec): min=1010, max=106979, avg=61847.04, stdev=10971.07 lat (usec): min=1754, max=107675, avg=62683.58, stdev=10976.35 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 46], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 77], 95.00th=[ 80], | 99.00th=[ 87], 99.50th=[ 89], 99.90th=[ 96], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 224, max= 440, per=24.81%, avg=340.83, stdev=55.92, samples=29 iops : min= 56, max= 110, avg=85.10, stdev=14.01, samples=29 write: IOPS=89, BW=360KiB/s (369kB/s)(5256KiB/14601msec) slat (usec): min=13, max=438, avg=56.15, stdev=34.06 clat (msec): min=43, max=193, avg=118.10, stdev=25.59 lat (msec): min=43, max=193, avg=118.16, stdev=25.59 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 78], 10.00th=[ 87], 20.00th=[ 96], | 30.00th=[ 104], 40.00th=[ 110], 50.00th=[ 118], 60.00th=[ 126], | 70.00th=[ 134], 80.00th=[ 142], 90.00th=[ 153], 95.00th=[ 159], | 99.00th=[ 174], 99.50th=[ 180], 99.90th=[ 188], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 288, max= 448, per=24.93%, avg=356.55, stdev=41.99, samples=29 iops : min= 72, max= 112, avg=89.03, stdev=10.51, samples=29 lat (msec) : 2=0.04%, 10=0.08%, 20=0.20%, 50=5.16%, 100=56.76% lat (msec) : 250=37.77% cpu : usr=0.27%, sys=1.65%, ctx=3013, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24057: Fri May 16 10:01:57 2025 read: IOPS=87, BW=349KiB/s (357kB/s)(5060KiB/14504msec) slat (usec): min=12, max=3568, avg=822.41, stdev=204.64 clat (usec): min=1516, max=97150, avg=61809.86, stdev=11442.76 lat (usec): min=1529, max=97687, avg=62633.93, stdev=11466.78 clat percentiles (usec): | 1.00th=[11994], 5.00th=[46924], 10.00th=[50070], 20.00th=[53740], | 30.00th=[56886], 40.00th=[59507], 50.00th=[61604], 60.00th=[64750], | 70.00th=[67634], 80.00th=[69731], 90.00th=[76022], 95.00th=[79168], | 99.00th=[86508], 99.50th=[88605], 99.90th=[94897], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 256, max= 432, per=25.31%, avg=347.72, stdev=42.75, samples=29 iops : min= 64, max= 108, avg=86.83, stdev=10.74, samples=29 write: IOPS=89, BW=357KiB/s (366kB/s)(5180KiB/14504msec) slat (usec): min=13, max=600, avg=55.72, stdev=35.37 clat (msec): min=24, max=187, avg=117.84, stdev=26.91 lat (msec): min=24, max=187, avg=117.90, stdev=26.91 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 78], 10.00th=[ 85], 20.00th=[ 93], | 30.00th=[ 104], 40.00th=[ 109], 50.00th=[ 118], 60.00th=[ 126], | 70.00th=[ 134], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 163], | 99.00th=[ 174], 99.50th=[ 180], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 272, max= 424, per=24.76%, avg=354.07, stdev=38.67, samples=29 iops : min= 68, max= 106, avg=88.41, stdev= 9.68, samples=29 lat (msec) : 2=0.04%, 4=0.16%, 20=0.35%, 50=4.77%, 100=58.20% lat (msec) : 250=36.48% cpu : usr=0.21%, sys=1.72%, ctx=2911, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1374KiB/s (1407kB/s), 341KiB/s-349KiB/s (349kB/s-357kB/s), io=19.6MiB (20.5MB), run=14504-14601msec WRITE: bw=1431KiB/s (1465kB/s), 356KiB/s-361KiB/s (364kB/s-369kB/s), io=20.4MiB (21.4MB), run=14504-14601msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=24073: Fri May 16 10:01:58 2025 read: IOPS=12, BW=488MiB/s (512MB/s)(40.0MiB/82msec) slat (nsec): min=7204.1k, max=7204.1k, avg=7204074.00, stdev= 0.00 clat (nsec): min=69665k, max=69665k, avg=69665062.00, stdev= 0.00 lat (nsec): min=76881k, max=76881k, avg=76880603.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[69731], 5.00th=[69731], 10.00th=[69731], 20.00th=[69731], | 30.00th=[69731], 40.00th=[69731], 50.00th=[69731], 60.00th=[69731], | 70.00th=[69731], 80.00th=[69731], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.88%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=488MiB/s (512MB/s), 488MiB/s-488MiB/s (512MB/s-512MB/s), io=40.0MiB (41.9MB), run=82-82msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck