== sanity test 398c: run fio to test AIO ================= 19:27:38 (1736987258) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0817 s, 513 MB/s osc.lustre-OST0000-osc-ffff88012d137000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=24341: Wed Jan 15 19:28:04 2025 write: IOPS=100, BW=402KiB/s (412kB/s)(10.0MiB/25442msec) slat (usec): min=11, max=438, avg=71.74, stdev=36.63 clat (msec): min=67, max=298, avg=158.85, stdev=28.57 lat (msec): min=67, max=298, avg=158.93, stdev=28.57 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 111], 10.00th=[ 125], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 194], 95.00th=[ 205], | 99.00th=[ 230], 99.50th=[ 247], 99.90th=[ 275], 99.95th=[ 288], | 99.99th=[ 300] bw ( KiB/s): min= 336, max= 640, per=25.03%, avg=402.78, stdev=53.20, samples=50 iops : min= 84, max= 160, avg=100.62, stdev=13.32, samples=50 lat (msec) : 100=2.58%, 250=97.07%, 500=0.35% cpu : usr=0.22%, sys=0.79%, ctx=2376, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24342: Wed Jan 15 19:28:04 2025 write: IOPS=100, BW=402KiB/s (412kB/s)(10.0MiB/25442msec) slat (usec): min=12, max=383, avg=70.26, stdev=38.60 clat (msec): min=26, max=260, avg=158.86, stdev=28.33 lat (msec): min=26, max=260, avg=158.93, stdev=28.33 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 110], 10.00th=[ 125], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 194], 95.00th=[ 205], | 99.00th=[ 222], 99.50th=[ 236], 99.90th=[ 253], 99.95th=[ 257], | 99.99th=[ 262] bw ( KiB/s): min= 312, max= 648, per=24.97%, avg=401.84, stdev=54.80, samples=50 iops : min= 78, max= 162, avg=100.40, stdev=13.72, samples=50 lat (msec) : 50=0.04%, 100=2.81%, 250=97.03%, 500=0.12% cpu : usr=0.17%, sys=0.83%, ctx=2281, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24343: Wed Jan 15 19:28:04 2025 write: IOPS=100, BW=403KiB/s (413kB/s)(10.0MiB/25399msec) slat (usec): min=11, max=369, avg=72.26, stdev=39.78 clat (msec): min=10, max=311, avg=158.59, stdev=30.02 lat (msec): min=10, max=311, avg=158.66, stdev=30.02 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 110], 10.00th=[ 123], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 194], 95.00th=[ 205], | 99.00th=[ 232], 99.50th=[ 247], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 313] bw ( KiB/s): min= 320, max= 664, per=25.06%, avg=403.26, stdev=57.10, samples=50 iops : min= 80, max= 166, avg=100.74, stdev=14.30, samples=50 lat (msec) : 20=0.04%, 50=0.04%, 100=3.59%, 250=95.90%, 500=0.43% cpu : usr=0.15%, sys=0.85%, ctx=2351, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24344: Wed Jan 15 19:28:04 2025 write: IOPS=100, BW=402KiB/s (412kB/s)(10.0MiB/25449msec) slat (usec): min=13, max=1322, avg=71.18, stdev=46.81 clat (msec): min=65, max=256, avg=158.90, stdev=28.22 lat (msec): min=65, max=257, avg=158.97, stdev=28.23 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 111], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 194], 95.00th=[ 205], | 99.00th=[ 230], 99.50th=[ 241], 99.90th=[ 251], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 328, max= 640, per=24.99%, avg=402.06, stdev=54.24, samples=50 iops : min= 82, max= 160, avg=100.50, stdev=13.57, samples=50 lat (msec) : 100=2.62%, 250=97.27%, 500=0.12% cpu : usr=0.13%, sys=0.86%, ctx=2377, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1609KiB/s (1648kB/s), 402KiB/s-403KiB/s (412kB/s-413kB/s), io=40.0MiB (41.9MB), run=25399-25449msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24374: Wed Jan 15 19:28:21 2025 read: IOPS=82, BW=330KiB/s (338kB/s)(5048KiB/15283msec) slat (usec): min=507, max=4121, avg=863.17, stdev=224.22 clat (usec): min=945, max=121444, avg=62891.32, stdev=10606.11 lat (usec): min=1887, max=122150, avg=63756.45, stdev=10599.35 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 55], | 30.00th=[ 58], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 66], | 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 77], 95.00th=[ 80], | 99.00th=[ 86], 99.50th=[ 89], 99.90th=[ 121], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 232, max= 408, per=25.03%, avg=328.13, stdev=47.06, samples=30 iops : min= 58, max= 102, avg=81.93, stdev=11.70, samples=30 write: IOPS=84, BW=340KiB/s (348kB/s)(5192KiB/15283msec) slat (usec): min=13, max=982, avg=62.49, stdev=51.37 clat (msec): min=36, max=191, avg=126.15, stdev=24.97 lat (msec): min=36, max=191, avg=126.21, stdev=24.97 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 85], 10.00th=[ 92], 20.00th=[ 104], | 30.00th=[ 113], 40.00th=[ 122], 50.00th=[ 127], 60.00th=[ 133], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 159], 95.00th=[ 167], | 99.00th=[ 184], 99.50th=[ 188], 99.90th=[ 190], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 280, max= 407, per=24.72%, avg=337.47, stdev=33.02, samples=30 iops : min= 70, max= 101, avg=84.27, stdev= 8.24, samples=30 lat (usec) : 1000=0.04% lat (msec) : 2=0.04%, 20=0.12%, 50=3.71%, 100=53.67%, 250=42.42% cpu : usr=0.34%, sys=1.68%, ctx=2991, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24375: Wed Jan 15 19:28:21 2025 read: IOPS=81, BW=325KiB/s (333kB/s)(4976KiB/15298msec) slat (usec): min=486, max=2635, avg=854.16, stdev=204.21 clat (usec): min=2650, max=90740, avg=62174.79, stdev=10406.81 lat (usec): min=3639, max=91464, avg=63030.88, stdev=10409.80 clat percentiles (usec): | 1.00th=[30802], 5.00th=[47449], 10.00th=[51119], 20.00th=[54264], | 30.00th=[57410], 40.00th=[60031], 50.00th=[62653], 60.00th=[64750], | 70.00th=[66847], 80.00th=[69731], 90.00th=[74974], 95.00th=[78119], | 99.00th=[84411], 99.50th=[85459], 99.90th=[89654], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 224, max= 488, per=24.66%, avg=323.33, stdev=60.16, samples=30 iops : min= 56, max= 122, avg=80.73, stdev=15.11, samples=30 write: IOPS=86, BW=344KiB/s (352kB/s)(5264KiB/15298msec) slat (usec): min=12, max=212, avg=60.11, stdev=29.50 clat (msec): min=29, max=191, avg=126.21, stdev=25.04 lat (msec): min=29, max=191, avg=126.27, stdev=25.04 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 84], 10.00th=[ 93], 20.00th=[ 106], | 30.00th=[ 113], 40.00th=[ 121], 50.00th=[ 127], 60.00th=[ 133], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 161], 95.00th=[ 169], | 99.00th=[ 180], 99.50th=[ 186], 99.90th=[ 190], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 280, max= 423, per=24.96%, avg=340.67, stdev=32.91, samples=30 iops : min= 70, max= 105, avg=85.07, stdev= 8.07, samples=30 lat (msec) : 4=0.04%, 10=0.08%, 20=0.16%, 50=3.79%, 100=52.62% lat (msec) : 250=43.32% cpu : usr=0.31%, sys=1.69%, ctx=2897, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24376: Wed Jan 15 19:28:21 2025 read: IOPS=81, BW=326KiB/s (334kB/s)(4984KiB/15266msec) slat (usec): min=500, max=4065, avg=850.31, stdev=207.85 clat (msec): min=2, max=128, avg=62.26, stdev=10.71 lat (msec): min=3, max=129, avg=63.11, stdev=10.72 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 77], 95.00th=[ 79], | 99.00th=[ 84], 99.50th=[ 87], 99.90th=[ 93], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 224, max= 439, per=24.95%, avg=327.07, stdev=51.99, samples=30 iops : min= 56, max= 109, avg=81.67, stdev=12.88, samples=30 write: IOPS=86, BW=344KiB/s (353kB/s)(5256KiB/15266msec) slat (usec): min=13, max=587, avg=59.81, stdev=35.21 clat (msec): min=47, max=208, avg=125.81, stdev=26.04 lat (msec): min=47, max=208, avg=125.87, stdev=26.04 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 85], 10.00th=[ 92], 20.00th=[ 102], | 30.00th=[ 111], 40.00th=[ 121], 50.00th=[ 127], 60.00th=[ 132], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 161], 95.00th=[ 171], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 188], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 256, max= 456, per=25.05%, avg=342.00, stdev=40.80, samples=30 iops : min= 64, max= 114, avg=85.40, stdev=10.23, samples=30 lat (msec) : 4=0.04%, 10=0.04%, 20=0.20%, 50=4.61%, 100=53.20% lat (msec) : 250=41.91% cpu : usr=0.27%, sys=1.74%, ctx=2829, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24377: Wed Jan 15 19:28:21 2025 read: IOPS=83, BW=333KiB/s (341kB/s)(5060KiB/15214msec) slat (usec): min=12, max=2843, avg=853.16, stdev=215.80 clat (usec): min=1388, max=119932, avg=62320.38, stdev=11548.59 lat (usec): min=1402, max=120781, avg=63175.56, stdev=11567.08 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 61], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 77], 95.00th=[ 80], | 99.00th=[ 87], 99.50th=[ 90], 99.90th=[ 120], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 240, max= 471, per=25.42%, avg=333.23, stdev=57.99, samples=30 iops : min= 60, max= 117, avg=83.23, stdev=14.38, samples=30 write: IOPS=85, BW=340KiB/s (349kB/s)(5180KiB/15214msec) slat (usec): min=12, max=304, avg=58.84, stdev=31.02 clat (msec): min=14, max=198, avg=126.07, stdev=26.63 lat (msec): min=14, max=198, avg=126.13, stdev=26.63 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 85], 10.00th=[ 92], 20.00th=[ 103], | 30.00th=[ 112], 40.00th=[ 121], 50.00th=[ 127], 60.00th=[ 134], | 70.00th=[ 140], 80.00th=[ 150], 90.00th=[ 161], 95.00th=[ 167], | 99.00th=[ 180], 99.50th=[ 188], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 272, max= 408, per=24.86%, avg=339.37, stdev=34.23, samples=30 iops : min= 68, max= 102, avg=84.77, stdev= 8.60, samples=30 lat (msec) : 2=0.16%, 4=0.08%, 10=0.04%, 20=0.23%, 50=4.88% lat (msec) : 100=52.58%, 250=42.03% cpu : usr=0.22%, sys=1.79%, ctx=2921, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1312KiB/s (1343kB/s), 325KiB/s-333KiB/s (333kB/s-341kB/s), io=19.6MiB (20.5MB), run=15214-15298msec WRITE: bw=1366KiB/s (1398kB/s), 340KiB/s-344KiB/s (348kB/s-353kB/s), io=20.4MiB (21.4MB), run=15214-15298msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=24393: Wed Jan 15 19:28:22 2025 read: IOPS=11, BW=476MiB/s (499MB/s)(40.0MiB/84msec) slat (nsec): min=7337.4k, max=7337.4k, avg=7337368.00, stdev= 0.00 clat (nsec): min=73231k, max=73231k, avg=73230805.00, stdev= 0.00 lat (nsec): min=80589k, max=80589k, avg=80588858.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[72877], 5.00th=[72877], 10.00th=[72877], 20.00th=[72877], | 30.00th=[72877], 40.00th=[72877], 50.00th=[72877], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=8.43%, ctx=9, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=476MiB/s (499MB/s), 476MiB/s-476MiB/s (499MB/s-499MB/s), io=40.0MiB (41.9MB), run=84-84msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck