== sanity test 398c: run fio to test AIO ================= 15:32:27 (1733171547) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0940856 s, 446 MB/s osc.lustre-OST0000-osc-ffff88009bf30000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=30871: Mon Dec 2 15:32:57 2024 write: IOPS=89, BW=356KiB/s (365kB/s)(10.0MiB/28749msec) slat (usec): min=12, max=3432, avg=85.10, stdev=89.88 clat (msec): min=49, max=357, avg=179.44, stdev=39.48 lat (msec): min=49, max=357, avg=179.53, stdev=39.49 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 123], 10.00th=[ 136], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 209], 90.00th=[ 228], 95.00th=[ 249], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 351], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 200, max= 512, per=24.88%, avg=354.30, stdev=58.69, samples=57 iops : min= 50, max= 128, avg=88.51, stdev=14.69, samples=57 lat (msec) : 50=0.04%, 100=1.37%, 250=93.67%, 500=4.92% cpu : usr=0.24%, sys=0.71%, ctx=2525, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=30872: Mon Dec 2 15:32:57 2024 write: IOPS=89, BW=356KiB/s (365kB/s)(10.0MiB/28737msec) slat (usec): min=10, max=1743, avg=90.12, stdev=72.66 clat (msec): min=39, max=366, avg=179.29, stdev=39.29 lat (msec): min=39, max=366, avg=179.38, stdev=39.30 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 124], 10.00th=[ 133], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 247], | 99.00th=[ 292], 99.50th=[ 313], 99.90th=[ 359], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 232, max= 520, per=24.87%, avg=354.16, stdev=59.59, samples=57 iops : min= 58, max= 130, avg=88.47, stdev=14.91, samples=57 lat (msec) : 50=0.04%, 100=0.82%, 250=94.34%, 500=4.80% cpu : usr=0.19%, sys=0.85%, ctx=2626, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=30873: Mon Dec 2 15:32:57 2024 write: IOPS=89, BW=357KiB/s (366kB/s)(10.0MiB/28656msec) slat (usec): min=10, max=566, avg=89.34, stdev=54.85 clat (msec): min=18, max=366, avg=178.89, stdev=40.93 lat (msec): min=18, max=366, avg=178.98, stdev=40.93 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 124], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 245], | 99.00th=[ 292], 99.50th=[ 317], 99.90th=[ 359], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 216, max= 568, per=25.04%, avg=356.54, stdev=62.48, samples=57 iops : min= 54, max= 142, avg=89.07, stdev=15.63, samples=57 lat (msec) : 20=0.27%, 50=0.31%, 100=1.13%, 250=93.67%, 500=4.61% cpu : usr=0.18%, sys=0.84%, ctx=2755, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=30874: Mon Dec 2 15:32:57 2024 write: IOPS=89, BW=356KiB/s (365kB/s)(10.0MiB/28726msec) slat (usec): min=12, max=1204, avg=89.91, stdev=64.20 clat (msec): min=33, max=366, avg=179.21, stdev=38.79 lat (msec): min=33, max=366, avg=179.30, stdev=38.80 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 126], 10.00th=[ 136], 20.00th=[ 148], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 249], | 99.00th=[ 292], 99.50th=[ 313], 99.90th=[ 342], 99.95th=[ 351], | 99.99th=[ 368] bw ( KiB/s): min= 224, max= 520, per=24.95%, avg=355.28, stdev=58.77, samples=57 iops : min= 56, max= 130, avg=88.75, stdev=14.69, samples=57 lat (msec) : 50=0.08%, 100=0.82%, 250=94.26%, 500=4.84% cpu : usr=0.21%, sys=0.80%, ctx=2728, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1425KiB/s (1459kB/s), 356KiB/s-357KiB/s (365kB/s-366kB/s), io=40.0MiB (41.9MB), run=28656-28749msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30907: Mon Dec 2 15:33:18 2024 read: IOPS=64, BW=258KiB/s (264kB/s)(5048KiB/19600msec) slat (usec): min=695, max=5878, avg=1454.07, stdev=542.33 clat (msec): min=10, max=167, avg=78.41, stdev=20.94 lat (msec): min=11, max=168, avg=79.86, stdev=21.10 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 63], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 81], | 70.00th=[ 87], 80.00th=[ 93], 90.00th=[ 104], 95.00th=[ 120], | 99.00th=[ 144], 99.50th=[ 146], 99.90th=[ 153], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 104, max= 392, per=25.17%, avg=256.95, stdev=67.52, samples=39 iops : min= 26, max= 98, avg=64.18, stdev=16.87, samples=39 write: IOPS=66, BW=265KiB/s (271kB/s)(5192KiB/19600msec) slat (usec): min=11, max=640, avg=71.27, stdev=49.93 clat (msec): min=67, max=364, avg=163.52, stdev=46.12 lat (msec): min=67, max=364, avg=163.60, stdev=46.12 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 100], 10.00th=[ 112], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 199], 90.00th=[ 222], 95.00th=[ 241], | 99.00th=[ 326], 99.50th=[ 347], 99.90th=[ 359], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 160, max= 368, per=24.79%, avg=263.51, stdev=52.60, samples=39 iops : min= 40, max= 92, avg=65.82, stdev=13.14, samples=39 lat (msec) : 20=0.51%, 50=1.68%, 100=43.91%, 250=51.88%, 500=2.03% cpu : usr=0.28%, sys=1.61%, ctx=3255, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30908: Mon Dec 2 15:33:18 2024 read: IOPS=63, BW=253KiB/s (260kB/s)(4976KiB/19633msec) slat (usec): min=676, max=15362, avg=1466.34, stdev=668.24 clat (usec): min=1746, max=169040, avg=78490.73, stdev=20716.87 lat (msec): min=3, max=171, avg=79.96, stdev=20.89 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 81], | 70.00th=[ 86], 80.00th=[ 93], 90.00th=[ 104], 95.00th=[ 116], | 99.00th=[ 146], 99.50th=[ 150], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 88, max= 384, per=24.54%, avg=250.59, stdev=65.81, samples=39 iops : min= 22, max= 96, avg=62.59, stdev=16.44, samples=39 write: IOPS=67, BW=268KiB/s (275kB/s)(5264KiB/19633msec) slat (usec): min=13, max=1627, avg=74.67, stdev=77.37 clat (msec): min=40, max=357, avg=162.81, stdev=46.87 lat (msec): min=41, max=357, avg=162.89, stdev=46.88 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 94], 10.00th=[ 112], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 157], 60.00th=[ 167], | 70.00th=[ 180], 80.00th=[ 199], 90.00th=[ 220], 95.00th=[ 245], | 99.00th=[ 317], 99.50th=[ 342], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 144, max= 344, per=25.06%, avg=266.38, stdev=52.69, samples=39 iops : min= 36, max= 86, avg=66.54, stdev=13.15, samples=39 lat (msec) : 2=0.04%, 10=0.04%, 20=0.39%, 50=1.13%, 100=43.79% lat (msec) : 250=52.58%, 500=2.03% cpu : usr=0.34%, sys=1.55%, ctx=3251, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30909: Mon Dec 2 15:33:18 2024 read: IOPS=63, BW=254KiB/s (260kB/s)(4984KiB/19643msec) slat (usec): min=694, max=7517, avg=1482.60, stdev=557.21 clat (msec): min=3, max=175, avg=79.00, stdev=19.89 lat (msec): min=4, max=177, avg=80.48, stdev=20.06 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 54], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 86], 80.00th=[ 93], 90.00th=[ 104], 95.00th=[ 114], | 99.00th=[ 146], 99.50th=[ 155], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 128, max= 416, per=24.74%, avg=252.64, stdev=62.24, samples=39 iops : min= 32, max= 104, avg=63.10, stdev=15.57, samples=39 write: IOPS=66, BW=268KiB/s (274kB/s)(5256KiB/19643msec) slat (usec): min=12, max=676, avg=72.76, stdev=54.07 clat (msec): min=53, max=370, avg=162.50, stdev=46.32 lat (msec): min=53, max=370, avg=162.57, stdev=46.33 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 97], 10.00th=[ 113], 20.00th=[ 127], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 157], 60.00th=[ 167], | 70.00th=[ 180], 80.00th=[ 199], 90.00th=[ 220], 95.00th=[ 247], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 347], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 152, max= 368, per=24.81%, avg=263.72, stdev=55.00, samples=39 iops : min= 38, max= 92, avg=65.87, stdev=13.72, samples=39 lat (msec) : 4=0.04%, 20=0.23%, 50=0.98%, 100=44.57%, 250=52.07% lat (msec) : 500=2.11% cpu : usr=0.21%, sys=1.67%, ctx=3383, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30910: Mon Dec 2 15:33:18 2024 read: IOPS=64, BW=259KiB/s (265kB/s)(5060KiB/19537msec) slat (usec): min=30, max=5442, avg=1445.45, stdev=506.79 clat (usec): min=1795, max=169223, avg=78487.13, stdev=21877.87 lat (msec): min=2, max=171, avg=79.93, stdev=22.04 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 52], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 86], 80.00th=[ 93], 90.00th=[ 104], 95.00th=[ 120], | 99.00th=[ 148], 99.50th=[ 153], 99.90th=[ 165], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 104, max= 376, per=25.29%, avg=258.18, stdev=64.50, samples=39 iops : min= 26, max= 94, avg=64.49, stdev=16.11, samples=39 write: IOPS=66, BW=265KiB/s (272kB/s)(5180KiB/19537msec) slat (usec): min=11, max=1078, avg=73.89, stdev=57.89 clat (msec): min=16, max=364, avg=163.05, stdev=47.32 lat (msec): min=16, max=364, avg=163.13, stdev=47.32 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 93], 10.00th=[ 112], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 199], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 321], 99.50th=[ 342], 99.90th=[ 359], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 152, max= 344, per=24.81%, avg=263.72, stdev=49.52, samples=39 iops : min= 38, max= 86, avg=65.87, stdev=12.39, samples=39 lat (msec) : 2=0.04%, 4=0.12%, 10=0.08%, 20=0.74%, 50=1.56% lat (msec) : 100=43.83%, 250=51.72%, 500=1.91% cpu : usr=0.32%, sys=1.57%, ctx=3228, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1022KiB/s (1046kB/s), 253KiB/s-259KiB/s (260kB/s-265kB/s), io=19.6MiB (20.5MB), run=19537-19643msec WRITE: bw=1064KiB/s (1089kB/s), 265KiB/s-268KiB/s (271kB/s-275kB/s), io=20.4MiB (21.4MB), run=19537-19643msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=30930: Mon Dec 2 15:33:19 2024 read: IOPS=10, BW=412MiB/s (432MB/s)(40.0MiB/97msec) slat (nsec): min=6553.7k, max=6553.7k, avg=6553652.00, stdev= 0.00 clat (nsec): min=85321k, max=85321k, avg=85320503.00, stdev= 0.00 lat (nsec): min=91899k, max=91899k, avg=91899448.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[85459], 5.00th=[85459], 10.00th=[85459], 20.00th=[85459], | 30.00th=[85459], 40.00th=[85459], 50.00th=[85459], 60.00th=[85459], | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.29%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=412MiB/s (432MB/s), 412MiB/s-412MiB/s (432MB/s-432MB/s), io=40.0MiB (41.9MB), run=97-97msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout