== sanity test 398c: run fio to test AIO ================= 16:20:54 (1709068854) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.129057 s, 325 MB/s osc.lustre-OST0000-osc-ffff8800aa072800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=16525: Tue Feb 27 16:22:37 2024 write: IOPS=25, BW=101KiB/s (103kB/s)(10.0MiB/101771msec) slat (usec): min=10, max=4604, avg=107.38, stdev=219.59 clat (msec): min=114, max=1464, avg=635.08, stdev=157.15 lat (msec): min=114, max=1464, avg=635.19, stdev=157.15 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 393], 10.00th=[ 443], 20.00th=[ 510], | 30.00th=[ 558], 40.00th=[ 592], 50.00th=[ 625], 60.00th=[ 659], | 70.00th=[ 709], 80.00th=[ 760], 90.00th=[ 844], 95.00th=[ 911], | 99.00th=[ 986], 99.50th=[ 1053], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 231, per=25.05%, avg=100.46, stdev=30.66, samples=202 iops : min= 1, max= 57, avg=24.88, stdev= 7.65, samples=202 lat (msec) : 250=0.62%, 500=17.46%, 750=59.57%, 1000=21.52% cpu : usr=0.18%, sys=0.34%, ctx=1385, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=16526: Tue Feb 27 16:22:37 2024 write: IOPS=25, BW=101KiB/s (103kB/s)(10.0MiB/101703msec) slat (usec): min=11, max=3203, avg=99.59, stdev=186.73 clat (msec): min=56, max=1396, avg=634.68, stdev=152.79 lat (msec): min=56, max=1396, avg=634.78, stdev=152.79 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 397], 10.00th=[ 456], 20.00th=[ 510], | 30.00th=[ 558], 40.00th=[ 592], 50.00th=[ 625], 60.00th=[ 659], | 70.00th=[ 701], 80.00th=[ 760], 90.00th=[ 844], 95.00th=[ 911], | 99.00th=[ 995], 99.50th=[ 1028], 99.90th=[ 1250], 99.95th=[ 1368], | 99.99th=[ 1401] bw ( KiB/s): min= 40, max= 200, per=25.07%, avg=100.54, stdev=26.84, samples=202 iops : min= 10, max= 50, avg=24.95, stdev= 6.71, samples=202 lat (msec) : 100=0.04%, 250=0.55%, 500=18.44%, 750=59.45%, 1000=20.70% cpu : usr=0.12%, sys=0.42%, ctx=1466, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=16527: Tue Feb 27 16:22:37 2024 write: IOPS=25, BW=101KiB/s (103kB/s)(10.0MiB/101830msec) slat (usec): min=13, max=2302, avg=96.32, stdev=170.23 clat (msec): min=56, max=1330, avg=635.52, stdev=150.97 lat (msec): min=56, max=1330, avg=635.62, stdev=150.98 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 393], 10.00th=[ 447], 20.00th=[ 514], | 30.00th=[ 558], 40.00th=[ 592], 50.00th=[ 625], 60.00th=[ 659], | 70.00th=[ 709], 80.00th=[ 751], 90.00th=[ 844], 95.00th=[ 911], | 99.00th=[ 1011], 99.50th=[ 1053], 99.90th=[ 1133], 99.95th=[ 1183], | 99.99th=[ 1334] bw ( KiB/s): min= 24, max= 208, per=24.96%, avg=100.07, stdev=29.86, samples=203 iops : min= 6, max= 52, avg=24.85, stdev= 7.49, samples=203 lat (msec) : 100=0.04%, 250=0.31%, 500=17.50%, 750=61.41%, 1000=19.73% cpu : usr=0.10%, sys=0.39%, ctx=1363, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=16528: Tue Feb 27 16:22:37 2024 write: IOPS=25, BW=100KiB/s (103kB/s)(10.0MiB/101898msec) slat (usec): min=9, max=3357, avg=98.99, stdev=207.65 clat (msec): min=208, max=1063, avg=636.08, stdev=148.49 lat (msec): min=208, max=1063, avg=636.18, stdev=148.50 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 393], 10.00th=[ 451], 20.00th=[ 510], | 30.00th=[ 558], 40.00th=[ 592], 50.00th=[ 625], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 760], 90.00th=[ 844], 95.00th=[ 894], | 99.00th=[ 995], 99.50th=[ 1020], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 15, max= 208, per=24.92%, avg=99.92, stdev=30.29, samples=203 iops : min= 3, max= 52, avg=24.82, stdev= 7.62, samples=203 lat (msec) : 250=0.27%, 500=18.01%, 750=59.45%, 1000=21.45% cpu : usr=0.11%, sys=0.37%, ctx=1348, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=402KiB/s (412kB/s), 100KiB/s-101KiB/s (103kB/s-103kB/s), io=40.0MiB (41.9MB), run=101703-101898msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16620: Tue Feb 27 16:23:40 2024 read: IOPS=20, BW=82.4KiB/s (84.4kB/s)(5048KiB/61270msec) slat (usec): min=23, max=42254, avg=4240.19, stdev=2459.98 clat (msec): min=4, max=801, avg=248.01, stdev=85.43 lat (msec): min=8, max=803, avg=252.26, stdev=86.20 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 131], 10.00th=[ 157], 20.00th=[ 188], | 30.00th=[ 207], 40.00th=[ 222], 50.00th=[ 241], 60.00th=[ 255], | 70.00th=[ 275], 80.00th=[ 305], 90.00th=[ 342], 95.00th=[ 384], | 99.00th=[ 584], 99.50th=[ 617], 99.90th=[ 735], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 15, max= 192, per=25.12%, avg=82.13, stdev=36.48, samples=122 iops : min= 3, max= 48, avg=20.26, stdev= 9.20, samples=122 write: IOPS=21, BW=84.7KiB/s (86.8kB/s)(5192KiB/61270msec) slat (usec): min=14, max=8022, avg=111.92, stdev=274.96 clat (msec): min=45, max=1328, avg=509.39, stdev=187.03 lat (msec): min=45, max=1328, avg=509.50, stdev=187.03 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 253], 10.00th=[ 300], 20.00th=[ 351], | 30.00th=[ 397], 40.00th=[ 439], 50.00th=[ 481], 60.00th=[ 535], | 70.00th=[ 600], 80.00th=[ 651], 90.00th=[ 768], 95.00th=[ 860], | 99.00th=[ 1020], 99.50th=[ 1045], 99.90th=[ 1234], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 16, max= 176, per=24.72%, avg=84.03, stdev=30.80, samples=122 iops : min= 4, max= 44, avg=20.74, stdev= 7.72, samples=122 lat (msec) : 10=0.04%, 50=0.35%, 100=0.82%, 250=29.22%, 500=45.51% lat (msec) : 750=18.16%, 1000=5.31% cpu : usr=0.09%, sys=1.28%, ctx=2668, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16621: Tue Feb 27 16:23:40 2024 read: IOPS=20, BW=81.2KiB/s (83.2kB/s)(4976KiB/61270msec) slat (usec): min=26, max=17452, avg=4311.36, stdev=2311.74 clat (msec): min=4, max=648, avg=244.65, stdev=79.65 lat (msec): min=7, max=656, avg=248.96, stdev=80.51 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 182], | 30.00th=[ 203], 40.00th=[ 220], 50.00th=[ 236], 60.00th=[ 255], | 70.00th=[ 275], 80.00th=[ 305], 90.00th=[ 342], 95.00th=[ 380], | 99.00th=[ 514], 99.50th=[ 535], 99.90th=[ 625], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 7, max= 191, per=24.85%, avg=81.25, stdev=35.80, samples=122 iops : min= 1, max= 47, avg=20.12, stdev= 9.02, samples=122 write: IOPS=21, BW=85.9KiB/s (87.0kB/s)(5264KiB/61270msec) slat (usec): min=10, max=2469, avg=115.08, stdev=189.67 clat (msec): min=46, max=1175, avg=509.05, stdev=185.11 lat (msec): min=46, max=1175, avg=509.17, stdev=185.12 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 245], 10.00th=[ 300], 20.00th=[ 355], | 30.00th=[ 397], 40.00th=[ 443], 50.00th=[ 485], 60.00th=[ 531], | 70.00th=[ 600], 80.00th=[ 651], 90.00th=[ 760], 95.00th=[ 869], | 99.00th=[ 1020], 99.50th=[ 1070], 99.90th=[ 1150], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8, max= 176, per=25.04%, avg=85.12, stdev=32.28, samples=122 iops : min= 2, max= 44, avg=21.09, stdev= 8.12, samples=122 lat (msec) : 10=0.04%, 50=0.47%, 100=0.43%, 250=29.80%, 500=45.20% lat (msec) : 750=18.59%, 1000=4.84% cpu : usr=0.15%, sys=1.25%, ctx=2723, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16622: Tue Feb 27 16:23:40 2024 read: IOPS=20, BW=81.3KiB/s (83.3kB/s)(4984KiB/61270msec) slat (usec): min=20, max=24304, avg=4267.92, stdev=2338.26 clat (msec): min=4, max=716, avg=246.03, stdev=76.72 lat (msec): min=7, max=722, avg=250.30, stdev=77.59 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 138], 10.00th=[ 157], 20.00th=[ 188], | 30.00th=[ 209], 40.00th=[ 224], 50.00th=[ 239], 60.00th=[ 255], | 70.00th=[ 275], 80.00th=[ 305], 90.00th=[ 342], 95.00th=[ 376], | 99.00th=[ 481], 99.50th=[ 531], 99.90th=[ 634], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 7, max= 208, per=24.89%, avg=81.40, stdev=36.46, samples=122 iops : min= 1, max= 52, avg=20.17, stdev= 9.21, samples=122 write: IOPS=21, BW=85.8KiB/s (87.8kB/s)(5256KiB/61270msec) slat (usec): min=10, max=2580, avg=108.76, stdev=182.78 clat (msec): min=45, max=1178, avg=508.11, stdev=188.21 lat (msec): min=45, max=1178, avg=508.22, stdev=188.22 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 234], 10.00th=[ 296], 20.00th=[ 351], | 30.00th=[ 393], 40.00th=[ 439], 50.00th=[ 485], 60.00th=[ 531], | 70.00th=[ 600], 80.00th=[ 659], 90.00th=[ 760], 95.00th=[ 844], | 99.00th=[ 1036], 99.50th=[ 1116], 99.90th=[ 1133], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 31, max= 216, per=25.00%, avg=85.01, stdev=31.72, samples=122 iops : min= 7, max= 54, avg=21.07, stdev= 7.96, samples=122 lat (msec) : 10=0.08%, 50=0.31%, 100=0.39%, 250=30.23%, 500=45.39% lat (msec) : 750=18.01%, 1000=4.84% cpu : usr=0.26%, sys=1.11%, ctx=2758, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16623: Tue Feb 27 16:23:40 2024 read: IOPS=20, BW=82.6KiB/s (84.6kB/s)(5060KiB/61270msec) slat (usec): min=21, max=27700, avg=4292.62, stdev=2441.99 clat (msec): min=4, max=715, avg=248.62, stdev=77.94 lat (msec): min=7, max=721, avg=252.92, stdev=78.77 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 140], 10.00th=[ 159], 20.00th=[ 190], | 30.00th=[ 209], 40.00th=[ 226], 50.00th=[ 243], 60.00th=[ 257], | 70.00th=[ 279], 80.00th=[ 309], 90.00th=[ 347], 95.00th=[ 384], | 99.00th=[ 514], 99.50th=[ 550], 99.90th=[ 609], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 7, max= 184, per=25.49%, avg=83.35, stdev=34.73, samples=121 iops : min= 1, max= 46, avg=20.67, stdev= 8.73, samples=121 write: IOPS=21, BW=84.5KiB/s (86.6kB/s)(5180KiB/61270msec) slat (usec): min=13, max=2735, avg=125.85, stdev=231.99 clat (msec): min=46, max=1336, avg=509.31, stdev=190.20 lat (msec): min=46, max=1336, avg=509.44, stdev=190.21 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 243], 10.00th=[ 292], 20.00th=[ 351], | 30.00th=[ 397], 40.00th=[ 435], 50.00th=[ 485], 60.00th=[ 535], | 70.00th=[ 592], 80.00th=[ 659], 90.00th=[ 760], 95.00th=[ 869], | 99.00th=[ 1062], 99.50th=[ 1133], 99.90th=[ 1234], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 16, max= 200, per=24.64%, avg=83.77, stdev=31.78, samples=122 iops : min= 4, max= 50, avg=20.77, stdev= 7.99, samples=122 lat (msec) : 10=0.12%, 50=0.31%, 100=0.23%, 250=29.88%, 500=45.31% lat (msec) : 750=18.91%, 1000=4.22% cpu : usr=0.25%, sys=1.18%, ctx=2786, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=328KiB/s (335kB/s), 81.2KiB/s-82.6KiB/s (83.2kB/s-84.6kB/s), io=19.6MiB (20.5MB), run=61270-61270msec WRITE: bw=341KiB/s (349kB/s), 84.5KiB/s-85.9KiB/s (86.6kB/s-87.0kB/s), io=20.4MiB (21.4MB), run=61270-61270msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=16675: Tue Feb 27 16:23:41 2024 read: IOPS=2, BW=112MiB/s (117MB/s)(40.0MiB/358msec) slat (nsec): min=10981k, max=10981k, avg=10981484.00, stdev= 0.00 clat (nsec): min=345877k, max=345877k, avg=345877103.00, stdev= 0.00 lat (nsec): min=356888k, max=356888k, avg=356888493.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.68%, ctx=44, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=112MiB/s (117MB/s), 112MiB/s-112MiB/s (117MB/s-117MB/s), io=40.0MiB (41.9MB), run=358-358msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout