== sanity test 398c: run fio to test AIO ================= 08:43:35 (1726836215) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.172616 s, 243 MB/s osc.lustre-OST0000-osc-ffff880088c73000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=14047: Fri Sep 20 08:45:36 2024 write: IOPS=21, BW=86.7KiB/s (88.8kB/s)(10.0MiB/118047msec) slat (usec): min=13, max=11849, avg=154.79, stdev=422.50 clat (msec): min=130, max=1358, avg=736.88, stdev=169.66 lat (msec): min=130, max=1358, avg=737.03, stdev=169.68 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 468], 10.00th=[ 542], 20.00th=[ 609], | 30.00th=[ 667], 40.00th=[ 701], 50.00th=[ 735], 60.00th=[ 776], | 70.00th=[ 827], 80.00th=[ 877], 90.00th=[ 936], 95.00th=[ 986], | 99.00th=[ 1133], 99.50th=[ 1217], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 16, max= 256, per=25.16%, avg=87.06, stdev=31.03, samples=232 iops : min= 4, max= 64, avg=21.56, stdev= 7.80, samples=232 lat (msec) : 250=0.62%, 500=5.82%, 750=47.62%, 1000=41.41% cpu : usr=0.14%, sys=0.40%, ctx=1338, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=14048: Fri Sep 20 08:45:36 2024 write: IOPS=21, BW=86.7KiB/s (88.8kB/s)(10.0MiB/118048msec) slat (usec): min=15, max=9132, avg=156.56, stdev=412.33 clat (msec): min=89, max=1373, avg=736.88, stdev=165.80 lat (msec): min=89, max=1373, avg=737.04, stdev=165.81 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 456], 10.00th=[ 542], 20.00th=[ 617], | 30.00th=[ 667], 40.00th=[ 709], 50.00th=[ 743], 60.00th=[ 785], | 70.00th=[ 818], 80.00th=[ 869], 90.00th=[ 936], 95.00th=[ 995], | 99.00th=[ 1083], 99.50th=[ 1099], 99.90th=[ 1318], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 15, max= 248, per=24.81%, avg=85.86, stdev=30.51, samples=236 iops : min= 3, max= 62, avg=21.17, stdev= 7.64, samples=236 lat (msec) : 100=0.04%, 250=0.31%, 500=6.17%, 750=45.35%, 1000=43.48% cpu : usr=0.10%, sys=0.47%, ctx=1427, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=14049: Fri Sep 20 08:45:36 2024 write: IOPS=21, BW=86.8KiB/s (88.9kB/s)(10.0MiB/117974msec) slat (usec): min=15, max=9030, avg=155.37, stdev=411.77 clat (msec): min=114, max=1356, avg=736.43, stdev=164.19 lat (msec): min=114, max=1356, avg=736.59, stdev=164.19 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 477], 10.00th=[ 542], 20.00th=[ 609], | 30.00th=[ 659], 40.00th=[ 701], 50.00th=[ 743], 60.00th=[ 785], | 70.00th=[ 818], 80.00th=[ 877], 90.00th=[ 936], 95.00th=[ 995], | 99.00th=[ 1099], 99.50th=[ 1133], 99.90th=[ 1200], 99.95th=[ 1334], | 99.99th=[ 1351] bw ( KiB/s): min= 15, max= 256, per=24.89%, avg=86.13, stdev=31.55, samples=235 iops : min= 3, max= 64, avg=21.31, stdev= 7.91, samples=235 lat (msec) : 250=0.16%, 500=6.02%, 750=45.94%, 1000=43.28% cpu : usr=0.15%, sys=0.42%, ctx=1456, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=14050: Fri Sep 20 08:45:36 2024 write: IOPS=21, BW=86.9KiB/s (88.9kB/s)(10.0MiB/117897msec) slat (usec): min=10, max=10876, avg=168.03, stdev=475.07 clat (msec): min=84, max=1342, avg=736.07, stdev=172.77 lat (msec): min=84, max=1342, avg=736.24, stdev=172.78 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 460], 10.00th=[ 535], 20.00th=[ 609], | 30.00th=[ 659], 40.00th=[ 701], 50.00th=[ 743], 60.00th=[ 785], | 70.00th=[ 827], 80.00th=[ 869], 90.00th=[ 944], 95.00th=[ 1011], | 99.00th=[ 1099], 99.50th=[ 1150], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1351] bw ( KiB/s): min= 8, max= 256, per=24.93%, avg=86.26, stdev=31.04, samples=235 iops : min= 2, max= 64, avg=21.34, stdev= 7.81, samples=235 lat (msec) : 100=0.62%, 500=5.98%, 750=46.29%, 1000=41.60% cpu : usr=0.10%, sys=0.45%, ctx=1348, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=347KiB/s (355kB/s), 86.7KiB/s-86.9KiB/s (88.8kB/s-88.9kB/s), io=40.0MiB (41.9MB), run=117897-118048msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14154: Fri Sep 20 08:46:43 2024 read: IOPS=19, BW=76.9KiB/s (78.7kB/s)(5048KiB/65646msec) slat (usec): min=1607, max=25987, avg=5908.93, stdev=2925.57 clat (msec): min=12, max=512, avg=268.13, stdev=65.03 lat (msec): min=16, max=520, avg=274.05, stdev=65.85 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 169], 10.00th=[ 190], 20.00th=[ 213], | 30.00th=[ 234], 40.00th=[ 251], 50.00th=[ 266], 60.00th=[ 284], | 70.00th=[ 300], 80.00th=[ 317], 90.00th=[ 347], 95.00th=[ 380], | 99.00th=[ 435], 99.50th=[ 447], 99.90th=[ 510], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 15, max= 183, per=25.09%, avg=76.53, stdev=30.62, samples=131 iops : min= 3, max= 45, avg=18.95, stdev= 7.65, samples=131 write: IOPS=19, BW=79.1KiB/s (80.0kB/s)(5192KiB/65646msec) slat (usec): min=16, max=4105, avg=157.53, stdev=305.09 clat (msec): min=193, max=1092, avg=541.57, stdev=144.49 lat (msec): min=193, max=1092, avg=541.73, stdev=144.51 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 330], 10.00th=[ 376], 20.00th=[ 422], | 30.00th=[ 460], 40.00th=[ 493], 50.00th=[ 523], 60.00th=[ 558], | 70.00th=[ 609], 80.00th=[ 659], 90.00th=[ 735], 95.00th=[ 818], | 99.00th=[ 978], 99.50th=[ 1045], 99.90th=[ 1083], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 160, per=24.66%, avg=78.42, stdev=24.56, samples=131 iops : min= 1, max= 40, avg=19.43, stdev= 6.19, samples=131 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=19.69%, 500=50.82% lat (msec) : 750=24.77%, 1000=4.02% cpu : usr=0.36%, sys=1.25%, ctx=2693, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14155: Fri Sep 20 08:46:43 2024 read: IOPS=18, BW=75.9KiB/s (77.8kB/s)(4976KiB/65525msec) slat (usec): min=17, max=28102, avg=5737.60, stdev=2741.22 clat (msec): min=5, max=781, avg=270.00, stdev=70.76 lat (msec): min=11, max=785, avg=275.75, stdev=71.56 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 171], 10.00th=[ 192], 20.00th=[ 215], | 30.00th=[ 236], 40.00th=[ 253], 50.00th=[ 268], 60.00th=[ 279], | 70.00th=[ 296], 80.00th=[ 317], 90.00th=[ 351], 95.00th=[ 388], | 99.00th=[ 468], 99.50th=[ 493], 99.90th=[ 726], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 8, max= 160, per=24.65%, avg=75.17, stdev=30.24, samples=131 iops : min= 2, max= 40, avg=18.60, stdev= 7.59, samples=131 write: IOPS=20, BW=80.3KiB/s (82.3kB/s)(5264KiB/65525msec) slat (usec): min=17, max=5125, avg=163.95, stdev=377.81 clat (msec): min=59, max=1112, avg=535.33, stdev=151.33 lat (msec): min=59, max=1112, avg=535.49, stdev=151.36 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 321], 10.00th=[ 355], 20.00th=[ 418], | 30.00th=[ 456], 40.00th=[ 485], 50.00th=[ 523], 60.00th=[ 558], | 70.00th=[ 600], 80.00th=[ 651], 90.00th=[ 726], 95.00th=[ 802], | 99.00th=[ 961], 99.50th=[ 1036], 99.90th=[ 1083], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 16, max= 160, per=25.08%, avg=79.75, stdev=26.40, samples=131 iops : min= 4, max= 40, avg=19.75, stdev= 6.62, samples=131 lat (msec) : 10=0.08%, 50=0.23%, 100=0.16%, 250=19.26%, 500=51.25% lat (msec) : 750=24.69%, 1000=3.95% cpu : usr=0.06%, sys=1.55%, ctx=2767, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14156: Fri Sep 20 08:46:43 2024 read: IOPS=18, BW=75.9KiB/s (77.7kB/s)(4984KiB/65645msec) slat (usec): min=1347, max=33019, avg=5785.76, stdev=2906.09 clat (msec): min=10, max=527, avg=267.63, stdev=65.33 lat (msec): min=15, max=541, avg=273.42, stdev=66.22 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 176], 10.00th=[ 192], 20.00th=[ 215], | 30.00th=[ 234], 40.00th=[ 249], 50.00th=[ 264], 60.00th=[ 279], | 70.00th=[ 292], 80.00th=[ 317], 90.00th=[ 351], 95.00th=[ 388], | 99.00th=[ 443], 99.50th=[ 477], 99.90th=[ 506], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 8, max= 159, per=24.79%, avg=75.60, stdev=30.98, samples=131 iops : min= 2, max= 39, avg=18.66, stdev= 7.76, samples=131 write: IOPS=20, BW=80.1KiB/s (81.0kB/s)(5256KiB/65645msec) slat (usec): min=16, max=3767, avg=164.65, stdev=339.82 clat (msec): min=170, max=1106, avg=539.11, stdev=146.80 lat (msec): min=170, max=1107, avg=539.28, stdev=146.80 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 326], 10.00th=[ 368], 20.00th=[ 422], | 30.00th=[ 456], 40.00th=[ 489], 50.00th=[ 523], 60.00th=[ 558], | 70.00th=[ 600], 80.00th=[ 651], 90.00th=[ 726], 95.00th=[ 793], | 99.00th=[ 978], 99.50th=[ 1045], 99.90th=[ 1083], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 32, max= 136, per=24.91%, avg=79.20, stdev=20.30, samples=131 iops : min= 8, max= 34, avg=19.56, stdev= 5.13, samples=131 lat (msec) : 20=0.04%, 50=0.12%, 100=0.12%, 250=20.16%, 500=50.04% lat (msec) : 750=25.51%, 1000=3.55% cpu : usr=0.23%, sys=1.36%, ctx=2761, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14157: Fri Sep 20 08:46:43 2024 read: IOPS=19, BW=77.1KiB/s (78.0kB/s)(5060KiB/65601msec) slat (usec): min=1371, max=32536, avg=5799.51, stdev=2766.59 clat (msec): min=3, max=665, avg=271.74, stdev=72.28 lat (msec): min=10, max=678, avg=277.55, stdev=73.07 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 171], 10.00th=[ 190], 20.00th=[ 211], | 30.00th=[ 236], 40.00th=[ 251], 50.00th=[ 266], 60.00th=[ 284], | 70.00th=[ 305], 80.00th=[ 330], 90.00th=[ 359], 95.00th=[ 393], | 99.00th=[ 485], 99.50th=[ 523], 99.90th=[ 617], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 16, max= 144, per=25.09%, avg=76.53, stdev=27.47, samples=131 iops : min= 4, max= 36, avg=18.87, stdev= 6.87, samples=131 write: IOPS=19, BW=78.0KiB/s (80.9kB/s)(5180KiB/65601msec) slat (usec): min=16, max=9932, avg=169.75, stdev=469.00 clat (msec): min=197, max=1092, avg=538.13, stdev=144.93 lat (msec): min=197, max=1092, avg=538.30, stdev=144.95 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 317], 10.00th=[ 368], 20.00th=[ 422], | 30.00th=[ 456], 40.00th=[ 489], 50.00th=[ 523], 60.00th=[ 558], | 70.00th=[ 609], 80.00th=[ 651], 90.00th=[ 726], 95.00th=[ 802], | 99.00th=[ 953], 99.50th=[ 1003], 99.90th=[ 1070], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 16, max= 152, per=24.59%, avg=78.18, stdev=24.77, samples=131 iops : min= 4, max= 38, avg=19.28, stdev= 6.25, samples=131 lat (msec) : 4=0.04%, 10=0.04%, 20=0.04%, 50=0.12%, 100=0.16% lat (msec) : 250=19.88%, 500=50.74%, 750=25.12%, 1000=3.55% cpu : usr=0.20%, sys=1.41%, ctx=2771, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=306KiB/s (313kB/s), 75.9KiB/s-77.1KiB/s (77.7kB/s-78.0kB/s), io=19.6MiB (20.5MB), run=65525-65646msec WRITE: bw=318KiB/s (326kB/s), 78.0KiB/s-80.3KiB/s (80.9kB/s-82.3kB/s), io=20.4MiB (21.4MB), run=65525-65646msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=14214: Fri Sep 20 08:46:44 2024 read: IOPS=4, BW=172MiB/s (181MB/s)(40.0MiB/232msec) slat (nsec): min=12732k, max=12732k, avg=12731905.00, stdev= 0.00 clat (nsec): min=209007k, max=209007k, avg=209007254.00, stdev= 0.00 lat (nsec): min=221765k, max=221765k, avg=221765404.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] lat (msec) : 250=100.00% cpu : usr=2.16%, sys=2.59%, ctx=16, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172MiB/s (181MB/s), 172MiB/s-172MiB/s (181MB/s-181MB/s), io=40.0MiB (41.9MB), run=232-232msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout