== sanity test 398c: run fio to test AIO ================= 04:07:22 (1734685642) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.100845 s, 416 MB/s osc.lustre-OST0000-osc-ffff8801302f7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=7519: Fri Dec 20 04:08:13 2024 write: IOPS=51, BW=206KiB/s (211kB/s)(10.0MiB/49759msec) slat (usec): min=11, max=1192, avg=89.22, stdev=74.97 clat (msec): min=84, max=740, avg=310.60, stdev=94.09 lat (msec): min=84, max=740, avg=310.69, stdev=94.10 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 180], 10.00th=[ 209], 20.00th=[ 234], | 30.00th=[ 257], 40.00th=[ 275], 50.00th=[ 300], 60.00th=[ 321], | 70.00th=[ 347], 80.00th=[ 376], 90.00th=[ 430], 95.00th=[ 485], | 99.00th=[ 600], 99.50th=[ 709], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 79, max= 384, per=24.89%, avg=204.82, stdev=54.50, samples=99 iops : min= 19, max= 96, avg=51.06, stdev=13.63, samples=99 lat (msec) : 100=0.16%, 250=26.76%, 500=68.98%, 750=4.10% cpu : usr=0.11%, sys=0.61%, ctx=2104, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=7520: Fri Dec 20 04:08:13 2024 write: IOPS=51, BW=206KiB/s (211kB/s)(10.0MiB/49725msec) slat (usec): min=12, max=920, avg=93.11, stdev=75.80 clat (msec): min=67, max=784, avg=310.30, stdev=94.52 lat (msec): min=67, max=784, avg=310.39, stdev=94.53 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 180], 10.00th=[ 211], 20.00th=[ 236], | 30.00th=[ 255], 40.00th=[ 275], 50.00th=[ 296], 60.00th=[ 317], | 70.00th=[ 347], 80.00th=[ 380], 90.00th=[ 426], 95.00th=[ 485], | 99.00th=[ 617], 99.50th=[ 684], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 785] bw ( KiB/s): min= 95, max= 392, per=24.97%, avg=205.46, stdev=55.44, samples=99 iops : min= 23, max= 98, avg=51.22, stdev=13.88, samples=99 lat (msec) : 100=0.16%, 250=27.03%, 500=68.79%, 750=3.91%, 1000=0.12% cpu : usr=0.16%, sys=0.59%, ctx=2033, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=7521: Fri Dec 20 04:08:13 2024 write: IOPS=51, BW=206KiB/s (211kB/s)(10.0MiB/49699msec) slat (usec): min=13, max=1240, avg=91.38, stdev=79.86 clat (msec): min=41, max=758, avg=310.25, stdev=96.59 lat (msec): min=41, max=758, avg=310.34, stdev=96.60 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 178], 10.00th=[ 207], 20.00th=[ 236], | 30.00th=[ 255], 40.00th=[ 271], 50.00th=[ 296], 60.00th=[ 321], | 70.00th=[ 351], 80.00th=[ 380], 90.00th=[ 435], 95.00th=[ 489], | 99.00th=[ 625], 99.50th=[ 684], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 760] bw ( KiB/s): min= 88, max= 384, per=24.98%, avg=205.56, stdev=54.66, samples=99 iops : min= 22, max= 96, avg=51.31, stdev=13.67, samples=99 lat (msec) : 50=0.62%, 250=26.45%, 500=68.87%, 750=3.95%, 1000=0.12% cpu : usr=0.17%, sys=0.57%, ctx=1810, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=7522: Fri Dec 20 04:08:13 2024 write: IOPS=51, BW=206KiB/s (211kB/s)(10.0MiB/49699msec) slat (usec): min=12, max=2755, avg=95.15, stdev=97.81 clat (msec): min=52, max=784, avg=310.23, stdev=96.74 lat (msec): min=52, max=784, avg=310.33, stdev=96.75 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 178], 10.00th=[ 209], 20.00th=[ 232], | 30.00th=[ 253], 40.00th=[ 271], 50.00th=[ 296], 60.00th=[ 317], | 70.00th=[ 351], 80.00th=[ 384], 90.00th=[ 426], 95.00th=[ 493], | 99.00th=[ 625], 99.50th=[ 684], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 785] bw ( KiB/s): min= 72, max= 400, per=24.97%, avg=205.51, stdev=57.51, samples=99 iops : min= 18, max= 100, avg=51.26, stdev=14.40, samples=99 lat (msec) : 100=0.74%, 250=27.54%, 500=67.38%, 750=4.18%, 1000=0.16% cpu : usr=0.22%, sys=0.52%, ctx=2182, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=823KiB/s (843kB/s), 206KiB/s-206KiB/s (211kB/s-211kB/s), io=40.0MiB (41.9MB), run=49699-49759msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7571: Fri Dec 20 04:08:48 2024 read: IOPS=37, BW=151KiB/s (155kB/s)(5048KiB/33425msec) slat (usec): min=1050, max=8646, avg=2320.55, stdev=804.12 clat (msec): min=3, max=424, avg=139.33, stdev=34.98 lat (msec): min=5, max=425, avg=141.65, stdev=35.17 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 112], | 30.00th=[ 122], 40.00th=[ 130], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 163], 90.00th=[ 182], 95.00th=[ 194], | 99.00th=[ 220], 99.50th=[ 259], 99.90th=[ 363], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 80, max= 256, per=25.28%, avg=151.41, stdev=42.93, samples=66 iops : min= 20, max= 64, avg=37.77, stdev=10.73, samples=66 write: IOPS=38, BW=155KiB/s (159kB/s)(5192KiB/33425msec) slat (usec): min=12, max=1097, avg=97.35, stdev=97.68 clat (msec): min=97, max=578, avg=273.55, stdev=79.37 lat (msec): min=97, max=579, avg=273.65, stdev=79.38 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 157], 10.00th=[ 180], 20.00th=[ 207], | 30.00th=[ 230], 40.00th=[ 251], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 334], 90.00th=[ 372], 95.00th=[ 414], | 99.00th=[ 527], 99.50th=[ 550], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 56, max= 264, per=24.95%, avg=155.41, stdev=36.74, samples=66 iops : min= 14, max= 66, avg=38.77, stdev= 9.17, samples=66 lat (msec) : 4=0.04%, 10=0.08%, 50=0.27%, 100=4.18%, 250=64.57% lat (msec) : 500=30.16%, 750=0.70% cpu : usr=0.11%, sys=1.68%, ctx=2936, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7572: Fri Dec 20 04:08:48 2024 read: IOPS=37, BW=149KiB/s (152kB/s)(4976KiB/33452msec) slat (usec): min=1105, max=7361, avg=2342.30, stdev=844.48 clat (usec): min=1583, max=450832, avg=139937.52, stdev=37918.20 lat (msec): min=2, max=452, avg=142.28, stdev=38.11 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 113], | 30.00th=[ 121], 40.00th=[ 129], 50.00th=[ 136], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 184], 95.00th=[ 197], | 99.00th=[ 239], 99.50th=[ 334], 99.90th=[ 393], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 40, max= 264, per=24.83%, avg=148.76, stdev=46.19, samples=66 iops : min= 10, max= 66, avg=37.03, stdev=11.62, samples=66 write: IOPS=39, BW=157KiB/s (161kB/s)(5264KiB/33452msec) slat (usec): min=14, max=2435, avg=105.45, stdev=121.55 clat (msec): min=101, max=575, avg=271.42, stdev=78.65 lat (msec): min=101, max=575, avg=271.52, stdev=78.66 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 155], 10.00th=[ 176], 20.00th=[ 203], | 30.00th=[ 228], 40.00th=[ 247], 50.00th=[ 268], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 334], 90.00th=[ 368], 95.00th=[ 414], | 99.00th=[ 498], 99.50th=[ 531], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 79, max= 256, per=25.30%, avg=157.61, stdev=36.65, samples=66 iops : min= 19, max= 64, avg=39.24, stdev= 9.16, samples=66 lat (msec) : 2=0.04%, 4=0.12%, 50=0.47%, 100=3.40%, 250=65.31% lat (msec) : 500=30.16%, 750=0.51% cpu : usr=0.32%, sys=1.54%, ctx=3026, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7573: Fri Dec 20 04:08:48 2024 read: IOPS=37, BW=149KiB/s (152kB/s)(4984KiB/33483msec) slat (usec): min=25, max=6299, avg=2284.01, stdev=787.03 clat (msec): min=3, max=263, avg=138.99, stdev=32.59 lat (msec): min=5, max=266, avg=141.28, stdev=32.80 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 114], | 30.00th=[ 122], 40.00th=[ 130], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 157], 80.00th=[ 165], 90.00th=[ 180], 95.00th=[ 194], | 99.00th=[ 215], 99.50th=[ 241], 99.90th=[ 259], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 48, max= 263, per=24.96%, avg=149.48, stdev=44.64, samples=66 iops : min= 12, max= 65, avg=37.21, stdev=11.20, samples=66 write: IOPS=39, BW=157KiB/s (161kB/s)(5256KiB/33483msec) slat (usec): min=14, max=2644, avg=104.17, stdev=129.67 clat (msec): min=33, max=571, avg=273.35, stdev=80.05 lat (msec): min=33, max=571, avg=273.45, stdev=80.06 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 159], 10.00th=[ 176], 20.00th=[ 205], | 30.00th=[ 228], 40.00th=[ 251], 50.00th=[ 268], 60.00th=[ 288], | 70.00th=[ 309], 80.00th=[ 338], 90.00th=[ 372], 95.00th=[ 422], | 99.00th=[ 498], 99.50th=[ 535], 99.90th=[ 550], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 64, max= 256, per=25.12%, avg=156.52, stdev=39.13, samples=66 iops : min= 16, max= 64, avg=38.97, stdev= 9.74, samples=66 lat (msec) : 4=0.04%, 20=0.20%, 50=0.47%, 100=3.87%, 250=64.45% lat (msec) : 500=30.51%, 750=0.47% cpu : usr=0.21%, sys=1.58%, ctx=2898, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7574: Fri Dec 20 04:08:48 2024 read: IOPS=37, BW=151KiB/s (155kB/s)(5060KiB/33483msec) slat (usec): min=999, max=7298, avg=2335.56, stdev=832.76 clat (msec): min=2, max=373, avg=138.38, stdev=34.45 lat (msec): min=5, max=376, avg=140.72, stdev=34.64 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 113], | 30.00th=[ 122], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 182], 95.00th=[ 197], | 99.00th=[ 234], 99.50th=[ 247], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 56, max= 264, per=25.40%, avg=152.17, stdev=46.23, samples=66 iops : min= 14, max= 66, avg=37.89, stdev=11.59, samples=66 write: IOPS=38, BW=155KiB/s (158kB/s)(5180KiB/33483msec) slat (usec): min=14, max=2269, avg=99.28, stdev=117.46 clat (msec): min=83, max=583, avg=275.49, stdev=80.42 lat (msec): min=84, max=583, avg=275.59, stdev=80.42 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 161], 10.00th=[ 178], 20.00th=[ 209], | 30.00th=[ 230], 40.00th=[ 249], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 309], 80.00th=[ 338], 90.00th=[ 372], 95.00th=[ 405], | 99.00th=[ 535], 99.50th=[ 550], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 88, max= 232, per=24.85%, avg=154.83, stdev=33.85, samples=66 iops : min= 22, max= 58, avg=38.56, stdev= 8.45, samples=66 lat (msec) : 4=0.04%, 10=0.12%, 20=0.04%, 50=0.39%, 100=4.45% lat (msec) : 250=64.65%, 500=29.30%, 750=1.02% cpu : usr=0.24%, sys=1.56%, ctx=3050, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=599KiB/s (614kB/s), 149KiB/s-151KiB/s (152kB/s-155kB/s), io=19.6MiB (20.5MB), run=33425-33483msec WRITE: bw=624KiB/s (639kB/s), 155KiB/s-157KiB/s (158kB/s-161kB/s), io=20.4MiB (21.4MB), run=33425-33483msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=7605: Fri Dec 20 04:08:50 2024 read: IOPS=4, BW=194MiB/s (204MB/s)(40.0MiB/206msec) slat (nsec): min=8088.6k, max=8088.6k, avg=8088586.00, stdev= 0.00 clat (nsec): min=189042k, max=189042k, avg=189041638.00, stdev= 0.00 lat (nsec): min=197159k, max=197159k, avg=197158884.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] lat (msec) : 250=100.00% cpu : usr=0.49%, sys=3.41%, ctx=41, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=194MiB/s (204MB/s), 194MiB/s-194MiB/s (204MB/s-204MB/s), io=40.0MiB (41.9MB), run=206-206msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout