== sanity test 398c: run fio to test AIO ================= 04:43:19 (1722242599) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.142268 s, 295 MB/s osc.lustre-OST0000-osc-ffff88012ab10800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=27454: Mon Jul 29 04:45:14 2024 write: IOPS=22, BW=91.6KiB/s (93.8kB/s)(10.0MiB/111755msec) slat (usec): min=11, max=4187, avg=123.28, stdev=273.97 clat (msec): min=173, max=1314, avg=698.07, stdev=163.77 lat (msec): min=173, max=1314, avg=698.19, stdev=163.78 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 439], 10.00th=[ 506], 20.00th=[ 575], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 693], 60.00th=[ 735], | 70.00th=[ 776], 80.00th=[ 827], 90.00th=[ 902], 95.00th=[ 961], | 99.00th=[ 1116], 99.50th=[ 1150], 99.90th=[ 1250], 99.95th=[ 1267], | 99.99th=[ 1318] bw ( KiB/s): min= 15, max= 255, per=24.93%, avg=91.00, stdev=31.78, samples=223 iops : min= 3, max= 63, avg=22.52, stdev= 7.97, samples=223 lat (msec) : 250=1.25%, 500=7.93%, 750=54.41%, 1000=32.89% cpu : usr=0.13%, sys=0.40%, ctx=1329, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27455: Mon Jul 29 04:45:14 2024 write: IOPS=22, BW=91.3KiB/s (93.5kB/s)(10.0MiB/112108msec) slat (usec): min=11, max=5486, avg=120.86, stdev=288.80 clat (msec): min=174, max=1255, avg=700.26, stdev=156.48 lat (msec): min=174, max=1256, avg=700.38, stdev=156.49 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 447], 10.00th=[ 514], 20.00th=[ 575], | 30.00th=[ 625], 40.00th=[ 659], 50.00th=[ 701], 60.00th=[ 735], | 70.00th=[ 776], 80.00th=[ 827], 90.00th=[ 902], 95.00th=[ 969], | 99.00th=[ 1045], 99.50th=[ 1116], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 256, per=25.04%, avg=91.41, stdev=31.16, samples=222 iops : min= 1, max= 64, avg=22.63, stdev= 7.79, samples=222 lat (msec) : 250=1.25%, 500=7.07%, 750=55.08%, 1000=33.87% cpu : usr=0.06%, sys=0.44%, ctx=1261, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27456: Mon Jul 29 04:45:14 2024 write: IOPS=22, BW=91.4KiB/s (93.6kB/s)(10.0MiB/112008msec) slat (usec): min=10, max=4552, avg=125.37, stdev=272.33 clat (msec): min=106, max=1292, avg=699.69, stdev=160.58 lat (msec): min=107, max=1292, avg=699.82, stdev=160.59 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 443], 10.00th=[ 514], 20.00th=[ 575], | 30.00th=[ 625], 40.00th=[ 659], 50.00th=[ 701], 60.00th=[ 735], | 70.00th=[ 776], 80.00th=[ 827], 90.00th=[ 902], 95.00th=[ 961], | 99.00th=[ 1083], 99.50th=[ 1133], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1301] bw ( KiB/s): min= 23, max= 280, per=24.82%, avg=90.58, stdev=32.04, samples=224 iops : min= 5, max= 70, avg=22.41, stdev= 8.06, samples=224 lat (msec) : 250=1.45%, 500=7.46%, 750=54.69%, 1000=33.67% cpu : usr=0.04%, sys=0.48%, ctx=1319, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27457: Mon Jul 29 04:45:14 2024 write: IOPS=22, BW=91.4KiB/s (93.6kB/s)(10.0MiB/112067msec) slat (usec): min=10, max=5291, avg=139.94, stdev=326.12 clat (msec): min=173, max=1204, avg=699.92, stdev=159.84 lat (msec): min=173, max=1204, avg=700.06, stdev=159.85 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 447], 10.00th=[ 514], 20.00th=[ 575], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 693], 60.00th=[ 735], | 70.00th=[ 776], 80.00th=[ 827], 90.00th=[ 902], 95.00th=[ 969], | 99.00th=[ 1099], 99.50th=[ 1133], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 24, max= 256, per=24.83%, avg=90.61, stdev=26.99, samples=224 iops : min= 6, max= 64, avg=22.44, stdev= 6.77, samples=224 lat (msec) : 250=1.25%, 500=7.34%, 750=54.49%, 1000=33.20% cpu : usr=0.06%, sys=0.50%, ctx=1390, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=365KiB/s (374kB/s), 91.3KiB/s-91.6KiB/s (93.5kB/s-93.8kB/s), io=40.0MiB (41.9MB), run=111755-112108msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27556: Mon Jul 29 04:46:13 2024 read: IOPS=21, BW=86.3KiB/s (88.4kB/s)(5048KiB/58501msec) slat (usec): min=1054, max=16873, avg=3685.83, stdev=1670.83 clat (msec): min=11, max=507, avg=245.63, stdev=57.18 lat (msec): min=17, max=509, avg=249.32, stdev=57.47 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 163], 10.00th=[ 180], 20.00th=[ 201], | 30.00th=[ 218], 40.00th=[ 230], 50.00th=[ 243], 60.00th=[ 255], | 70.00th=[ 271], 80.00th=[ 288], 90.00th=[ 313], 95.00th=[ 351], | 99.00th=[ 393], 99.50th=[ 422], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 24, max= 192, per=25.07%, avg=85.73, stdev=29.57, samples=116 iops : min= 6, max= 48, avg=21.18, stdev= 7.42, samples=116 write: IOPS=22, BW=88.8KiB/s (90.9kB/s)(5192KiB/58501msec) slat (usec): min=14, max=3541, avg=122.12, stdev=241.65 clat (msec): min=179, max=869, avg=477.68, stdev=125.17 lat (msec): min=179, max=869, avg=477.80, stdev=125.19 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 439], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 659], 95.00th=[ 693], | 99.00th=[ 776], 99.50th=[ 810], 99.90th=[ 844], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 24, max= 176, per=24.80%, avg=88.28, stdev=29.26, samples=116 iops : min= 6, max= 44, avg=21.82, stdev= 7.38, samples=116 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=27.93%, 500=51.91% lat (msec) : 750=18.98%, 1000=0.90% cpu : usr=0.42%, sys=1.12%, ctx=2600, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27557: Mon Jul 29 04:46:13 2024 read: IOPS=21, BW=84.8KiB/s (86.9kB/s)(4976KiB/58657msec) slat (usec): min=32, max=19181, avg=3738.18, stdev=1806.31 clat (msec): min=3, max=428, avg=242.19, stdev=51.85 lat (msec): min=3, max=436, avg=245.93, stdev=52.21 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 167], 10.00th=[ 184], 20.00th=[ 201], | 30.00th=[ 215], 40.00th=[ 228], 50.00th=[ 239], 60.00th=[ 251], | 70.00th=[ 268], 80.00th=[ 284], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 372], 99.50th=[ 388], 99.90th=[ 409], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 24, max= 151, per=24.58%, avg=84.07, stdev=29.67, samples=117 iops : min= 6, max= 37, avg=20.84, stdev= 7.39, samples=117 write: IOPS=22, BW=89.7KiB/s (91.9kB/s)(5264KiB/58657msec) slat (usec): min=14, max=3754, avg=128.61, stdev=253.42 clat (msec): min=57, max=883, avg=480.25, stdev=131.76 lat (msec): min=57, max=883, avg=480.38, stdev=131.76 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 279], 10.00th=[ 317], 20.00th=[ 368], | 30.00th=[ 405], 40.00th=[ 443], 50.00th=[ 472], 60.00th=[ 506], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 659], 95.00th=[ 701], | 99.00th=[ 785], 99.50th=[ 810], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 23, max= 168, per=25.06%, avg=89.20, stdev=27.59, samples=117 iops : min= 5, max= 42, avg=22.12, stdev= 6.99, samples=117 lat (msec) : 4=0.08%, 10=0.08%, 20=0.04%, 50=0.04%, 100=0.23% lat (msec) : 250=29.34%, 500=48.52%, 750=20.74%, 1000=0.94% cpu : usr=0.17%, sys=1.35%, ctx=2598, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27558: Mon Jul 29 04:46:13 2024 read: IOPS=21, BW=84.0KiB/s (87.0kB/s)(4984KiB/58653msec) slat (usec): min=1015, max=14915, avg=3725.58, stdev=1712.68 clat (msec): min=2, max=428, avg=241.41, stdev=52.04 lat (msec): min=4, max=436, avg=245.14, stdev=52.33 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 167], 10.00th=[ 182], 20.00th=[ 199], | 30.00th=[ 215], 40.00th=[ 228], 50.00th=[ 236], 60.00th=[ 249], | 70.00th=[ 266], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 330], | 99.00th=[ 380], 99.50th=[ 393], 99.90th=[ 422], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 23, max= 168, per=24.75%, avg=84.66, stdev=28.82, samples=117 iops : min= 5, max= 42, avg=20.91, stdev= 7.21, samples=117 write: IOPS=22, BW=89.6KiB/s (91.8kB/s)(5256KiB/58653msec) slat (usec): min=11, max=3373, avg=114.81, stdev=216.21 clat (msec): min=155, max=845, avg=480.97, stdev=127.25 lat (msec): min=155, max=845, avg=481.09, stdev=127.26 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 292], 10.00th=[ 321], 20.00th=[ 368], | 30.00th=[ 405], 40.00th=[ 439], 50.00th=[ 477], 60.00th=[ 510], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 651], 95.00th=[ 709], | 99.00th=[ 785], 99.50th=[ 810], 99.90th=[ 827], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 23, max= 192, per=24.91%, avg=88.69, stdev=27.19, samples=117 iops : min= 5, max= 48, avg=21.92, stdev= 6.87, samples=117 lat (msec) : 4=0.04%, 50=0.12%, 100=0.04%, 250=30.00%, 500=48.12% lat (msec) : 750=20.59%, 1000=1.09% cpu : usr=0.24%, sys=1.29%, ctx=2620, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27559: Mon Jul 29 04:46:13 2024 read: IOPS=21, BW=86.5KiB/s (88.6kB/s)(5060KiB/58500msec) slat (usec): min=1230, max=16894, avg=3736.21, stdev=1707.47 clat (msec): min=3, max=503, avg=242.02, stdev=53.42 lat (msec): min=6, max=507, avg=245.76, stdev=53.71 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 167], 10.00th=[ 184], 20.00th=[ 203], | 30.00th=[ 215], 40.00th=[ 226], 50.00th=[ 236], 60.00th=[ 251], | 70.00th=[ 266], 80.00th=[ 279], 90.00th=[ 313], 95.00th=[ 334], | 99.00th=[ 384], 99.50th=[ 405], 99.90th=[ 498], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 16, max= 160, per=25.26%, avg=86.41, stdev=29.45, samples=116 iops : min= 4, max= 40, avg=21.44, stdev= 7.37, samples=116 write: IOPS=22, BW=88.5KiB/s (90.7kB/s)(5180KiB/58500msec) slat (usec): min=14, max=2899, avg=117.25, stdev=227.76 clat (msec): min=155, max=880, avg=481.84, stdev=127.22 lat (msec): min=155, max=880, avg=481.96, stdev=127.21 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 372], | 30.00th=[ 405], 40.00th=[ 443], 50.00th=[ 477], 60.00th=[ 506], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 651], 95.00th=[ 709], | 99.00th=[ 785], 99.50th=[ 810], 99.90th=[ 844], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 32, max= 184, per=24.72%, avg=87.99, stdev=28.39, samples=116 iops : min= 8, max= 46, avg=21.84, stdev= 7.14, samples=116 lat (msec) : 4=0.04%, 50=0.12%, 100=0.04%, 250=30.04%, 500=48.55% lat (msec) : 750=20.20%, 1000=1.02% cpu : usr=0.23%, sys=1.32%, ctx=2659, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=342KiB/s (350kB/s), 84.8KiB/s-86.5KiB/s (86.9kB/s-88.6kB/s), io=19.6MiB (20.5MB), run=58500-58657msec WRITE: bw=356KiB/s (365kB/s), 88.5KiB/s-89.7KiB/s (90.7kB/s-91.9kB/s), io=20.4MiB (21.4MB), run=58500-58657msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=27610: Mon Jul 29 04:46:15 2024 read: IOPS=3, BW=143MiB/s (150MB/s)(40.0MiB/279msec) slat (nsec): min=12670k, max=12670k, avg=12670392.00, stdev= 0.00 clat (nsec): min=253932k, max=253932k, avg=253932258.00, stdev= 0.00 lat (nsec): min=266624k, max=266624k, avg=266624290.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] lat (msec) : 500=100.00% cpu : usr=0.72%, sys=3.24%, ctx=8, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=40.0MiB (41.9MB), run=279-279msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck