== sanity test 398c: run fio to test AIO ================= 12:12:00 (1730826720) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0849761 s, 494 MB/s osc.lustre-OST0000-osc-ffff8800b6123000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=15510: Tue Nov 5 12:12:04 2024 write: IOPS=1014, BW=4057KiB/s (4154kB/s)(10.0MiB/2524msec) slat (usec): min=12, max=291, avg=41.55, stdev=22.82 clat (usec): min=1772, max=28448, avg=15703.15, stdev=1756.72 lat (usec): min=1828, max=28461, avg=15745.80, stdev=1755.76 clat percentiles (usec): | 1.00th=[11863], 5.00th=[13173], 10.00th=[13698], 20.00th=[14484], | 30.00th=[15008], 40.00th=[15270], 50.00th=[15664], 60.00th=[15926], | 70.00th=[16450], 80.00th=[16909], 90.00th=[17695], 95.00th=[18220], | 99.00th=[19268], 99.50th=[22152], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 3864, max= 4312, per=25.14%, avg=4052.80, stdev=163.62, samples=5 iops : min= 966, max= 1078, avg=1013.20, stdev=40.90, samples=5 lat (msec) : 2=0.04%, 4=0.04%, 10=0.12%, 20=98.83%, 50=0.98% cpu : usr=0.67%, sys=5.15%, ctx=1578, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=15511: Tue Nov 5 12:12:04 2024 write: IOPS=1007, BW=4030KiB/s (4127kB/s)(10.0MiB/2541msec) slat (usec): min=12, max=17546, avg=47.56, stdev=346.65 clat (usec): min=2094, max=27440, avg=15673.32, stdev=1608.29 lat (usec): min=4609, max=27462, avg=15721.91, stdev=1585.96 clat percentiles (usec): | 1.00th=[11863], 5.00th=[13042], 10.00th=[13698], 20.00th=[14484], | 30.00th=[15008], 40.00th=[15401], 50.00th=[15664], 60.00th=[15926], | 70.00th=[16319], 80.00th=[16909], 90.00th=[17695], 95.00th=[18220], | 99.00th=[18744], 99.50th=[19006], 99.90th=[25560], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 3736, max= 4312, per=24.98%, avg=4027.20, stdev=206.50, samples=5 iops : min= 934, max= 1078, avg=1006.80, stdev=51.63, samples=5 lat (msec) : 4=0.04%, 10=0.12%, 20=99.61%, 50=0.23% cpu : usr=0.98%, sys=5.35%, ctx=1612, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=15512: Tue Nov 5 12:12:04 2024 write: IOPS=1009, BW=4036KiB/s (4133kB/s)(10.0MiB/2537msec) slat (usec): min=12, max=17585, avg=46.07, stdev=347.42 clat (usec): min=1772, max=28072, avg=15650.18, stdev=1709.41 lat (usec): min=1827, max=28124, avg=15697.32, stdev=1700.21 clat percentiles (usec): | 1.00th=[11731], 5.00th=[13042], 10.00th=[13698], 20.00th=[14484], | 30.00th=[15008], 40.00th=[15270], 50.00th=[15664], 60.00th=[15926], | 70.00th=[16319], 80.00th=[16909], 90.00th=[17695], 95.00th=[18220], | 99.00th=[18744], 99.50th=[19792], 99.90th=[27919], 99.95th=[27919], | 99.99th=[28181] bw ( KiB/s): min= 3752, max= 4280, per=25.03%, avg=4035.20, stdev=188.60, samples=5 iops : min= 938, max= 1070, avg=1008.80, stdev=47.15, samples=5 lat (msec) : 2=0.04%, 4=0.08%, 10=0.12%, 20=99.34%, 50=0.43% cpu : usr=0.67%, sys=5.64%, ctx=1318, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=15513: Tue Nov 5 12:12:04 2024 write: IOPS=1007, BW=4030KiB/s (4127kB/s)(10.0MiB/2541msec) slat (usec): min=11, max=12472, avg=43.20, stdev=246.76 clat (usec): min=4192, max=23879, avg=15675.51, stdev=1523.70 lat (usec): min=10101, max=23920, avg=15719.68, stdev=1506.19 clat percentiles (usec): | 1.00th=[11863], 5.00th=[13042], 10.00th=[13698], 20.00th=[14484], | 30.00th=[15008], 40.00th=[15401], 50.00th=[15664], 60.00th=[15926], | 70.00th=[16319], 80.00th=[16909], 90.00th=[17695], 95.00th=[18220], | 99.00th=[18744], 99.50th=[18744], 99.90th=[20841], 99.95th=[22414], | 99.99th=[23987] bw ( KiB/s): min= 3720, max= 4280, per=24.94%, avg=4020.80, stdev=203.22, samples=5 iops : min= 930, max= 1070, avg=1005.20, stdev=50.81, samples=5 lat (msec) : 10=0.04%, 20=99.84%, 50=0.12% cpu : usr=0.87%, sys=4.49%, ctx=1291, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=15.7MiB/s (16.5MB/s), 4030KiB/s-4057KiB/s (4127kB/s-4154kB/s), io=40.0MiB (41.9MB), run=2524-2541msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15525: Tue Nov 5 12:12:07 2024 read: IOPS=625, BW=2500KiB/s (2560kB/s)(5048KiB/2019msec) slat (usec): min=459, max=1449, avg=755.33, stdev=144.01 clat (usec): min=2096, max=20645, avg=11969.03, stdev=1845.42 lat (usec): min=2905, max=21239, avg=12726.05, stdev=1863.50 clat percentiles (usec): | 1.00th=[ 8029], 5.00th=[ 9110], 10.00th=[ 9765], 20.00th=[10421], | 30.00th=[10945], 40.00th=[11469], 50.00th=[11994], 60.00th=[12387], | 70.00th=[12911], 80.00th=[13435], 90.00th=[14222], 95.00th=[15008], | 99.00th=[16319], 99.50th=[16909], 99.90th=[17957], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 2344, max= 2616, per=25.08%, avg=2490.50, stdev=111.78, samples=4 iops : min= 586, max= 654, avg=622.50, stdev=27.92, samples=4 write: IOPS=642, BW=2572KiB/s (2633kB/s)(5192KiB/2019msec) slat (usec): min=12, max=382, avg=37.56, stdev=22.89 clat (usec): min=1753, max=18005, avg=12341.03, stdev=1492.44 lat (usec): min=1779, max=18037, avg=12379.68, stdev=1491.32 clat percentiles (usec): | 1.00th=[ 8848], 5.00th=[ 9896], 10.00th=[10552], 20.00th=[11076], | 30.00th=[11600], 40.00th=[11994], 50.00th=[12256], 60.00th=[12649], | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353], 95.00th=[14877], | 99.00th=[15926], 99.50th=[16057], 99.90th=[16909], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 2514, max= 2656, per=24.85%, avg=2568.50, stdev=61.37, samples=4 iops : min= 628, max= 664, avg=642.00, stdev=15.49, samples=4 lat (msec) : 2=0.04%, 4=0.08%, 10=9.77%, 20=90.08%, 50=0.04% cpu : usr=1.39%, sys=8.82%, ctx=2229, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15526: Tue Nov 5 12:12:07 2024 read: IOPS=617, BW=2468KiB/s (2527kB/s)(4976KiB/2016msec) slat (usec): min=454, max=1428, avg=748.36, stdev=146.02 clat (usec): min=5631, max=20380, avg=11907.79, stdev=1849.59 lat (usec): min=6741, max=21212, avg=12657.94, stdev=1869.57 clat percentiles (usec): | 1.00th=[ 7898], 5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10421], | 30.00th=[10814], 40.00th=[11338], 50.00th=[11863], 60.00th=[12387], | 70.00th=[12911], 80.00th=[13435], 90.00th=[14222], 95.00th=[15008], | 99.00th=[16450], 99.50th=[17695], 99.90th=[19792], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 2368, max= 2544, per=24.74%, avg=2456.75, stdev=74.81, samples=4 iops : min= 592, max= 636, avg=614.00, stdev=18.62, samples=4 write: IOPS=652, BW=2611KiB/s (2674kB/s)(5264KiB/2016msec) slat (usec): min=12, max=841, avg=37.52, stdev=28.88 clat (usec): min=6382, max=21867, avg=12412.71, stdev=1552.92 lat (usec): min=6408, max=21913, avg=12451.15, stdev=1553.40 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[10028], 10.00th=[10552], 20.00th=[11207], | 30.00th=[11600], 40.00th=[11994], 50.00th=[12387], 60.00th=[12649], | 70.00th=[13173], 80.00th=[13698], 90.00th=[14353], 95.00th=[14877], | 99.00th=[16319], 99.50th=[17433], 99.90th=[21103], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 2568, max= 2712, per=25.29%, avg=2614.50, stdev=66.36, samples=4 iops : min= 642, max= 678, avg=653.50, stdev=16.68, samples=4 lat (msec) : 10=9.69%, 20=90.20%, 50=0.12% cpu : usr=0.55%, sys=9.73%, ctx=2237, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15527: Tue Nov 5 12:12:07 2024 read: IOPS=616, BW=2466KiB/s (2525kB/s)(4984KiB/2021msec) slat (usec): min=466, max=1390, avg=752.29, stdev=142.43 clat (usec): min=6105, max=17935, avg=11959.81, stdev=1741.72 lat (usec): min=6967, max=18836, avg=12713.71, stdev=1755.72 clat percentiles (usec): | 1.00th=[ 7898], 5.00th=[ 9241], 10.00th=[ 9765], 20.00th=[10421], | 30.00th=[10945], 40.00th=[11600], 50.00th=[11994], 60.00th=[12518], | 70.00th=[12911], 80.00th=[13435], 90.00th=[14091], 95.00th=[14746], | 99.00th=[16188], 99.50th=[16909], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 2395, max= 2576, per=24.74%, avg=2456.75, stdev=82.64, samples=4 iops : min= 598, max= 644, avg=614.00, stdev=20.85, samples=4 write: IOPS=650, BW=2601KiB/s (2663kB/s)(5256KiB/2021msec) slat (usec): min=12, max=947, avg=39.06, stdev=38.36 clat (usec): min=5155, max=18221, avg=12396.50, stdev=1511.86 lat (usec): min=5204, max=18351, avg=12436.48, stdev=1509.16 clat percentiles (usec): | 1.00th=[ 8717], 5.00th=[ 9896], 10.00th=[10552], 20.00th=[11207], | 30.00th=[11600], 40.00th=[11994], 50.00th=[12387], 60.00th=[12780], | 70.00th=[13173], 80.00th=[13698], 90.00th=[14222], 95.00th=[14746], | 99.00th=[15795], 99.50th=[16319], 99.90th=[17171], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 2424, max= 2672, per=24.96%, avg=2580.50, stdev=110.54, samples=4 iops : min= 606, max= 668, avg=645.00, stdev=27.54, samples=4 lat (msec) : 10=9.45%, 20=90.55% cpu : usr=1.14%, sys=9.16%, ctx=2212, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15528: Tue Nov 5 12:12:07 2024 read: IOPS=628, BW=2512KiB/s (2573kB/s)(5060KiB/2014msec) slat (usec): min=20, max=1410, avg=754.46, stdev=147.83 clat (usec): min=1508, max=18405, avg=11843.46, stdev=1870.05 lat (usec): min=2208, max=19059, avg=12599.51, stdev=1890.83 clat percentiles (usec): | 1.00th=[ 7373], 5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10290], | 30.00th=[10814], 40.00th=[11338], 50.00th=[11863], 60.00th=[12256], | 70.00th=[12780], 80.00th=[13304], 90.00th=[14222], 95.00th=[14877], | 99.00th=[16712], 99.50th=[17171], 99.90th=[18220], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 2384, max= 2576, per=25.28%, avg=2510.50, stdev=87.32, samples=4 iops : min= 596, max= 644, avg=627.50, stdev=21.81, samples=4 write: IOPS=642, BW=2572KiB/s (2634kB/s)(5180KiB/2014msec) slat (usec): min=12, max=647, avg=37.42, stdev=29.23 clat (usec): min=2655, max=20800, avg=12467.59, stdev=1612.07 lat (usec): min=2702, max=20818, avg=12506.06, stdev=1609.95 clat percentiles (usec): | 1.00th=[ 8455], 5.00th=[10159], 10.00th=[10552], 20.00th=[11207], | 30.00th=[11600], 40.00th=[11994], 50.00th=[12387], 60.00th=[12780], | 70.00th=[13304], 80.00th=[13829], 90.00th=[14484], 95.00th=[15008], | 99.00th=[15926], 99.50th=[16450], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 2530, max= 2600, per=24.77%, avg=2560.50, stdev=33.32, samples=4 iops : min= 632, max= 650, avg=640.00, stdev= 8.49, samples=4 lat (msec) : 2=0.04%, 4=0.20%, 10=9.49%, 20=90.16%, 50=0.12% cpu : usr=1.04%, sys=9.14%, ctx=2299, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9930KiB/s (10.2MB/s), 2466KiB/s-2512KiB/s (2525kB/s-2573kB/s), io=19.6MiB (20.5MB), run=2014-2021msec WRITE: bw=10.1MiB/s (10.6MB/s), 2572KiB/s-2611KiB/s (2633kB/s-2674kB/s), io=20.4MiB (21.4MB), run=2014-2021msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=15533: Tue Nov 5 12:12:09 2024 read: IOPS=6, BW=267MiB/s (280MB/s)(40.0MiB/150msec) slat (nsec): min=10465k, max=10465k, avg=10465214.00, stdev= 0.00 clat (nsec): min=135145k, max=135145k, avg=135145467.00, stdev= 0.00 lat (nsec): min=145620k, max=145620k, avg=145620034.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.71%, ctx=9, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=40.0MiB (41.9MB), run=150-150msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck