== sanity test 398c: run fio to test AIO ================= 11:45:30 (1736268330) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0845893 s, 496 MB/s osc.lustre-OST0000-osc-ffff8800b5c25000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=20294: Tue Jan 7 11:45:35 2025 write: IOPS=767, BW=3070KiB/s (3144kB/s)(10.0MiB/3335msec) slat (usec): min=12, max=21338, avg=60.85, stdev=422.48 clat (usec): min=3453, max=40113, avg=20603.41, stdev=6102.00 lat (usec): min=4511, max=40149, avg=20665.47, stdev=6108.80 clat percentiles (usec): | 1.00th=[13042], 5.00th=[14746], 10.00th=[15401], 20.00th=[16188], | 30.00th=[16909], 40.00th=[17695], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20841], 80.00th=[23987], 90.00th=[31327], 95.00th=[34866], | 99.00th=[37487], 99.50th=[37487], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 1872, max= 3904, per=25.66%, avg=3147.17, stdev=695.68, samples=6 iops : min= 468, max= 976, avg=786.67, stdev=173.93, samples=6 lat (msec) : 4=0.04%, 10=0.12%, 20=64.26%, 50=35.59% cpu : usr=0.87%, sys=5.31%, ctx=1445, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=20295: Tue Jan 7 11:45:35 2025 write: IOPS=766, BW=3067KiB/s (3140kB/s)(10.0MiB/3339msec) slat (usec): min=12, max=21303, avg=59.47, stdev=421.42 clat (usec): min=3491, max=40351, avg=20631.92, stdev=6059.08 lat (usec): min=11133, max=40615, avg=20692.67, stdev=6064.28 clat percentiles (usec): | 1.00th=[13042], 5.00th=[14746], 10.00th=[15401], 20.00th=[16188], | 30.00th=[16909], 40.00th=[17695], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20841], 80.00th=[23987], 90.00th=[31589], 95.00th=[34866], | 99.00th=[36963], 99.50th=[38011], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 1876, max= 3864, per=25.59%, avg=3139.33, stdev=690.50, samples=6 iops : min= 469, max= 966, avg=784.83, stdev=172.62, samples=6 lat (msec) : 4=0.04%, 20=64.10%, 50=35.86% cpu : usr=0.69%, sys=5.42%, ctx=1353, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=20296: Tue Jan 7 11:45:35 2025 write: IOPS=771, BW=3084KiB/s (3158kB/s)(10.0MiB/3320msec) slat (usec): min=11, max=584, avg=54.43, stdev=38.09 clat (usec): min=1748, max=46300, avg=20664.02, stdev=6197.57 lat (usec): min=1813, max=46471, avg=20719.77, stdev=6212.22 clat percentiles (usec): | 1.00th=[13042], 5.00th=[14746], 10.00th=[15401], 20.00th=[16188], | 30.00th=[16909], 40.00th=[17695], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20841], 80.00th=[24511], 90.00th=[31589], 95.00th=[34866], | 99.00th=[37487], 99.50th=[38011], 99.90th=[43254], 99.95th=[43779], | 99.99th=[46400] bw ( KiB/s): min= 1856, max= 3872, per=25.79%, avg=3163.17, stdev=706.99, samples=6 iops : min= 464, max= 968, avg=790.67, stdev=176.77, samples=6 lat (msec) : 2=0.04%, 4=0.12%, 10=0.08%, 20=64.14%, 50=35.62% cpu : usr=0.84%, sys=4.94%, ctx=1525, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=20297: Tue Jan 7 11:45:35 2025 write: IOPS=766, BW=3067KiB/s (3140kB/s)(10.0MiB/3339msec) slat (usec): min=12, max=16551, avg=57.32, stdev=327.90 clat (usec): min=3446, max=39995, avg=20634.15, stdev=6083.14 lat (usec): min=8060, max=40075, avg=20692.66, stdev=6087.31 clat percentiles (usec): | 1.00th=[12911], 5.00th=[14746], 10.00th=[15401], 20.00th=[16188], | 30.00th=[16909], 40.00th=[17695], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20841], 80.00th=[23987], 90.00th=[31327], 95.00th=[34866], | 99.00th=[37487], 99.50th=[37487], 99.90th=[38536], 99.95th=[38536], | 99.99th=[40109] bw ( KiB/s): min= 1888, max= 3864, per=25.63%, avg=3144.00, stdev=685.91, samples=6 iops : min= 472, max= 966, avg=786.00, stdev=171.48, samples=6 lat (msec) : 4=0.04%, 10=0.04%, 20=65.23%, 50=34.69% cpu : usr=0.72%, sys=4.67%, ctx=1317, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=11.0MiB/s (12.6MB/s), 3067KiB/s-3084KiB/s (3140kB/s-3158kB/s), io=40.0MiB (41.9MB), run=3320-3339msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20309: Tue Jan 7 11:45:39 2025 read: IOPS=523, BW=2093KiB/s (2143kB/s)(5048KiB/2412msec) slat (usec): min=13, max=2103, avg=872.38, stdev=213.51 clat (usec): min=1581, max=33173, avg=14408.81, stdev=3204.75 lat (usec): min=1665, max=34182, avg=15283.01, stdev=3296.32 clat percentiles (usec): | 1.00th=[ 8586], 5.00th=[10159], 10.00th=[10945], 20.00th=[11863], | 30.00th=[12649], 40.00th=[13304], 50.00th=[13960], 60.00th=[14746], | 70.00th=[15664], 80.00th=[16909], 90.00th=[18744], 95.00th=[20317], | 99.00th=[22676], 99.50th=[24249], 99.90th=[32113], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 2064, max= 2424, per=26.33%, avg=2168.00, stdev=171.21, samples=4 iops : min= 516, max= 606, avg=542.00, stdev=42.80, samples=4 write: IOPS=538, BW=2153KiB/s (2204kB/s)(5192KiB/2412msec) slat (usec): min=12, max=1653, avg=41.67, stdev=50.05 clat (usec): min=5355, max=31953, avg=14747.33, stdev=2775.59 lat (usec): min=5369, max=32000, avg=14789.93, stdev=2779.30 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[11076], 10.00th=[11863], 20.00th=[12649], | 30.00th=[13304], 40.00th=[13829], 50.00th=[14353], 60.00th=[14877], | 70.00th=[15533], 80.00th=[16581], 90.00th=[18744], 95.00th=[20055], | 99.00th=[22938], 99.50th=[24249], 99.90th=[30802], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 2048, max= 2584, per=26.39%, avg=2262.00, stdev=235.00, samples=4 iops : min= 512, max= 646, avg=565.50, stdev=58.75, samples=4 lat (msec) : 2=0.08%, 4=0.16%, 10=2.50%, 20=92.11%, 50=5.16% cpu : usr=1.12%, sys=8.21%, ctx=2409, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20310: Tue Jan 7 11:45:39 2025 read: IOPS=511, BW=2044KiB/s (2093kB/s)(4976KiB/2434msec) slat (usec): min=407, max=2239, avg=874.36, stdev=201.61 clat (usec): min=4243, max=40123, avg=14505.29, stdev=3228.89 lat (usec): min=5345, max=41131, avg=15381.44, stdev=3297.65 clat percentiles (usec): | 1.00th=[ 8848], 5.00th=[10421], 10.00th=[11076], 20.00th=[11994], | 30.00th=[12649], 40.00th=[13435], 50.00th=[14091], 60.00th=[14615], | 70.00th=[15533], 80.00th=[16909], 90.00th=[18482], 95.00th=[20055], | 99.00th=[24249], 99.50th=[27132], 99.90th=[34341], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 2024, max= 2392, per=25.75%, avg=2120.00, stdev=181.49, samples=4 iops : min= 506, max= 598, avg=530.00, stdev=45.37, samples=4 write: IOPS=540, BW=2163KiB/s (2215kB/s)(5264KiB/2434msec) slat (usec): min=12, max=264, avg=40.16, stdev=21.26 clat (usec): min=7720, max=34254, avg=14861.86, stdev=2879.46 lat (usec): min=7756, max=34299, avg=14902.97, stdev=2879.74 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11338], 10.00th=[11994], 20.00th=[12780], | 30.00th=[13173], 40.00th=[13829], 50.00th=[14353], 60.00th=[14877], | 70.00th=[15533], 80.00th=[16581], 90.00th=[19006], 95.00th=[20579], | 99.00th=[23462], 99.50th=[25560], 99.90th=[31327], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 2120, max= 2616, per=26.69%, avg=2288.00, stdev=223.43, samples=4 iops : min= 530, max= 654, avg=572.00, stdev=55.86, samples=4 lat (msec) : 10=1.95%, 20=92.34%, 50=5.70% cpu : usr=1.23%, sys=7.93%, ctx=2325, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20311: Tue Jan 7 11:45:39 2025 read: IOPS=512, BW=2048KiB/s (2098kB/s)(4984KiB/2433msec) slat (usec): min=436, max=2182, avg=871.01, stdev=203.12 clat (usec): min=4234, max=40135, avg=14385.87, stdev=3074.17 lat (usec): min=5073, max=41157, avg=15258.66, stdev=3162.14 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[10421], 10.00th=[11076], 20.00th=[11994], | 30.00th=[12518], 40.00th=[13173], 50.00th=[13960], 60.00th=[14615], | 70.00th=[15401], 80.00th=[16581], 90.00th=[18220], 95.00th=[19792], | 99.00th=[23725], 99.50th=[25297], 99.90th=[31589], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 1792, max= 2528, per=25.87%, avg=2130.00, stdev=303.89, samples=4 iops : min= 448, max= 632, avg=532.50, stdev=75.97, samples=4 write: IOPS=540, BW=2160KiB/s (2212kB/s)(5256KiB/2433msec) slat (usec): min=13, max=195, avg=39.70, stdev=18.83 clat (usec): min=3372, max=38397, avg=14934.93, stdev=3014.06 lat (usec): min=3399, max=38426, avg=14975.61, stdev=3014.16 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11338], 10.00th=[11863], 20.00th=[12780], | 30.00th=[13304], 40.00th=[13829], 50.00th=[14353], 60.00th=[14877], | 70.00th=[15664], 80.00th=[16909], 90.00th=[19006], 95.00th=[20317], | 99.00th=[24773], 99.50th=[28181], 99.90th=[35914], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 2088, max= 2472, per=26.50%, avg=2272.00, stdev=170.08, samples=4 iops : min= 522, max= 618, avg=568.00, stdev=42.52, samples=4 lat (msec) : 4=0.04%, 10=1.56%, 20=93.36%, 50=5.04% cpu : usr=0.82%, sys=8.39%, ctx=2341, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20312: Tue Jan 7 11:45:39 2025 read: IOPS=519, BW=2076KiB/s (2126kB/s)(5060KiB/2437msec) slat (usec): min=446, max=2049, avg=873.60, stdev=201.96 clat (usec): min=2540, max=31215, avg=14461.70, stdev=2973.78 lat (usec): min=3532, max=32254, avg=15337.00, stdev=3047.97 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[10421], 10.00th=[11076], 20.00th=[11994], | 30.00th=[12780], 40.00th=[13435], 50.00th=[14091], 60.00th=[14746], | 70.00th=[15533], 80.00th=[16712], 90.00th=[18482], 95.00th=[20055], | 99.00th=[23200], 99.50th=[23725], 99.90th=[30016], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 2003, max= 2440, per=26.19%, avg=2156.75, stdev=193.69, samples=4 iops : min= 500, max= 610, avg=539.00, stdev=48.62, samples=4 write: IOPS=531, BW=2126KiB/s (2177kB/s)(5180KiB/2437msec) slat (usec): min=12, max=578, avg=42.03, stdev=28.06 clat (usec): min=8033, max=32350, avg=14900.96, stdev=2848.04 lat (usec): min=8066, max=32428, avg=14943.94, stdev=2848.33 clat percentiles (usec): | 1.00th=[10028], 5.00th=[11338], 10.00th=[11863], 20.00th=[12649], | 30.00th=[13304], 40.00th=[13829], 50.00th=[14353], 60.00th=[15008], | 70.00th=[15664], 80.00th=[17171], 90.00th=[19006], 95.00th=[20317], | 99.00th=[22938], 99.50th=[25035], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 2088, max= 2568, per=26.02%, avg=2230.75, stdev=225.90, samples=4 iops : min= 522, max= 642, avg=557.50, stdev=56.58, samples=4 lat (msec) : 4=0.04%, 10=1.88%, 20=92.54%, 50=5.55% cpu : usr=0.66%, sys=8.58%, ctx=2415, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8235KiB/s (8432kB/s), 2044KiB/s-2093KiB/s (2093kB/s-2143kB/s), io=19.6MiB (20.5MB), run=2412-2437msec WRITE: bw=8573KiB/s (8779kB/s), 2126KiB/s-2163KiB/s (2177kB/s-2215kB/s), io=20.4MiB (21.4MB), run=2412-2437msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=20318: Tue Jan 7 11:45:40 2025 read: IOPS=7, BW=301MiB/s (315MB/s)(40.0MiB/133msec) slat (nsec): min=15845k, max=15845k, avg=15844609.00, stdev= 0.00 clat (nsec): min=111987k, max=111987k, avg=111987188.00, stdev= 0.00 lat (nsec): min=127860k, max=127860k, avg=127859891.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 112], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=11.36%, ctx=11, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=301MiB/s (315MB/s), 301MiB/s-301MiB/s (315MB/s-315MB/s), io=40.0MiB (41.9MB), run=133-133msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck