== sanity test 398c: run fio to test AIO ================= 12:02:18 (1730826138) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.149491 s, 281 MB/s osc.lustre-OST0000-osc-ffff88009beb6800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=20325: Tue Nov 5 12:02:51 2024 write: IOPS=81, BW=324KiB/s (332kB/s)(10.0MiB/31601msec) slat (usec): min=12, max=561, avg=69.25, stdev=45.02 clat (msec): min=24, max=523, avg=197.29, stdev=42.10 lat (msec): min=24, max=523, avg=197.36, stdev=42.10 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 127], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 220], 80.00th=[ 230], 90.00th=[ 243], 95.00th=[ 253], | 99.00th=[ 284], 99.50th=[ 313], 99.90th=[ 510], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 168, max= 560, per=25.01%, avg=323.10, stdev=59.74, samples=63 iops : min= 42, max= 140, avg=80.71, stdev=14.91, samples=63 lat (msec) : 50=0.12%, 100=2.38%, 250=91.80%, 500=5.55%, 750=0.16% cpu : usr=0.11%, sys=0.76%, ctx=1870, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=20326: Tue Nov 5 12:02:51 2024 write: IOPS=80, BW=323KiB/s (331kB/s)(10.0MiB/31671msec) slat (usec): min=12, max=467, avg=69.75, stdev=44.98 clat (msec): min=69, max=468, avg=197.71, stdev=40.65 lat (msec): min=69, max=468, avg=197.79, stdev=40.65 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 209], | 70.00th=[ 220], 80.00th=[ 232], 90.00th=[ 243], 95.00th=[ 255], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 200, max= 640, per=25.01%, avg=323.17, stdev=62.82, samples=63 iops : min= 50, max= 160, avg=80.79, stdev=15.71, samples=63 lat (msec) : 100=1.95%, 250=91.72%, 500=6.33% cpu : usr=0.11%, sys=0.72%, ctx=2078, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=20327: Tue Nov 5 12:02:51 2024 write: IOPS=80, BW=323KiB/s (331kB/s)(10.0MiB/31686msec) slat (usec): min=12, max=496, avg=70.42, stdev=47.33 clat (msec): min=71, max=425, avg=197.85, stdev=38.81 lat (msec): min=71, max=425, avg=197.93, stdev=38.81 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 220], 80.00th=[ 232], 90.00th=[ 243], 95.00th=[ 253], | 99.00th=[ 284], 99.50th=[ 288], 99.90th=[ 380], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 224, max= 640, per=24.99%, avg=322.92, stdev=61.03, samples=63 iops : min= 56, max= 160, avg=80.73, stdev=15.26, samples=63 lat (msec) : 100=1.95%, 250=91.72%, 500=6.33% cpu : usr=0.11%, sys=0.72%, ctx=1926, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=20328: Tue Nov 5 12:02:51 2024 write: IOPS=80, BW=323KiB/s (331kB/s)(10.0MiB/31686msec) slat (usec): min=12, max=475, avg=71.05, stdev=48.03 clat (msec): min=73, max=411, avg=197.89, stdev=38.73 lat (msec): min=73, max=411, avg=197.96, stdev=38.73 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 134], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 220], 80.00th=[ 230], 90.00th=[ 243], 95.00th=[ 255], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 414] bw ( KiB/s): min= 256, max= 640, per=24.99%, avg=322.87, stdev=60.65, samples=63 iops : min= 64, max= 160, avg=80.68, stdev=15.17, samples=63 lat (msec) : 100=1.95%, 250=91.17%, 500=6.88% cpu : usr=0.12%, sys=0.71%, ctx=2274, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1293KiB/s (1324kB/s), 323KiB/s-324KiB/s (331kB/s-332kB/s), io=40.0MiB (41.9MB), run=31601-31686msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20364: Tue Nov 5 12:03:10 2024 read: IOPS=72, BW=288KiB/s (295kB/s)(5048KiB/17511msec) slat (usec): min=467, max=2838, avg=1011.11, stdev=292.54 clat (usec): min=1214, max=160401, avg=73890.87, stdev=18737.24 lat (msec): min=2, max=161, avg=74.90, stdev=18.84 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 60], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 82], 80.00th=[ 88], 90.00th=[ 99], 95.00th=[ 107], | 99.00th=[ 140], 99.50th=[ 142], 99.90th=[ 150], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 152, max= 408, per=25.11%, avg=287.06, stdev=71.58, samples=35 iops : min= 38, max= 102, avg=71.74, stdev=17.88, samples=35 write: IOPS=74, BW=296KiB/s (304kB/s)(5192KiB/17511msec) slat (usec): min=12, max=738, avg=62.99, stdev=40.68 clat (msec): min=63, max=270, avg=142.64, stdev=36.82 lat (msec): min=63, max=270, avg=142.70, stdev=36.83 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 93], 10.00th=[ 102], 20.00th=[ 110], | 30.00th=[ 120], 40.00th=[ 127], 50.00th=[ 136], 60.00th=[ 146], | 70.00th=[ 159], 80.00th=[ 176], 90.00th=[ 194], 95.00th=[ 209], | 99.00th=[ 241], 99.50th=[ 257], 99.90th=[ 268], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 192, max= 391, per=24.76%, avg=294.60, stdev=50.48, samples=35 iops : min= 48, max= 97, avg=73.63, stdev=12.58, samples=35 lat (msec) : 2=0.04%, 10=0.04%, 20=0.16%, 50=2.54%, 100=46.52% lat (msec) : 250=50.27%, 500=0.43% cpu : usr=0.31%, sys=1.69%, ctx=3138, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20365: Tue Nov 5 12:03:10 2024 read: IOPS=70, BW=284KiB/s (290kB/s)(4976KiB/17543msec) slat (usec): min=436, max=3839, avg=991.07, stdev=297.13 clat (usec): min=1066, max=157333, avg=73298.30, stdev=19075.96 lat (usec): min=1947, max=159053, avg=74291.97, stdev=19184.90 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 48], 10.00th=[ 54], 20.00th=[ 59], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 87], 90.00th=[ 97], 95.00th=[ 106], | 99.00th=[ 136], 99.50th=[ 142], 99.90th=[ 157], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 152, max= 440, per=24.57%, avg=280.86, stdev=70.00, samples=35 iops : min= 38, max= 110, avg=70.17, stdev=17.52, samples=35 write: IOPS=75, BW=300KiB/s (307kB/s)(5264KiB/17543msec) slat (usec): min=13, max=721, avg=64.17, stdev=40.54 clat (msec): min=20, max=272, avg=142.79, stdev=37.73 lat (msec): min=20, max=272, avg=142.85, stdev=37.74 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 92], 10.00th=[ 101], 20.00th=[ 110], | 30.00th=[ 118], 40.00th=[ 128], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 161], 80.00th=[ 176], 90.00th=[ 197], 95.00th=[ 211], | 99.00th=[ 243], 99.50th=[ 253], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 207, max= 408, per=25.00%, avg=297.54, stdev=53.63, samples=35 iops : min= 51, max= 102, avg=74.34, stdev=13.42, samples=35 lat (msec) : 2=0.08%, 10=0.12%, 20=0.20%, 50=3.12%, 100=45.90% lat (msec) : 250=50.12%, 500=0.47% cpu : usr=0.25%, sys=1.73%, ctx=3183, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20366: Tue Nov 5 12:03:10 2024 read: IOPS=71, BW=285KiB/s (292kB/s)(4984KiB/17496msec) slat (usec): min=23, max=2977, avg=1003.79, stdev=307.59 clat (usec): min=1051, max=205800, avg=73660.23, stdev=19745.90 lat (msec): min=2, max=206, avg=74.67, stdev=19.86 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 49], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 88], 90.00th=[ 97], 95.00th=[ 108], | 99.00th=[ 140], 99.50th=[ 142], 99.90th=[ 182], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 192, max= 400, per=25.13%, avg=287.24, stdev=53.77, samples=34 iops : min= 48, max= 100, avg=71.76, stdev=13.50, samples=34 write: IOPS=75, BW=300KiB/s (308kB/s)(5256KiB/17496msec) slat (usec): min=13, max=471, avg=63.71, stdev=37.66 clat (msec): min=27, max=275, avg=141.98, stdev=38.90 lat (msec): min=27, max=275, avg=142.05, stdev=38.91 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 109], | 30.00th=[ 118], 40.00th=[ 128], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 176], 90.00th=[ 194], 95.00th=[ 211], | 99.00th=[ 251], 99.50th=[ 266], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 176, max= 424, per=25.15%, avg=299.24, stdev=61.49, samples=34 iops : min= 44, max= 106, avg=74.76, stdev=15.39, samples=34 lat (msec) : 2=0.08%, 4=0.08%, 10=0.20%, 20=0.12%, 50=3.01% lat (msec) : 100=47.46%, 250=48.48%, 500=0.59% cpu : usr=0.27%, sys=1.79%, ctx=2973, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20367: Tue Nov 5 12:03:10 2024 read: IOPS=72, BW=289KiB/s (296kB/s)(5060KiB/17522msec) slat (usec): min=451, max=2969, avg=999.25, stdev=301.83 clat (msec): min=9, max=158, avg=73.96, stdev=19.15 lat (msec): min=10, max=160, avg=74.96, stdev=19.25 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 49], 10.00th=[ 54], 20.00th=[ 59], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 88], 90.00th=[ 99], 95.00th=[ 107], | 99.00th=[ 140], 99.50th=[ 150], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 176, max= 440, per=25.17%, avg=287.71, stdev=65.18, samples=35 iops : min= 44, max= 110, avg=71.89, stdev=16.30, samples=35 write: IOPS=73, BW=296KiB/s (303kB/s)(5180KiB/17522msec) slat (usec): min=12, max=809, avg=61.00, stdev=40.25 clat (msec): min=60, max=272, avg=142.94, stdev=36.88 lat (msec): min=60, max=272, avg=143.00, stdev=36.88 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 111], | 30.00th=[ 120], 40.00th=[ 128], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 161], 80.00th=[ 176], 90.00th=[ 194], 95.00th=[ 209], | 99.00th=[ 253], 99.50th=[ 264], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 200, max= 400, per=24.70%, avg=293.89, stdev=55.71, samples=35 iops : min= 50, max= 100, avg=73.43, stdev=13.95, samples=35 lat (msec) : 10=0.04%, 20=0.27%, 50=2.89%, 100=46.09%, 250=50.16% lat (msec) : 500=0.55% cpu : usr=0.25%, sys=1.71%, ctx=3138, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1144KiB/s (1171kB/s), 284KiB/s-289KiB/s (290kB/s-296kB/s), io=19.6MiB (20.5MB), run=17496-17543msec WRITE: bw=1191KiB/s (1219kB/s), 296KiB/s-300KiB/s (303kB/s-308kB/s), io=20.4MiB (21.4MB), run=17496-17543msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=20384: Tue Nov 5 12:03:11 2024 read: IOPS=10, BW=426MiB/s (446MB/s)(40.0MiB/94msec) slat (nsec): min=7940.2k, max=7940.2k, avg=7940152.00, stdev= 0.00 clat (nsec): min=82412k, max=82412k, avg=82411859.00, stdev= 0.00 lat (nsec): min=90362k, max=90362k, avg=90362463.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[82314], 5.00th=[82314], 10.00th=[82314], 20.00th=[82314], | 30.00th=[82314], 40.00th=[82314], 50.00th=[82314], 60.00th=[82314], | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.68%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=426MiB/s (446MB/s), 426MiB/s-426MiB/s (446MB/s-446MB/s), io=40.0MiB (41.9MB), run=94-94msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck