== sanity test 398c: run fio to test AIO ================= 11:32:12 (1736267532) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.129591 s, 324 MB/s osc.lustre-OST0000-osc-ffff88008b2bc000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=21974: Tue Jan 7 11:32:38 2025 write: IOPS=103, BW=414KiB/s (424kB/s)(10.0MiB/24715msec) slat (usec): min=12, max=315, avg=66.06, stdev=32.98 clat (msec): min=34, max=252, avg=154.29, stdev=27.17 lat (msec): min=34, max=252, avg=154.36, stdev=27.17 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 109], 10.00th=[ 123], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 148], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 167], 80.00th=[ 176], 90.00th=[ 188], 95.00th=[ 199], | 99.00th=[ 218], 99.50th=[ 226], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 312, max= 640, per=24.99%, avg=414.02, stdev=53.03, samples=49 iops : min= 78, max= 160, avg=103.49, stdev=13.24, samples=49 lat (msec) : 50=0.08%, 100=3.63%, 250=96.13%, 500=0.16% cpu : usr=0.22%, sys=0.80%, ctx=2239, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21975: Tue Jan 7 11:32:38 2025 write: IOPS=103, BW=414KiB/s (424kB/s)(10.0MiB/24715msec) slat (usec): min=12, max=341, avg=63.59, stdev=34.28 clat (msec): min=34, max=236, avg=154.32, stdev=26.83 lat (msec): min=35, max=236, avg=154.39, stdev=26.83 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 169], 80.00th=[ 176], 90.00th=[ 188], 95.00th=[ 199], | 99.00th=[ 211], 99.50th=[ 215], 99.90th=[ 226], 99.95th=[ 232], | 99.99th=[ 236] bw ( KiB/s): min= 304, max= 640, per=24.92%, avg=412.94, stdev=53.45, samples=49 iops : min= 76, max= 160, avg=103.14, stdev=13.36, samples=49 lat (msec) : 50=0.04%, 100=3.40%, 250=96.56% cpu : usr=0.13%, sys=0.83%, ctx=2082, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21976: Tue Jan 7 11:32:38 2025 write: IOPS=103, BW=414KiB/s (424kB/s)(10.0MiB/24715msec) slat (usec): min=11, max=337, avg=64.33, stdev=34.85 clat (msec): min=59, max=233, avg=154.31, stdev=26.75 lat (msec): min=59, max=233, avg=154.37, stdev=26.75 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 110], 10.00th=[ 123], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 169], 80.00th=[ 178], 90.00th=[ 188], 95.00th=[ 197], | 99.00th=[ 213], 99.50th=[ 218], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 234] bw ( KiB/s): min= 312, max= 640, per=24.92%, avg=412.92, stdev=55.01, samples=49 iops : min= 78, max= 160, avg=103.12, stdev=13.74, samples=49 lat (msec) : 100=3.79%, 250=96.21% cpu : usr=0.16%, sys=0.80%, ctx=2301, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21977: Tue Jan 7 11:32:38 2025 write: IOPS=103, BW=415KiB/s (425kB/s)(10.0MiB/24689msec) slat (usec): min=12, max=297, avg=66.22, stdev=34.55 clat (msec): min=14, max=310, avg=154.14, stdev=29.01 lat (msec): min=14, max=310, avg=154.21, stdev=29.01 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 105], 10.00th=[ 120], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 169], 80.00th=[ 178], 90.00th=[ 188], 95.00th=[ 199], | 99.00th=[ 215], 99.50th=[ 224], 99.90th=[ 292], 99.95th=[ 300], | 99.99th=[ 309] bw ( KiB/s): min= 336, max= 688, per=25.01%, avg=414.39, stdev=57.54, samples=49 iops : min= 84, max= 172, avg=103.49, stdev=14.38, samples=49 lat (msec) : 20=0.04%, 50=0.12%, 100=4.02%, 250=95.51%, 500=0.31% cpu : usr=0.18%, sys=0.80%, ctx=2428, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1657KiB/s (1697kB/s), 414KiB/s-415KiB/s (424kB/s-425kB/s), io=40.0MiB (41.9MB), run=24689-24715msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22007: Tue Jan 7 11:32:53 2025 read: IOPS=91, BW=368KiB/s (377kB/s)(5048KiB/13720msec) slat (usec): min=448, max=2990, avg=823.20, stdev=213.49 clat (msec): min=9, max=125, avg=57.55, stdev=10.17 lat (msec): min=10, max=125, avg=58.38, stdev=10.17 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 66], 90.00th=[ 70], 95.00th=[ 74], | 99.00th=[ 81], 99.50th=[ 90], 99.90th=[ 96], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 264, max= 512, per=25.09%, avg=366.52, stdev=53.71, samples=27 iops : min= 66, max= 128, avg=91.63, stdev=13.43, samples=27 write: IOPS=94, BW=378KiB/s (388kB/s)(5192KiB/13720msec) slat (usec): min=13, max=852, avg=56.31, stdev=36.38 clat (msec): min=24, max=177, avg=112.12, stdev=24.51 lat (msec): min=24, max=177, avg=112.18, stdev=24.51 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 75], 10.00th=[ 83], 20.00th=[ 92], | 30.00th=[ 99], 40.00th=[ 105], 50.00th=[ 110], 60.00th=[ 116], | 70.00th=[ 124], 80.00th=[ 134], 90.00th=[ 146], 95.00th=[ 157], | 99.00th=[ 169], 99.50th=[ 174], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 328, max= 448, per=24.80%, avg=377.19, stdev=35.48, samples=27 iops : min= 82, max= 112, avg=94.30, stdev= 8.87, samples=27 lat (msec) : 10=0.08%, 20=0.20%, 50=10.51%, 100=54.88%, 250=34.34% cpu : usr=0.28%, sys=1.80%, ctx=3205, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22008: Tue Jan 7 11:32:53 2025 read: IOPS=90, BW=363KiB/s (372kB/s)(4976KiB/13704msec) slat (usec): min=477, max=2409, avg=816.12, stdev=191.31 clat (usec): min=878, max=109322, avg=56979.77, stdev=10123.75 lat (msec): min=2, max=109, avg=57.80, stdev=10.13 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 57], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 70], 95.00th=[ 73], | 99.00th=[ 80], 99.50th=[ 89], 99.90th=[ 91], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 224, max= 464, per=24.75%, avg=361.67, stdev=61.31, samples=27 iops : min= 56, max= 116, avg=90.33, stdev=15.36, samples=27 write: IOPS=96, BW=384KiB/s (393kB/s)(5264KiB/13704msec) slat (usec): min=12, max=506, avg=55.71, stdev=30.25 clat (msec): min=33, max=182, avg=111.77, stdev=25.05 lat (msec): min=34, max=182, avg=111.83, stdev=25.05 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 91], | 30.00th=[ 97], 40.00th=[ 105], 50.00th=[ 111], 60.00th=[ 116], | 70.00th=[ 124], 80.00th=[ 133], 90.00th=[ 148], 95.00th=[ 159], | 99.00th=[ 171], 99.50th=[ 174], 99.90th=[ 178], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 319, max= 456, per=25.10%, avg=381.81, stdev=34.14, samples=27 iops : min= 79, max= 114, avg=95.37, stdev= 8.55, samples=27 lat (usec) : 1000=0.04% lat (msec) : 2=0.04%, 10=0.04%, 20=0.16%, 50=11.37%, 100=53.79% lat (msec) : 250=34.57% cpu : usr=0.31%, sys=1.77%, ctx=3191, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22009: Tue Jan 7 11:32:53 2025 read: IOPS=90, BW=363KiB/s (372kB/s)(4984KiB/13729msec) slat (usec): min=439, max=2940, avg=818.42, stdev=192.80 clat (usec): min=560, max=89735, avg=57172.46, stdev=10056.08 lat (usec): min=1432, max=90932, avg=57992.83, stdev=10056.12 clat percentiles (usec): | 1.00th=[32113], 5.00th=[42730], 10.00th=[45876], 20.00th=[49546], | 30.00th=[52167], 40.00th=[54264], 50.00th=[56886], 60.00th=[59507], | 70.00th=[62129], 80.00th=[65274], 90.00th=[69731], 95.00th=[72877], | 99.00th=[80217], 99.50th=[82314], 99.90th=[88605], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 256, max= 496, per=24.75%, avg=361.67, stdev=65.27, samples=27 iops : min= 64, max= 124, avg=90.33, stdev=16.29, samples=27 write: IOPS=95, BW=383KiB/s (392kB/s)(5256KiB/13729msec) slat (usec): min=12, max=1498, avg=57.27, stdev=51.38 clat (msec): min=24, max=182, avg=111.98, stdev=25.53 lat (msec): min=24, max=182, avg=112.04, stdev=25.53 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 72], 10.00th=[ 81], 20.00th=[ 91], | 30.00th=[ 100], 40.00th=[ 105], 50.00th=[ 111], 60.00th=[ 117], | 70.00th=[ 124], 80.00th=[ 134], 90.00th=[ 148], 95.00th=[ 159], | 99.00th=[ 171], 99.50th=[ 176], 99.90th=[ 180], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 312, max= 440, per=24.83%, avg=377.67, stdev=36.07, samples=27 iops : min= 78, max= 110, avg=94.33, stdev= 9.07, samples=27 lat (usec) : 750=0.04% lat (msec) : 4=0.04%, 10=0.04%, 20=0.16%, 50=11.13%, 100=53.95% lat (msec) : 250=34.65% cpu : usr=0.36%, sys=1.69%, ctx=3310, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22010: Tue Jan 7 11:32:53 2025 read: IOPS=92, BW=371KiB/s (380kB/s)(5060KiB/13646msec) slat (usec): min=31, max=3115, avg=810.20, stdev=222.49 clat (usec): min=968, max=108472, avg=57136.00, stdev=10783.54 lat (usec): min=1001, max=108976, avg=57948.27, stdev=10805.45 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 66], 90.00th=[ 70], 95.00th=[ 73], | 99.00th=[ 81], 99.50th=[ 83], 99.90th=[ 99], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 280, max= 472, per=25.49%, avg=372.37, stdev=58.94, samples=27 iops : min= 70, max= 118, avg=93.04, stdev=14.75, samples=27 write: IOPS=94, BW=380KiB/s (389kB/s)(5180KiB/13646msec) slat (usec): min=12, max=338, avg=57.23, stdev=30.50 clat (msec): min=13, max=185, avg=111.83, stdev=25.80 lat (msec): min=13, max=185, avg=111.89, stdev=25.80 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 73], 10.00th=[ 81], 20.00th=[ 92], | 30.00th=[ 99], 40.00th=[ 105], 50.00th=[ 110], 60.00th=[ 116], | 70.00th=[ 124], 80.00th=[ 132], 90.00th=[ 146], 95.00th=[ 159], | 99.00th=[ 178], 99.50th=[ 182], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 320, max= 504, per=24.85%, avg=378.00, stdev=38.49, samples=27 iops : min= 80, max= 126, avg=94.44, stdev= 9.64, samples=27 lat (usec) : 1000=0.04% lat (msec) : 2=0.12%, 4=0.08%, 10=0.04%, 20=0.27%, 50=10.23% lat (msec) : 100=55.08%, 250=34.14% cpu : usr=0.32%, sys=1.77%, ctx=3318, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1462KiB/s (1497kB/s), 363KiB/s-371KiB/s (372kB/s-380kB/s), io=19.6MiB (20.5MB), run=13646-13729msec WRITE: bw=1522KiB/s (1558kB/s), 378KiB/s-384KiB/s (388kB/s-393kB/s), io=20.4MiB (21.4MB), run=13646-13729msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=22025: Tue Jan 7 11:32:54 2025 read: IOPS=12, BW=513MiB/s (538MB/s)(40.0MiB/78msec) slat (nsec): min=7715.2k, max=7715.2k, avg=7715185.00, stdev= 0.00 clat (nsec): min=66876k, max=66876k, avg=66875640.00, stdev= 0.00 lat (nsec): min=74610k, max=74610k, avg=74609777.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[66847], 5.00th=[66847], 10.00th=[66847], 20.00th=[66847], | 30.00th=[66847], 40.00th=[66847], 50.00th=[66847], 60.00th=[66847], | 70.00th=[66847], 80.00th=[66847], 90.00th=[66847], 95.00th=[66847], | 99.00th=[66847], 99.50th=[66847], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=10.39%, ctx=14, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=513MiB/s (538MB/s), 513MiB/s-513MiB/s (538MB/s-538MB/s), io=40.0MiB (41.9MB), run=78-78msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck