== sanity test 398c: run fio to test AIO ================= 09:59:44 (1747403984) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0722934 s, 580 MB/s osc.lustre-OST0000-osc-ffff8800b60ac000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=24097: Fri May 16 10:00:11 2025 write: IOPS=97, BW=389KiB/s (398kB/s)(10.0MiB/26338msec) slat (usec): min=13, max=644, avg=80.89, stdev=56.47 clat (msec): min=62, max=289, avg=164.46, stdev=38.36 lat (msec): min=62, max=289, avg=164.54, stdev=38.37 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 132], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 197], 90.00th=[ 215], 95.00th=[ 232], | 99.00th=[ 259], 99.50th=[ 271], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 264, max= 640, per=24.86%, avg=386.29, stdev=81.18, samples=52 iops : min= 66, max= 160, avg=96.46, stdev=20.25, samples=52 lat (msec) : 100=4.34%, 250=93.67%, 500=1.99% cpu : usr=0.15%, sys=0.93%, ctx=2303, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24098: Fri May 16 10:00:11 2025 write: IOPS=97, BW=390KiB/s (399kB/s)(10.0MiB/26263msec) slat (usec): min=12, max=794, avg=82.50, stdev=59.12 clat (msec): min=8, max=290, avg=164.00, stdev=39.92 lat (msec): min=9, max=290, avg=164.08, stdev=39.94 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 101], 10.00th=[ 116], 20.00th=[ 132], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 215], 95.00th=[ 232], | 99.00th=[ 259], 99.50th=[ 268], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 292] bw ( KiB/s): min= 256, max= 816, per=25.01%, avg=388.62, stdev=89.14, samples=52 iops : min= 64, max= 204, avg=97.06, stdev=22.26, samples=52 lat (msec) : 10=0.04%, 20=0.08%, 50=0.31%, 100=4.53%, 250=93.12% lat (msec) : 500=1.91% cpu : usr=0.18%, sys=0.95%, ctx=2308, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24099: Fri May 16 10:00:11 2025 write: IOPS=97, BW=389KiB/s (398kB/s)(10.0MiB/26348msec) slat (usec): min=13, max=1276, avg=80.46, stdev=58.39 clat (msec): min=62, max=292, avg=164.51, stdev=38.66 lat (msec): min=62, max=292, avg=164.59, stdev=38.68 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 106], 10.00th=[ 115], 20.00th=[ 133], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 215], 95.00th=[ 232], | 99.00th=[ 266], 99.50th=[ 279], 99.90th=[ 288], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 256, max= 640, per=24.89%, avg=386.75, stdev=79.38, samples=52 iops : min= 64, max= 160, avg=96.58, stdev=19.81, samples=52 lat (msec) : 100=4.26%, 250=93.20%, 500=2.54% cpu : usr=0.16%, sys=0.92%, ctx=2286, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24100: Fri May 16 10:00:11 2025 write: IOPS=97, BW=389KiB/s (398kB/s)(10.0MiB/26348msec) slat (usec): min=13, max=1331, avg=84.22, stdev=67.28 clat (msec): min=62, max=292, avg=164.51, stdev=38.55 lat (msec): min=62, max=292, avg=164.60, stdev=38.57 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 104], 10.00th=[ 114], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 213], 95.00th=[ 230], | 99.00th=[ 268], 99.50th=[ 271], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 292] bw ( KiB/s): min= 264, max= 640, per=24.87%, avg=386.46, stdev=77.07, samples=52 iops : min= 66, max= 160, avg=96.52, stdev=19.23, samples=52 lat (msec) : 100=4.65%, 250=93.05%, 500=2.30% cpu : usr=0.20%, sys=0.88%, ctx=2539, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1555KiB/s (1592kB/s), 389KiB/s-390KiB/s (398kB/s-399kB/s), io=40.0MiB (41.9MB), run=26263-26348msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24131: Fri May 16 10:00:29 2025 read: IOPS=76, BW=304KiB/s (311kB/s)(5048KiB/16596msec) slat (usec): min=477, max=3805, avg=977.65, stdev=310.95 clat (msec): min=3, max=149, avg=69.37, stdev=14.99 lat (msec): min=3, max=150, avg=70.35, stdev=15.10 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 87], 95.00th=[ 94], | 99.00th=[ 118], 99.50th=[ 126], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 168, max= 424, per=25.18%, avg=304.39, stdev=64.45, samples=33 iops : min= 42, max= 106, avg=76.03, stdev=16.12, samples=33 write: IOPS=78, BW=313KiB/s (320kB/s)(5192KiB/16596msec) slat (usec): min=13, max=788, avg=64.30, stdev=43.86 clat (msec): min=57, max=240, avg=135.79, stdev=31.12 lat (msec): min=57, max=241, avg=135.86, stdev=31.12 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 88], 10.00th=[ 99], 20.00th=[ 111], | 30.00th=[ 120], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 178], 95.00th=[ 190], | 99.00th=[ 224], 99.50th=[ 232], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 231, max= 392, per=24.83%, avg=312.39, stdev=44.29, samples=33 iops : min= 57, max= 98, avg=78.03, stdev=11.09, samples=33 lat (msec) : 4=0.04%, 20=0.16%, 50=2.77%, 100=50.51%, 250=46.52% cpu : usr=0.27%, sys=1.71%, ctx=2995, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24132: Fri May 16 10:00:29 2025 read: IOPS=74, BW=300KiB/s (307kB/s)(4976KiB/16596msec) slat (usec): min=480, max=2701, avg=988.79, stdev=312.73 clat (usec): min=635, max=126111, avg=68952.29, stdev=14814.30 lat (usec): min=1449, max=127473, avg=69943.38, stdev=14933.40 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 49], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 77], 80.00th=[ 81], 90.00th=[ 89], 95.00th=[ 94], | 99.00th=[ 106], 99.50th=[ 117], 99.90th=[ 122], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 200, max= 416, per=24.66%, avg=298.15, stdev=60.92, samples=33 iops : min= 50, max= 104, avg=74.52, stdev=15.25, samples=33 write: IOPS=79, BW=317KiB/s (325kB/s)(5264KiB/16596msec) slat (usec): min=13, max=318, avg=64.35, stdev=36.46 clat (msec): min=10, max=240, avg=135.36, stdev=30.18 lat (msec): min=10, max=240, avg=135.42, stdev=30.18 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 88], 10.00th=[ 100], 20.00th=[ 110], | 30.00th=[ 118], 40.00th=[ 127], 50.00th=[ 136], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 218], 99.50th=[ 224], 99.90th=[ 234], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 224, max= 384, per=25.09%, avg=315.61, stdev=44.45, samples=33 iops : min= 56, max= 96, avg=78.88, stdev=11.14, samples=33 lat (usec) : 750=0.04% lat (msec) : 2=0.08%, 4=0.04%, 20=0.20%, 50=2.85%, 100=50.08% lat (msec) : 250=46.72% cpu : usr=0.24%, sys=1.72%, ctx=3201, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24133: Fri May 16 10:00:29 2025 read: IOPS=75, BW=300KiB/s (308kB/s)(4984KiB/16596msec) slat (usec): min=18, max=5011, avg=980.54, stdev=336.42 clat (usec): min=1095, max=149145, avg=68149.06, stdev=14949.00 lat (usec): min=1975, max=149945, avg=69131.93, stdev=15089.94 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 86], 95.00th=[ 94], | 99.00th=[ 106], 99.50th=[ 121], 99.90th=[ 148], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 184, max= 464, per=24.90%, avg=301.06, stdev=72.83, samples=33 iops : min= 46, max= 116, avg=75.24, stdev=18.21, samples=33 write: IOPS=79, BW=317KiB/s (324kB/s)(5256KiB/16596msec) slat (usec): min=13, max=521, avg=64.57, stdev=40.27 clat (msec): min=24, max=242, avg=136.31, stdev=31.48 lat (msec): min=24, max=242, avg=136.38, stdev=31.49 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 88], 10.00th=[ 99], 20.00th=[ 111], | 30.00th=[ 118], 40.00th=[ 128], 50.00th=[ 136], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 163], 90.00th=[ 178], 95.00th=[ 190], | 99.00th=[ 220], 99.50th=[ 230], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 240, max= 392, per=25.07%, avg=315.36, stdev=38.73, samples=33 iops : min= 60, max= 98, avg=78.82, stdev= 9.69, samples=33 lat (msec) : 2=0.08%, 4=0.04%, 10=0.20%, 20=0.08%, 50=3.40% lat (msec) : 100=49.45%, 250=46.76% cpu : usr=0.25%, sys=1.70%, ctx=3196, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24134: Fri May 16 10:00:29 2025 read: IOPS=76, BW=305KiB/s (312kB/s)(5060KiB/16596msec) slat (usec): min=416, max=3075, avg=982.44, stdev=308.67 clat (usec): min=1080, max=149961, avg=68936.32, stdev=14664.90 lat (msec): min=2, max=150, avg=69.92, stdev=14.78 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 87], 95.00th=[ 94], | 99.00th=[ 118], 99.50th=[ 122], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 152, max= 424, per=25.28%, avg=305.61, stdev=61.07, samples=33 iops : min= 38, max= 106, avg=76.33, stdev=15.24, samples=33 write: IOPS=78, BW=312KiB/s (320kB/s)(5180KiB/16596msec) slat (usec): min=13, max=600, avg=67.04, stdev=43.54 clat (msec): min=54, max=252, avg=136.36, stdev=30.61 lat (msec): min=54, max=253, avg=136.43, stdev=30.62 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 88], 10.00th=[ 100], 20.00th=[ 112], | 30.00th=[ 120], 40.00th=[ 128], 50.00th=[ 136], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 178], 95.00th=[ 192], | 99.00th=[ 220], 99.50th=[ 224], 99.90th=[ 243], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 231, max= 376, per=24.74%, avg=311.18, stdev=39.47, samples=33 iops : min= 57, max= 94, avg=77.73, stdev= 9.91, samples=33 lat (msec) : 2=0.04%, 4=0.04%, 20=0.20%, 50=2.50%, 100=50.66% lat (msec) : 250=46.52%, 500=0.04% cpu : usr=0.27%, sys=1.73%, ctx=3185, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1209KiB/s (1238kB/s), 300KiB/s-305KiB/s (307kB/s-312kB/s), io=19.6MiB (20.5MB), run=16596-16596msec WRITE: bw=1259KiB/s (1289kB/s), 312KiB/s-317KiB/s (320kB/s-325kB/s), io=20.4MiB (21.4MB), run=16596-16596msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=24151: Fri May 16 10:00:30 2025 read: IOPS=14, BW=588MiB/s (617MB/s)(40.0MiB/68msec) slat (nsec): min=6713.6k, max=6713.6k, avg=6713566.00, stdev= 0.00 clat (nsec): min=57041k, max=57041k, avg=57040577.00, stdev= 0.00 lat (nsec): min=63771k, max=63771k, avg=63770655.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[56886], 5.00th=[56886], 10.00th=[56886], 20.00th=[56886], | 30.00th=[56886], 40.00th=[56886], 50.00th=[56886], 60.00th=[56886], | 70.00th=[56886], 80.00th=[56886], 90.00th=[56886], 95.00th=[56886], | 99.00th=[56886], 99.50th=[56886], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=10.45%, ctx=15, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=588MiB/s (617MB/s), 588MiB/s-588MiB/s (617MB/s-617MB/s), io=40.0MiB (41.9MB), run=68-68msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck