== sanity test 398c: run fio to test AIO ================= 03:25:20 (1753428320) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.13517 s, 310 MB/s osc.lustre-OST0000-osc-ffffa03f4a025800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=157467: Fri Jul 25 03:25:55 2025 write: IOPS=79, BW=317KiB/s (324kB/s)(10.0MiB/32318msec); 0 zone resets slat (usec): min=26, max=1708, avg=142.65, stdev=77.91 clat (msec): min=45, max=357, avg=201.65, stdev=39.63 lat (msec): min=45, max=357, avg=201.80, stdev=39.63 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 136], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 222], 80.00th=[ 234], 90.00th=[ 249], 95.00th=[ 262], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 321], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 256, max= 560, per=24.95%, avg=315.86, stdev=47.86, samples=64 iops : min= 64, max= 140, avg=78.95, stdev=11.97, samples=64 lat (msec) : 50=0.12%, 100=1.37%, 250=89.38%, 500=9.14% cpu : usr=0.19%, sys=1.06%, ctx=2127, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=157468: Fri Jul 25 03:25:55 2025 write: IOPS=79, BW=317KiB/s (324kB/s)(10.0MiB/32341msec); 0 zone resets slat (usec): min=32, max=569, avg=137.84, stdev=72.43 clat (msec): min=45, max=305, avg=201.80, stdev=39.31 lat (msec): min=45, max=305, avg=201.94, stdev=39.32 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 136], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 213], | 70.00th=[ 222], 80.00th=[ 234], 90.00th=[ 249], 95.00th=[ 262], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 256, max= 573, per=24.93%, avg=315.56, stdev=49.14, samples=64 iops : min= 64, max= 143, avg=78.87, stdev=12.27, samples=64 lat (msec) : 50=0.04%, 100=1.48%, 250=89.38%, 500=9.10% cpu : usr=0.15%, sys=1.05%, ctx=2041, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=157469: Fri Jul 25 03:25:55 2025 write: IOPS=79, BW=317KiB/s (324kB/s)(10.0MiB/32341msec); 0 zone resets slat (usec): min=34, max=1060, avg=141.62, stdev=77.87 clat (msec): min=56, max=329, avg=201.77, stdev=38.97 lat (msec): min=57, max=330, avg=201.91, stdev=38.97 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 136], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 213], | 70.00th=[ 222], 80.00th=[ 232], 90.00th=[ 251], 95.00th=[ 262], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 330] bw ( KiB/s): min= 256, max= 566, per=24.92%, avg=315.45, stdev=48.85, samples=64 iops : min= 64, max= 141, avg=78.84, stdev=12.18, samples=64 lat (msec) : 100=1.41%, 250=88.71%, 500=9.88% cpu : usr=0.21%, sys=1.01%, ctx=2111, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=157470: Fri Jul 25 03:25:55 2025 write: IOPS=79, BW=318KiB/s (325kB/s)(10.0MiB/32217msec); 0 zone resets slat (usec): min=28, max=822, avg=144.09, stdev=74.75 clat (msec): min=21, max=359, avg=201.08, stdev=41.86 lat (msec): min=21, max=359, avg=201.23, stdev=41.86 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 129], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 222], 80.00th=[ 232], 90.00th=[ 249], 95.00th=[ 264], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 256, max= 631, per=24.94%, avg=315.72, stdev=56.26, samples=64 iops : min= 64, max= 157, avg=78.91, stdev=14.01, samples=64 lat (msec) : 50=0.62%, 100=1.37%, 250=88.40%, 500=9.61% cpu : usr=0.17%, sys=1.08%, ctx=2161, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1267KiB/s (1297kB/s), 317KiB/s-318KiB/s (324kB/s-325kB/s), io=40.0MiB (41.9MB), run=32217-32341msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=157484: Fri Jul 25 03:26:19 2025 read: IOPS=61, BW=246KiB/s (252kB/s)(5048KiB/20544msec) slat (usec): min=1027, max=8828, avg=2435.41, stdev=740.56 clat (msec): min=2, max=131, avg=82.44, stdev=15.03 lat (msec): min=5, max=134, avg=84.88, stdev=15.10 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 66], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 83], 60.00th=[ 87], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 102], 95.00th=[ 107], | 99.00th=[ 115], 99.50th=[ 124], 99.90th=[ 130], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 144, max= 344, per=25.00%, avg=244.03, stdev=51.29, samples=40 iops : min= 36, max= 86, avg=60.97, stdev=12.81, samples=40 write: IOPS=63, BW=253KiB/s (259kB/s)(5192KiB/20544msec); 0 zone resets slat (usec): min=24, max=1124, avg=114.88, stdev=69.00 clat (msec): min=39, max=292, avg=170.04, stdev=37.25 lat (msec): min=39, max=292, avg=170.16, stdev=37.25 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 110], 10.00th=[ 123], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 218], 95.00th=[ 236], | 99.00th=[ 259], 99.50th=[ 266], 99.90th=[ 288], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 192, max= 304, per=24.74%, avg=251.40, stdev=28.39, samples=40 iops : min= 48, max= 76, avg=62.83, stdev= 7.11, samples=40 lat (msec) : 4=0.08%, 10=0.08%, 20=0.12%, 50=0.35%, 100=44.41% lat (msec) : 250=53.67%, 500=1.29% cpu : usr=0.29%, sys=1.91%, ctx=3280, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157485: Fri Jul 25 03:26:19 2025 read: IOPS=60, BW=243KiB/s (248kB/s)(4976KiB/20513msec) slat (usec): min=986, max=5967, avg=2440.75, stdev=714.73 clat (msec): min=4, max=154, avg=81.84, stdev=15.58 lat (msec): min=6, max=159, avg=84.29, stdev=15.70 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 101], 95.00th=[ 105], | 99.00th=[ 120], 99.50th=[ 130], 99.90th=[ 153], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 152, max= 344, per=24.82%, avg=242.27, stdev=54.18, samples=40 iops : min= 38, max= 86, avg=60.55, stdev=13.55, samples=40 write: IOPS=64, BW=257KiB/s (263kB/s)(5264KiB/20513msec); 0 zone resets slat (usec): min=29, max=1105, avg=115.32, stdev=63.90 clat (msec): min=66, max=282, avg=169.17, stdev=36.70 lat (msec): min=66, max=282, avg=169.29, stdev=36.70 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 111], 10.00th=[ 123], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 199], 90.00th=[ 215], 95.00th=[ 234], | 99.00th=[ 262], 99.50th=[ 268], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 184, max= 358, per=25.17%, avg=255.73, stdev=33.04, samples=40 iops : min= 46, max= 89, avg=63.90, stdev= 8.24, samples=40 lat (msec) : 10=0.16%, 20=0.27%, 50=0.31%, 100=44.30%, 250=53.63% lat (msec) : 500=1.33% cpu : usr=0.31%, sys=1.90%, ctx=3268, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157486: Fri Jul 25 03:26:19 2025 read: IOPS=60, BW=244KiB/s (250kB/s)(4984KiB/20446msec) slat (usec): min=60, max=11086, avg=2448.06, stdev=771.77 clat (msec): min=2, max=190, avg=81.85, stdev=16.03 lat (msec): min=4, max=194, avg=84.30, stdev=16.14 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 70], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 101], 95.00th=[ 106], | 99.00th=[ 117], 99.50th=[ 125], 99.90th=[ 190], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 128, max= 360, per=25.10%, avg=244.95, stdev=55.39, samples=40 iops : min= 32, max= 90, avg=61.22, stdev=13.83, samples=40 write: IOPS=64, BW=257KiB/s (263kB/s)(5256KiB/20446msec); 0 zone resets slat (usec): min=31, max=1005, avg=113.82, stdev=64.61 clat (msec): min=26, max=289, avg=168.70, stdev=36.40 lat (msec): min=26, max=289, avg=168.81, stdev=36.40 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 111], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 213], 95.00th=[ 228], | 99.00th=[ 264], 99.50th=[ 266], 99.90th=[ 279], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 192, max= 312, per=25.21%, avg=256.12, stdev=30.38, samples=40 iops : min= 48, max= 78, avg=64.00, stdev= 7.60, samples=40 lat (msec) : 4=0.08%, 10=0.23%, 20=0.20%, 50=0.47%, 100=43.91% lat (msec) : 250=54.14%, 500=0.98% cpu : usr=0.35%, sys=1.84%, ctx=3336, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157487: Fri Jul 25 03:26:19 2025 read: IOPS=61, BW=247KiB/s (253kB/s)(5060KiB/20479msec) slat (usec): min=985, max=6041, avg=2428.86, stdev=749.34 clat (msec): min=4, max=184, avg=82.24, stdev=15.56 lat (msec): min=6, max=186, avg=84.68, stdev=15.64 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 89], 80.00th=[ 94], 90.00th=[ 101], 95.00th=[ 106], | 99.00th=[ 123], 99.50th=[ 128], 99.90th=[ 182], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 144, max= 352, per=25.39%, avg=247.77, stdev=49.63, samples=40 iops : min= 36, max= 88, avg=61.92, stdev=12.39, samples=40 write: IOPS=63, BW=253KiB/s (259kB/s)(5180KiB/20479msec); 0 zone resets slat (usec): min=35, max=561, avg=113.85, stdev=55.94 clat (msec): min=67, max=288, avg=169.75, stdev=36.04 lat (msec): min=67, max=289, avg=169.86, stdev=36.04 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 215], 95.00th=[ 232], | 99.00th=[ 264], 99.50th=[ 268], 99.90th=[ 275], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 200, max= 328, per=24.83%, avg=252.30, stdev=27.92, samples=40 iops : min= 50, max= 82, avg=63.05, stdev= 6.98, samples=40 lat (msec) : 10=0.08%, 20=0.23%, 50=0.27%, 100=44.61%, 250=53.75% lat (msec) : 500=1.05% cpu : usr=0.34%, sys=1.90%, ctx=3310, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=977KiB/s (1000kB/s), 243KiB/s-247KiB/s (248kB/s-253kB/s), io=19.6MiB (20.5MB), run=20446-20544msec WRITE: bw=1017KiB/s (1041kB/s), 253KiB/s-257KiB/s (259kB/s-263kB/s), io=20.4MiB (21.4MB), run=20446-20544msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=157494: Fri Jul 25 03:26:23 2025 read: IOPS=10, BW=417MiB/s (437MB/s)(40.0MiB/96msec) slat (nsec): min=9146.9k, max=9146.9k, avg=9146924.00, stdev= 0.00 clat (nsec): min=81843k, max=81843k, avg=81843402.00, stdev= 0.00 lat (nsec): min=90994k, max=90994k, avg=90993576.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[82314], 5.00th=[82314], 10.00th=[82314], 20.00th=[82314], | 30.00th=[82314], 40.00th=[82314], 50.00th=[82314], 60.00th=[82314], | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.47%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=417MiB/s (437MB/s), 417MiB/s-417MiB/s (437MB/s-437MB/s), io=40.0MiB (41.9MB), run=96-96msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout