== sanity test 398c: run fio to test AIO ================= 04:13:14 (1752480794) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.428412 s, 97.9 MB/s osc.lustre-OST0000-osc-ffff8a33cbe59000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=158282: Mon Jul 14 04:15:28 2025 write: IOPS=20, BW=81.5KiB/s (83.4kB/s)(10.0MiB/125705msec); 0 zone resets slat (usec): min=27, max=7709, avg=303.24, stdev=639.89 clat (msec): min=264, max=1455, avg=784.46, stdev=174.05 lat (msec): min=264, max=1455, avg=784.76, stdev=174.07 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 527], 10.00th=[ 584], 20.00th=[ 642], | 30.00th=[ 684], 40.00th=[ 726], 50.00th=[ 768], 60.00th=[ 827], | 70.00th=[ 877], 80.00th=[ 927], 90.00th=[ 1011], 95.00th=[ 1083], | 99.00th=[ 1217], 99.50th=[ 1334], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 7, max= 159, per=25.20%, avg=81.89, stdev=29.04, samples=246 iops : min= 1, max= 39, avg=19.92, stdev= 7.26, samples=246 lat (msec) : 500=3.75%, 750=42.38%, 1000=43.24%, 2000=10.62% cpu : usr=0.07%, sys=0.41%, ctx=1581, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158283: Mon Jul 14 04:15:28 2025 write: IOPS=20, BW=81.5KiB/s (83.4kB/s)(10.0MiB/125708msec); 0 zone resets slat (usec): min=28, max=9691, avg=283.78, stdev=630.94 clat (msec): min=293, max=1428, avg=784.59, stdev=172.21 lat (msec): min=293, max=1428, avg=784.88, stdev=172.25 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 531], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 726], 50.00th=[ 768], 60.00th=[ 818], | 70.00th=[ 877], 80.00th=[ 927], 90.00th=[ 1003], 95.00th=[ 1083], | 99.00th=[ 1200], 99.50th=[ 1284], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 135, per=25.00%, avg=81.26, stdev=28.83, samples=248 iops : min= 1, max= 33, avg=19.76, stdev= 7.22, samples=248 lat (msec) : 500=3.71%, 750=41.68%, 1000=44.06%, 2000=10.55% cpu : usr=0.06%, sys=0.41%, ctx=1613, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158284: Mon Jul 14 04:15:28 2025 write: IOPS=20, BW=81.9KiB/s (83.8kB/s)(10.0MiB/125093msec); 0 zone resets slat (usec): min=29, max=14784, avg=334.28, stdev=737.85 clat (msec): min=95, max=1393, avg=781.16, stdev=176.03 lat (msec): min=95, max=1393, avg=781.49, stdev=176.03 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 523], 10.00th=[ 567], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 726], 50.00th=[ 776], 60.00th=[ 810], | 70.00th=[ 869], 80.00th=[ 927], 90.00th=[ 1011], 95.00th=[ 1083], | 99.00th=[ 1200], 99.50th=[ 1284], 99.90th=[ 1385], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 23, max= 172, per=24.98%, avg=81.19, stdev=28.31, samples=248 iops : min= 5, max= 43, avg=19.75, stdev= 7.09, samples=248 lat (msec) : 100=0.08%, 250=0.47%, 500=3.28%, 750=41.05%, 1000=44.26% lat (msec) : 2000=10.86% cpu : usr=0.08%, sys=0.45%, ctx=1913, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158285: Mon Jul 14 04:15:28 2025 write: IOPS=20, BW=81.5KiB/s (83.4kB/s)(10.0MiB/125707msec); 0 zone resets slat (usec): min=29, max=12156, avg=281.88, stdev=602.67 clat (msec): min=311, max=1374, avg=784.59, stdev=170.00 lat (msec): min=311, max=1374, avg=784.87, stdev=170.02 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 523], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 726], 50.00th=[ 768], 60.00th=[ 818], | 70.00th=[ 869], 80.00th=[ 927], 90.00th=[ 1003], 95.00th=[ 1083], | 99.00th=[ 1200], 99.50th=[ 1284], 99.90th=[ 1351], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 8, max= 151, per=25.20%, avg=81.90, stdev=28.68, samples=246 iops : min= 2, max= 37, avg=19.92, stdev= 7.15, samples=246 lat (msec) : 500=3.87%, 750=41.52%, 1000=44.80%, 2000=9.80% cpu : usr=0.06%, sys=0.41%, ctx=1710, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=326KiB/s (334kB/s), 81.5KiB/s-81.9KiB/s (83.4kB/s-83.8kB/s), io=40.0MiB (41.9MB), run=125093-125708msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=158312: Mon Jul 14 04:16:48 2025 read: IOPS=17, BW=68.2KiB/s (69.8kB/s)(5048KiB/74036msec) slat (usec): min=2717, max=25425, avg=9212.53, stdev=3764.03 clat (msec): min=16, max=516, avg=298.69, stdev=66.87 lat (msec): min=24, max=524, avg=307.91, stdev=67.58 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 199], 10.00th=[ 215], 20.00th=[ 243], | 30.00th=[ 264], 40.00th=[ 284], 50.00th=[ 300], 60.00th=[ 317], | 70.00th=[ 334], 80.00th=[ 351], 90.00th=[ 384], 95.00th=[ 409], | 99.00th=[ 456], 99.50th=[ 468], 99.90th=[ 489], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 7, max= 128, per=24.97%, avg=67.43, stdev=25.95, samples=147 iops : min= 1, max= 32, avg=16.31, stdev= 6.57, samples=147 write: IOPS=17, BW=70.1KiB/s (71.8kB/s)(5192KiB/74036msec); 0 zone resets slat (usec): min=29, max=4468, avg=269.26, stdev=432.56 clat (msec): min=183, max=1212, avg=611.66, stdev=172.00 lat (msec): min=183, max=1212, avg=611.93, stdev=172.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 355], 10.00th=[ 393], 20.00th=[ 447], | 30.00th=[ 510], 40.00th=[ 558], 50.00th=[ 609], 60.00th=[ 651], | 70.00th=[ 701], 80.00th=[ 760], 90.00th=[ 844], 95.00th=[ 919], | 99.00th=[ 1020], 99.50th=[ 1053], 99.90th=[ 1116], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 23, max= 128, per=24.72%, avg=69.46, stdev=24.38, samples=147 iops : min= 5, max= 32, avg=16.82, stdev= 6.11, samples=147 lat (msec) : 20=0.04%, 50=0.12%, 100=0.27%, 250=11.13%, 500=51.95% lat (msec) : 750=25.98%, 1000=9.69%, 2000=0.82% cpu : usr=0.13%, sys=1.08%, ctx=2925, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158313: Mon Jul 14 04:16:48 2025 read: IOPS=16, BW=67.2KiB/s (68.8kB/s)(4976KiB/74097msec) slat (usec): min=40, max=28747, avg=9064.21, stdev=3885.47 clat (msec): min=12, max=517, avg=297.31, stdev=66.06 lat (msec): min=14, max=527, avg=306.38, stdev=66.86 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 203], 10.00th=[ 218], 20.00th=[ 243], | 30.00th=[ 264], 40.00th=[ 284], 50.00th=[ 296], 60.00th=[ 313], | 70.00th=[ 330], 80.00th=[ 355], 90.00th=[ 380], 95.00th=[ 401], | 99.00th=[ 456], 99.50th=[ 468], 99.90th=[ 514], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 16, max= 159, per=24.57%, avg=66.34, stdev=29.86, samples=147 iops : min= 4, max= 39, avg=16.05, stdev= 7.51, samples=147 write: IOPS=17, BW=71.0KiB/s (72.7kB/s)(5264KiB/74097msec); 0 zone resets slat (usec): min=29, max=6732, avg=303.56, stdev=532.84 clat (msec): min=66, max=1214, avg=610.46, stdev=177.77 lat (msec): min=66, max=1215, avg=610.77, stdev=177.79 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 338], 10.00th=[ 397], 20.00th=[ 460], | 30.00th=[ 506], 40.00th=[ 558], 50.00th=[ 609], 60.00th=[ 659], | 70.00th=[ 701], 80.00th=[ 760], 90.00th=[ 835], 95.00th=[ 919], | 99.00th=[ 1070], 99.50th=[ 1083], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 16, max= 138, per=25.03%, avg=70.35, stdev=25.36, samples=147 iops : min= 4, max= 34, avg=17.05, stdev= 6.35, samples=147 lat (msec) : 20=0.08%, 50=0.12%, 100=0.31%, 250=11.64%, 500=51.13% lat (msec) : 750=25.78%, 1000=9.88%, 2000=1.05% cpu : usr=0.15%, sys=1.08%, ctx=3000, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158314: Mon Jul 14 04:16:48 2025 read: IOPS=16, BW=67.2KiB/s (68.8kB/s)(4984KiB/74188msec) slat (usec): min=2818, max=27137, avg=9406.81, stdev=3809.52 clat (msec): min=12, max=514, avg=297.29, stdev=64.04 lat (msec): min=19, max=525, avg=306.71, stdev=64.85 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 201], 10.00th=[ 220], 20.00th=[ 245], | 30.00th=[ 264], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 313], | 70.00th=[ 330], 80.00th=[ 351], 90.00th=[ 380], 95.00th=[ 401], | 99.00th=[ 464], 99.50th=[ 472], 99.90th=[ 498], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 7, max= 128, per=24.58%, avg=66.36, stdev=27.06, samples=147 iops : min= 1, max= 32, avg=16.05, stdev= 6.79, samples=147 write: IOPS=17, BW=70.8KiB/s (72.5kB/s)(5256KiB/74188msec); 0 zone resets slat (usec): min=28, max=6417, avg=273.39, stdev=480.70 clat (msec): min=52, max=1203, avg=611.06, stdev=181.60 lat (msec): min=53, max=1204, avg=611.33, stdev=181.60 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 342], 10.00th=[ 393], 20.00th=[ 443], | 30.00th=[ 502], 40.00th=[ 550], 50.00th=[ 609], 60.00th=[ 659], | 70.00th=[ 709], 80.00th=[ 768], 90.00th=[ 852], 95.00th=[ 911], | 99.00th=[ 1045], 99.50th=[ 1083], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 15, max= 135, per=24.75%, avg=69.55, stdev=23.19, samples=147 iops : min= 3, max= 33, avg=16.84, stdev= 5.84, samples=147 lat (msec) : 20=0.04%, 50=0.12%, 100=0.35%, 250=10.94%, 500=52.38% lat (msec) : 750=24.41%, 1000=10.70%, 2000=1.05% cpu : usr=0.14%, sys=1.09%, ctx=2901, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158315: Mon Jul 14 04:16:48 2025 read: IOPS=17, BW=68.3KiB/s (69.9kB/s)(5060KiB/74130msec) slat (usec): min=2812, max=34122, avg=9215.10, stdev=3914.27 clat (msec): min=37, max=490, avg=299.25, stdev=63.56 lat (msec): min=42, max=508, avg=308.47, stdev=64.31 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 205], 10.00th=[ 220], 20.00th=[ 245], | 30.00th=[ 264], 40.00th=[ 284], 50.00th=[ 296], 60.00th=[ 313], | 70.00th=[ 334], 80.00th=[ 355], 90.00th=[ 380], 95.00th=[ 405], | 99.00th=[ 447], 99.50th=[ 460], 99.90th=[ 477], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 7, max= 143, per=25.12%, avg=67.82, stdev=25.47, samples=147 iops : min= 1, max= 35, avg=16.41, stdev= 6.40, samples=147 write: IOPS=17, BW=69.9KiB/s (71.6kB/s)(5180KiB/74130msec); 0 zone resets slat (usec): min=31, max=9474, avg=304.23, stdev=563.83 clat (msec): min=107, max=1126, avg=613.01, stdev=169.25 lat (msec): min=108, max=1126, avg=613.32, stdev=169.25 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 363], 10.00th=[ 401], 20.00th=[ 456], | 30.00th=[ 514], 40.00th=[ 558], 50.00th=[ 609], 60.00th=[ 651], | 70.00th=[ 701], 80.00th=[ 760], 90.00th=[ 835], 95.00th=[ 919], | 99.00th=[ 1028], 99.50th=[ 1070], 99.90th=[ 1099], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 15, max= 136, per=24.57%, avg=69.05, stdev=23.72, samples=147 iops : min= 3, max= 34, avg=16.72, stdev= 5.96, samples=147 lat (msec) : 50=0.08%, 100=0.20%, 250=11.45%, 500=51.41%, 750=26.09% lat (msec) : 1000=9.88%, 2000=0.90% cpu : usr=0.14%, sys=1.11%, ctx=3113, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=271KiB/s (277kB/s), 67.2KiB/s-68.3KiB/s (68.8kB/s-69.9kB/s), io=19.6MiB (20.5MB), run=74036-74188msec WRITE: bw=282KiB/s (288kB/s), 69.9KiB/s-71.0KiB/s (71.6kB/s-72.7kB/s), io=20.4MiB (21.4MB), run=74036-74188msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=158329: Mon Jul 14 04:16:53 2025 read: IOPS=3, BW=133MiB/s (140MB/s)(40.0MiB/300msec) slat (nsec): min=29193k, max=29193k, avg=29192584.00, stdev= 0.00 clat (nsec): min=255914k, max=255914k, avg=255913652.00, stdev= 0.00 lat (nsec): min=285112k, max=285112k, avg=285111963.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 257], | 70.00th=[ 257], 80.00th=[ 257], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=5.02%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=40.0MiB (41.9MB), run=300-300msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout