== sanity test 398c: run fio to test AIO ================= 13:44:15 (1761241455) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.252089 s, 166 MB/s osc.lustre-OST0000-osc-ffff9e6d4631c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=123728: Thu Oct 23 13:46:33 2025 write: IOPS=19, BW=77.0KiB/s (79.9kB/s)(10.0MiB/131290msec); 0 zone resets slat (usec): min=20, max=11474, avg=195.69, stdev=543.80 clat (msec): min=229, max=1914, avg=820.05, stdev=221.69 lat (msec): min=229, max=1914, avg=820.25, stdev=221.71 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 506], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 986], 90.00th=[ 1099], 95.00th=[ 1217], | 99.00th=[ 1452], 99.50th=[ 1653], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 15, max= 246, per=24.83%, avg=77.22, stdev=32.65, samples=261 iops : min= 3, max= 61, avg=18.82, stdev= 8.22, samples=261 lat (msec) : 250=0.12%, 500=4.80%, 750=36.64%, 1000=40.16%, 2000=18.28% cpu : usr=0.04%, sys=0.29%, ctx=1479, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123729: Thu Oct 23 13:46:33 2025 write: IOPS=19, BW=78.1KiB/s (79.9kB/s)(10.0MiB/131189msec); 0 zone resets slat (usec): min=25, max=8406, avg=175.94, stdev=469.88 clat (msec): min=186, max=1851, avg=819.53, stdev=223.73 lat (msec): min=187, max=1851, avg=819.71, stdev=223.75 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 514], 10.00th=[ 584], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 986], 90.00th=[ 1099], 95.00th=[ 1200], | 99.00th=[ 1502], 99.50th=[ 1653], 99.90th=[ 1838], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 214, per=24.81%, avg=77.15, stdev=29.89, samples=261 iops : min= 1, max= 53, avg=18.80, stdev= 7.48, samples=261 lat (msec) : 250=0.94%, 500=3.52%, 750=36.60%, 1000=40.82%, 2000=18.12% cpu : usr=0.04%, sys=0.28%, ctx=1549, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123730: Thu Oct 23 13:46:33 2025 write: IOPS=19, BW=78.1KiB/s (79.9kB/s)(10.0MiB/131196msec); 0 zone resets slat (usec): min=20, max=16525, avg=181.07, stdev=562.42 clat (msec): min=208, max=1851, avg=819.48, stdev=221.99 lat (msec): min=209, max=1851, avg=819.66, stdev=222.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 523], 10.00th=[ 584], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 802], 60.00th=[ 844], | 70.00th=[ 911], 80.00th=[ 986], 90.00th=[ 1099], 95.00th=[ 1217], | 99.00th=[ 1502], 99.50th=[ 1653], 99.90th=[ 1787], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8, max= 212, per=24.90%, avg=77.45, stdev=32.75, samples=260 iops : min= 2, max= 53, avg=18.87, stdev= 8.23, samples=260 lat (msec) : 250=0.62%, 500=4.02%, 750=37.27%, 1000=40.12%, 2000=17.97% cpu : usr=0.05%, sys=0.27%, ctx=1408, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123731: Thu Oct 23 13:46:33 2025 write: IOPS=19, BW=78.1KiB/s (80.0kB/s)(10.0MiB/131063msec); 0 zone resets slat (usec): min=25, max=22260, avg=175.44, stdev=617.07 clat (msec): min=73, max=1944, avg=818.66, stdev=222.81 lat (msec): min=73, max=1944, avg=818.83, stdev=222.82 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 523], 10.00th=[ 592], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 751], 50.00th=[ 802], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 978], 90.00th=[ 1099], 95.00th=[ 1217], | 99.00th=[ 1502], 99.50th=[ 1552], 99.90th=[ 1854], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 211, per=24.90%, avg=77.45, stdev=33.02, samples=260 iops : min= 1, max= 52, avg=18.87, stdev= 8.26, samples=260 lat (msec) : 100=0.04%, 250=0.86%, 500=3.63%, 750=35.39%, 1000=42.27% lat (msec) : 2000=17.81% cpu : usr=0.05%, sys=0.28%, ctx=1528, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=312KiB/s (319kB/s), 77.0KiB/s-78.1KiB/s (79.9kB/s-80.0kB/s), io=40.0MiB (41.9MB), run=131063-131290msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=123759: Thu Oct 23 13:47:48 2025 read: IOPS=18, BW=73.2KiB/s (74.0kB/s)(5048KiB/68959msec) slat (usec): min=1729, max=25981, avg=7296.62, stdev=3235.02 clat (msec): min=9, max=878, avg=279.50, stdev=72.96 lat (msec): min=19, max=885, avg=286.81, stdev=73.62 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 184], 10.00th=[ 203], 20.00th=[ 222], | 30.00th=[ 243], 40.00th=[ 259], 50.00th=[ 275], 60.00th=[ 288], | 70.00th=[ 309], 80.00th=[ 330], 90.00th=[ 363], 95.00th=[ 393], | 99.00th=[ 451], 99.50th=[ 550], 99.90th=[ 802], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 15, max= 159, per=25.02%, avg=72.55, stdev=27.53, samples=137 iops : min= 3, max= 39, avg=17.66, stdev= 6.88, samples=137 write: IOPS=18, BW=75.3KiB/s (77.1kB/s)(5192KiB/68959msec); 0 zone resets slat (usec): min=25, max=3248, avg=199.44, stdev=360.16 clat (msec): min=204, max=1210, avg=569.65, stdev=147.00 lat (msec): min=204, max=1210, avg=569.85, stdev=147.01 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 359], 10.00th=[ 384], 20.00th=[ 430], | 30.00th=[ 481], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 693], 90.00th=[ 751], 95.00th=[ 818], | 99.00th=[ 961], 99.50th=[ 995], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 135, per=24.67%, avg=74.50, stdev=25.21, samples=137 iops : min= 1, max= 33, avg=18.15, stdev= 6.36, samples=137 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 250=17.11%, 500=49.10% lat (msec) : 750=28.16%, 1000=5.20%, 2000=0.23% cpu : usr=0.11%, sys=0.85%, ctx=3110, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123760: Thu Oct 23 13:47:48 2025 read: IOPS=17, BW=71.0KiB/s (73.7kB/s)(4976KiB/69134msec) slat (usec): min=1802, max=26917, avg=7344.75, stdev=3444.00 clat (msec): min=39, max=562, avg=276.96, stdev=65.01 lat (msec): min=50, max=574, avg=284.32, stdev=65.72 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 180], 10.00th=[ 199], 20.00th=[ 220], | 30.00th=[ 241], 40.00th=[ 257], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 330], 90.00th=[ 363], 95.00th=[ 393], | 99.00th=[ 443], 99.50th=[ 481], 99.90th=[ 558], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 16, max= 151, per=24.78%, avg=71.87, stdev=27.01, samples=136 iops : min= 4, max= 37, avg=17.49, stdev= 6.72, samples=136 write: IOPS=19, BW=76.1KiB/s (77.0kB/s)(5264KiB/69134msec); 0 zone resets slat (usec): min=25, max=3863, avg=180.37, stdev=321.28 clat (msec): min=122, max=994, avg=570.81, stdev=138.52 lat (msec): min=122, max=998, avg=570.99, stdev=138.53 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 359], 10.00th=[ 393], 20.00th=[ 439], | 30.00th=[ 493], 40.00th=[ 531], 50.00th=[ 575], 60.00th=[ 609], | 70.00th=[ 642], 80.00th=[ 684], 90.00th=[ 751], 95.00th=[ 802], | 99.00th=[ 911], 99.50th=[ 919], 99.90th=[ 978], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 31, max= 136, per=24.95%, avg=75.35, stdev=22.52, samples=137 iops : min= 7, max= 34, avg=18.36, stdev= 5.69, samples=137 lat (msec) : 50=0.04%, 100=0.08%, 250=17.58%, 500=47.19%, 750=30.12% lat (msec) : 1000=5.00% cpu : usr=0.09%, sys=0.85%, ctx=2886, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123761: Thu Oct 23 13:47:48 2025 read: IOPS=18, BW=72.2KiB/s (73.9kB/s)(4984KiB/69020msec) slat (usec): min=42, max=28048, avg=7223.74, stdev=3360.20 clat (msec): min=9, max=848, avg=279.21, stdev=71.99 lat (msec): min=9, max=855, avg=286.44, stdev=72.76 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 182], 10.00th=[ 199], 20.00th=[ 226], | 30.00th=[ 243], 40.00th=[ 257], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 368], 95.00th=[ 401], | 99.00th=[ 472], 99.50th=[ 506], 99.90th=[ 584], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 15, max= 136, per=24.95%, avg=72.36, stdev=27.20, samples=136 iops : min= 3, max= 34, avg=17.60, stdev= 6.81, samples=136 write: IOPS=19, BW=76.2KiB/s (77.0kB/s)(5256KiB/69020msec); 0 zone resets slat (usec): min=25, max=6171, avg=205.50, stdev=423.77 clat (msec): min=64, max=1205, avg=567.92, stdev=145.87 lat (msec): min=64, max=1205, avg=568.13, stdev=145.87 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 355], 10.00th=[ 388], 20.00th=[ 439], | 30.00th=[ 485], 40.00th=[ 523], 50.00th=[ 567], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 684], 90.00th=[ 760], 95.00th=[ 818], | 99.00th=[ 911], 99.50th=[ 944], 99.90th=[ 1070], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 15, max= 143, per=24.87%, avg=75.10, stdev=24.22, samples=137 iops : min= 3, max= 35, avg=18.29, stdev= 6.14, samples=137 lat (msec) : 10=0.08%, 20=0.08%, 50=0.12%, 100=0.39%, 250=17.27% lat (msec) : 500=47.73%, 750=28.79%, 1000=5.47%, 2000=0.08% cpu : usr=0.11%, sys=0.84%, ctx=3154, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123762: Thu Oct 23 13:47:48 2025 read: IOPS=18, BW=73.4KiB/s (75.1kB/s)(5060KiB/68958msec) slat (usec): min=1751, max=23276, avg=7202.15, stdev=3235.01 clat (msec): min=7, max=668, avg=276.08, stdev=67.41 lat (msec): min=11, max=677, avg=283.28, stdev=68.02 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 180], 10.00th=[ 199], 20.00th=[ 224], | 30.00th=[ 239], 40.00th=[ 255], 50.00th=[ 275], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 359], 95.00th=[ 397], | 99.00th=[ 447], 99.50th=[ 460], 99.90th=[ 617], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 15, max= 144, per=25.14%, avg=72.91, stdev=27.13, samples=137 iops : min= 3, max= 36, avg=17.74, stdev= 6.82, samples=137 write: IOPS=18, BW=75.1KiB/s (76.9kB/s)(5180KiB/68958msec); 0 zone resets slat (usec): min=27, max=6995, avg=196.04, stdev=390.39 clat (msec): min=124, max=1209, avg=574.06, stdev=149.78 lat (msec): min=124, max=1209, avg=574.26, stdev=149.80 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 355], 10.00th=[ 388], 20.00th=[ 439], | 30.00th=[ 485], 40.00th=[ 531], 50.00th=[ 567], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 701], 90.00th=[ 760], 95.00th=[ 827], | 99.00th=[ 961], 99.50th=[ 1083], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 143, per=24.54%, avg=74.12, stdev=22.85, samples=137 iops : min= 1, max= 35, avg=18.05, stdev= 5.72, samples=137 lat (msec) : 10=0.08%, 50=0.16%, 100=0.12%, 250=18.05%, 500=47.38% lat (msec) : 750=28.75%, 1000=5.12%, 2000=0.35% cpu : usr=0.12%, sys=0.85%, ctx=3132, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=290KiB/s (297kB/s), 71.0KiB/s-73.4KiB/s (73.7kB/s-75.1kB/s), io=19.6MiB (20.5MB), run=68958-69134msec WRITE: bw=302KiB/s (309kB/s), 75.1KiB/s-76.2KiB/s (76.9kB/s-77.0kB/s), io=20.4MiB (21.4MB), run=68958-69134msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=123778: Thu Oct 23 13:47:54 2025 read: IOPS=4, BW=177MiB/s (186MB/s)(40.0MiB/226msec) slat (nsec): min=18904k, max=18904k, avg=18904476.00, stdev= 0.00 clat (nsec): min=196276k, max=196276k, avg=196276254.00, stdev= 0.00 lat (nsec): min=215185k, max=215185k, avg=215185333.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] lat (msec) : 250=100.00% cpu : usr=0.44%, sys=5.78%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177MiB/s (186MB/s), 177MiB/s-177MiB/s (186MB/s-186MB/s), io=40.0MiB (41.9MB), run=226-226msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck