== sanity test 398c: run fio to test AIO ================= 10:04:29 (1761228269) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.352282 s, 119 MB/s osc.lustre-OST0000-osc-ffff8915c66f0000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144485: Thu Oct 23 10:06:59 2025 write: IOPS=17, BW=71.3KiB/s (72.0kB/s)(10.0MiB/143696msec); 0 zone resets slat (usec): min=25, max=8301, avg=238.64, stdev=523.10 clat (msec): min=63, max=1509, avg=897.46, stdev=206.37 lat (msec): min=63, max=1509, avg=897.70, stdev=206.36 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 542], 10.00th=[ 642], 20.00th=[ 760], | 30.00th=[ 818], 40.00th=[ 860], 50.00th=[ 902], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1351], 99.50th=[ 1418], 99.90th=[ 1435], 99.95th=[ 1485], | 99.99th=[ 1502] bw ( KiB/s): min= 8, max= 183, per=24.88%, avg=70.67, stdev=27.61, samples=285 iops : min= 2, max= 45, avg=17.21, stdev= 6.90, samples=285 lat (msec) : 100=0.23%, 250=0.62%, 500=2.73%, 750=14.84%, 1000=53.09% lat (msec) : 2000=28.48% cpu : usr=0.05%, sys=0.32%, ctx=1798, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144486: Thu Oct 23 10:06:59 2025 write: IOPS=17, BW=71.0KiB/s (72.7kB/s)(10.0MiB/144157msec); 0 zone resets slat (usec): min=26, max=9356, avg=249.03, stdev=640.84 clat (msec): min=211, max=1525, avg=900.20, stdev=199.92 lat (msec): min=211, max=1526, avg=900.45, stdev=199.93 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 567], 10.00th=[ 651], 20.00th=[ 760], | 30.00th=[ 818], 40.00th=[ 860], 50.00th=[ 894], 60.00th=[ 936], | 70.00th=[ 986], 80.00th=[ 1053], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1351], 99.50th=[ 1385], 99.90th=[ 1435], 99.95th=[ 1502], | 99.99th=[ 1519] bw ( KiB/s): min= 15, max= 207, per=24.81%, avg=70.46, stdev=28.05, samples=286 iops : min= 3, max= 51, avg=17.16, stdev= 7.03, samples=286 lat (msec) : 250=0.62%, 500=2.38%, 750=15.66%, 1000=52.77%, 2000=28.55% cpu : usr=0.05%, sys=0.31%, ctx=1791, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144487: Thu Oct 23 10:06:59 2025 write: IOPS=17, BW=71.1KiB/s (72.8kB/s)(10.0MiB/144068msec); 0 zone resets slat (usec): min=24, max=8137, avg=228.83, stdev=491.54 clat (msec): min=208, max=1531, avg=899.69, stdev=197.95 lat (msec): min=211, max=1531, avg=899.92, stdev=197.93 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 567], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 902], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1351], 99.50th=[ 1385], 99.90th=[ 1435], 99.95th=[ 1485], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 143, per=24.81%, avg=70.46, stdev=26.35, samples=286 iops : min= 1, max= 35, avg=17.16, stdev= 6.62, samples=286 lat (msec) : 250=0.62%, 500=2.42%, 750=16.13%, 1000=51.84%, 2000=28.98% cpu : usr=0.05%, sys=0.32%, ctx=1738, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144488: Thu Oct 23 10:06:59 2025 write: IOPS=17, BW=71.1KiB/s (72.8kB/s)(10.0MiB/144108msec); 0 zone resets slat (usec): min=25, max=10388, avg=241.83, stdev=591.08 clat (msec): min=209, max=1480, avg=899.89, stdev=200.54 lat (msec): min=209, max=1480, avg=900.14, stdev=200.57 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 558], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1351], 99.50th=[ 1401], 99.90th=[ 1435], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 8, max= 239, per=24.81%, avg=70.47, stdev=26.59, samples=286 iops : min= 2, max= 59, avg=17.16, stdev= 6.67, samples=286 lat (msec) : 250=0.62%, 500=2.30%, 750=16.56%, 1000=50.82%, 2000=29.69% cpu : usr=0.04%, sys=0.31%, ctx=1749, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=284KiB/s (291kB/s), 71.0KiB/s-71.3KiB/s (72.7kB/s-72.0kB/s), io=40.0MiB (41.9MB), run=143696-144157msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144520: Thu Oct 23 10:08:33 2025 read: IOPS=14, BW=58.4KiB/s (59.8kB/s)(5048KiB/86423msec) slat (usec): min=49, max=32993, avg=9430.23, stdev=4809.23 clat (msec): min=9, max=774, avg=362.88, stdev=93.27 lat (msec): min=9, max=785, avg=372.32, stdev=94.39 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 230], 10.00th=[ 257], 20.00th=[ 292], | 30.00th=[ 313], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 414], 80.00th=[ 439], 90.00th=[ 481], 95.00th=[ 518], | 99.00th=[ 567], 99.50th=[ 600], 99.90th=[ 634], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 16, max= 134, per=25.14%, avg=57.83, stdev=23.90, samples=171 iops : min= 4, max= 33, avg=13.94, stdev= 5.99, samples=171 write: IOPS=15, BW=60.1KiB/s (61.5kB/s)(5192KiB/86423msec); 0 zone resets slat (usec): min=28, max=11761, avg=295.46, stdev=657.65 clat (msec): min=43, max=1159, avg=702.44, stdev=187.47 lat (msec): min=43, max=1159, avg=702.74, stdev=187.46 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 414], 10.00th=[ 477], 20.00th=[ 542], | 30.00th=[ 600], 40.00th=[ 651], 50.00th=[ 693], 60.00th=[ 743], | 70.00th=[ 802], 80.00th=[ 877], 90.00th=[ 961], 95.00th=[ 1020], | 99.00th=[ 1099], 99.50th=[ 1116], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 16, max= 142, per=24.84%, avg=59.36, stdev=20.87, samples=171 iops : min= 4, max= 35, avg=14.32, stdev= 5.23, samples=171 lat (msec) : 10=0.04%, 20=0.20%, 50=0.20%, 100=0.35%, 250=3.91% lat (msec) : 500=47.97%, 750=27.70%, 1000=16.68%, 2000=2.97% cpu : usr=0.13%, sys=0.90%, ctx=3362, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144521: Thu Oct 23 10:08:33 2025 read: IOPS=14, BW=57.2KiB/s (58.6kB/s)(4976KiB/87021msec) slat (usec): min=1928, max=29708, avg=9353.11, stdev=4411.46 clat (msec): min=11, max=583, avg=363.02, stdev=82.58 lat (msec): min=22, max=590, avg=372.39, stdev=83.34 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 234], 10.00th=[ 262], 20.00th=[ 296], | 30.00th=[ 317], 40.00th=[ 342], 50.00th=[ 355], 60.00th=[ 380], | 70.00th=[ 409], 80.00th=[ 430], 90.00th=[ 472], 95.00th=[ 510], | 99.00th=[ 550], 99.50th=[ 558], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 7, max= 120, per=24.54%, avg=56.44, stdev=24.41, samples=173 iops : min= 1, max= 30, avg=13.59, stdev= 6.11, samples=173 write: IOPS=15, BW=60.5KiB/s (61.9kB/s)(5264KiB/87021msec); 0 zone resets slat (usec): min=25, max=8351, avg=306.78, stdev=720.72 clat (msec): min=187, max=1164, avg=704.58, stdev=179.81 lat (msec): min=187, max=1164, avg=704.89, stdev=179.90 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 426], 10.00th=[ 485], 20.00th=[ 550], | 30.00th=[ 592], 40.00th=[ 651], 50.00th=[ 701], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 877], 90.00th=[ 953], 95.00th=[ 995], | 99.00th=[ 1083], 99.50th=[ 1099], 99.90th=[ 1133], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 119, per=24.97%, avg=59.68, stdev=21.64, samples=173 iops : min= 1, max= 29, avg=14.40, stdev= 5.41, samples=173 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=4.02%, 500=47.66% lat (msec) : 750=27.30%, 1000=18.24%, 2000=2.50% cpu : usr=0.12%, sys=0.87%, ctx=3257, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144522: Thu Oct 23 10:08:33 2025 read: IOPS=14, BW=57.2KiB/s (58.6kB/s)(4984KiB/87115msec) slat (usec): min=2230, max=28968, avg=9376.89, stdev=4388.88 clat (msec): min=10, max=610, avg=363.73, stdev=84.99 lat (msec): min=14, max=616, avg=373.12, stdev=85.76 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 232], 10.00th=[ 259], 20.00th=[ 296], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 384], | 70.00th=[ 409], 80.00th=[ 439], 90.00th=[ 477], 95.00th=[ 510], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 7, max= 136, per=24.66%, avg=56.71, stdev=24.40, samples=173 iops : min= 1, max= 34, avg=13.66, stdev= 6.13, samples=173 write: IOPS=15, BW=60.3KiB/s (61.8kB/s)(5256KiB/87115msec); 0 zone resets slat (usec): min=27, max=8178, avg=287.20, stdev=663.91 clat (msec): min=128, max=1140, avg=705.39, stdev=184.35 lat (msec): min=129, max=1141, avg=705.68, stdev=184.35 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 409], 10.00th=[ 477], 20.00th=[ 550], | 30.00th=[ 600], 40.00th=[ 651], 50.00th=[ 701], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 877], 90.00th=[ 944], 95.00th=[ 1003], | 99.00th=[ 1083], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 119, per=24.86%, avg=59.42, stdev=19.99, samples=173 iops : min= 1, max= 29, avg=14.34, stdev= 5.02, samples=173 lat (msec) : 20=0.08%, 50=0.08%, 100=0.04%, 250=4.18%, 500=47.66% lat (msec) : 750=27.23%, 1000=18.12%, 2000=2.62% cpu : usr=0.12%, sys=0.83%, ctx=2983, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144523: Thu Oct 23 10:08:33 2025 read: IOPS=14, BW=58.4KiB/s (59.8kB/s)(5060KiB/86641msec) slat (usec): min=2064, max=35033, avg=9474.17, stdev=4583.89 clat (msec): min=49, max=627, avg=367.10, stdev=83.84 lat (msec): min=69, max=641, avg=376.58, stdev=84.44 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 239], 10.00th=[ 266], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 342], 50.00th=[ 359], 60.00th=[ 388], | 70.00th=[ 414], 80.00th=[ 443], 90.00th=[ 481], 95.00th=[ 514], | 99.00th=[ 550], 99.50th=[ 567], 99.90th=[ 575], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 7, max= 111, per=25.31%, avg=58.22, stdev=22.51, samples=171 iops : min= 1, max= 27, avg=14.04, stdev= 5.63, samples=171 write: IOPS=14, BW=59.8KiB/s (61.2kB/s)(5180KiB/86641msec); 0 zone resets slat (usec): min=27, max=17161, avg=315.86, stdev=830.18 clat (msec): min=164, max=1306, avg=700.76, stdev=184.12 lat (msec): min=165, max=1306, avg=701.08, stdev=184.14 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 418], 10.00th=[ 477], 20.00th=[ 542], | 30.00th=[ 592], 40.00th=[ 642], 50.00th=[ 693], 60.00th=[ 743], | 70.00th=[ 802], 80.00th=[ 869], 90.00th=[ 953], 95.00th=[ 1011], | 99.00th=[ 1099], 99.50th=[ 1167], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 16, max= 127, per=24.65%, avg=58.92, stdev=21.04, samples=172 iops : min= 4, max= 31, avg=14.22, stdev= 5.31, samples=172 lat (msec) : 50=0.04%, 100=0.04%, 250=3.87%, 500=49.26%, 750=27.11% lat (msec) : 1000=16.80%, 2000=2.89% cpu : usr=0.11%, sys=0.89%, ctx=3177, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=230KiB/s (236kB/s), 57.2KiB/s-58.4KiB/s (58.6kB/s-59.8kB/s), io=19.6MiB (20.5MB), run=86423-87115msec WRITE: bw=240KiB/s (246kB/s), 59.8KiB/s-60.5KiB/s (61.2kB/s-61.9kB/s), io=20.4MiB (21.4MB), run=86423-87115msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144539: Thu Oct 23 10:08:41 2025 read: IOPS=2, BW=83.7MiB/s (87.7MB/s)(40.0MiB/478msec) slat (nsec): min=60038k, max=60038k, avg=60037695.00, stdev= 0.00 clat (nsec): min=392047k, max=392047k, avg=392047059.00, stdev= 0.00 lat (nsec): min=452090k, max=452090k, avg=452089874.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.89%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.7MiB/s (87.7MB/s), 83.7MiB/s-83.7MiB/s (87.7MB/s-87.7MB/s), io=40.0MiB (41.9MB), run=478-478msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck