== sanity test 398c: run fio to test AIO ================= 16:24:09 (1760127849) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.305441 s, 137 MB/s osc.lustre-OST0000-osc-ffff93698619e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=153907: Fri Oct 10 16:26:41 2025 write: IOPS=17, BW=71.3KiB/s (72.0kB/s)(10.0MiB/143654msec); 0 zone resets slat (usec): min=27, max=14744, avg=196.60, stdev=645.71 clat (msec): min=151, max=1849, avg=897.25, stdev=207.61 lat (msec): min=151, max=1849, avg=897.44, stdev=207.63 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 592], 10.00th=[ 676], 20.00th=[ 751], | 30.00th=[ 802], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1133], 95.00th=[ 1217], | 99.00th=[ 1435], 99.50th=[ 1536], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1854] bw ( KiB/s): min= 15, max= 234, per=24.88%, avg=70.66, stdev=26.13, samples=285 iops : min= 3, max= 58, avg=17.18, stdev= 6.52, samples=285 lat (msec) : 250=1.29%, 500=2.34%, 750=16.41%, 1000=51.48%, 2000=28.48% cpu : usr=0.06%, sys=0.28%, ctx=1800, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153908: Fri Oct 10 16:26:41 2025 write: IOPS=17, BW=71.0KiB/s (72.7kB/s)(10.0MiB/144137msec); 0 zone resets slat (usec): min=26, max=9733, avg=178.27, stdev=507.62 clat (msec): min=211, max=1637, avg=899.67, stdev=198.99 lat (msec): min=211, max=1637, avg=899.85, stdev=198.99 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 609], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 810], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 936], | 70.00th=[ 986], 80.00th=[ 1053], 90.00th=[ 1133], 95.00th=[ 1234], | 99.00th=[ 1418], 99.50th=[ 1469], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1636] bw ( KiB/s): min= 23, max= 254, per=24.99%, avg=70.96, stdev=25.39, samples=284 iops : min= 5, max= 63, avg=17.25, stdev= 6.38, samples=284 lat (msec) : 250=0.78%, 500=2.30%, 750=15.86%, 1000=53.36%, 2000=27.70% cpu : usr=0.04%, sys=0.28%, ctx=1724, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153909: Fri Oct 10 16:26:41 2025 write: IOPS=17, BW=71.1KiB/s (72.8kB/s)(10.0MiB/144096msec); 0 zone resets slat (usec): min=27, max=11166, avg=158.82, stdev=438.72 clat (msec): min=230, max=1649, avg=899.59, stdev=197.79 lat (msec): min=230, max=1650, avg=899.75, stdev=197.78 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 592], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 810], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1217], | 99.00th=[ 1401], 99.50th=[ 1502], 99.90th=[ 1603], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 183, per=24.89%, avg=70.70, stdev=27.80, samples=285 iops : min= 1, max= 45, avg=17.19, stdev= 6.96, samples=285 lat (msec) : 250=0.90%, 500=2.23%, 750=15.00%, 1000=53.63%, 2000=28.24% cpu : usr=0.06%, sys=0.26%, ctx=1715, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153910: Fri Oct 10 16:26:41 2025 write: IOPS=17, BW=71.2KiB/s (72.9kB/s)(10.0MiB/143832msec); 0 zone resets slat (usec): min=27, max=12523, avg=194.95, stdev=543.26 clat (msec): min=75, max=1698, avg=897.96, stdev=204.63 lat (msec): min=75, max=1698, avg=898.16, stdev=204.64 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 584], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 802], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1217], | 99.00th=[ 1418], 99.50th=[ 1485], 99.90th=[ 1569], 99.95th=[ 1620], | 99.99th=[ 1703] bw ( KiB/s): min= 8, max= 199, per=24.89%, avg=70.67, stdev=23.46, samples=285 iops : min= 2, max= 49, avg=17.18, stdev= 5.88, samples=285 lat (msec) : 100=0.04%, 250=1.29%, 500=1.88%, 750=16.05%, 1000=52.34% lat (msec) : 2000=28.40% cpu : usr=0.04%, sys=0.29%, ctx=1703, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=284KiB/s (291kB/s), 71.0KiB/s-71.3KiB/s (72.7kB/s-72.0kB/s), io=40.0MiB (41.9MB), run=143654-144137msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=153940: Fri Oct 10 16:28:37 2025 read: IOPS=11, BW=46.8KiB/s (47.0kB/s)(5048KiB/107786msec) slat (usec): min=55, max=483083, avg=13610.61, stdev=14698.55 clat (msec): min=27, max=1335, avg=428.90, stdev=110.87 lat (msec): min=29, max=1344, avg=442.52, stdev=111.17 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 443], | 70.00th=[ 472], 80.00th=[ 510], 90.00th=[ 558], 95.00th=[ 600], | 99.00th=[ 676], 99.50th=[ 760], 99.90th=[ 1217], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 103, per=25.22%, avg=46.41, stdev=20.38, samples=213 iops : min= 1, max= 25, avg=11.09, stdev= 5.15, samples=213 write: IOPS=12, BW=48.2KiB/s (49.3kB/s)(5192KiB/107786msec); 0 zone resets slat (usec): min=27, max=12510, avg=260.20, stdev=684.77 clat (msec): min=168, max=1959, avg=896.89, stdev=238.79 lat (msec): min=170, max=1959, avg=897.15, stdev=238.78 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 527], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 986], 80.00th=[ 1083], 90.00th=[ 1183], 95.00th=[ 1318], | 99.00th=[ 1569], 99.50th=[ 1670], 99.90th=[ 1905], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 112, per=24.90%, avg=47.82, stdev=20.67, samples=212 iops : min= 1, max= 28, avg=11.44, stdev= 5.23, samples=212 lat (msec) : 50=0.23%, 100=0.16%, 250=1.13%, 500=38.91%, 750=21.91% lat (msec) : 1000=22.93%, 2000=14.73% cpu : usr=0.10%, sys=0.62%, ctx=4091, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=153941: Fri Oct 10 16:28:37 2025 read: IOPS=11, BW=45.8KiB/s (46.9kB/s)(4976KiB/108675msec) slat (usec): min=2461, max=62726, avg=13609.25, stdev=6257.46 clat (msec): min=15, max=783, avg=429.49, stdev=91.97 lat (msec): min=21, max=797, avg=443.11, stdev=92.80 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 355], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 447], | 70.00th=[ 472], 80.00th=[ 502], 90.00th=[ 550], 95.00th=[ 584], | 99.00th=[ 659], 99.50th=[ 676], 99.90th=[ 718], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 7, max= 112, per=24.69%, avg=45.43, stdev=21.38, samples=212 iops : min= 1, max= 28, avg=10.84, stdev= 5.43, samples=212 write: IOPS=12, BW=48.4KiB/s (49.6kB/s)(5264KiB/108675msec); 0 zone resets slat (usec): min=30, max=6294, avg=232.12, stdev=534.38 clat (msec): min=201, max=1658, avg=899.98, stdev=229.71 lat (msec): min=202, max=1658, avg=900.22, stdev=229.72 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 527], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 844], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 1011], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1301], | 99.00th=[ 1519], 99.50th=[ 1569], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 104, per=24.86%, avg=47.73, stdev=19.89, samples=215 iops : min= 1, max= 26, avg=11.42, stdev= 5.03, samples=215 lat (msec) : 20=0.04%, 100=0.20%, 250=0.51%, 500=39.57%, 750=20.98% lat (msec) : 1000=22.73%, 2000=15.98% cpu : usr=0.11%, sys=0.62%, ctx=3979, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=153942: Fri Oct 10 16:28:37 2025 read: IOPS=11, BW=45.0KiB/s (47.1kB/s)(4984KiB/108462msec) slat (usec): min=2780, max=56850, avg=13494.00, stdev=6020.87 clat (msec): min=9, max=745, avg=430.56, stdev=96.55 lat (msec): min=17, max=756, avg=444.06, stdev=97.13 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 447], | 70.00th=[ 477], 80.00th=[ 510], 90.00th=[ 558], 95.00th=[ 592], | 99.00th=[ 667], 99.50th=[ 684], 99.90th=[ 709], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 7, max= 104, per=25.18%, avg=46.32, stdev=20.97, samples=211 iops : min= 1, max= 26, avg=11.08, stdev= 5.30, samples=211 write: IOPS=12, BW=48.5KiB/s (49.6kB/s)(5256KiB/108462msec); 0 zone resets slat (usec): min=29, max=11394, avg=234.48, stdev=617.26 clat (msec): min=313, max=1641, avg=896.33, stdev=221.08 lat (msec): min=314, max=1642, avg=896.57, stdev=221.11 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 986], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1485], 99.50th=[ 1569], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 112, per=24.97%, avg=47.95, stdev=19.35, samples=214 iops : min= 1, max= 28, avg=11.47, stdev= 4.89, samples=214 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=0.04%, 250=0.82% lat (msec) : 500=38.12%, 750=23.01%, 1000=23.36%, 2000=14.53% cpu : usr=0.09%, sys=0.65%, ctx=4038, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=153943: Fri Oct 10 16:28:37 2025 read: IOPS=11, BW=46.9KiB/s (48.0kB/s)(5060KiB/107943msec) slat (usec): min=2971, max=63944, avg=13305.35, stdev=6083.50 clat (msec): min=15, max=1002, avg=432.47, stdev=98.60 lat (msec): min=20, max=1016, avg=445.79, stdev=99.53 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 355], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 451], | 70.00th=[ 477], 80.00th=[ 518], 90.00th=[ 558], 95.00th=[ 600], | 99.00th=[ 667], 99.50th=[ 676], 99.90th=[ 743], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 112, per=25.55%, avg=47.01, stdev=21.17, samples=211 iops : min= 1, max= 28, avg=11.24, stdev= 5.36, samples=211 write: IOPS=11, BW=47.0KiB/s (49.1kB/s)(5180KiB/107943msec); 0 zone resets slat (usec): min=29, max=6135, avg=210.46, stdev=434.75 clat (msec): min=351, max=1618, avg=895.46, stdev=221.24 lat (msec): min=351, max=1619, avg=895.67, stdev=221.25 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 550], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1183], 95.00th=[ 1284], | 99.00th=[ 1485], 99.50th=[ 1519], 99.90th=[ 1603], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 15, max= 104, per=24.61%, avg=47.25, stdev=18.58, samples=214 iops : min= 3, max= 26, avg=11.30, stdev= 4.70, samples=214 lat (msec) : 20=0.04%, 50=0.12%, 100=0.16%, 250=0.66%, 500=38.36% lat (msec) : 750=23.44%, 1000=21.99%, 2000=15.23% cpu : usr=0.10%, sys=0.69%, ctx=4290, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=185KiB/s (189kB/s), 45.8KiB/s-46.9KiB/s (46.9kB/s-48.0kB/s), io=19.6MiB (20.5MB), run=107786-108675msec WRITE: bw=192KiB/s (197kB/s), 47.0KiB/s-48.5KiB/s (49.1kB/s-49.6kB/s), io=20.4MiB (21.4MB), run=107786-108675msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=153969: Fri Oct 10 16:28:48 2025 read: IOPS=1, BW=68.5MiB/s (71.8MB/s)(40.0MiB/584msec) slat (nsec): min=25064k, max=25064k, avg=25064229.00, stdev= 0.00 clat (nsec): min=526391k, max=526391k, avg=526390638.00, stdev= 0.00 lat (nsec): min=551461k, max=551461k, avg=551461087.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.23%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=68.5MiB/s (71.8MB/s), 68.5MiB/s-68.5MiB/s (71.8MB/s-71.8MB/s), io=40.0MiB (41.9MB), run=584-584msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout