== sanity test 398c: run fio to test AIO ================= 18:35:45 (1763336145) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.742792 s, 56.5 MB/s osc.lustre-OST0000-osc-ffff8ad7c91cf000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147901: Sun Nov 16 18:38:28 2025 write: IOPS=16, BW=66.9KiB/s (68.5kB/s)(10.0MiB/152993msec); 0 zone resets slat (usec): min=27, max=17227, avg=172.88, stdev=639.03 clat (msec): min=195, max=2004, avg=954.84, stdev=252.17 lat (msec): min=195, max=2004, avg=955.02, stdev=252.18 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 567], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1670], 99.50th=[ 1871], 99.90th=[ 1955], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 144, per=25.06%, avg=66.91, stdev=28.57, samples=301 iops : min= 1, max= 36, avg=16.23, stdev= 7.18, samples=301 lat (msec) : 250=0.62%, 500=2.81%, 750=15.20%, 1000=41.84%, 2000=39.45% lat (msec) : >=2000=0.08% cpu : usr=0.04%, sys=0.23%, ctx=1765, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147902: Sun Nov 16 18:38:28 2025 write: IOPS=16, BW=66.9KiB/s (68.5kB/s)(10.0MiB/153070msec); 0 zone resets slat (usec): min=23, max=8862, avg=148.74, stdev=409.11 clat (msec): min=84, max=2013, avg=955.59, stdev=252.35 lat (msec): min=84, max=2013, avg=955.75, stdev=252.33 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 600], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1687], 99.50th=[ 1888], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 152, per=24.81%, avg=66.23, stdev=26.38, samples=304 iops : min= 1, max= 38, avg=16.06, stdev= 6.65, samples=304 lat (msec) : 100=0.08%, 250=0.70%, 500=2.66%, 750=14.88%, 1000=42.34% lat (msec) : 2000=39.30%, >=2000=0.04% cpu : usr=0.04%, sys=0.24%, ctx=1742, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147903: Sun Nov 16 18:38:28 2025 write: IOPS=16, BW=66.9KiB/s (68.5kB/s)(10.0MiB/153071msec); 0 zone resets slat (usec): min=27, max=19856, avg=157.66, stdev=613.49 clat (msec): min=211, max=1990, avg=955.44, stdev=248.81 lat (msec): min=211, max=1990, avg=955.60, stdev=248.80 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 600], 10.00th=[ 659], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1401], | 99.00th=[ 1636], 99.50th=[ 1854], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 128, per=25.31%, avg=67.57, stdev=30.92, samples=298 iops : min= 1, max= 32, avg=16.40, stdev= 7.78, samples=298 lat (msec) : 250=0.62%, 500=2.58%, 750=16.05%, 1000=40.90%, 2000=39.84% cpu : usr=0.04%, sys=0.23%, ctx=1705, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147904: Sun Nov 16 18:38:28 2025 write: IOPS=16, BW=67.0KiB/s (68.6kB/s)(10.0MiB/152755msec); 0 zone resets slat (usec): min=27, max=7257, avg=138.02, stdev=387.22 clat (msec): min=169, max=2012, avg=954.24, stdev=257.44 lat (msec): min=170, max=2012, avg=954.38, stdev=257.45 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 575], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 244, per=25.29%, avg=67.53, stdev=30.56, samples=298 iops : min= 1, max= 61, avg=16.40, stdev= 7.69, samples=298 lat (msec) : 250=1.25%, 500=2.62%, 750=15.47%, 1000=40.70%, 2000=39.84% lat (msec) : >=2000=0.12% cpu : usr=0.05%, sys=0.22%, ctx=1578, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=268KiB/s (274kB/s), 66.9KiB/s-67.0KiB/s (68.5kB/s-68.6kB/s), io=40.0MiB (41.9MB), run=152755-153071msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147937: Sun Nov 16 18:40:13 2025 read: IOPS=13, BW=52.4KiB/s (53.6kB/s)(5048KiB/96380msec) slat (usec): min=2296, max=62707, avg=11690.36, stdev=5361.11 clat (msec): min=13, max=1089, avg=376.37, stdev=88.78 lat (msec): min=22, max=1102, avg=388.06, stdev=89.24 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 257], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 330], 40.00th=[ 351], 50.00th=[ 376], 60.00th=[ 393], | 70.00th=[ 414], 80.00th=[ 439], 90.00th=[ 481], 95.00th=[ 514], | 99.00th=[ 600], 99.50th=[ 642], 99.90th=[ 936], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 15, max= 111, per=25.06%, avg=52.12, stdev=20.60, samples=190 iops : min= 3, max= 27, avg=12.51, stdev= 5.12, samples=190 write: IOPS=13, BW=53.9KiB/s (55.2kB/s)(5192KiB/96380msec); 0 zone resets slat (usec): min=28, max=5064, avg=204.24, stdev=408.00 clat (msec): min=217, max=1727, avg=807.90, stdev=190.76 lat (msec): min=217, max=1727, avg=808.11, stdev=190.77 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 523], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 969], 90.00th=[ 1062], 95.00th=[ 1133], | 99.00th=[ 1301], 99.50th=[ 1334], 99.90th=[ 1687], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 95, per=24.65%, avg=53.24, stdev=17.15, samples=191 iops : min= 1, max= 23, avg=12.80, stdev= 4.30, samples=191 lat (msec) : 20=0.08%, 50=0.08%, 100=0.12%, 250=1.91%, 500=45.98% lat (msec) : 750=21.68%, 1000=21.64%, 2000=8.52% cpu : usr=0.09%, sys=0.62%, ctx=4707, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147938: Sun Nov 16 18:40:13 2025 read: IOPS=12, BW=51.6KiB/s (52.8kB/s)(4976KiB/96425msec) slat (usec): min=2627, max=58526, avg=11517.65, stdev=5429.96 clat (msec): min=15, max=907, avg=380.58, stdev=84.82 lat (msec): min=23, max=912, avg=392.10, stdev=85.41 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 259], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 376], 60.00th=[ 397], | 70.00th=[ 418], 80.00th=[ 443], 90.00th=[ 485], 95.00th=[ 518], | 99.00th=[ 600], 99.50th=[ 642], 99.90th=[ 902], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 7, max= 104, per=24.55%, avg=51.06, stdev=20.48, samples=191 iops : min= 1, max= 26, avg=12.25, stdev= 5.10, samples=191 write: IOPS=13, BW=54.6KiB/s (55.9kB/s)(5264KiB/96425msec); 0 zone resets slat (usec): min=31, max=7752, avg=209.50, stdev=526.60 clat (msec): min=213, max=1367, avg=799.89, stdev=185.32 lat (msec): min=213, max=1367, avg=800.10, stdev=185.36 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 527], 10.00th=[ 584], 20.00th=[ 634], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 961], 90.00th=[ 1053], 95.00th=[ 1116], | 99.00th=[ 1267], 99.50th=[ 1318], 99.90th=[ 1334], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 15, max= 95, per=25.00%, avg=54.01, stdev=19.01, samples=191 iops : min= 3, max= 23, avg=12.99, stdev= 4.75, samples=191 lat (msec) : 20=0.04%, 50=0.04%, 100=0.12%, 250=1.64%, 500=44.84% lat (msec) : 750=23.75%, 1000=22.15%, 2000=7.42% cpu : usr=0.09%, sys=0.59%, ctx=4447, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147939: Sun Nov 16 18:40:13 2025 read: IOPS=12, BW=51.8KiB/s (53.0kB/s)(4984KiB/96281msec) slat (usec): min=36, max=55860, avg=11470.19, stdev=5430.41 clat (msec): min=14, max=1191, avg=376.79, stdev=90.59 lat (msec): min=31, max=1204, avg=388.27, stdev=91.40 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 253], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 376], 60.00th=[ 393], | 70.00th=[ 414], 80.00th=[ 439], 90.00th=[ 481], 95.00th=[ 514], | 99.00th=[ 592], 99.50th=[ 609], 99.90th=[ 1183], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 136, per=24.78%, avg=51.55, stdev=23.71, samples=190 iops : min= 1, max= 34, avg=12.37, stdev= 5.94, samples=190 write: IOPS=13, BW=54.6KiB/s (55.9kB/s)(5256KiB/96281msec); 0 zone resets slat (usec): min=28, max=18728, avg=226.68, stdev=718.73 clat (msec): min=165, max=1660, avg=803.03, stdev=198.83 lat (msec): min=165, max=1660, avg=803.26, stdev=198.83 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 518], 10.00th=[ 567], 20.00th=[ 634], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 885], 80.00th=[ 961], 90.00th=[ 1070], 95.00th=[ 1150], | 99.00th=[ 1284], 99.50th=[ 1318], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 119, per=25.02%, avg=54.04, stdev=20.01, samples=190 iops : min= 1, max= 29, avg=12.99, stdev= 4.98, samples=190 lat (msec) : 20=0.04%, 50=0.12%, 100=0.35%, 250=2.19%, 500=44.69% lat (msec) : 750=22.66%, 1000=21.41%, 2000=8.55% cpu : usr=0.08%, sys=0.66%, ctx=4494, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147940: Sun Nov 16 18:40:13 2025 read: IOPS=13, BW=52.5KiB/s (53.7kB/s)(5060KiB/96417msec) slat (usec): min=2763, max=58138, avg=11474.63, stdev=5413.86 clat (msec): min=8, max=820, avg=379.74, stdev=84.91 lat (msec): min=14, max=838, avg=391.22, stdev=85.57 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 264], 10.00th=[ 284], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 393], | 70.00th=[ 422], 80.00th=[ 443], 90.00th=[ 485], 95.00th=[ 514], | 99.00th=[ 617], 99.50th=[ 642], 99.90th=[ 793], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 7, max= 136, per=25.04%, avg=52.08, stdev=21.54, samples=191 iops : min= 1, max= 34, avg=12.50, stdev= 5.38, samples=191 write: IOPS=13, BW=53.7KiB/s (55.0kB/s)(5180KiB/96417msec); 0 zone resets slat (usec): min=30, max=11351, avg=212.82, stdev=571.60 clat (msec): min=234, max=1435, avg=806.65, stdev=189.32 lat (msec): min=234, max=1435, avg=806.87, stdev=189.34 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 542], 10.00th=[ 584], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 751], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 894], 80.00th=[ 969], 90.00th=[ 1053], 95.00th=[ 1150], | 99.00th=[ 1318], 99.50th=[ 1334], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 111, per=24.55%, avg=53.04, stdev=20.06, samples=191 iops : min= 1, max= 27, avg=12.74, stdev= 4.98, samples=191 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=0.12%, 250=1.64% lat (msec) : 500=45.74%, 750=22.11%, 1000=21.88%, 2000=8.40% cpu : usr=0.09%, sys=0.66%, ctx=4584, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=208KiB/s (213kB/s), 51.6KiB/s-52.5KiB/s (52.8kB/s-53.7kB/s), io=19.6MiB (20.5MB), run=96281-96425msec WRITE: bw=217KiB/s (222kB/s), 53.7KiB/s-54.6KiB/s (55.0kB/s-55.9kB/s), io=20.4MiB (21.4MB), run=96281-96425msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147957: Sun Nov 16 18:40:19 2025 read: IOPS=2, BW=120MiB/s (126MB/s)(40.0MiB/334msec) slat (nsec): min=15649k, max=15649k, avg=15649032.00, stdev= 0.00 clat (nsec): min=299189k, max=299189k, avg=299188505.00, stdev= 0.00 lat (nsec): min=314842k, max=314842k, avg=314842269.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.20%, ctx=4, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=40.0MiB (41.9MB), run=334-334msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout