== sanity test 398c: run fio to test AIO ================= 23:32:15 (1764736335) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.284315 s, 148 MB/s osc.lustre-OST0000-osc-ffff929517170000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147401: Tue Dec 2 23:34:10 2025 write: IOPS=23, BW=92.4KiB/s (94.7kB/s)(10.0MiB/110775msec); 0 zone resets slat (usec): min=23, max=4661, avg=125.54, stdev=260.37 clat (msec): min=242, max=1210, avg=691.54, stdev=157.20 lat (msec): min=242, max=1210, avg=691.67, stdev=157.20 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 422], 10.00th=[ 502], 20.00th=[ 575], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 684], 60.00th=[ 718], | 70.00th=[ 760], 80.00th=[ 818], 90.00th=[ 911], 95.00th=[ 953], | 99.00th=[ 1070], 99.50th=[ 1133], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 31, max= 247, per=24.90%, avg=91.87, stdev=28.77, samples=220 iops : min= 7, max= 61, avg=22.55, stdev= 7.24, samples=220 lat (msec) : 250=0.04%, 500=9.84%, 750=58.09%, 1000=29.34%, 2000=2.70% cpu : usr=0.05%, sys=0.29%, ctx=1807, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147402: Tue Dec 2 23:34:10 2025 write: IOPS=23, BW=92.6KiB/s (94.9kB/s)(10.0MiB/110546msec); 0 zone resets slat (usec): min=27, max=4631, avg=134.67, stdev=298.70 clat (msec): min=169, max=1675, avg=690.17, stdev=166.83 lat (msec): min=169, max=1675, avg=690.30, stdev=166.84 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 422], 10.00th=[ 485], 20.00th=[ 567], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 684], 60.00th=[ 718], | 70.00th=[ 760], 80.00th=[ 818], 90.00th=[ 902], 95.00th=[ 953], | 99.00th=[ 1116], 99.50th=[ 1183], 99.90th=[ 1552], 99.95th=[ 1620], | 99.99th=[ 1670] bw ( KiB/s): min= 16, max= 255, per=25.10%, avg=92.62, stdev=29.98, samples=218 iops : min= 4, max= 63, avg=22.74, stdev= 7.50, samples=218 lat (msec) : 250=1.25%, 500=9.80%, 750=56.99%, 1000=28.91%, 2000=3.05% cpu : usr=0.04%, sys=0.32%, ctx=1809, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147403: Tue Dec 2 23:34:10 2025 write: IOPS=23, BW=92.7KiB/s (94.9kB/s)(10.0MiB/110479msec); 0 zone resets slat (usec): min=24, max=5323, avg=128.17, stdev=253.42 clat (msec): min=190, max=1608, avg=689.98, stdev=169.24 lat (msec): min=190, max=1608, avg=690.11, stdev=169.24 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 418], 10.00th=[ 498], 20.00th=[ 567], | 30.00th=[ 609], 40.00th=[ 651], 50.00th=[ 684], 60.00th=[ 718], | 70.00th=[ 768], 80.00th=[ 818], 90.00th=[ 902], 95.00th=[ 961], | 99.00th=[ 1116], 99.50th=[ 1167], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 23, max= 247, per=24.99%, avg=92.20, stdev=31.44, samples=219 iops : min= 5, max= 61, avg=22.63, stdev= 7.92, samples=219 lat (msec) : 250=1.25%, 500=9.14%, 750=56.56%, 1000=29.96%, 2000=3.09% cpu : usr=0.05%, sys=0.30%, ctx=1982, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147404: Tue Dec 2 23:34:10 2025 write: IOPS=23, BW=92.6KiB/s (94.8kB/s)(10.0MiB/110627msec); 0 zone resets slat (usec): min=26, max=6968, avg=127.86, stdev=281.29 clat (msec): min=183, max=1334, avg=690.76, stdev=160.61 lat (msec): min=183, max=1334, avg=690.89, stdev=160.61 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 439], 10.00th=[ 489], 20.00th=[ 567], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 684], 60.00th=[ 718], | 70.00th=[ 760], 80.00th=[ 818], 90.00th=[ 894], 95.00th=[ 953], | 99.00th=[ 1150], 99.50th=[ 1167], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1334] bw ( KiB/s): min= 15, max= 200, per=24.89%, avg=91.84, stdev=31.25, samples=220 iops : min= 3, max= 50, avg=22.54, stdev= 7.85, samples=220 lat (msec) : 250=0.35%, 500=10.55%, 750=56.25%, 1000=29.96%, 2000=2.89% cpu : usr=0.05%, sys=0.27%, ctx=1947, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=370KiB/s (379kB/s), 92.4KiB/s-92.7KiB/s (94.7kB/s-94.9kB/s), io=40.0MiB (41.9MB), run=110479-110775msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147431: Tue Dec 2 23:35:35 2025 read: IOPS=16, BW=64.6KiB/s (66.2kB/s)(5048KiB/78091msec) slat (usec): min=2228, max=34399, avg=9126.27, stdev=3936.04 clat (msec): min=9, max=575, avg=311.31, stdev=61.21 lat (msec): min=19, max=584, avg=320.44, stdev=61.39 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 222], 10.00th=[ 239], 20.00th=[ 262], | 30.00th=[ 279], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 326], | 70.00th=[ 342], 80.00th=[ 359], 90.00th=[ 388], 95.00th=[ 409], | 99.00th=[ 464], 99.50th=[ 489], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 15, max= 150, per=25.04%, avg=64.11, stdev=25.27, samples=155 iops : min= 3, max= 37, avg=15.63, stdev= 6.35, samples=155 write: IOPS=16, BW=66.5KiB/s (68.1kB/s)(5192KiB/78091msec); 0 zone resets slat (usec): min=32, max=6970, avg=180.93, stdev=432.93 clat (msec): min=255, max=1009, avg=648.57, stdev=146.22 lat (msec): min=256, max=1009, avg=648.76, stdev=146.22 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 393], 10.00th=[ 460], 20.00th=[ 527], | 30.00th=[ 567], 40.00th=[ 609], 50.00th=[ 651], 60.00th=[ 684], | 70.00th=[ 735], 80.00th=[ 776], 90.00th=[ 844], 95.00th=[ 894], | 99.00th=[ 969], 99.50th=[ 986], 99.90th=[ 1003], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 16, max= 136, per=24.71%, avg=65.97, stdev=21.46, samples=155 iops : min= 4, max= 34, avg=16.08, stdev= 5.41, samples=155 lat (msec) : 10=0.04%, 100=0.20%, 250=6.99%, 500=49.92%, 750=29.49% lat (msec) : 1000=13.24%, 2000=0.12% cpu : usr=0.11%, sys=0.78%, ctx=4968, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147432: Tue Dec 2 23:35:35 2025 read: IOPS=15, BW=63.7KiB/s (65.3kB/s)(4976KiB/78083msec) slat (usec): min=2093, max=23544, avg=9251.12, stdev=3876.94 clat (msec): min=14, max=637, avg=309.42, stdev=61.74 lat (msec): min=22, max=644, avg=318.68, stdev=62.15 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 220], 10.00th=[ 236], 20.00th=[ 259], | 30.00th=[ 279], 40.00th=[ 296], 50.00th=[ 309], 60.00th=[ 321], | 70.00th=[ 338], 80.00th=[ 355], 90.00th=[ 384], 95.00th=[ 405], | 99.00th=[ 481], 99.50th=[ 523], 99.90th=[ 617], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 7, max= 159, per=24.70%, avg=63.23, stdev=25.19, samples=155 iops : min= 1, max= 39, avg=15.40, stdev= 6.32, samples=155 write: IOPS=16, BW=67.4KiB/s (69.0kB/s)(5264KiB/78083msec); 0 zone resets slat (usec): min=32, max=6213, avg=165.31, stdev=325.17 clat (msec): min=254, max=1121, avg=645.73, stdev=149.68 lat (msec): min=254, max=1121, avg=645.90, stdev=149.68 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 397], 10.00th=[ 456], 20.00th=[ 523], | 30.00th=[ 567], 40.00th=[ 600], 50.00th=[ 642], 60.00th=[ 684], | 70.00th=[ 735], 80.00th=[ 776], 90.00th=[ 835], 95.00th=[ 894], | 99.00th=[ 986], 99.50th=[ 1053], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 16, max= 136, per=25.04%, avg=66.86, stdev=21.32, samples=155 iops : min= 4, max= 34, avg=16.31, stdev= 5.37, samples=155 lat (msec) : 20=0.08%, 100=0.12%, 250=7.46%, 500=48.75%, 750=30.08% lat (msec) : 1000=13.12%, 2000=0.39% cpu : usr=0.08%, sys=0.75%, ctx=4757, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147433: Tue Dec 2 23:35:35 2025 read: IOPS=15, BW=63.8KiB/s (65.3kB/s)(4984KiB/78151msec) slat (usec): min=2004, max=29798, avg=9218.13, stdev=3943.54 clat (msec): min=52, max=527, avg=309.97, stdev=62.41 lat (msec): min=59, max=535, avg=319.20, stdev=62.66 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 222], 10.00th=[ 239], 20.00th=[ 262], | 30.00th=[ 279], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 321], | 70.00th=[ 338], 80.00th=[ 359], 90.00th=[ 393], 95.00th=[ 418], | 99.00th=[ 468], 99.50th=[ 498], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 15, max= 160, per=24.77%, avg=63.41, stdev=25.50, samples=155 iops : min= 3, max= 40, avg=15.45, stdev= 6.42, samples=155 write: IOPS=16, BW=67.3KiB/s (68.9kB/s)(5256KiB/78151msec); 0 zone resets slat (usec): min=29, max=5993, avg=173.82, stdev=366.53 clat (msec): min=38, max=1048, avg=646.33, stdev=147.54 lat (msec): min=38, max=1049, avg=646.50, stdev=147.54 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 397], 10.00th=[ 464], 20.00th=[ 527], | 30.00th=[ 575], 40.00th=[ 609], 50.00th=[ 642], 60.00th=[ 684], | 70.00th=[ 726], 80.00th=[ 776], 90.00th=[ 835], 95.00th=[ 894], | 99.00th=[ 978], 99.50th=[ 995], 99.90th=[ 1036], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 16, max= 127, per=24.89%, avg=66.46, stdev=20.49, samples=155 iops : min= 4, max= 31, avg=16.21, stdev= 5.16, samples=155 lat (msec) : 50=0.04%, 100=0.31%, 250=7.30%, 500=49.30%, 750=30.39% lat (msec) : 1000=12.42%, 2000=0.23% cpu : usr=0.09%, sys=0.77%, ctx=4495, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147434: Tue Dec 2 23:35:35 2025 read: IOPS=16, BW=65.3KiB/s (66.9kB/s)(5060KiB/77442msec) slat (usec): min=56, max=36903, avg=8944.56, stdev=4108.53 clat (msec): min=14, max=676, avg=307.73, stdev=65.98 lat (msec): min=17, max=681, avg=316.68, stdev=66.49 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 215], 10.00th=[ 234], 20.00th=[ 259], | 30.00th=[ 275], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 321], | 70.00th=[ 338], 80.00th=[ 359], 90.00th=[ 384], 95.00th=[ 414], | 99.00th=[ 464], 99.50th=[ 485], 99.90th=[ 659], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 16, max= 152, per=25.50%, avg=65.27, stdev=25.01, samples=153 iops : min= 4, max= 38, avg=15.91, stdev= 6.26, samples=153 write: IOPS=16, BW=66.9KiB/s (68.5kB/s)(5180KiB/77442msec); 0 zone resets slat (usec): min=31, max=4857, avg=160.37, stdev=300.27 clat (msec): min=135, max=1085, avg=646.42, stdev=149.56 lat (msec): min=135, max=1085, avg=646.58, stdev=149.55 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 393], 10.00th=[ 451], 20.00th=[ 527], | 30.00th=[ 567], 40.00th=[ 609], 50.00th=[ 651], 60.00th=[ 693], | 70.00th=[ 735], 80.00th=[ 768], 90.00th=[ 844], 95.00th=[ 885], | 99.00th=[ 978], 99.50th=[ 1028], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 23, max= 120, per=24.74%, avg=66.05, stdev=18.67, samples=154 iops : min= 5, max= 30, avg=16.10, stdev= 4.77, samples=154 lat (msec) : 20=0.12%, 50=0.16%, 100=0.23%, 250=7.19%, 500=49.65% lat (msec) : 750=30.08%, 1000=12.23%, 2000=0.35% cpu : usr=0.08%, sys=0.75%, ctx=4643, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=257KiB/s (263kB/s), 63.7KiB/s-65.3KiB/s (65.3kB/s-66.9kB/s), io=19.6MiB (20.5MB), run=77442-78151msec WRITE: bw=267KiB/s (274kB/s), 66.5KiB/s-67.4KiB/s (68.1kB/s-69.0kB/s), io=20.4MiB (21.4MB), run=77442-78151msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147450: Tue Dec 2 23:35:40 2025 read: IOPS=3, BW=129MiB/s (135MB/s)(40.0MiB/311msec) slat (nsec): min=26685k, max=26685k, avg=26685038.00, stdev= 0.00 clat (nsec): min=266737k, max=266737k, avg=266736956.00, stdev= 0.00 lat (nsec): min=293426k, max=293426k, avg=293425729.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 268], 10.00th=[ 268], 20.00th=[ 268], | 30.00th=[ 268], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 268], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.23%, ctx=43, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=40.0MiB (41.9MB), run=311-311msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout