== sanity test 398c: run fio to test AIO ================= 23:19:30 (1788837570) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.462563 s, 90.7 MB/s osc.lustre-OST0000-osc-ffff99c28397a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145049: Mon Sep 7 23:22:01 2026 write: IOPS=18, BW=73.9KiB/s (75.6kB/s)(10.0MiB/138624msec); 0 zone resets slat (usec): min=21, max=14065, avg=185.98, stdev=660.83 clat (msec): min=213, max=1680, avg=865.77, stdev=220.63 lat (msec): min=213, max=1680, avg=865.95, stdev=220.64 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 567], 10.00th=[ 609], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 885], | 70.00th=[ 953], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1469], 99.50th=[ 1519], 99.90th=[ 1636], 99.95th=[ 1653], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 128, per=25.15%, avg=73.93, stdev=29.10, samples=273 iops : min= 1, max= 32, avg=18.04, stdev= 7.31, samples=273 lat (msec) : 250=0.62%, 500=1.09%, 750=30.98%, 1000=42.85%, 2000=24.45% cpu : usr=0.05%, sys=0.25%, ctx=1880, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145050: Mon Sep 7 23:22:01 2026 write: IOPS=18, BW=73.7KiB/s (75.5kB/s)(10.0MiB/138951msec); 0 zone resets slat (usec): min=21, max=17132, avg=197.16, stdev=801.70 clat (msec): min=421, max=1684, avg=867.29, stdev=210.19 lat (msec): min=421, max=1685, avg=867.49, stdev=210.20 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 567], 10.00th=[ 617], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1133], 95.00th=[ 1267], | 99.00th=[ 1452], 99.50th=[ 1502], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 128, per=25.06%, avg=73.68, stdev=30.90, samples=274 iops : min= 1, max= 32, avg=17.98, stdev= 7.79, samples=274 lat (msec) : 500=1.09%, 750=29.65%, 1000=45.20%, 2000=24.06% cpu : usr=0.06%, sys=0.23%, ctx=2052, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145051: Mon Sep 7 23:22:01 2026 write: IOPS=18, BW=73.8KiB/s (75.5kB/s)(10.0MiB/138825msec); 0 zone resets slat (usec): min=23, max=13885, avg=204.65, stdev=800.67 clat (msec): min=146, max=1659, avg=866.60, stdev=215.20 lat (msec): min=146, max=1660, avg=866.80, stdev=215.23 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 567], 10.00th=[ 609], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1452], 99.50th=[ 1502], 99.90th=[ 1586], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 135, per=24.96%, avg=73.39, stdev=27.71, samples=275 iops : min= 1, max= 33, avg=17.91, stdev= 6.95, samples=275 lat (msec) : 250=0.12%, 500=1.45%, 750=29.73%, 1000=43.75%, 2000=24.96% cpu : usr=0.07%, sys=0.23%, ctx=2259, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145052: Mon Sep 7 23:22:01 2026 write: IOPS=18, BW=73.7KiB/s (75.5kB/s)(10.0MiB/138879msec); 0 zone resets slat (usec): min=22, max=15792, avg=183.83, stdev=648.86 clat (msec): min=354, max=1624, avg=866.97, stdev=211.91 lat (msec): min=354, max=1624, avg=867.15, stdev=211.91 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 567], 10.00th=[ 609], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 835], 60.00th=[ 885], | 70.00th=[ 961], 80.00th=[ 1053], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1452], 99.50th=[ 1519], 99.90th=[ 1536], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 135, per=25.16%, avg=73.97, stdev=29.79, samples=273 iops : min= 1, max= 33, avg=18.05, stdev= 7.49, samples=273 lat (msec) : 500=0.94%, 750=30.78%, 1000=43.20%, 2000=25.08% cpu : usr=0.05%, sys=0.25%, ctx=1971, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=295KiB/s (302kB/s), 73.7KiB/s-73.9KiB/s (75.5kB/s-75.6kB/s), io=40.0MiB (41.9MB), run=138624-138951msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145082: Mon Sep 7 23:23:45 2026 read: IOPS=13, BW=53.0KiB/s (54.3kB/s)(5048KiB/95193msec) slat (usec): min=2716, max=76144, avg=13119.60, stdev=7597.69 clat (msec): min=16, max=1112, avg=369.24, stdev=97.39 lat (msec): min=22, max=1122, avg=382.37, stdev=98.62 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 239], 10.00th=[ 271], 20.00th=[ 300], | 30.00th=[ 321], 40.00th=[ 338], 50.00th=[ 355], 60.00th=[ 380], | 70.00th=[ 401], 80.00th=[ 426], 90.00th=[ 485], 95.00th=[ 542], | 99.00th=[ 659], 99.50th=[ 743], 99.90th=[ 995], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 128, per=24.95%, avg=52.39, stdev=23.73, samples=189 iops : min= 1, max= 32, avg=12.65, stdev= 5.99, samples=189 write: IOPS=13, BW=54.5KiB/s (55.8kB/s)(5192KiB/95193msec); 0 zone resets slat (usec): min=27, max=12968, avg=216.93, stdev=713.07 clat (msec): min=273, max=1733, avg=798.70, stdev=206.80 lat (msec): min=273, max=1733, avg=798.92, stdev=206.83 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 498], 10.00th=[ 550], 20.00th=[ 634], | 30.00th=[ 693], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 953], 90.00th=[ 1045], 95.00th=[ 1167], | 99.00th=[ 1401], 99.50th=[ 1586], 99.90th=[ 1720], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 119, per=24.79%, avg=54.28, stdev=20.87, samples=188 iops : min= 1, max= 29, avg=13.11, stdev= 5.29, samples=188 lat (msec) : 20=0.04%, 50=0.08%, 100=0.12%, 250=3.01%, 500=44.65% lat (msec) : 750=22.93%, 1000=22.15%, 2000=7.03% cpu : usr=0.13%, sys=0.66%, ctx=4901, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145083: Mon Sep 7 23:23:45 2026 read: IOPS=13, BW=52.2KiB/s (53.5kB/s)(4976KiB/95273msec) slat (usec): min=2924, max=59090, avg=13077.45, stdev=7342.49 clat (msec): min=6, max=1147, avg=370.04, stdev=100.79 lat (msec): min=15, max=1151, avg=383.12, stdev=101.69 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 234], 10.00th=[ 264], 20.00th=[ 300], | 30.00th=[ 321], 40.00th=[ 342], 50.00th=[ 359], 60.00th=[ 376], | 70.00th=[ 405], 80.00th=[ 443], 90.00th=[ 485], 95.00th=[ 527], | 99.00th=[ 667], 99.50th=[ 793], 99.90th=[ 1116], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 127, per=24.82%, avg=52.12, stdev=24.34, samples=187 iops : min= 1, max= 31, avg=12.58, stdev= 6.09, samples=187 write: IOPS=13, BW=55.3KiB/s (56.6kB/s)(5264KiB/95273msec); 0 zone resets slat (usec): min=24, max=20296, avg=210.50, stdev=801.46 clat (msec): min=232, max=1782, avg=793.41, stdev=206.53 lat (msec): min=232, max=1782, avg=793.62, stdev=206.53 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 498], 10.00th=[ 550], 20.00th=[ 617], | 30.00th=[ 684], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 827], | 70.00th=[ 877], 80.00th=[ 953], 90.00th=[ 1036], 95.00th=[ 1150], | 99.00th=[ 1401], 99.50th=[ 1452], 99.90th=[ 1703], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 8, max= 128, per=25.03%, avg=54.81, stdev=20.23, samples=189 iops : min= 2, max= 32, avg=13.25, stdev= 5.13, samples=189 lat (msec) : 10=0.04%, 50=0.16%, 100=0.20%, 250=2.93%, 500=44.06% lat (msec) : 750=22.66%, 1000=22.85%, 2000=7.11% cpu : usr=0.10%, sys=0.62%, ctx=4123, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145084: Mon Sep 7 23:23:45 2026 read: IOPS=13, BW=52.4KiB/s (53.6kB/s)(4984KiB/95196msec) slat (usec): min=2652, max=59493, avg=13344.52, stdev=7683.45 clat (msec): min=27, max=806, avg=368.30, stdev=97.27 lat (msec): min=44, max=817, avg=381.65, stdev=98.18 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 239], 10.00th=[ 266], 20.00th=[ 300], | 30.00th=[ 321], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 430], 90.00th=[ 485], 95.00th=[ 542], | 99.00th=[ 676], 99.50th=[ 709], 99.90th=[ 793], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 7, max= 127, per=24.72%, avg=51.91, stdev=24.29, samples=189 iops : min= 1, max= 31, avg=12.52, stdev= 6.14, samples=189 write: IOPS=13, BW=55.2KiB/s (56.5kB/s)(5256KiB/95196msec); 0 zone resets slat (usec): min=27, max=22413, avg=223.84, stdev=891.42 clat (msec): min=240, max=1713, avg=794.69, stdev=203.76 lat (msec): min=240, max=1713, avg=794.92, stdev=203.74 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 489], 10.00th=[ 550], 20.00th=[ 634], | 30.00th=[ 684], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 953], 90.00th=[ 1020], 95.00th=[ 1116], | 99.00th=[ 1418], 99.50th=[ 1586], 99.90th=[ 1687], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 112, per=24.88%, avg=54.49, stdev=19.96, samples=189 iops : min= 1, max= 28, avg=13.16, stdev= 5.02, samples=189 lat (msec) : 50=0.08%, 100=0.43%, 250=2.70%, 500=44.41%, 750=22.19% lat (msec) : 1000=23.40%, 2000=6.80% cpu : usr=0.10%, sys=0.68%, ctx=4805, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145085: Mon Sep 7 23:23:45 2026 read: IOPS=13, BW=53.5KiB/s (54.8kB/s)(5060KiB/94551msec) slat (usec): min=48, max=75278, avg=12991.28, stdev=7422.79 clat (msec): min=17, max=941, avg=368.93, stdev=100.76 lat (msec): min=18, max=958, avg=381.92, stdev=101.91 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 239], 10.00th=[ 268], 20.00th=[ 296], | 30.00th=[ 321], 40.00th=[ 342], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 401], 80.00th=[ 430], 90.00th=[ 481], 95.00th=[ 542], | 99.00th=[ 709], 99.50th=[ 768], 99.90th=[ 844], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 112, per=25.64%, avg=53.84, stdev=21.28, samples=185 iops : min= 1, max= 28, avg=13.01, stdev= 5.37, samples=185 write: IOPS=13, BW=54.8KiB/s (56.1kB/s)(5180KiB/94551msec); 0 zone resets slat (usec): min=24, max=15433, avg=215.65, stdev=818.51 clat (msec): min=95, max=1986, avg=794.11, stdev=218.74 lat (msec): min=96, max=1986, avg=794.33, stdev=218.78 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 485], 10.00th=[ 542], 20.00th=[ 625], | 30.00th=[ 684], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 953], 90.00th=[ 1036], 95.00th=[ 1150], | 99.00th=[ 1435], 99.50th=[ 1620], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 103, per=24.64%, avg=53.96, stdev=19.94, samples=188 iops : min= 1, max= 25, avg=13.04, stdev= 5.02, samples=188 lat (msec) : 20=0.08%, 50=0.35%, 100=0.23%, 250=2.97%, 500=44.61% lat (msec) : 750=22.23%, 1000=22.93%, 2000=6.60% cpu : usr=0.12%, sys=0.64%, ctx=4346, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=211KiB/s (216kB/s), 52.2KiB/s-53.5KiB/s (53.5kB/s-54.8kB/s), io=19.6MiB (20.5MB), run=94551-95273msec WRITE: bw=219KiB/s (225kB/s), 54.5KiB/s-55.3KiB/s (55.8kB/s-56.6kB/s), io=20.4MiB (21.4MB), run=94551-95273msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145103: Mon Sep 7 23:23:56 2026 read: IOPS=1, BW=54.8MiB/s (57.5MB/s)(40.0MiB/730msec) slat (nsec): min=50279k, max=50279k, avg=50278770.00, stdev= 0.00 clat (nsec): min=646692k, max=646692k, avg=646691677.00, stdev= 0.00 lat (nsec): min=696974k, max=696974k, avg=696974366.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 651], 5.00th=[ 651], 10.00th=[ 651], 20.00th=[ 651], | 30.00th=[ 651], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 651], | 70.00th=[ 651], 80.00th=[ 651], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.51%, ctx=11, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=54.8MiB/s (57.5MB/s), 54.8MiB/s-54.8MiB/s (57.5MB/s-57.5MB/s), io=40.0MiB (41.9MB), run=730-730msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout