== sanity test 398c: run fio to test AIO ================= 08:27:40 (1787574460) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.448939 s, 93.4 MB/s osc.lustre-OST0000-osc-ffff91b050758000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145147: Mon Aug 24 08:30:53 2026 write: IOPS=13, BW=55.7KiB/s (57.0kB/s)(10.0MiB/183937msec); 0 zone resets slat (usec): min=27, max=19085, avg=184.78, stdev=734.23 clat (msec): min=160, max=2620, avg=1148.45, stdev=440.79 lat (msec): min=160, max=2620, avg=1148.63, stdev=440.79 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 550], 10.00th=[ 684], 20.00th=[ 802], | 30.00th=[ 902], 40.00th=[ 986], 50.00th=[ 1083], 60.00th=[ 1183], | 70.00th=[ 1301], 80.00th=[ 1502], 90.00th=[ 1804], 95.00th=[ 2005], | 99.00th=[ 2265], 99.50th=[ 2333], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 383, per=25.09%, avg=55.69, stdev=34.18, samples=361 iops : min= 1, max= 95, avg=13.43, stdev= 8.57, samples=361 lat (msec) : 250=2.50%, 500=2.03%, 750=10.98%, 1000=25.78%, 2000=53.63% lat (msec) : >=2000=5.08% cpu : usr=0.03%, sys=0.18%, ctx=1633, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145148: Mon Aug 24 08:30:53 2026 write: IOPS=13, BW=55.6KiB/s (56.0kB/s)(10.0MiB/184028msec); 0 zone resets slat (usec): min=28, max=17560, avg=173.95, stdev=721.72 clat (msec): min=157, max=2621, avg=1149.08, stdev=443.02 lat (msec): min=157, max=2621, avg=1149.25, stdev=443.05 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 502], 10.00th=[ 684], 20.00th=[ 793], | 30.00th=[ 894], 40.00th=[ 995], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1301], 80.00th=[ 1502], 90.00th=[ 1804], 95.00th=[ 1972], | 99.00th=[ 2299], 99.50th=[ 2400], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 383, per=26.04%, avg=57.81, stdev=36.94, samples=348 iops : min= 1, max= 95, avg=13.96, stdev= 9.25, samples=348 lat (msec) : 250=2.50%, 500=2.66%, 750=11.09%, 1000=24.22%, 2000=55.04% lat (msec) : >=2000=4.49% cpu : usr=0.04%, sys=0.18%, ctx=1711, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145149: Mon Aug 24 08:30:53 2026 write: IOPS=13, BW=55.6KiB/s (56.0kB/s)(10.0MiB/184025msec); 0 zone resets slat (usec): min=28, max=18256, avg=202.88, stdev=820.25 clat (msec): min=89, max=2617, avg=1149.01, stdev=441.87 lat (msec): min=90, max=2617, avg=1149.21, stdev=441.88 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 542], 10.00th=[ 684], 20.00th=[ 802], | 30.00th=[ 902], 40.00th=[ 995], 50.00th=[ 1083], 60.00th=[ 1183], | 70.00th=[ 1318], 80.00th=[ 1519], 90.00th=[ 1804], 95.00th=[ 2005], | 99.00th=[ 2299], 99.50th=[ 2366], 99.90th=[ 2467], 99.95th=[ 2534], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 391, per=25.23%, avg=56.02, stdev=34.68, samples=359 iops : min= 1, max= 97, avg=13.51, stdev= 8.69, samples=359 lat (msec) : 100=0.04%, 250=2.50%, 500=1.29%, 750=12.81%, 1000=23.83% lat (msec) : 2000=54.38%, >=2000=5.16% cpu : usr=0.04%, sys=0.17%, ctx=1693, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145150: Mon Aug 24 08:30:53 2026 write: IOPS=13, BW=55.7KiB/s (57.0kB/s)(10.0MiB/183886msec); 0 zone resets slat (usec): min=28, max=28846, avg=204.00, stdev=1069.25 clat (msec): min=85, max=2432, avg=1148.62, stdev=441.75 lat (msec): min=85, max=2432, avg=1148.82, stdev=441.77 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 518], 10.00th=[ 684], 20.00th=[ 802], | 30.00th=[ 894], 40.00th=[ 986], 50.00th=[ 1083], 60.00th=[ 1183], | 70.00th=[ 1318], 80.00th=[ 1519], 90.00th=[ 1804], 95.00th=[ 1972], | 99.00th=[ 2232], 99.50th=[ 2366], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 319, per=25.29%, avg=56.15, stdev=35.02, samples=358 iops : min= 1, max= 79, avg=13.55, stdev= 8.80, samples=358 lat (msec) : 100=0.12%, 250=2.11%, 500=2.34%, 750=10.55%, 1000=26.60% lat (msec) : 2000=53.83%, >=2000=4.45% cpu : usr=0.05%, sys=0.18%, ctx=1830, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=223KiB/s (228kB/s), 55.6KiB/s-55.7KiB/s (56.0kB/s-57.0kB/s), io=40.0MiB (41.9MB), run=183886-184028msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145187: Mon Aug 24 08:32:49 2026 read: IOPS=11, BW=47.5KiB/s (48.6kB/s)(5048KiB/106303msec) slat (usec): min=2870, max=63476, avg=14016.36, stdev=7372.95 clat (msec): min=12, max=831, avg=418.86, stdev=95.31 lat (msec): min=18, max=847, avg=432.89, stdev=96.11 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 393], 50.00th=[ 414], 60.00th=[ 439], | 70.00th=[ 460], 80.00th=[ 489], 90.00th=[ 542], 95.00th=[ 575], | 99.00th=[ 667], 99.50th=[ 709], 99.90th=[ 785], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7, max= 119, per=24.92%, avg=46.85, stdev=21.46, samples=211 iops : min= 1, max= 29, avg=11.24, stdev= 5.37, samples=211 write: IOPS=12, BW=48.8KiB/s (50.0kB/s)(5192KiB/106303msec); 0 zone resets slat (usec): min=29, max=14819, avg=270.84, stdev=854.62 clat (msec): min=152, max=1536, avg=886.68, stdev=213.02 lat (msec): min=152, max=1536, avg=886.96, stdev=213.05 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 567], 10.00th=[ 642], 20.00th=[ 701], | 30.00th=[ 768], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1070], 90.00th=[ 1183], 95.00th=[ 1267], | 99.00th=[ 1452], 99.50th=[ 1469], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 112, per=24.69%, avg=48.40, stdev=20.61, samples=210 iops : min= 1, max= 28, avg=11.62, stdev= 5.24, samples=210 lat (msec) : 20=0.04%, 50=0.12%, 100=0.12%, 250=0.98%, 500=40.82% lat (msec) : 750=20.35%, 1000=23.87%, 2000=13.71% cpu : usr=0.09%, sys=0.57%, ctx=4346, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145188: Mon Aug 24 08:32:49 2026 read: IOPS=11, BW=46.8KiB/s (47.9kB/s)(4976KiB/106266msec) slat (usec): min=48, max=72004, avg=14419.87, stdev=7795.23 clat (msec): min=28, max=821, avg=419.60, stdev=93.87 lat (msec): min=29, max=826, avg=434.03, stdev=95.33 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 388], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 498], 90.00th=[ 550], 95.00th=[ 575], | 99.00th=[ 659], 99.50th=[ 693], 99.90th=[ 802], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 7, max= 104, per=24.69%, avg=46.42, stdev=21.19, samples=210 iops : min= 1, max= 26, avg=11.13, stdev= 5.32, samples=210 write: IOPS=12, BW=49.5KiB/s (50.7kB/s)(5264KiB/106266msec); 0 zone resets slat (usec): min=28, max=11544, avg=280.03, stdev=733.39 clat (msec): min=126, max=1598, avg=880.66, stdev=220.42 lat (msec): min=126, max=1598, avg=880.94, stdev=220.41 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1062], 90.00th=[ 1183], 95.00th=[ 1284], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1552], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 112, per=24.93%, avg=48.85, stdev=20.46, samples=211 iops : min= 1, max= 28, avg=11.74, stdev= 5.15, samples=211 lat (msec) : 50=0.08%, 100=0.16%, 250=0.90%, 500=39.30%, 750=22.46% lat (msec) : 1000=23.01%, 2000=14.10% cpu : usr=0.08%, sys=0.59%, ctx=4374, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145189: Mon Aug 24 08:32:49 2026 read: IOPS=11, BW=46.9KiB/s (48.0kB/s)(4984KiB/106265msec) slat (usec): min=2757, max=60953, avg=14455.50, stdev=7533.89 clat (msec): min=71, max=940, avg=420.83, stdev=96.41 lat (msec): min=77, max=956, avg=435.29, stdev=97.16 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 368], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 439], | 70.00th=[ 464], 80.00th=[ 493], 90.00th=[ 550], 95.00th=[ 592], | 99.00th=[ 667], 99.50th=[ 684], 99.90th=[ 919], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 119, per=24.80%, avg=46.62, stdev=19.44, samples=210 iops : min= 1, max= 29, avg=11.19, stdev= 4.87, samples=210 write: IOPS=12, BW=49.5KiB/s (50.6kB/s)(5256KiB/106265msec); 0 zone resets slat (usec): min=31, max=12159, avg=225.52, stdev=603.14 clat (msec): min=337, max=1553, avg=878.52, stdev=210.26 lat (msec): min=337, max=1553, avg=878.75, stdev=210.29 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 584], 10.00th=[ 634], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 793], 50.00th=[ 852], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1053], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1452], 99.50th=[ 1485], 99.90th=[ 1502], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 111, per=24.86%, avg=48.73, stdev=17.78, samples=211 iops : min= 1, max= 27, avg=11.71, stdev= 4.50, samples=211 lat (msec) : 100=0.16%, 250=0.78%, 500=39.69%, 750=23.20%, 1000=23.16% lat (msec) : 2000=13.01% cpu : usr=0.09%, sys=0.62%, ctx=4350, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145190: Mon Aug 24 08:32:49 2026 read: IOPS=11, BW=47.7KiB/s (48.8kB/s)(5060KiB/106138msec) slat (usec): min=2735, max=48386, avg=13968.86, stdev=7616.24 clat (msec): min=13, max=878, avg=421.50, stdev=95.50 lat (msec): min=23, max=886, avg=435.48, stdev=96.68 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 393], 50.00th=[ 414], 60.00th=[ 439], | 70.00th=[ 464], 80.00th=[ 498], 90.00th=[ 550], 95.00th=[ 584], | 99.00th=[ 659], 99.50th=[ 693], 99.90th=[ 835], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 104, per=25.21%, avg=47.39, stdev=18.64, samples=210 iops : min= 1, max= 26, avg=11.37, stdev= 4.70, samples=210 write: IOPS=12, BW=48.8KiB/s (49.0kB/s)(5180KiB/106138msec); 0 zone resets slat (usec): min=31, max=15231, avg=231.56, stdev=814.40 clat (msec): min=297, max=1566, avg=883.35, stdev=215.93 lat (msec): min=297, max=1566, avg=883.58, stdev=215.94 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 567], 10.00th=[ 625], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1070], 90.00th=[ 1183], 95.00th=[ 1301], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1536], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 95, per=24.58%, avg=48.18, stdev=17.21, samples=210 iops : min= 1, max= 23, avg=11.57, stdev= 4.37, samples=210 lat (msec) : 20=0.08%, 100=0.12%, 250=0.59%, 500=39.92%, 750=23.83% lat (msec) : 1000=21.60%, 2000=13.87% cpu : usr=0.10%, sys=0.59%, ctx=4489, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=189KiB/s (193kB/s), 46.8KiB/s-47.7KiB/s (47.9kB/s-48.8kB/s), io=19.6MiB (20.5MB), run=106138-106303msec WRITE: bw=197KiB/s (201kB/s), 48.8KiB/s-49.5KiB/s (49.0kB/s-50.7kB/s), io=20.4MiB (21.4MB), run=106138-106303msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145209: Mon Aug 24 08:32:59 2026 read: IOPS=1, BW=71.2MiB/s (74.6MB/s)(40.0MiB/562msec) slat (nsec): min=38448k, max=38448k, avg=38448258.00, stdev= 0.00 clat (nsec): min=492624k, max=492624k, avg=492624298.00, stdev= 0.00 lat (nsec): min=531076k, max=531076k, avg=531076479.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 493], 10.00th=[ 493], 20.00th=[ 493], | 30.00th=[ 493], 40.00th=[ 493], 50.00th=[ 493], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.32%, ctx=45, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=71.2MiB/s (74.6MB/s), 71.2MiB/s-71.2MiB/s (74.6MB/s-74.6MB/s), io=40.0MiB (41.9MB), run=562-562msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout