== sanity test 398c: run fio to test AIO ================= 15:15:11 (1782760511) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.329528 s, 127 MB/s osc.lustre-OST0000-osc-ffff995fc4bed800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147053: Mon Jun 29 15:18:17 2026 write: IOPS=14, BW=58.4KiB/s (59.8kB/s)(10.0MiB/175418msec); 0 zone resets slat (usec): min=36, max=23774, avg=258.04, stdev=937.27 clat (msec): min=316, max=2433, avg=1095.45, stdev=318.23 lat (msec): min=316, max=2433, avg=1095.71, stdev=318.22 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 676], 10.00th=[ 760], 20.00th=[ 852], | 30.00th=[ 927], 40.00th=[ 986], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1502], 95.00th=[ 1703], | 99.00th=[ 2140], 99.50th=[ 2232], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2433] bw ( KiB/s): min= 8, max= 128, per=26.12%, avg=60.87, stdev=28.55, samples=331 iops : min= 2, max= 32, avg=14.79, stdev= 7.18, samples=331 lat (msec) : 500=1.72%, 750=7.34%, 1000=33.79%, 2000=55.43%, >=2000=1.72% cpu : usr=0.04%, sys=0.24%, ctx=1652, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147054: Mon Jun 29 15:18:17 2026 write: IOPS=14, BW=58.3KiB/s (59.7kB/s)(10.0MiB/175522msec); 0 zone resets slat (usec): min=35, max=23439, avg=278.17, stdev=1027.49 clat (msec): min=369, max=2417, avg=1095.73, stdev=315.87 lat (msec): min=369, max=2417, avg=1096.01, stdev=315.87 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 684], 10.00th=[ 768], 20.00th=[ 852], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1519], 95.00th=[ 1720], | 99.00th=[ 2056], 99.50th=[ 2165], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 144, per=26.04%, avg=60.67, stdev=29.18, samples=332 iops : min= 1, max= 36, avg=14.75, stdev= 7.33, samples=332 lat (msec) : 500=1.72%, 750=6.91%, 1000=34.06%, 2000=55.66%, >=2000=1.64% cpu : usr=0.04%, sys=0.24%, ctx=2010, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147055: Mon Jun 29 15:18:17 2026 write: IOPS=14, BW=58.4KiB/s (59.8kB/s)(10.0MiB/175355msec); 0 zone resets slat (usec): min=37, max=28867, avg=270.69, stdev=1012.58 clat (msec): min=236, max=2417, avg=1095.10, stdev=321.06 lat (msec): min=237, max=2417, avg=1095.37, stdev=321.05 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 676], 10.00th=[ 760], 20.00th=[ 844], | 30.00th=[ 927], 40.00th=[ 986], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1519], 95.00th=[ 1703], | 99.00th=[ 2089], 99.50th=[ 2198], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 136, per=25.73%, avg=59.94, stdev=33.76, samples=336 iops : min= 1, max= 34, avg=14.56, stdev= 8.47, samples=336 lat (msec) : 250=0.62%, 500=0.74%, 750=7.81%, 1000=33.40%, 2000=55.43% lat (msec) : >=2000=1.99% cpu : usr=0.02%, sys=0.26%, ctx=2040, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147056: Mon Jun 29 15:18:17 2026 write: IOPS=14, BW=58.3KiB/s (59.7kB/s)(10.0MiB/175520msec); 0 zone resets slat (usec): min=36, max=25619, avg=300.23, stdev=1064.91 clat (msec): min=287, max=2418, avg=1095.60, stdev=315.52 lat (msec): min=288, max=2418, avg=1095.90, stdev=315.48 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 693], 10.00th=[ 768], 20.00th=[ 852], | 30.00th=[ 927], 40.00th=[ 978], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1502], 95.00th=[ 1703], | 99.00th=[ 2165], 99.50th=[ 2232], 99.90th=[ 2299], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 136, per=26.70%, avg=62.21, stdev=31.79, samples=324 iops : min= 1, max= 34, avg=15.12, stdev= 7.97, samples=324 lat (msec) : 500=1.84%, 750=6.09%, 1000=35.47%, 2000=54.53%, >=2000=2.07% cpu : usr=0.03%, sys=0.24%, ctx=2251, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=233KiB/s (239kB/s), 58.3KiB/s-58.4KiB/s (59.7kB/s-59.8kB/s), io=40.0MiB (41.9MB), run=175355-175522msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147092: Mon Jun 29 15:20:05 2026 read: IOPS=12, BW=51.3KiB/s (52.6kB/s)(5048KiB/98350msec) slat (usec): min=2206, max=79660, avg=14100.20, stdev=8888.30 clat (msec): min=28, max=1099, avg=377.23, stdev=130.37 lat (msec): min=35, max=1160, avg=391.34, stdev=132.88 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 203], 10.00th=[ 232], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 334], 50.00th=[ 368], 60.00th=[ 393], | 70.00th=[ 422], 80.00th=[ 460], 90.00th=[ 527], 95.00th=[ 609], | 99.00th=[ 768], 99.50th=[ 995], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 119, per=25.41%, avg=51.57, stdev=24.29, samples=192 iops : min= 1, max= 29, avg=12.43, stdev= 6.12, samples=192 write: IOPS=13, BW=52.8KiB/s (54.1kB/s)(5192KiB/98350msec); 0 zone resets slat (usec): min=44, max=14995, avg=352.78, stdev=942.18 clat (msec): min=268, max=2146, avg=828.65, stdev=284.80 lat (msec): min=269, max=2146, avg=829.01, stdev=284.87 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 468], 10.00th=[ 523], 20.00th=[ 600], | 30.00th=[ 676], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 1020], 90.00th=[ 1133], 95.00th=[ 1334], | 99.00th=[ 1905], 99.50th=[ 2056], 99.90th=[ 2106], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 112, per=24.64%, avg=52.24, stdev=23.88, samples=195 iops : min= 1, max= 28, avg=12.59, stdev= 6.05, samples=195 lat (msec) : 50=0.12%, 100=0.12%, 250=6.29%, 500=39.69%, 750=24.02% lat (msec) : 1000=18.95%, 2000=10.47%, >=2000=0.35% cpu : usr=0.10%, sys=0.78%, ctx=5056, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147093: Mon Jun 29 15:20:05 2026 read: IOPS=12, BW=50.6KiB/s (51.8kB/s)(4976KiB/98418msec) slat (usec): min=2369, max=95887, avg=14144.85, stdev=9307.38 clat (msec): min=32, max=1088, avg=385.64, stdev=132.26 lat (msec): min=42, max=1145, avg=399.80, stdev=134.80 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 207], 10.00th=[ 241], 20.00th=[ 284], | 30.00th=[ 313], 40.00th=[ 342], 50.00th=[ 376], 60.00th=[ 401], | 70.00th=[ 426], 80.00th=[ 464], 90.00th=[ 550], 95.00th=[ 634], | 99.00th=[ 818], 99.50th=[ 919], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 160, per=24.60%, avg=49.93, stdev=24.71, samples=195 iops : min= 1, max= 40, avg=12.01, stdev= 6.24, samples=195 write: IOPS=13, BW=53.5KiB/s (54.8kB/s)(5264KiB/98418msec); 0 zone resets slat (usec): min=54, max=35888, avg=403.87, stdev=1606.06 clat (msec): min=264, max=2057, avg=815.93, stdev=280.62 lat (msec): min=265, max=2057, avg=816.34, stdev=280.70 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 451], 10.00th=[ 502], 20.00th=[ 592], | 30.00th=[ 659], 40.00th=[ 726], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 902], 80.00th=[ 1011], 90.00th=[ 1167], 95.00th=[ 1301], | 99.00th=[ 1821], 99.50th=[ 2005], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 136, per=25.34%, avg=53.72, stdev=22.84, samples=192 iops : min= 1, max= 34, avg=12.97, stdev= 5.78, samples=192 lat (msec) : 50=0.08%, 100=0.12%, 250=5.20%, 500=41.45%, 750=24.22% lat (msec) : 1000=18.20%, 2000=10.47%, >=2000=0.27% cpu : usr=0.11%, sys=0.82%, ctx=4972, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147094: Mon Jun 29 15:20:05 2026 read: IOPS=12, BW=50.6KiB/s (51.8kB/s)(4984KiB/98505msec) slat (usec): min=57, max=63625, avg=14150.42, stdev=8677.02 clat (msec): min=26, max=1087, avg=391.02, stdev=140.14 lat (msec): min=32, max=1148, avg=405.18, stdev=142.76 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 201], 10.00th=[ 249], 20.00th=[ 284], | 30.00th=[ 313], 40.00th=[ 347], 50.00th=[ 376], 60.00th=[ 405], | 70.00th=[ 435], 80.00th=[ 481], 90.00th=[ 567], 95.00th=[ 642], | 99.00th=[ 877], 99.50th=[ 1003], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 120, per=24.64%, avg=50.02, stdev=24.13, samples=195 iops : min= 1, max= 30, avg=12.04, stdev= 6.05, samples=195 write: IOPS=13, BW=53.4KiB/s (54.6kB/s)(5256KiB/98505msec); 0 zone resets slat (usec): min=46, max=29447, avg=461.06, stdev=1679.68 clat (msec): min=111, max=2080, avg=814.09, stdev=277.94 lat (msec): min=111, max=2081, avg=814.55, stdev=277.97 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 456], 10.00th=[ 510], 20.00th=[ 600], | 30.00th=[ 667], 40.00th=[ 718], 50.00th=[ 776], 60.00th=[ 835], | 70.00th=[ 902], 80.00th=[ 978], 90.00th=[ 1150], 95.00th=[ 1318], | 99.00th=[ 1871], 99.50th=[ 1955], 99.90th=[ 2072], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 111, per=24.82%, avg=52.63, stdev=23.58, samples=194 iops : min= 1, max= 27, avg=12.69, stdev= 5.95, samples=194 lat (msec) : 50=0.27%, 100=0.20%, 250=4.80%, 500=39.73%, 750=26.13% lat (msec) : 1000=18.98%, 2000=9.69%, >=2000=0.20% cpu : usr=0.10%, sys=0.84%, ctx=5020, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147095: Mon Jun 29 15:20:05 2026 read: IOPS=12, BW=51.7KiB/s (52.9kB/s)(5060KiB/97907msec) slat (usec): min=2419, max=76628, avg=13961.08, stdev=8583.03 clat (msec): min=41, max=1108, avg=380.99, stdev=135.48 lat (msec): min=51, max=1147, avg=394.95, stdev=138.10 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 201], 10.00th=[ 239], 20.00th=[ 279], | 30.00th=[ 309], 40.00th=[ 338], 50.00th=[ 372], 60.00th=[ 397], | 70.00th=[ 426], 80.00th=[ 464], 90.00th=[ 542], 95.00th=[ 642], | 99.00th=[ 877], 99.50th=[ 978], 99.90th=[ 1045], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 127, per=25.66%, avg=52.10, stdev=22.67, samples=191 iops : min= 1, max= 31, avg=12.56, stdev= 5.71, samples=191 write: IOPS=13, BW=52.9KiB/s (54.2kB/s)(5180KiB/97907msec); 0 zone resets slat (usec): min=40, max=12467, avg=332.15, stdev=878.74 clat (msec): min=214, max=2062, avg=820.97, stdev=281.34 lat (msec): min=215, max=2063, avg=821.30, stdev=281.34 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 443], 10.00th=[ 510], 20.00th=[ 584], | 30.00th=[ 659], 40.00th=[ 735], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 1003], 90.00th=[ 1167], 95.00th=[ 1318], | 99.00th=[ 1854], 99.50th=[ 1921], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 120, per=24.89%, avg=52.77, stdev=22.45, samples=192 iops : min= 1, max= 30, avg=12.73, stdev= 5.66, samples=192 lat (msec) : 50=0.08%, 100=0.27%, 250=6.05%, 500=40.78%, 750=22.77% lat (msec) : 1000=19.57%, 2000=10.31%, >=2000=0.16% cpu : usr=0.11%, sys=0.70%, ctx=4817, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=204KiB/s (209kB/s), 50.6KiB/s-51.7KiB/s (51.8kB/s-52.9kB/s), io=19.6MiB (20.5MB), run=97907-98505msec WRITE: bw=212KiB/s (217kB/s), 52.8KiB/s-53.5KiB/s (54.1kB/s-54.8kB/s), io=20.4MiB (21.4MB), run=97907-98505msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147113: Mon Jun 29 15:20:16 2026 read: IOPS=1, BW=70.7MiB/s (74.1MB/s)(40.0MiB/566msec) slat (nsec): min=35488k, max=35488k, avg=35487592.00, stdev= 0.00 clat (nsec): min=515427k, max=515427k, avg=515426602.00, stdev= 0.00 lat (nsec): min=550977k, max=550977k, avg=550977308.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 514], 10.00th=[ 514], 20.00th=[ 514], | 30.00th=[ 514], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.48%, ctx=10, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=70.7MiB/s (74.1MB/s), 70.7MiB/s-70.7MiB/s (74.1MB/s-74.1MB/s), io=40.0MiB (41.9MB), run=566-566msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout