== sanity test 398c: run fio to test AIO ================= 15:57:57 (1787860677) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.434314 s, 96.6 MB/s osc.lustre-OST0000-osc-ffff9602e631b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146586: Thu Aug 27 16:00:37 2026 write: IOPS=16, BW=67.1KiB/s (68.7kB/s)(10.0MiB/152714msec); 0 zone resets slat (usec): min=27, max=18406, avg=217.71, stdev=831.47 clat (msec): min=268, max=2026, avg=952.99, stdev=254.42 lat (msec): min=268, max=2027, avg=953.20, stdev=254.45 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 609], 10.00th=[ 659], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1401], | 99.00th=[ 1670], 99.50th=[ 1871], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 143, per=25.24%, avg=67.64, stdev=26.74, samples=298 iops : min= 1, max= 35, avg=16.47, stdev= 6.72, samples=298 lat (msec) : 500=2.19%, 750=19.45%, 1000=39.49%, 2000=38.83%, >=2000=0.04% cpu : usr=0.04%, sys=0.23%, ctx=1733, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146587: Thu Aug 27 16:00:37 2026 write: IOPS=16, BW=67.2KiB/s (68.9kB/s)(10.0MiB/152290msec); 0 zone resets slat (usec): min=27, max=13727, avg=229.64, stdev=792.51 clat (msec): min=224, max=2027, avg=951.19, stdev=261.48 lat (msec): min=224, max=2028, avg=951.42, stdev=261.52 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1401], | 99.00th=[ 1670], 99.50th=[ 1821], 99.90th=[ 1921], 99.95th=[ 1955], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 128, per=25.07%, avg=67.19, stdev=25.08, samples=300 iops : min= 1, max= 32, avg=16.36, stdev= 6.30, samples=300 lat (msec) : 250=0.62%, 500=1.72%, 750=20.08%, 1000=38.12%, 2000=39.41% lat (msec) : >=2000=0.04% cpu : usr=0.05%, sys=0.22%, ctx=2190, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146588: Thu Aug 27 16:00:37 2026 write: IOPS=16, BW=67.1KiB/s (68.7kB/s)(10.0MiB/152715msec); 0 zone resets slat (usec): min=26, max=16065, avg=207.28, stdev=767.70 clat (msec): min=92, max=2027, avg=953.38, stdev=258.89 lat (msec): min=93, max=2027, avg=953.59, stdev=258.92 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1720], 99.50th=[ 1905], 99.90th=[ 2005], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 167, per=25.07%, avg=67.19, stdev=27.23, samples=300 iops : min= 1, max= 41, avg=16.35, stdev= 6.83, samples=300 lat (msec) : 100=0.04%, 250=0.20%, 500=2.15%, 750=18.95%, 1000=39.92% lat (msec) : 2000=38.63%, >=2000=0.12% cpu : usr=0.05%, sys=0.19%, ctx=2022, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146589: Thu Aug 27 16:00:37 2026 write: IOPS=16, BW=67.1KiB/s (68.7kB/s)(10.0MiB/152709msec); 0 zone resets slat (usec): min=29, max=12606, avg=207.99, stdev=743.41 clat (msec): min=357, max=2028, avg=953.22, stdev=256.53 lat (msec): min=357, max=2028, avg=953.43, stdev=256.54 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 609], 10.00th=[ 659], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1720], 99.50th=[ 1838], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 135, per=24.82%, avg=66.52, stdev=24.89, samples=303 iops : min= 1, max= 33, avg=16.19, stdev= 6.24, samples=303 lat (msec) : 500=2.93%, 750=19.61%, 1000=38.16%, 2000=39.14%, >=2000=0.16% cpu : usr=0.04%, sys=0.22%, ctx=2160, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=268KiB/s (275kB/s), 67.1KiB/s-67.2KiB/s (68.7kB/s-68.9kB/s), io=40.0MiB (41.9MB), run=152290-152715msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146620: Thu Aug 27 16:02:39 2026 read: IOPS=11, BW=44.7KiB/s (45.7kB/s)(5048KiB/113032msec) slat (usec): min=2674, max=58493, avg=13958.77, stdev=8232.30 clat (msec): min=18, max=1001, avg=438.63, stdev=125.19 lat (msec): min=30, max=1009, avg=452.59, stdev=126.49 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 266], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 456], | 70.00th=[ 489], 80.00th=[ 527], 90.00th=[ 592], 95.00th=[ 659], | 99.00th=[ 827], 99.50th=[ 911], 99.90th=[ 978], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 128, per=25.38%, avg=44.92, stdev=22.01, samples=221 iops : min= 1, max= 32, avg=10.80, stdev= 5.53, samples=221 write: IOPS=11, BW=45.9KiB/s (47.0kB/s)(5192KiB/113032msec); 0 zone resets slat (usec): min=38, max=17577, avg=237.06, stdev=913.17 clat (msec): min=270, max=2010, avg=949.29, stdev=275.61 lat (msec): min=270, max=2010, avg=949.52, stdev=275.66 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 718], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1053], 80.00th=[ 1167], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1770], 99.50th=[ 1871], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 104, per=24.76%, avg=45.56, stdev=21.28, samples=223 iops : min= 1, max= 26, avg=10.96, stdev= 5.34, samples=223 lat (msec) : 20=0.04%, 50=0.04%, 100=0.35%, 250=1.56%, 500=35.66% lat (msec) : 750=23.12%, 1000=20.62%, 2000=18.55%, >=2000=0.04% cpu : usr=0.10%, sys=0.51%, ctx=4528, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146621: Thu Aug 27 16:02:39 2026 read: IOPS=10, BW=43.0KiB/s (45.0kB/s)(4976KiB/113172msec) slat (usec): min=50, max=60970, avg=13915.29, stdev=8141.99 clat (msec): min=12, max=996, avg=439.09, stdev=128.49 lat (msec): min=17, max=1010, avg=453.01, stdev=129.96 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 264], 10.00th=[ 296], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 489], 80.00th=[ 531], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 793], 99.50th=[ 919], 99.90th=[ 978], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 116, per=24.65%, avg=43.64, stdev=21.80, samples=224 iops : min= 1, max= 29, avg=10.49, stdev= 5.47, samples=224 write: IOPS=11, BW=46.5KiB/s (47.6kB/s)(5264KiB/113172msec); 0 zone resets slat (usec): min=31, max=15045, avg=215.04, stdev=869.04 clat (msec): min=145, max=2009, avg=946.75, stdev=270.93 lat (msec): min=145, max=2009, avg=946.97, stdev=270.90 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1062], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1653], 99.50th=[ 1737], 99.90th=[ 1854], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 103, per=24.89%, avg=45.80, stdev=20.42, samples=225 iops : min= 1, max= 25, avg=11.03, stdev= 5.12, samples=225 lat (msec) : 20=0.08%, 50=0.12%, 100=0.20%, 250=1.45%, 500=35.04% lat (msec) : 750=22.93%, 1000=21.25%, 2000=18.91%, >=2000=0.04% cpu : usr=0.08%, sys=0.53%, ctx=4580, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146622: Thu Aug 27 16:02:39 2026 read: IOPS=11, BW=44.1KiB/s (45.1kB/s)(4984KiB/113126msec) slat (usec): min=2231, max=75871, avg=13822.26, stdev=8066.68 clat (msec): min=25, max=1010, avg=436.20, stdev=122.80 lat (msec): min=37, max=1023, avg=450.03, stdev=124.14 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 264], 10.00th=[ 296], 20.00th=[ 342], | 30.00th=[ 368], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 485], 80.00th=[ 523], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 768], 99.50th=[ 835], 99.90th=[ 995], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 7, max= 142, per=24.78%, avg=43.86, stdev=21.25, samples=224 iops : min= 1, max= 35, avg=10.54, stdev= 5.31, samples=224 write: IOPS=11, BW=46.5KiB/s (47.6kB/s)(5256KiB/113126msec); 0 zone resets slat (usec): min=33, max=16159, avg=244.01, stdev=988.26 clat (msec): min=299, max=1989, avg=947.29, stdev=269.26 lat (msec): min=299, max=1989, avg=947.54, stdev=269.27 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1653], 99.50th=[ 1737], 99.90th=[ 1921], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 95, per=24.90%, avg=45.82, stdev=19.63, samples=224 iops : min= 1, max= 23, avg=11.03, stdev= 4.93, samples=224 lat (msec) : 50=0.08%, 100=0.27%, 250=1.45%, 500=35.98%, 750=22.93% lat (msec) : 1000=19.80%, 2000=19.49% cpu : usr=0.07%, sys=0.49%, ctx=4513, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146623: Thu Aug 27 16:02:39 2026 read: IOPS=11, BW=44.7KiB/s (45.8kB/s)(5060KiB/113221msec) slat (usec): min=2574, max=62358, avg=13943.90, stdev=8420.22 clat (msec): min=24, max=980, avg=437.42, stdev=123.94 lat (msec): min=33, max=985, avg=451.37, stdev=125.56 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 368], 40.00th=[ 397], 50.00th=[ 426], 60.00th=[ 460], | 70.00th=[ 489], 80.00th=[ 527], 90.00th=[ 600], 95.00th=[ 651], | 99.00th=[ 802], 99.50th=[ 911], 99.90th=[ 936], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7, max= 127, per=25.30%, avg=44.79, stdev=22.18, samples=222 iops : min= 1, max= 31, avg=10.77, stdev= 5.54, samples=222 write: IOPS=11, BW=45.8KiB/s (46.8kB/s)(5180KiB/113221msec); 0 zone resets slat (usec): min=25, max=12334, avg=197.08, stdev=598.42 clat (msec): min=310, max=1886, avg=954.29, stdev=272.11 lat (msec): min=310, max=1886, avg=954.49, stdev=272.10 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 718], | 30.00th=[ 793], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1737], 99.50th=[ 1804], 99.90th=[ 1854], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 8, max= 95, per=24.61%, avg=45.28, stdev=19.62, samples=224 iops : min= 2, max= 23, avg=10.90, stdev= 4.96, samples=224 lat (msec) : 50=0.16%, 100=0.20%, 250=1.21%, 500=36.17%, 750=23.40% lat (msec) : 1000=19.38%, 2000=19.49% cpu : usr=0.09%, sys=0.47%, ctx=4216, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177KiB/s (182kB/s), 43.0KiB/s-44.7KiB/s (45.0kB/s-45.8kB/s), io=19.6MiB (20.5MB), run=113032-113221msec WRITE: bw=185KiB/s (189kB/s), 45.8KiB/s-46.5KiB/s (46.8kB/s-47.6kB/s), io=20.4MiB (21.4MB), run=113032-113221msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146655: Thu Aug 27 16:02:51 2026 read: IOPS=1, BW=71.9MiB/s (75.4MB/s)(40.0MiB/556msec) slat (nsec): min=45308k, max=45308k, avg=45307815.00, stdev= 0.00 clat (nsec): min=461844k, max=461844k, avg=461843780.00, stdev= 0.00 lat (nsec): min=507157k, max=507157k, avg=507156959.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.16%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=71.9MiB/s (75.4MB/s), 71.9MiB/s-71.9MiB/s (75.4MB/s-75.4MB/s), io=40.0MiB (41.9MB), run=556-556msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout