== sanity test 398c: run fio to test AIO ================= 21:06:29 (1767319589) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.484266 s, 86.6 MB/s osc.lustre-OST0000-osc-ffff8925c3eac800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=151434: Thu Jan 1 21:08:48 2026 write: IOPS=19, BW=78.8KiB/s (80.7kB/s)(10.0MiB/129918msec); 0 zone resets slat (usec): min=20, max=17843, avg=184.72, stdev=667.16 clat (msec): min=110, max=1521, avg=810.38, stdev=197.38 lat (msec): min=110, max=1522, avg=810.57, stdev=197.43 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 550], 10.00th=[ 592], 20.00th=[ 651], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 953], 90.00th=[ 1070], 95.00th=[ 1183], | 99.00th=[ 1368], 99.50th=[ 1435], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 135, per=25.15%, avg=79.24, stdev=31.11, samples=255 iops : min= 1, max= 33, avg=19.42, stdev= 7.81, samples=255 lat (msec) : 250=0.62%, 500=2.46%, 750=36.80%, 1000=45.35%, 2000=14.77% cpu : usr=0.04%, sys=0.26%, ctx=2024, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151435: Thu Jan 1 21:08:48 2026 write: IOPS=19, BW=78.8KiB/s (80.7kB/s)(10.0MiB/130001msec); 0 zone resets slat (usec): min=22, max=14027, avg=175.95, stdev=664.35 clat (msec): min=166, max=1550, avg=810.99, stdev=196.88 lat (msec): min=166, max=1551, avg=811.17, stdev=196.89 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 523], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 760], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 894], 80.00th=[ 961], 90.00th=[ 1062], 95.00th=[ 1183], | 99.00th=[ 1334], 99.50th=[ 1401], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 144, per=24.96%, avg=78.63, stdev=30.60, samples=257 iops : min= 1, max= 36, avg=19.28, stdev= 7.69, samples=257 lat (msec) : 250=0.62%, 500=3.16%, 750=35.35%, 1000=45.51%, 2000=15.35% cpu : usr=0.05%, sys=0.24%, ctx=1715, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151436: Thu Jan 1 21:08:48 2026 write: IOPS=19, BW=78.8KiB/s (80.7kB/s)(10.0MiB/129874msec); 0 zone resets slat (usec): min=21, max=9445, avg=173.73, stdev=562.58 clat (msec): min=128, max=1538, avg=810.86, stdev=197.02 lat (msec): min=129, max=1538, avg=811.04, stdev=197.05 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 523], 10.00th=[ 575], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 894], 80.00th=[ 953], 90.00th=[ 1062], 95.00th=[ 1183], | 99.00th=[ 1351], 99.50th=[ 1401], 99.90th=[ 1502], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 174, per=24.95%, avg=78.58, stdev=26.95, samples=257 iops : min= 1, max= 43, avg=19.26, stdev= 6.77, samples=257 lat (msec) : 250=0.98%, 500=2.77%, 750=35.23%, 1000=45.62%, 2000=15.39% cpu : usr=0.05%, sys=0.25%, ctx=1822, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151437: Thu Jan 1 21:08:48 2026 write: IOPS=19, BW=79.1KiB/s (80.0kB/s)(10.0MiB/129508msec); 0 zone resets slat (usec): min=22, max=15525, avg=192.52, stdev=707.77 clat (msec): min=126, max=1960, avg=807.87, stdev=203.16 lat (msec): min=126, max=1960, avg=808.07, stdev=203.19 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 527], 10.00th=[ 584], 20.00th=[ 651], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 953], 90.00th=[ 1070], 95.00th=[ 1183], | 99.00th=[ 1368], 99.50th=[ 1418], 99.90th=[ 1519], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 8, max= 153, per=25.04%, avg=78.89, stdev=28.66, samples=256 iops : min= 2, max= 38, avg=19.34, stdev= 7.16, samples=256 lat (msec) : 250=0.74%, 500=3.01%, 750=36.29%, 1000=44.69%, 2000=15.27% cpu : usr=0.06%, sys=0.28%, ctx=2167, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=315KiB/s (323kB/s), 78.8KiB/s-79.1KiB/s (80.7kB/s-80.0kB/s), io=40.0MiB (41.9MB), run=129508-130001msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=151466: Thu Jan 1 21:10:28 2026 read: IOPS=13, BW=55.5KiB/s (56.8kB/s)(5048KiB/90925msec) slat (usec): min=2269, max=44182, avg=12162.77, stdev=6799.49 clat (msec): min=6, max=865, avg=356.37, stdev=100.73 lat (msec): min=11, max=881, avg=368.54, stdev=102.30 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 211], 10.00th=[ 243], 20.00th=[ 271], | 30.00th=[ 305], 40.00th=[ 326], 50.00th=[ 351], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 435], 90.00th=[ 477], 95.00th=[ 523], | 99.00th=[ 642], 99.50th=[ 676], 99.90th=[ 793], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 7, max= 119, per=24.98%, avg=54.97, stdev=23.45, samples=180 iops : min= 1, max= 29, avg=13.24, stdev= 5.89, samples=180 write: IOPS=14, BW=57.1KiB/s (58.5kB/s)(5192KiB/90925msec); 0 zone resets slat (usec): min=28, max=16269, avg=238.90, stdev=822.31 clat (msec): min=296, max=1423, avg=759.48, stdev=191.23 lat (msec): min=296, max=1423, avg=759.72, stdev=191.25 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 451], 10.00th=[ 527], 20.00th=[ 600], | 30.00th=[ 659], 40.00th=[ 709], 50.00th=[ 751], 60.00th=[ 802], | 70.00th=[ 852], 80.00th=[ 911], 90.00th=[ 1011], 95.00th=[ 1099], | 99.00th=[ 1284], 99.50th=[ 1368], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 128, per=24.73%, avg=56.63, stdev=22.30, samples=180 iops : min= 1, max= 32, avg=13.65, stdev= 5.62, samples=180 lat (msec) : 10=0.04%, 50=0.08%, 100=0.35%, 250=5.62%, 500=43.98% lat (msec) : 750=24.65%, 1000=19.92%, 2000=5.35% cpu : usr=0.10%, sys=0.73%, ctx=4770, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151467: Thu Jan 1 21:10:28 2026 read: IOPS=13, BW=54.7KiB/s (56.0kB/s)(4976KiB/90980msec) slat (usec): min=2312, max=48816, avg=12297.73, stdev=6859.65 clat (msec): min=10, max=726, avg=352.21, stdev=100.40 lat (msec): min=15, max=747, avg=364.52, stdev=101.76 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 207], 10.00th=[ 236], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 326], 50.00th=[ 351], 60.00th=[ 368], | 70.00th=[ 393], 80.00th=[ 426], 90.00th=[ 472], 95.00th=[ 535], | 99.00th=[ 642], 99.50th=[ 651], 99.90th=[ 701], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 7, max= 135, per=24.53%, avg=53.97, stdev=25.63, samples=180 iops : min= 1, max= 33, avg=12.98, stdev= 6.44, samples=180 write: IOPS=14, BW=57.9KiB/s (59.2kB/s)(5264KiB/90980msec); 0 zone resets slat (usec): min=29, max=17282, avg=255.02, stdev=854.03 clat (msec): min=116, max=1412, avg=759.20, stdev=191.81 lat (msec): min=116, max=1412, avg=759.46, stdev=191.81 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 435], 10.00th=[ 527], 20.00th=[ 600], | 30.00th=[ 651], 40.00th=[ 701], 50.00th=[ 760], 60.00th=[ 810], | 70.00th=[ 860], 80.00th=[ 919], 90.00th=[ 995], 95.00th=[ 1083], | 99.00th=[ 1234], 99.50th=[ 1334], 99.90th=[ 1401], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 135, per=25.11%, avg=57.51, stdev=20.59, samples=179 iops : min= 1, max= 33, avg=13.88, stdev= 5.20, samples=179 lat (msec) : 20=0.12%, 50=0.12%, 100=0.39%, 250=5.94%, 500=43.12% lat (msec) : 750=24.06%, 1000=21.33%, 2000=4.92% cpu : usr=0.11%, sys=0.65%, ctx=4699, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151468: Thu Jan 1 21:10:28 2026 read: IOPS=13, BW=54.9KiB/s (56.2kB/s)(4984KiB/90816msec) slat (usec): min=29, max=48210, avg=11972.84, stdev=6864.46 clat (msec): min=12, max=834, avg=353.91, stdev=97.05 lat (msec): min=16, max=851, avg=365.89, stdev=98.72 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 211], 10.00th=[ 241], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 326], 50.00th=[ 351], 60.00th=[ 372], | 70.00th=[ 397], 80.00th=[ 430], 90.00th=[ 477], 95.00th=[ 514], | 99.00th=[ 634], 99.50th=[ 659], 99.90th=[ 827], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7, max= 128, per=24.90%, avg=54.77, stdev=25.00, samples=179 iops : min= 1, max= 32, avg=13.19, stdev= 6.26, samples=179 write: IOPS=14, BW=57.9KiB/s (59.3kB/s)(5256KiB/90816msec); 0 zone resets slat (usec): min=27, max=18068, avg=280.31, stdev=1074.72 clat (msec): min=156, max=1393, avg=757.71, stdev=195.14 lat (msec): min=156, max=1393, avg=757.99, stdev=195.12 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 451], 10.00th=[ 523], 20.00th=[ 600], | 30.00th=[ 659], 40.00th=[ 701], 50.00th=[ 751], 60.00th=[ 802], | 70.00th=[ 852], 80.00th=[ 919], 90.00th=[ 995], 95.00th=[ 1099], | 99.00th=[ 1250], 99.50th=[ 1301], 99.90th=[ 1351], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 15, max= 128, per=25.08%, avg=57.43, stdev=21.83, samples=179 iops : min= 3, max= 32, avg=13.85, stdev= 5.54, samples=179 lat (msec) : 20=0.12%, 50=0.04%, 100=0.20%, 250=5.82%, 500=43.48% lat (msec) : 750=24.18%, 1000=21.13%, 2000=5.04% cpu : usr=0.12%, sys=0.75%, ctx=4956, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151469: Thu Jan 1 21:10:28 2026 read: IOPS=14, BW=56.1KiB/s (57.4kB/s)(5060KiB/90272msec) slat (usec): min=2438, max=50282, avg=12235.48, stdev=6869.39 clat (msec): min=13, max=901, avg=356.74, stdev=99.78 lat (msec): min=22, max=910, avg=368.99, stdev=101.39 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 213], 10.00th=[ 241], 20.00th=[ 271], | 30.00th=[ 305], 40.00th=[ 330], 50.00th=[ 355], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 435], 90.00th=[ 477], 95.00th=[ 518], | 99.00th=[ 625], 99.50th=[ 676], 99.90th=[ 844], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 152, per=25.24%, avg=55.53, stdev=23.72, samples=179 iops : min= 1, max= 38, avg=13.37, stdev= 5.94, samples=179 write: IOPS=14, BW=57.4KiB/s (58.8kB/s)(5180KiB/90272msec); 0 zone resets slat (usec): min=26, max=29329, avg=287.67, stdev=1248.83 clat (msec): min=195, max=1443, avg=752.24, stdev=194.37 lat (msec): min=195, max=1443, avg=752.53, stdev=194.36 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 443], 10.00th=[ 518], 20.00th=[ 592], | 30.00th=[ 642], 40.00th=[ 693], 50.00th=[ 751], 60.00th=[ 802], | 70.00th=[ 852], 80.00th=[ 911], 90.00th=[ 995], 95.00th=[ 1083], | 99.00th=[ 1234], 99.50th=[ 1318], 99.90th=[ 1401], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 15, max= 136, per=24.78%, avg=56.74, stdev=21.99, samples=179 iops : min= 3, max= 34, avg=13.67, stdev= 5.55, samples=179 lat (msec) : 20=0.04%, 50=0.08%, 100=0.31%, 250=6.02%, 500=43.71% lat (msec) : 750=24.26%, 1000=20.74%, 2000=4.84% cpu : usr=0.10%, sys=0.73%, ctx=4432, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=221KiB/s (226kB/s), 54.7KiB/s-56.1KiB/s (56.0kB/s-57.4kB/s), io=19.6MiB (20.5MB), run=90272-90980msec WRITE: bw=230KiB/s (235kB/s), 57.1KiB/s-57.9KiB/s (58.5kB/s-59.3kB/s), io=20.4MiB (21.4MB), run=90272-90980msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=151486: Thu Jan 1 21:10:37 2026 read: IOPS=1, BW=74.2MiB/s (77.8MB/s)(40.0MiB/539msec) slat (nsec): min=30861k, max=30861k, avg=30860916.00, stdev= 0.00 clat (nsec): min=485371k, max=485371k, avg=485370765.00, stdev= 0.00 lat (nsec): min=516314k, max=516314k, avg=516314325.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 485], 10.00th=[ 485], 20.00th=[ 485], | 30.00th=[ 485], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 500=100.00% cpu : usr=0.37%, sys=2.60%, ctx=10, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.2MiB/s (77.8MB/s), 74.2MiB/s-74.2MiB/s (77.8MB/s-77.8MB/s), io=40.0MiB (41.9MB), run=539-539msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout