== sanity test 398c: run fio to test AIO ================= 21:31:35 (1763346695) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.33359 s, 126 MB/s osc.lustre-OST0000-osc-ffff905d914be800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148019: Sun Nov 16 21:34:15 2025 write: IOPS=16, BW=66.2KiB/s (67.8kB/s)(10.0MiB/154580msec); 0 zone resets slat (usec): min=27, max=16971, avg=207.05, stdev=752.05 clat (msec): min=96, max=1652, avg=965.14, stdev=229.33 lat (msec): min=96, max=1652, avg=965.35, stdev=229.33 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1334], | 99.00th=[ 1452], 99.50th=[ 1502], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 209, per=24.88%, avg=65.70, stdev=30.47, samples=306 iops : min= 1, max= 52, avg=15.93, stdev= 7.62, samples=306 lat (msec) : 100=0.62%, 500=1.95%, 750=12.58%, 1000=40.00%, 2000=44.84% cpu : usr=0.05%, sys=0.26%, ctx=1654, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148020: Sun Nov 16 21:34:15 2025 write: IOPS=16, BW=66.3KiB/s (67.9kB/s)(10.0MiB/154410msec); 0 zone resets slat (usec): min=26, max=13787, avg=224.50, stdev=686.56 clat (msec): min=82, max=1627, avg=964.44, stdev=234.55 lat (msec): min=82, max=1627, avg=964.67, stdev=234.55 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 575], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1028], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1334], | 99.00th=[ 1485], 99.50th=[ 1519], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 229, per=24.88%, avg=65.69, stdev=28.60, samples=306 iops : min= 1, max= 57, avg=15.92, stdev= 7.18, samples=306 lat (msec) : 100=0.23%, 250=0.74%, 500=2.15%, 750=12.81%, 1000=39.53% lat (msec) : 2000=44.53% cpu : usr=0.06%, sys=0.27%, ctx=1933, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148021: Sun Nov 16 21:34:15 2025 write: IOPS=16, BW=66.2KiB/s (67.8kB/s)(10.0MiB/154578msec); 0 zone resets slat (usec): min=26, max=10252, avg=181.35, stdev=537.48 clat (msec): min=103, max=1626, avg=965.29, stdev=230.49 lat (msec): min=103, max=1626, avg=965.47, stdev=230.48 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 584], 10.00th=[ 693], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1334], | 99.00th=[ 1452], 99.50th=[ 1502], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 209, per=25.05%, avg=66.12, stdev=30.24, samples=304 iops : min= 1, max= 52, avg=16.04, stdev= 7.56, samples=304 lat (msec) : 250=0.66%, 500=2.11%, 750=13.24%, 1000=40.20%, 2000=43.79% cpu : usr=0.05%, sys=0.28%, ctx=1611, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148022: Sun Nov 16 21:34:15 2025 write: IOPS=16, BW=66.2KiB/s (67.8kB/s)(10.0MiB/154577msec); 0 zone resets slat (usec): min=28, max=12681, avg=199.46, stdev=588.54 clat (msec): min=89, max=1684, avg=965.06, stdev=226.66 lat (msec): min=89, max=1684, avg=965.26, stdev=226.68 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 600], 10.00th=[ 701], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1020], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1334], | 99.00th=[ 1469], 99.50th=[ 1502], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 210, per=24.80%, avg=65.47, stdev=25.23, samples=307 iops : min= 1, max= 52, avg=15.87, stdev= 6.33, samples=307 lat (msec) : 100=0.62%, 500=2.07%, 750=11.41%, 1000=42.42%, 2000=43.48% cpu : usr=0.04%, sys=0.27%, ctx=1691, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=265KiB/s (271kB/s), 66.2KiB/s-66.3KiB/s (67.8kB/s-67.9kB/s), io=40.0MiB (41.9MB), run=154410-154580msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148054: Sun Nov 16 21:36:02 2025 read: IOPS=12, BW=50.7KiB/s (51.9kB/s)(5048KiB/99559msec) slat (usec): min=56, max=581457, avg=13022.18, stdev=17231.23 clat (msec): min=9, max=1018, avg=402.20, stdev=107.06 lat (msec): min=18, max=1028, avg=415.24, stdev=107.47 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 262], 10.00th=[ 296], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 443], 80.00th=[ 477], 90.00th=[ 518], 95.00th=[ 567], | 99.00th=[ 709], 99.50th=[ 885], 99.90th=[ 995], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 119, per=25.34%, avg=50.69, stdev=22.61, samples=195 iops : min= 1, max= 29, avg=12.16, stdev= 5.67, samples=195 write: IOPS=13, BW=52.1KiB/s (53.4kB/s)(5192KiB/99559msec); 0 zone resets slat (usec): min=28, max=11587, avg=250.37, stdev=674.95 clat (msec): min=107, max=1541, avg=822.29, stdev=201.19 lat (msec): min=107, max=1541, avg=822.54, stdev=201.20 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 506], 10.00th=[ 575], 20.00th=[ 659], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 818], 60.00th=[ 869], | 70.00th=[ 927], 80.00th=[ 1003], 90.00th=[ 1083], 95.00th=[ 1150], | 99.00th=[ 1284], 99.50th=[ 1385], 99.90th=[ 1435], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 102, per=24.78%, avg=51.80, stdev=19.53, samples=196 iops : min= 1, max= 25, avg=12.43, stdev= 4.88, samples=196 lat (msec) : 10=0.04%, 20=0.08%, 50=0.20%, 100=0.35%, 250=1.60% lat (msec) : 500=43.12%, 750=21.72%, 1000=22.73%, 2000=10.16% cpu : usr=0.11%, sys=0.71%, ctx=3709, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148055: Sun Nov 16 21:36:02 2025 read: IOPS=12, BW=49.9KiB/s (51.1kB/s)(4976KiB/99750msec) slat (usec): min=2270, max=37471, avg=12835.76, stdev=5637.58 clat (msec): min=13, max=778, avg=399.05, stdev=95.18 lat (msec): min=21, max=792, avg=411.90, stdev=95.93 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 259], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 397], 60.00th=[ 418], | 70.00th=[ 443], 80.00th=[ 472], 90.00th=[ 518], 95.00th=[ 558], | 99.00th=[ 667], 99.50th=[ 676], 99.90th=[ 760], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 7, max= 111, per=24.63%, avg=49.25, stdev=21.30, samples=198 iops : min= 1, max= 27, avg=11.80, stdev= 5.32, samples=198 write: IOPS=13, BW=52.8KiB/s (54.0kB/s)(5264KiB/99750msec); 0 zone resets slat (usec): min=28, max=16853, avg=241.28, stdev=745.94 clat (msec): min=205, max=1559, avg=821.26, stdev=199.06 lat (msec): min=205, max=1559, avg=821.51, stdev=199.08 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 514], 10.00th=[ 575], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 768], 50.00th=[ 818], 60.00th=[ 869], | 70.00th=[ 936], 80.00th=[ 995], 90.00th=[ 1083], 95.00th=[ 1150], | 99.00th=[ 1250], 99.50th=[ 1368], 99.90th=[ 1536], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 96, per=24.88%, avg=51.99, stdev=19.72, samples=198 iops : min= 1, max= 24, avg=12.49, stdev= 4.95, samples=198 lat (msec) : 20=0.08%, 50=0.04%, 100=0.27%, 250=1.88%, 500=41.72% lat (msec) : 750=24.02%, 1000=21.88%, 2000=10.12% cpu : usr=0.12%, sys=0.73%, ctx=4389, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148056: Sun Nov 16 21:36:02 2025 read: IOPS=12, BW=49.9KiB/s (51.1kB/s)(4984KiB/99925msec) slat (msec): min=2, max=115, avg=12.99, stdev= 6.28 clat (msec): min=8, max=705, avg=399.16, stdev=93.09 lat (msec): min=20, max=718, avg=412.15, stdev=93.81 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 397], 60.00th=[ 418], | 70.00th=[ 443], 80.00th=[ 477], 90.00th=[ 514], 95.00th=[ 550], | 99.00th=[ 667], 99.50th=[ 676], 99.90th=[ 701], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 7, max= 119, per=24.73%, avg=49.45, stdev=22.17, samples=197 iops : min= 1, max= 29, avg=11.85, stdev= 5.57, samples=197 write: IOPS=13, BW=52.6KiB/s (53.9kB/s)(5256KiB/99925msec); 0 zone resets slat (usec): min=29, max=6686, avg=229.61, stdev=504.44 clat (msec): min=156, max=1543, avg=823.53, stdev=204.29 lat (msec): min=156, max=1543, avg=823.76, stdev=204.31 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 493], 10.00th=[ 567], 20.00th=[ 651], | 30.00th=[ 709], 40.00th=[ 768], 50.00th=[ 818], 60.00th=[ 877], | 70.00th=[ 936], 80.00th=[ 1003], 90.00th=[ 1083], 95.00th=[ 1150], | 99.00th=[ 1284], 99.50th=[ 1385], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 103, per=24.71%, avg=51.65, stdev=18.53, samples=198 iops : min= 1, max= 25, avg=12.39, stdev= 4.64, samples=198 lat (msec) : 10=0.04%, 50=0.08%, 100=0.27%, 250=1.56%, 500=43.16% lat (msec) : 750=22.54%, 1000=21.91%, 2000=10.43% cpu : usr=0.11%, sys=0.77%, ctx=4035, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148057: Sun Nov 16 21:36:02 2025 read: IOPS=12, BW=50.8KiB/s (51.0kB/s)(5060KiB/99690msec) slat (msec): min=2, max=110, avg=13.16, stdev= 6.44 clat (msec): min=53, max=712, avg=403.50, stdev=91.87 lat (msec): min=67, max=725, avg=416.67, stdev=92.73 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 266], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 422], | 70.00th=[ 447], 80.00th=[ 477], 90.00th=[ 518], 95.00th=[ 558], | 99.00th=[ 651], 99.50th=[ 684], 99.90th=[ 701], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 7, max= 120, per=25.20%, avg=50.39, stdev=21.08, samples=197 iops : min= 1, max= 30, avg=12.08, stdev= 5.29, samples=197 write: IOPS=12, BW=51.0KiB/s (53.2kB/s)(5180KiB/99690msec); 0 zone resets slat (usec): min=29, max=7592, avg=229.45, stdev=496.30 clat (msec): min=224, max=1525, avg=822.42, stdev=197.23 lat (msec): min=224, max=1525, avg=822.65, stdev=197.23 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 506], 10.00th=[ 575], 20.00th=[ 651], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 810], 60.00th=[ 869], | 70.00th=[ 927], 80.00th=[ 1003], 90.00th=[ 1083], 95.00th=[ 1150], | 99.00th=[ 1284], 99.50th=[ 1385], 99.90th=[ 1469], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 127, per=24.48%, avg=51.16, stdev=18.87, samples=198 iops : min= 1, max= 31, avg=12.27, stdev= 4.69, samples=198 lat (msec) : 100=0.16%, 250=1.80%, 500=43.01%, 750=22.62%, 1000=22.54% lat (msec) : 2000=9.88% cpu : usr=0.11%, sys=0.81%, ctx=4432, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=201KiB/s (206kB/s), 49.9KiB/s-50.8KiB/s (51.1kB/s-51.0kB/s), io=19.6MiB (20.5MB), run=99559-99925msec WRITE: bw=209KiB/s (214kB/s), 51.0KiB/s-52.8KiB/s (53.2kB/s-54.0kB/s), io=20.4MiB (21.4MB), run=99559-99925msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148076: Sun Nov 16 21:36:09 2025 read: IOPS=1, BW=79.5MiB/s (83.4MB/s)(40.0MiB/503msec) slat (nsec): min=32537k, max=32537k, avg=32537335.00, stdev= 0.00 clat (nsec): min=437041k, max=437041k, avg=437040901.00, stdev= 0.00 lat (nsec): min=469584k, max=469584k, avg=469583664.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 439], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.59%, ctx=24, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=79.5MiB/s (83.4MB/s), 79.5MiB/s-79.5MiB/s (83.4MB/s-83.4MB/s), io=40.0MiB (41.9MB), run=503-503msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout