== sanity test 398c: run fio to test AIO ================= 18:55:55 (1786748155) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.422407 s, 99.3 MB/s osc.lustre-OST0000-osc-ffff99bda0ebd000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145955: Fri Aug 14 18:58:24 2026 write: IOPS=18, BW=72.8KiB/s (74.5kB/s)(10.0MiB/140665msec); 0 zone resets slat (usec): min=38, max=27772, avg=257.91, stdev=822.59 clat (msec): min=176, max=1535, avg=878.55, stdev=228.72 lat (msec): min=176, max=1535, avg=878.81, stdev=228.77 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 558], 10.00th=[ 617], 20.00th=[ 701], | 30.00th=[ 743], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1062], 90.00th=[ 1167], 95.00th=[ 1284], | 99.00th=[ 1452], 99.50th=[ 1485], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 222, per=25.12%, avg=72.84, stdev=32.64, samples=277 iops : min= 1, max= 55, avg=17.83, stdev= 8.20, samples=277 lat (msec) : 250=1.25%, 500=2.58%, 750=27.07%, 1000=41.13%, 2000=27.97% cpu : usr=0.04%, sys=0.29%, ctx=2053, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145956: Fri Aug 14 18:58:24 2026 write: IOPS=18, BW=72.6KiB/s (74.4kB/s)(10.0MiB/140978msec); 0 zone resets slat (usec): min=52, max=19097, avg=270.12, stdev=854.66 clat (msec): min=207, max=1534, avg=880.01, stdev=222.12 lat (msec): min=207, max=1534, avg=880.28, stdev=222.17 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1062], 90.00th=[ 1167], 95.00th=[ 1284], | 99.00th=[ 1435], 99.50th=[ 1469], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 8, max= 128, per=25.23%, avg=73.16, stdev=28.70, samples=275 iops : min= 2, max= 32, avg=17.91, stdev= 7.22, samples=275 lat (msec) : 250=0.62%, 500=2.97%, 750=25.82%, 1000=42.89%, 2000=27.70% cpu : usr=0.05%, sys=0.27%, ctx=1889, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145957: Fri Aug 14 18:58:24 2026 write: IOPS=18, BW=72.8KiB/s (74.5kB/s)(10.0MiB/140753msec); 0 zone resets slat (usec): min=39, max=19040, avg=258.86, stdev=787.24 clat (msec): min=105, max=1542, avg=878.72, stdev=228.66 lat (msec): min=105, max=1542, avg=878.98, stdev=228.68 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 693], | 30.00th=[ 743], 40.00th=[ 802], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 995], 80.00th=[ 1062], 90.00th=[ 1183], 95.00th=[ 1267], | 99.00th=[ 1401], 99.50th=[ 1485], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 222, per=25.21%, avg=73.11, stdev=32.09, samples=276 iops : min= 1, max= 55, avg=17.89, stdev= 8.05, samples=276 lat (msec) : 250=0.62%, 500=3.01%, 750=27.07%, 1000=40.04%, 2000=29.26% cpu : usr=0.04%, sys=0.28%, ctx=1730, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145958: Fri Aug 14 18:58:24 2026 write: IOPS=18, BW=72.7KiB/s (74.4kB/s)(10.0MiB/140898msec); 0 zone resets slat (usec): min=45, max=14241, avg=229.10, stdev=608.11 clat (msec): min=74, max=1578, avg=879.62, stdev=228.40 lat (msec): min=74, max=1578, avg=879.85, stdev=228.39 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 550], 10.00th=[ 625], 20.00th=[ 693], | 30.00th=[ 743], 40.00th=[ 818], 50.00th=[ 877], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1062], 90.00th=[ 1167], 95.00th=[ 1284], | 99.00th=[ 1452], 99.50th=[ 1485], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 183, per=25.13%, avg=72.88, stdev=34.17, samples=277 iops : min= 1, max= 45, avg=17.84, stdev= 8.61, samples=277 lat (msec) : 100=0.16%, 250=0.66%, 500=2.73%, 750=27.38%, 1000=40.31% lat (msec) : 2000=28.75% cpu : usr=0.04%, sys=0.27%, ctx=1928, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=291KiB/s (298kB/s), 72.6KiB/s-72.8KiB/s (74.4kB/s-74.5kB/s), io=40.0MiB (41.9MB), run=140665-140978msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145988: Fri Aug 14 19:00:01 2026 read: IOPS=13, BW=55.7KiB/s (57.1kB/s)(5048KiB/90570msec) slat (usec): min=2522, max=48276, avg=11425.42, stdev=5735.84 clat (msec): min=25, max=601, avg=349.05, stdev=79.50 lat (msec): min=34, max=616, avg=360.48, stdev=80.38 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 230], 10.00th=[ 259], 20.00th=[ 288], | 30.00th=[ 309], 40.00th=[ 326], 50.00th=[ 347], 60.00th=[ 363], | 70.00th=[ 384], 80.00th=[ 414], 90.00th=[ 456], 95.00th=[ 489], | 99.00th=[ 535], 99.50th=[ 542], 99.90th=[ 584], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 7, max= 119, per=24.97%, avg=55.18, stdev=24.60, samples=180 iops : min= 1, max= 29, avg=13.36, stdev= 6.15, samples=180 write: IOPS=14, BW=57.3KiB/s (58.7kB/s)(5192KiB/90570msec); 0 zone resets slat (usec): min=50, max=18154, avg=327.58, stdev=850.68 clat (msec): min=287, max=1327, avg=763.05, stdev=182.36 lat (msec): min=288, max=1327, avg=763.38, stdev=182.33 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 472], 10.00th=[ 527], 20.00th=[ 625], | 30.00th=[ 667], 40.00th=[ 709], 50.00th=[ 751], 60.00th=[ 793], | 70.00th=[ 844], 80.00th=[ 894], 90.00th=[ 1011], 95.00th=[ 1099], | 99.00th=[ 1234], 99.50th=[ 1284], 99.90th=[ 1318], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8, max= 128, per=24.64%, avg=56.68, stdev=20.66, samples=180 iops : min= 2, max= 32, avg=13.73, stdev= 5.18, samples=180 lat (msec) : 50=0.20%, 100=0.20%, 250=3.44%, 500=47.54%, 750=23.44% lat (msec) : 1000=19.38%, 2000=5.82% cpu : usr=0.10%, sys=0.69%, ctx=4710, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145989: Fri Aug 14 19:00:01 2026 read: IOPS=13, BW=54.0KiB/s (56.3kB/s)(4976KiB/90509msec) slat (usec): min=2802, max=45684, avg=11209.37, stdev=5769.69 clat (msec): min=17, max=662, avg=348.42, stdev=83.54 lat (msec): min=24, max=669, avg=359.63, stdev=84.46 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 234], 10.00th=[ 255], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 326], 50.00th=[ 342], 60.00th=[ 363], | 70.00th=[ 384], 80.00th=[ 414], 90.00th=[ 460], 95.00th=[ 498], | 99.00th=[ 558], 99.50th=[ 567], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 7, max= 127, per=24.60%, avg=54.37, stdev=24.02, samples=180 iops : min= 1, max= 31, avg=13.16, stdev= 6.02, samples=180 write: IOPS=14, BW=58.2KiB/s (59.6kB/s)(5264KiB/90509msec); 0 zone resets slat (usec): min=55, max=18427, avg=347.36, stdev=907.45 clat (msec): min=251, max=1370, avg=758.39, stdev=187.69 lat (msec): min=251, max=1370, avg=758.74, stdev=187.67 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 439], 10.00th=[ 523], 20.00th=[ 609], | 30.00th=[ 676], 40.00th=[ 718], 50.00th=[ 751], 60.00th=[ 793], | 70.00th=[ 827], 80.00th=[ 894], 90.00th=[ 1011], 95.00th=[ 1099], | 99.00th=[ 1250], 99.50th=[ 1301], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 7, max= 144, per=25.00%, avg=57.49, stdev=21.74, samples=180 iops : min= 1, max= 36, avg=13.94, stdev= 5.42, samples=180 lat (msec) : 20=0.08%, 50=0.27%, 100=0.23%, 250=3.67%, 500=46.05% lat (msec) : 750=23.44%, 1000=20.86%, 2000=5.39% cpu : usr=0.07%, sys=0.70%, ctx=4571, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145990: Fri Aug 14 19:00:02 2026 read: IOPS=13, BW=55.0KiB/s (56.3kB/s)(4984KiB/90575msec) slat (usec): min=2866, max=49804, avg=11640.83, stdev=6209.25 clat (msec): min=21, max=741, avg=349.08, stdev=82.80 lat (msec): min=28, max=753, avg=360.73, stdev=83.73 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 232], 10.00th=[ 253], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 326], 50.00th=[ 347], 60.00th=[ 368], | 70.00th=[ 388], 80.00th=[ 418], 90.00th=[ 456], 95.00th=[ 489], | 99.00th=[ 542], 99.50th=[ 575], 99.90th=[ 726], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 7, max= 128, per=24.85%, avg=54.92, stdev=24.13, samples=179 iops : min= 1, max= 32, avg=13.30, stdev= 6.04, samples=179 write: IOPS=14, BW=58.0KiB/s (59.4kB/s)(5256KiB/90575msec); 0 zone resets slat (usec): min=52, max=14881, avg=316.50, stdev=638.39 clat (msec): min=106, max=1381, avg=758.45, stdev=184.62 lat (msec): min=107, max=1381, avg=758.77, stdev=184.62 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 472], 10.00th=[ 527], 20.00th=[ 617], | 30.00th=[ 667], 40.00th=[ 709], 50.00th=[ 751], 60.00th=[ 793], | 70.00th=[ 844], 80.00th=[ 902], 90.00th=[ 1003], 95.00th=[ 1083], | 99.00th=[ 1217], 99.50th=[ 1318], 99.90th=[ 1368], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 8, max= 120, per=24.88%, avg=57.23, stdev=20.08, samples=180 iops : min= 2, max= 30, avg=13.87, stdev= 5.01, samples=180 lat (msec) : 50=0.16%, 100=0.31%, 250=3.98%, 500=46.25%, 750=23.91% lat (msec) : 1000=20.16%, 2000=5.23% cpu : usr=0.08%, sys=0.72%, ctx=4870, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145991: Fri Aug 14 19:00:02 2026 read: IOPS=14, BW=56.0KiB/s (57.4kB/s)(5060KiB/90278msec) slat (usec): min=54, max=50152, avg=11312.97, stdev=5872.99 clat (msec): min=20, max=1040, avg=352.14, stdev=86.73 lat (msec): min=20, max=1049, avg=363.46, stdev=87.99 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 236], 10.00th=[ 257], 20.00th=[ 288], | 30.00th=[ 305], 40.00th=[ 330], 50.00th=[ 347], 60.00th=[ 368], | 70.00th=[ 388], 80.00th=[ 414], 90.00th=[ 456], 95.00th=[ 493], | 99.00th=[ 575], 99.50th=[ 609], 99.90th=[ 1003], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 7, max= 119, per=25.31%, avg=55.93, stdev=23.12, samples=178 iops : min= 1, max= 29, avg=13.54, stdev= 5.76, samples=178 write: IOPS=14, BW=57.4KiB/s (58.8kB/s)(5180KiB/90278msec); 0 zone resets slat (usec): min=40, max=12111, avg=321.90, stdev=680.48 clat (msec): min=134, max=1424, avg=759.15, stdev=195.52 lat (msec): min=134, max=1425, avg=759.48, stdev=195.53 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 464], 10.00th=[ 518], 20.00th=[ 609], | 30.00th=[ 667], 40.00th=[ 701], 50.00th=[ 751], 60.00th=[ 793], | 70.00th=[ 835], 80.00th=[ 902], 90.00th=[ 1020], 95.00th=[ 1116], | 99.00th=[ 1267], 99.50th=[ 1301], 99.90th=[ 1368], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 111, per=24.70%, avg=56.82, stdev=19.31, samples=179 iops : min= 1, max= 27, avg=13.78, stdev= 4.88, samples=179 lat (msec) : 50=0.20%, 100=0.08%, 250=4.22%, 500=46.68%, 750=23.79% lat (msec) : 1000=19.26%, 2000=5.78% cpu : usr=0.08%, sys=0.72%, ctx=4918, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=222KiB/s (227kB/s), 54.0KiB/s-56.0KiB/s (56.3kB/s-57.4kB/s), io=19.6MiB (20.5MB), run=90278-90575msec WRITE: bw=231KiB/s (236kB/s), 57.3KiB/s-58.2KiB/s (58.7kB/s-59.6kB/s), io=20.4MiB (21.4MB), run=90278-90575msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146009: Fri Aug 14 19:00:12 2026 read: IOPS=1, BW=69.9MiB/s (73.3MB/s)(40.0MiB/572msec) slat (nsec): min=30500k, max=30500k, avg=30499857.00, stdev= 0.00 clat (nsec): min=515103k, max=515103k, avg=515103123.00, stdev= 0.00 lat (nsec): min=545608k, max=545608k, avg=545607573.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 514], 10.00th=[ 514], 20.00th=[ 514], | 30.00th=[ 514], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] lat (msec) : 750=100.00% cpu : usr=0.18%, sys=2.45%, ctx=44, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=69.9MiB/s (73.3MB/s), 69.9MiB/s-69.9MiB/s (73.3MB/s-73.3MB/s), io=40.0MiB (41.9MB), run=572-572msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout