== sanity test 398c: run fio to test AIO ================= 19:13:06 (1763338386) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.554327 s, 75.7 MB/s osc.lustre-OST0000-osc-ffff8d8208c2d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149788: Sun Nov 16 19:15:59 2025 write: IOPS=15, BW=62.3KiB/s (63.8kB/s)(10.0MiB/164406msec); 0 zone resets slat (usec): min=27, max=13205, avg=202.31, stdev=593.84 clat (msec): min=179, max=1708, avg=1027.01, stdev=245.49 lat (msec): min=179, max=1708, avg=1027.21, stdev=245.51 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 609], 10.00th=[ 751], 20.00th=[ 835], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1150], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1401], | 99.00th=[ 1603], 99.50th=[ 1636], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 239, per=25.27%, avg=62.67, stdev=30.61, samples=321 iops : min= 1, max= 59, avg=15.16, stdev= 7.66, samples=321 lat (msec) : 250=0.62%, 500=3.12%, 750=6.09%, 1000=34.73%, 2000=55.43% cpu : usr=0.04%, sys=0.25%, ctx=1499, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149789: Sun Nov 16 19:15:59 2025 write: IOPS=15, BW=62.2KiB/s (63.7kB/s)(10.0MiB/164645msec); 0 zone resets slat (usec): min=27, max=16200, avg=211.08, stdev=692.35 clat (msec): min=149, max=1784, avg=1028.07, stdev=244.91 lat (msec): min=149, max=1784, avg=1028.28, stdev=244.95 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 625], 10.00th=[ 743], 20.00th=[ 835], | 30.00th=[ 911], 40.00th=[ 986], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1150], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1418], | 99.00th=[ 1552], 99.50th=[ 1620], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 255, per=25.60%, avg=63.48, stdev=28.75, samples=317 iops : min= 1, max= 63, avg=15.37, stdev= 7.20, samples=317 lat (msec) : 250=0.62%, 500=3.16%, 750=6.60%, 1000=32.42%, 2000=57.19% cpu : usr=0.04%, sys=0.26%, ctx=1645, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149790: Sun Nov 16 19:15:59 2025 write: IOPS=15, BW=62.2KiB/s (63.7kB/s)(10.0MiB/164717msec); 0 zone resets slat (usec): min=26, max=12089, avg=201.71, stdev=570.34 clat (msec): min=272, max=1766, avg=1028.31, stdev=239.36 lat (msec): min=272, max=1766, avg=1028.51, stdev=239.37 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 642], 10.00th=[ 751], 20.00th=[ 844], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1036], 60.00th=[ 1083], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1418], | 99.00th=[ 1569], 99.50th=[ 1636], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 135, per=25.12%, avg=62.30, stdev=29.40, samples=323 iops : min= 1, max= 33, avg=15.07, stdev= 7.37, samples=323 lat (msec) : 500=3.12%, 750=6.91%, 1000=34.49%, 2000=55.47% cpu : usr=0.04%, sys=0.26%, ctx=1628, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149791: Sun Nov 16 19:15:59 2025 write: IOPS=15, BW=62.2KiB/s (63.6kB/s)(10.0MiB/164751msec); 0 zone resets slat (usec): min=26, max=6591, avg=196.37, stdev=483.61 clat (msec): min=57, max=1781, avg=1028.64, stdev=244.48 lat (msec): min=58, max=1781, avg=1028.84, stdev=244.48 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 625], 10.00th=[ 743], 20.00th=[ 844], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1036], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1586], 99.50th=[ 1636], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 167, per=25.36%, avg=62.88, stdev=29.60, samples=320 iops : min= 1, max= 41, avg=15.22, stdev= 7.43, samples=320 lat (msec) : 100=0.08%, 250=0.27%, 500=2.77%, 750=7.34%, 1000=33.63% lat (msec) : 2000=55.90% cpu : usr=0.06%, sys=0.25%, ctx=1616, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=249KiB/s (255kB/s), 62.2KiB/s-62.3KiB/s (63.6kB/s-63.8kB/s), io=40.0MiB (41.9MB), run=164406-164751msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149825: Sun Nov 16 19:17:52 2025 read: IOPS=11, BW=47.6KiB/s (48.8kB/s)(5048KiB/105957msec) slat (usec): min=2952, max=45569, avg=13254.42, stdev=6041.90 clat (msec): min=17, max=729, avg=422.53, stdev=85.37 lat (msec): min=28, max=763, avg=435.79, stdev=85.94 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 305], 10.00th=[ 326], 20.00th=[ 355], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 414], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 493], 90.00th=[ 531], 95.00th=[ 575], | 99.00th=[ 642], 99.50th=[ 667], 99.90th=[ 726], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 8, max= 111, per=24.99%, avg=47.23, stdev=19.95, samples=209 iops : min= 2, max= 27, avg=11.29, stdev= 5.01, samples=209 write: IOPS=12, BW=49.0KiB/s (50.2kB/s)(5192KiB/105957msec); 0 zone resets slat (usec): min=31, max=6068, avg=281.34, stdev=565.74 clat (msec): min=307, max=1509, avg=879.67, stdev=200.57 lat (msec): min=307, max=1509, avg=879.96, stdev=200.57 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 558], 10.00th=[ 617], 20.00th=[ 701], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1200], | 99.00th=[ 1334], 99.50th=[ 1385], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 104, per=24.68%, avg=48.61, stdev=18.08, samples=209 iops : min= 1, max= 26, avg=11.63, stdev= 4.56, samples=209 lat (msec) : 20=0.04%, 100=0.12%, 250=0.59%, 500=41.13%, 750=20.70% lat (msec) : 1000=22.85%, 2000=14.57% cpu : usr=0.11%, sys=0.74%, ctx=4903, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149826: Sun Nov 16 19:17:52 2025 read: IOPS=11, BW=46.9KiB/s (48.1kB/s)(4976KiB/106040msec) slat (usec): min=3454, max=40787, avg=13103.05, stdev=5408.00 clat (msec): min=42, max=702, avg=422.12, stdev=86.82 lat (msec): min=55, max=728, avg=435.23, stdev=87.49 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 300], 10.00th=[ 326], 20.00th=[ 355], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 493], 90.00th=[ 542], 95.00th=[ 575], | 99.00th=[ 634], 99.50th=[ 642], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 7, max= 119, per=24.43%, avg=46.17, stdev=20.72, samples=210 iops : min= 1, max= 29, avg=11.02, stdev= 5.21, samples=210 write: IOPS=12, BW=49.6KiB/s (50.8kB/s)(5264KiB/106040msec); 0 zone resets slat (usec): min=33, max=14495, avg=300.79, stdev=761.91 clat (msec): min=177, max=1463, avg=875.55, stdev=209.29 lat (msec): min=182, max=1464, avg=875.86, stdev=209.26 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 535], 10.00th=[ 600], 20.00th=[ 693], | 30.00th=[ 768], 40.00th=[ 818], 50.00th=[ 877], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1062], 90.00th=[ 1150], 95.00th=[ 1200], | 99.00th=[ 1334], 99.50th=[ 1401], 99.90th=[ 1435], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 8, max= 104, per=24.80%, avg=48.85, stdev=18.16, samples=210 iops : min= 2, max= 26, avg=11.70, stdev= 4.55, samples=210 lat (msec) : 50=0.04%, 100=0.27%, 250=0.43%, 500=40.82%, 750=21.09% lat (msec) : 1000=22.42%, 2000=14.92% cpu : usr=0.11%, sys=0.73%, ctx=4586, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149827: Sun Nov 16 19:17:53 2025 read: IOPS=11, BW=47.1KiB/s (48.3kB/s)(4984KiB/105724msec) slat (usec): min=30, max=49191, avg=13087.78, stdev=5912.17 clat (msec): min=20, max=880, avg=416.08, stdev=94.00 lat (msec): min=27, max=896, avg=429.18, stdev=95.01 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 292], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 435], | 70.00th=[ 460], 80.00th=[ 485], 90.00th=[ 527], 95.00th=[ 567], | 99.00th=[ 634], 99.50th=[ 659], 99.90th=[ 726], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 112, per=24.78%, avg=46.83, stdev=21.32, samples=209 iops : min= 1, max= 28, avg=11.19, stdev= 5.32, samples=209 write: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5256KiB/105724msec); 0 zone resets slat (usec): min=29, max=7222, avg=288.04, stdev=631.08 clat (msec): min=178, max=1838, avg=879.19, stdev=214.59 lat (msec): min=178, max=1838, avg=879.48, stdev=214.59 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 550], 10.00th=[ 609], 20.00th=[ 693], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 1003], 80.00th=[ 1053], 90.00th=[ 1133], 95.00th=[ 1217], | 99.00th=[ 1418], 99.50th=[ 1536], 99.90th=[ 1720], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 96, per=24.85%, avg=48.95, stdev=19.09, samples=209 iops : min= 1, max= 24, avg=11.72, stdev= 4.82, samples=209 lat (msec) : 50=0.27%, 100=0.43%, 250=1.02%, 500=40.86%, 750=19.69% lat (msec) : 1000=22.30%, 2000=15.43% cpu : usr=0.11%, sys=0.71%, ctx=4570, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149828: Sun Nov 16 19:17:53 2025 read: IOPS=11, BW=47.9KiB/s (49.0kB/s)(5060KiB/105671msec) slat (usec): min=3759, max=54445, avg=13237.29, stdev=5836.14 clat (msec): min=24, max=1231, avg=423.69, stdev=90.41 lat (msec): min=41, max=1250, avg=436.94, stdev=91.12 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 300], 10.00th=[ 326], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 393], 50.00th=[ 414], 60.00th=[ 439], | 70.00th=[ 464], 80.00th=[ 489], 90.00th=[ 531], 95.00th=[ 575], | 99.00th=[ 659], 99.50th=[ 701], 99.90th=[ 1036], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 117, per=25.27%, avg=47.75, stdev=19.22, samples=208 iops : min= 1, max= 29, avg=11.42, stdev= 4.80, samples=208 write: IOPS=12, BW=49.0KiB/s (50.2kB/s)(5180KiB/105671msec); 0 zone resets slat (usec): min=32, max=16960, avg=322.18, stdev=808.56 clat (msec): min=151, max=2079, avg=875.38, stdev=219.85 lat (msec): min=152, max=2080, avg=875.71, stdev=219.81 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 550], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1045], 90.00th=[ 1150], 95.00th=[ 1217], | 99.00th=[ 1385], 99.50th=[ 1888], 99.90th=[ 1972], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 95, per=24.64%, avg=48.55, stdev=17.96, samples=208 iops : min= 1, max= 23, avg=11.62, stdev= 4.55, samples=208 lat (msec) : 50=0.04%, 100=0.12%, 250=0.47%, 500=42.19%, 750=21.05% lat (msec) : 1000=22.27%, 2000=13.83%, >=2000=0.04% cpu : usr=0.11%, sys=0.74%, ctx=4750, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=189KiB/s (194kB/s), 46.9KiB/s-47.9KiB/s (48.1kB/s-49.0kB/s), io=19.6MiB (20.5MB), run=105671-106040msec WRITE: bw=197KiB/s (202kB/s), 49.0KiB/s-49.7KiB/s (50.2kB/s-50.9kB/s), io=20.4MiB (21.4MB), run=105671-106040msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149848: Sun Nov 16 19:18:02 2025 read: IOPS=1, BW=64.1MiB/s (67.2MB/s)(40.0MiB/624msec) slat (nsec): min=85506k, max=85506k, avg=85505991.00, stdev= 0.00 clat (nsec): min=516158k, max=516158k, avg=516158090.00, stdev= 0.00 lat (nsec): min=601669k, max=601669k, avg=601668956.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 518], 10.00th=[ 518], 20.00th=[ 518], | 30.00th=[ 518], 40.00th=[ 518], 50.00th=[ 518], 60.00th=[ 518], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.09%, ctx=15, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=64.1MiB/s (67.2MB/s), 64.1MiB/s-64.1MiB/s (67.2MB/s-67.2MB/s), io=40.0MiB (41.9MB), run=624-624msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout