== sanity test 398c: run fio to test AIO ================= 10:46:04 (1786632364) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.549459 s, 76.3 MB/s osc.lustre-OST0000-osc-ffff88a6a6a1c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146325: Thu Aug 13 10:48:43 2026 write: IOPS=17, BW=68.8KiB/s (70.4kB/s)(10.0MiB/148853msec); 0 zone resets slat (usec): min=35, max=18832, avg=238.51, stdev=869.12 clat (msec): min=361, max=2029, avg=928.66, stdev=234.86 lat (msec): min=361, max=2029, avg=928.90, stdev=234.92 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 550], 10.00th=[ 684], 20.00th=[ 760], | 30.00th=[ 810], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1234], 95.00th=[ 1351], | 99.00th=[ 1536], 99.50th=[ 1838], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 256, per=25.39%, avg=69.83, stdev=32.95, samples=289 iops : min= 1, max= 64, avg=17.01, stdev= 8.28, samples=289 lat (msec) : 500=2.97%, 750=15.43%, 1000=49.61%, 2000=31.88%, >=2000=0.12% cpu : usr=0.04%, sys=0.27%, ctx=1870, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146326: Thu Aug 13 10:48:43 2026 write: IOPS=17, BW=68.8KiB/s (70.4kB/s)(10.0MiB/148920msec); 0 zone resets slat (usec): min=37, max=20566, avg=265.87, stdev=894.98 clat (msec): min=334, max=1914, avg=928.92, stdev=228.29 lat (msec): min=334, max=1914, avg=929.19, stdev=228.33 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 600], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 810], 40.00th=[ 852], 50.00th=[ 902], 60.00th=[ 944], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1234], 95.00th=[ 1368], | 99.00th=[ 1552], 99.50th=[ 1737], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 128, per=25.26%, avg=69.47, stdev=30.38, samples=290 iops : min= 1, max= 32, avg=16.92, stdev= 7.65, samples=290 lat (msec) : 500=2.97%, 750=15.82%, 1000=49.77%, 2000=31.45% cpu : usr=0.05%, sys=0.26%, ctx=2396, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146327: Thu Aug 13 10:48:43 2026 write: IOPS=17, BW=68.9KiB/s (70.5kB/s)(10.0MiB/148701msec); 0 zone resets slat (usec): min=37, max=13933, avg=249.54, stdev=807.57 clat (msec): min=82, max=1948, avg=927.61, stdev=236.64 lat (msec): min=83, max=1948, avg=927.86, stdev=236.64 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 810], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 953], | 70.00th=[ 1020], 80.00th=[ 1099], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1603], 99.50th=[ 1804], 99.90th=[ 1938], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 8, max= 176, per=25.21%, avg=69.32, stdev=27.69, samples=291 iops : min= 2, max= 44, avg=16.89, stdev= 6.96, samples=291 lat (msec) : 100=0.08%, 250=0.08%, 500=2.54%, 750=16.17%, 1000=48.75% lat (msec) : 2000=32.38% cpu : usr=0.04%, sys=0.27%, ctx=1832, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146328: Thu Aug 13 10:48:43 2026 write: IOPS=17, BW=68.0KiB/s (70.6kB/s)(10.0MiB/148424msec); 0 zone resets slat (usec): min=36, max=19279, avg=253.12, stdev=926.54 clat (msec): min=150, max=2068, avg=926.68, stdev=237.69 lat (msec): min=169, max=2074, avg=926.94, stdev=237.70 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 575], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 1003], 80.00th=[ 1099], 90.00th=[ 1234], 95.00th=[ 1368], | 99.00th=[ 1569], 99.50th=[ 1838], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 128, per=24.94%, avg=68.60, stdev=28.85, samples=294 iops : min= 1, max= 32, avg=16.70, stdev= 7.27, samples=294 lat (msec) : 250=0.08%, 500=2.73%, 750=16.41%, 1000=49.96%, 2000=30.78% lat (msec) : >=2000=0.04% cpu : usr=0.05%, sys=0.28%, ctx=1925, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=275KiB/s (282kB/s), 68.8KiB/s-68.0KiB/s (70.4kB/s-70.6kB/s), io=40.0MiB (41.9MB), run=148424-148920msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146360: Thu Aug 13 10:50:31 2026 read: IOPS=12, BW=49.6KiB/s (50.8kB/s)(5048KiB/101765msec) slat (usec): min=2629, max=81683, avg=14187.40, stdev=8071.64 clat (msec): min=28, max=1035, avg=388.84, stdev=95.36 lat (msec): min=36, max=1065, avg=403.04, stdev=96.97 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 266], 10.00th=[ 292], 20.00th=[ 321], | 30.00th=[ 342], 40.00th=[ 363], 50.00th=[ 380], 60.00th=[ 401], | 70.00th=[ 426], 80.00th=[ 456], 90.00th=[ 498], 95.00th=[ 542], | 99.00th=[ 667], 99.50th=[ 751], 99.90th=[ 969], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 120, per=24.91%, avg=49.07, stdev=20.89, samples=202 iops : min= 1, max= 30, avg=11.79, stdev= 5.24, samples=202 write: IOPS=12, BW=51.0KiB/s (52.2kB/s)(5192KiB/101765msec); 0 zone resets slat (usec): min=54, max=17562, avg=323.68, stdev=1011.17 clat (msec): min=315, max=1524, avg=857.71, stdev=208.18 lat (msec): min=316, max=1524, avg=858.03, stdev=208.18 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 542], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 885], | 70.00th=[ 944], 80.00th=[ 1020], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1452], 99.50th=[ 1485], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 103, per=24.68%, avg=50.59, stdev=18.50, samples=201 iops : min= 1, max= 25, avg=12.16, stdev= 4.65, samples=201 lat (msec) : 50=0.20%, 100=0.20%, 250=1.48%, 500=44.02%, 750=18.71% lat (msec) : 1000=24.02%, 2000=11.37% cpu : usr=0.09%, sys=0.74%, ctx=4862, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146361: Thu Aug 13 10:50:31 2026 read: IOPS=12, BW=48.9KiB/s (50.0kB/s)(4976KiB/101826msec) slat (usec): min=61, max=79120, avg=13796.30, stdev=8002.48 clat (msec): min=16, max=970, avg=394.94, stdev=94.36 lat (msec): min=24, max=982, avg=408.75, stdev=95.90 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 271], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 347], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 435], 80.00th=[ 460], 90.00th=[ 506], 95.00th=[ 558], | 99.00th=[ 651], 99.50th=[ 751], 99.90th=[ 944], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 96, per=24.51%, avg=48.29, stdev=20.16, samples=202 iops : min= 1, max= 24, avg=11.59, stdev= 5.10, samples=202 write: IOPS=12, BW=51.7KiB/s (52.9kB/s)(5264KiB/101826msec); 0 zone resets slat (usec): min=41, max=7747, avg=248.79, stdev=440.11 clat (msec): min=227, max=1541, avg=850.53, stdev=210.58 lat (msec): min=227, max=1542, avg=850.78, stdev=210.59 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 542], 10.00th=[ 592], 20.00th=[ 684], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 877], | 70.00th=[ 936], 80.00th=[ 1011], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1435], 99.50th=[ 1469], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 111, per=24.83%, avg=50.90, stdev=21.50, samples=202 iops : min= 1, max= 27, avg=12.25, stdev= 5.38, samples=202 lat (msec) : 20=0.04%, 50=0.12%, 100=0.04%, 250=1.33%, 500=43.12% lat (msec) : 750=21.05%, 1000=23.44%, 2000=10.86% cpu : usr=0.08%, sys=0.66%, ctx=4628, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146362: Thu Aug 13 10:50:31 2026 read: IOPS=12, BW=48.0KiB/s (50.2kB/s)(4984KiB/101721msec) slat (usec): min=2447, max=60016, avg=14320.76, stdev=8067.02 clat (msec): min=14, max=971, avg=389.39, stdev=93.42 lat (msec): min=21, max=980, avg=403.72, stdev=94.48 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 259], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 384], 60.00th=[ 401], | 70.00th=[ 430], 80.00th=[ 456], 90.00th=[ 502], 95.00th=[ 550], | 99.00th=[ 634], 99.50th=[ 667], 99.90th=[ 944], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 112, per=24.80%, avg=48.86, stdev=22.07, samples=201 iops : min= 1, max= 28, avg=11.74, stdev= 5.55, samples=201 write: IOPS=12, BW=51.7KiB/s (52.9kB/s)(5256KiB/101721msec); 0 zone resets slat (usec): min=43, max=8661, avg=270.81, stdev=529.31 clat (msec): min=255, max=1503, avg=852.41, stdev=212.47 lat (msec): min=257, max=1503, avg=852.68, stdev=212.45 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 531], 10.00th=[ 592], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 835], 60.00th=[ 877], | 70.00th=[ 936], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1418], 99.50th=[ 1452], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 103, per=24.77%, avg=50.79, stdev=19.96, samples=202 iops : min= 1, max= 25, avg=12.22, stdev= 5.01, samples=202 lat (msec) : 20=0.12%, 50=0.12%, 100=0.20%, 250=1.48%, 500=43.28% lat (msec) : 750=20.12%, 1000=23.01%, 2000=11.68% cpu : usr=0.11%, sys=0.69%, ctx=4807, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146363: Thu Aug 13 10:50:31 2026 read: IOPS=12, BW=49.8KiB/s (50.0kB/s)(5060KiB/101615msec) slat (usec): min=2933, max=75430, avg=14373.72, stdev=8032.87 clat (msec): min=30, max=902, avg=392.29, stdev=95.39 lat (msec): min=42, max=929, avg=406.67, stdev=96.38 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 266], 10.00th=[ 292], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 401], | 70.00th=[ 426], 80.00th=[ 456], 90.00th=[ 510], 95.00th=[ 550], | 99.00th=[ 701], 99.50th=[ 802], 99.90th=[ 877], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 102, per=25.25%, avg=49.75, stdev=20.14, samples=200 iops : min= 1, max= 25, avg=11.96, stdev= 5.09, samples=200 write: IOPS=12, BW=50.0KiB/s (52.2kB/s)(5180KiB/101615msec); 0 zone resets slat (usec): min=44, max=22751, avg=316.35, stdev=1122.73 clat (msec): min=312, max=1524, avg=854.15, stdev=209.53 lat (msec): min=312, max=1524, avg=854.47, stdev=209.51 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 542], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 835], 60.00th=[ 877], | 70.00th=[ 936], 80.00th=[ 1020], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 104, per=24.46%, avg=50.15, stdev=20.07, samples=202 iops : min= 1, max= 26, avg=12.06, stdev= 5.03, samples=202 lat (msec) : 50=0.23%, 100=0.04%, 250=1.45%, 500=43.63%, 750=20.16% lat (msec) : 1000=23.71%, 2000=10.78% cpu : usr=0.12%, sys=0.74%, ctx=4995, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=197KiB/s (202kB/s), 48.9KiB/s-49.8KiB/s (50.0kB/s-50.0kB/s), io=19.6MiB (20.5MB), run=101615-101826msec WRITE: bw=205KiB/s (210kB/s), 50.0KiB/s-51.7KiB/s (52.2kB/s-52.9kB/s), io=20.4MiB (21.4MB), run=101615-101826msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146382: Thu Aug 13 10:50:42 2026 read: IOPS=1, BW=69.4MiB/s (72.8MB/s)(40.0MiB/576msec) slat (nsec): min=28483k, max=28483k, avg=28483054.00, stdev= 0.00 clat (nsec): min=520606k, max=520606k, avg=520605682.00, stdev= 0.00 lat (nsec): min=549094k, max=549094k, avg=549093618.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 523], | 30.00th=[ 523], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 523], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.22%, ctx=5, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=69.4MiB/s (72.8MB/s), 69.4MiB/s-69.4MiB/s (72.8MB/s-72.8MB/s), io=40.0MiB (41.9MB), run=576-576msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout