== sanity test 398c: run fio to test AIO ================= 20:24:51 (1789086291) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.227863 s, 184 MB/s osc.lustre-OST0000-osc-ffff9f1b5019b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144863: Thu Sep 10 20:27:43 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/163046msec); 0 zone resets slat (usec): min=35, max=14769, avg=270.80, stdev=769.97 clat (msec): min=300, max=2757, avg=1017.50, stdev=349.56 lat (msec): min=300, max=2757, avg=1017.77, stdev=349.61 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 451], 10.00th=[ 617], 20.00th=[ 751], | 30.00th=[ 844], 40.00th=[ 919], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1418], 95.00th=[ 1620], | 99.00th=[ 2198], 99.50th=[ 2400], 99.90th=[ 2534], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 256, per=25.33%, avg=63.58, stdev=34.28, samples=317 iops : min= 1, max= 64, avg=15.48, stdev= 8.62, samples=317 lat (msec) : 500=6.09%, 750=13.63%, 1000=30.94%, 2000=47.85%, >=2000=1.48% cpu : usr=0.04%, sys=0.26%, ctx=1920, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144864: Thu Sep 10 20:27:43 2026 write: IOPS=15, BW=62.9KiB/s (64.4kB/s)(10.0MiB/162754msec); 0 zone resets slat (usec): min=36, max=15503, avg=316.57, stdev=1024.15 clat (msec): min=87, max=2634, avg=1016.38, stdev=351.84 lat (msec): min=88, max=2634, avg=1016.70, stdev=351.80 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 439], 10.00th=[ 592], 20.00th=[ 751], | 30.00th=[ 844], 40.00th=[ 911], 50.00th=[ 995], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1620], | 99.00th=[ 2123], 99.50th=[ 2232], 99.90th=[ 2433], 99.95th=[ 2534], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 216, per=25.01%, avg=62.78, stdev=33.46, samples=321 iops : min= 1, max= 54, avg=15.28, stdev= 8.39, samples=321 lat (msec) : 100=0.20%, 250=0.59%, 500=5.55%, 750=13.79%, 1000=30.62% lat (msec) : 2000=47.89%, >=2000=1.37% cpu : usr=0.05%, sys=0.24%, ctx=2277, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144865: Thu Sep 10 20:27:43 2026 write: IOPS=15, BW=62.9KiB/s (64.4kB/s)(10.0MiB/162891msec); 0 zone resets slat (usec): min=35, max=17380, avg=291.46, stdev=919.23 clat (msec): min=61, max=2635, avg=1016.64, stdev=347.00 lat (msec): min=62, max=2635, avg=1016.94, stdev=347.04 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 435], 10.00th=[ 625], 20.00th=[ 760], | 30.00th=[ 844], 40.00th=[ 919], 50.00th=[ 995], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1653], | 99.00th=[ 2056], 99.50th=[ 2232], 99.90th=[ 2534], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 247, per=25.09%, avg=62.98, stdev=33.85, samples=320 iops : min= 1, max= 61, avg=15.33, stdev= 8.49, samples=320 lat (msec) : 100=0.04%, 250=0.59%, 500=5.55%, 750=13.32%, 1000=31.37% lat (msec) : 2000=47.81%, >=2000=1.33% cpu : usr=0.04%, sys=0.27%, ctx=2486, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144866: Thu Sep 10 20:27:43 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/163045msec); 0 zone resets slat (usec): min=36, max=20633, avg=282.54, stdev=967.54 clat (msec): min=312, max=2671, avg=1017.57, stdev=350.50 lat (msec): min=312, max=2672, avg=1017.85, stdev=350.52 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 451], 10.00th=[ 617], 20.00th=[ 743], | 30.00th=[ 835], 40.00th=[ 919], 50.00th=[ 995], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1418], 95.00th=[ 1653], | 99.00th=[ 2123], 99.50th=[ 2400], 99.90th=[ 2668], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 256, per=25.17%, avg=63.18, stdev=30.40, samples=319 iops : min= 1, max= 64, avg=15.38, stdev= 7.62, samples=319 lat (msec) : 500=6.02%, 750=14.61%, 1000=30.47%, 2000=47.62%, >=2000=1.29% cpu : usr=0.05%, sys=0.26%, ctx=2153, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=251KiB/s (257kB/s), 62.8KiB/s-62.9KiB/s (64.3kB/s-64.4kB/s), io=40.0MiB (41.9MB), run=162754-163046msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144899: Thu Sep 10 20:29:55 2026 read: IOPS=10, BW=41.5KiB/s (42.5kB/s)(5048KiB/121770msec) slat (usec): min=3194, max=72101, avg=16341.56, stdev=9838.01 clat (msec): min=11, max=1105, avg=470.14, stdev=145.11 lat (msec): min=20, max=1136, avg=486.49, stdev=147.42 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 359], | 30.00th=[ 393], 40.00th=[ 418], 50.00th=[ 447], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 567], 90.00th=[ 667], 95.00th=[ 768], | 99.00th=[ 894], 99.50th=[ 969], 99.90th=[ 1036], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 112, per=24.96%, avg=40.94, stdev=20.91, samples=241 iops : min= 1, max= 28, avg= 9.76, stdev= 5.24, samples=241 write: IOPS=10, BW=42.6KiB/s (43.7kB/s)(5192KiB/121770msec); 0 zone resets slat (usec): min=47, max=22700, avg=332.36, stdev=887.50 clat (msec): min=277, max=2030, avg=1024.06, stdev=313.95 lat (msec): min=278, max=2031, avg=1024.40, stdev=313.96 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 735], | 30.00th=[ 827], 40.00th=[ 919], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1452], 95.00th=[ 1586], | 99.00th=[ 1821], 99.50th=[ 1888], 99.90th=[ 1972], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 111, per=24.77%, avg=42.35, stdev=19.27, samples=240 iops : min= 1, max= 27, avg=10.11, stdev= 4.83, samples=240 lat (msec) : 20=0.04%, 50=0.08%, 100=0.08%, 250=1.29%, 500=32.54% lat (msec) : 750=23.44%, 1000=16.45%, 2000=26.05%, >=2000=0.04% cpu : usr=0.09%, sys=0.64%, ctx=4591, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144900: Thu Sep 10 20:29:55 2026 read: IOPS=10, BW=40.9KiB/s (41.8kB/s)(4976KiB/121771msec) slat (usec): min=3234, max=77901, avg=16508.68, stdev=9475.08 clat (msec): min=78, max=1077, avg=472.53, stdev=143.99 lat (msec): min=84, max=1088, avg=489.05, stdev=146.40 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 271], 10.00th=[ 313], 20.00th=[ 363], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 481], | 70.00th=[ 531], 80.00th=[ 584], 90.00th=[ 659], 95.00th=[ 735], | 99.00th=[ 877], 99.50th=[ 944], 99.90th=[ 1053], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 119, per=24.65%, avg=40.42, stdev=20.44, samples=240 iops : min= 1, max= 29, avg= 9.63, stdev= 5.14, samples=240 write: IOPS=10, BW=43.2KiB/s (44.3kB/s)(5264KiB/121771msec); 0 zone resets slat (usec): min=54, max=23365, avg=358.52, stdev=1076.58 clat (msec): min=315, max=2080, avg=1014.45, stdev=314.99 lat (msec): min=315, max=2080, avg=1014.81, stdev=315.06 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 531], 10.00th=[ 609], 20.00th=[ 726], | 30.00th=[ 818], 40.00th=[ 919], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1301], 90.00th=[ 1435], 95.00th=[ 1536], | 99.00th=[ 1838], 99.50th=[ 1905], 99.90th=[ 2039], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 119, per=25.28%, avg=43.23, stdev=21.40, samples=239 iops : min= 1, max= 29, avg=10.33, stdev= 5.38, samples=239 lat (msec) : 100=0.27%, 250=1.68%, 500=31.29%, 750=24.10%, 1000=16.17% lat (msec) : 2000=26.37%, >=2000=0.12% cpu : usr=0.10%, sys=0.67%, ctx=4663, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144901: Thu Sep 10 20:29:55 2026 read: IOPS=10, BW=40.9KiB/s (41.9kB/s)(4984KiB/121770msec) slat (usec): min=3078, max=71332, avg=16288.81, stdev=9749.14 clat (msec): min=30, max=1063, avg=465.25, stdev=147.61 lat (msec): min=48, max=1074, avg=481.54, stdev=149.58 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 271], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 447], 60.00th=[ 481], | 70.00th=[ 514], 80.00th=[ 567], 90.00th=[ 667], 95.00th=[ 743], | 99.00th=[ 919], 99.50th=[ 986], 99.90th=[ 1036], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 120, per=25.26%, avg=41.43, stdev=22.61, samples=236 iops : min= 1, max= 30, avg= 9.88, stdev= 5.70, samples=236 write: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5256KiB/121770msec); 0 zone resets slat (usec): min=38, max=32601, avg=402.30, stdev=1503.82 clat (msec): min=299, max=2026, avg=1022.32, stdev=315.89 lat (msec): min=299, max=2026, avg=1022.73, stdev=315.88 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 835], 40.00th=[ 927], 50.00th=[ 1003], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1452], 95.00th=[ 1586], | 99.00th=[ 1804], 99.50th=[ 1955], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 128, per=24.89%, avg=42.57, stdev=21.43, samples=241 iops : min= 1, max= 32, avg=10.17, stdev= 5.37, samples=241 lat (msec) : 50=0.08%, 100=0.35%, 250=1.48%, 500=32.15%, 750=23.20% lat (msec) : 1000=16.88%, 2000=25.70%, >=2000=0.16% cpu : usr=0.10%, sys=0.65%, ctx=4527, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144902: Thu Sep 10 20:29:55 2026 read: IOPS=10, BW=41.7KiB/s (42.7kB/s)(5060KiB/121270msec) slat (usec): min=95, max=69141, avg=16595.51, stdev=9836.20 clat (msec): min=18, max=990, avg=470.09, stdev=147.40 lat (msec): min=28, max=1044, avg=486.69, stdev=150.20 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 279], 10.00th=[ 317], 20.00th=[ 355], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 443], 60.00th=[ 481], | 70.00th=[ 518], 80.00th=[ 575], 90.00th=[ 676], 95.00th=[ 751], | 99.00th=[ 902], 99.50th=[ 953], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 119, per=25.59%, avg=41.96, stdev=20.91, samples=236 iops : min= 1, max= 29, avg=10.00, stdev= 5.25, samples=236 write: IOPS=10, BW=42.7KiB/s (43.7kB/s)(5180KiB/121270msec); 0 zone resets slat (usec): min=57, max=13956, avg=375.00, stdev=997.78 clat (msec): min=191, max=2108, avg=1021.84, stdev=309.60 lat (msec): min=191, max=2116, avg=1022.22, stdev=309.59 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 550], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 827], 40.00th=[ 936], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1435], 95.00th=[ 1536], | 99.00th=[ 1821], 99.50th=[ 1905], 99.90th=[ 1955], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 103, per=24.78%, avg=42.38, stdev=18.39, samples=239 iops : min= 1, max= 25, avg=10.12, stdev= 4.63, samples=239 lat (msec) : 20=0.04%, 50=0.16%, 100=0.20%, 250=1.68%, 500=31.95% lat (msec) : 750=23.20%, 1000=16.76%, 2000=25.98%, >=2000=0.04% cpu : usr=0.08%, sys=0.66%, ctx=4589, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=165KiB/s (169kB/s), 40.9KiB/s-41.7KiB/s (41.8kB/s-42.7kB/s), io=19.6MiB (20.5MB), run=121270-121771msec WRITE: bw=172KiB/s (176kB/s), 42.6KiB/s-43.2KiB/s (43.7kB/s-44.3kB/s), io=20.4MiB (21.4MB), run=121270-121771msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144924: Thu Sep 10 20:30:07 2026 read: IOPS=1, BW=69.1MiB/s (72.4MB/s)(40.0MiB/579msec) slat (nsec): min=34589k, max=34589k, avg=34589001.00, stdev= 0.00 clat (nsec): min=515332k, max=515332k, avg=515332286.00, stdev= 0.00 lat (nsec): min=549926k, max=549926k, avg=549926326.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 514], 10.00th=[ 514], 20.00th=[ 514], | 30.00th=[ 514], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.56%, ctx=25, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=69.1MiB/s (72.4MB/s), 69.1MiB/s-69.1MiB/s (72.4MB/s-72.4MB/s), io=40.0MiB (41.9MB), run=579-579msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout