== sanity test 398c: run fio to test AIO ================= 08:22:10 (1787833330) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.452793 s, 92.6 MB/s osc.lustre-OST0000-osc-ffff99e334917800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146089: Thu Aug 27 08:24:57 2026 write: IOPS=16, BW=65.0KiB/s (66.6kB/s)(10.0MiB/157519msec); 0 zone resets slat (usec): min=22, max=17722, avg=159.43, stdev=644.12 clat (msec): min=231, max=2261, avg=984.04, stdev=292.86 lat (msec): min=231, max=2261, avg=984.20, stdev=292.87 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 575], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1469], | 99.00th=[ 2056], 99.50th=[ 2165], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 252, per=25.10%, avg=65.00, stdev=30.35, samples=310 iops : min= 1, max= 63, avg=15.82, stdev= 7.63, samples=310 lat (msec) : 250=0.62%, 500=3.28%, 750=13.28%, 1000=40.23%, 2000=41.48% lat (msec) : >=2000=1.09% cpu : usr=0.04%, sys=0.22%, ctx=1716, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146090: Thu Aug 27 08:24:57 2026 write: IOPS=16, BW=64.0KiB/s (66.5kB/s)(10.0MiB/157590msec); 0 zone resets slat (usec): min=22, max=16837, avg=164.15, stdev=671.38 clat (msec): min=344, max=2433, avg=983.35, stdev=294.39 lat (msec): min=344, max=2434, avg=983.52, stdev=294.42 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 852], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1469], | 99.00th=[ 2056], 99.50th=[ 2165], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=25.48%, avg=65.99, stdev=31.83, samples=305 iops : min= 1, max= 32, avg=16.06, stdev= 7.96, samples=305 lat (msec) : 500=3.75%, 750=14.69%, 1000=40.62%, 2000=39.53%, >=2000=1.41% cpu : usr=0.04%, sys=0.20%, ctx=1537, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146091: Thu Aug 27 08:24:57 2026 write: IOPS=16, BW=65.1KiB/s (66.6kB/s)(10.0MiB/157368msec); 0 zone resets slat (usec): min=21, max=17203, avg=150.13, stdev=556.43 clat (msec): min=220, max=2587, avg=982.28, stdev=299.00 lat (msec): min=220, max=2588, avg=982.44, stdev=299.01 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 567], 10.00th=[ 676], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1469], | 99.00th=[ 2056], 99.50th=[ 2165], 99.90th=[ 2467], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 252, per=26.03%, avg=67.41, stdev=32.70, samples=299 iops : min= 1, max= 63, avg=16.42, stdev= 8.21, samples=299 lat (msec) : 250=0.62%, 500=3.44%, 750=13.36%, 1000=41.33%, 2000=39.73% lat (msec) : >=2000=1.52% cpu : usr=0.05%, sys=0.21%, ctx=1757, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146092: Thu Aug 27 08:24:57 2026 write: IOPS=16, BW=65.0KiB/s (66.6kB/s)(10.0MiB/157449msec); 0 zone resets slat (usec): min=21, max=11240, avg=160.58, stdev=571.21 clat (msec): min=39, max=2256, avg=982.68, stdev=296.91 lat (msec): min=39, max=2256, avg=982.84, stdev=296.90 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 550], 10.00th=[ 676], 20.00th=[ 776], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1469], | 99.00th=[ 2056], 99.50th=[ 2165], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 216, per=26.03%, avg=67.42, stdev=32.92, samples=299 iops : min= 1, max= 54, avg=16.42, stdev= 8.27, samples=299 lat (msec) : 50=0.04%, 250=0.12%, 500=4.06%, 750=12.58%, 1000=42.11% lat (msec) : 2000=39.49%, >=2000=1.60% cpu : usr=0.05%, sys=0.20%, ctx=1773, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=260KiB/s (266kB/s), 64.0KiB/s-65.1KiB/s (66.5kB/s-66.6kB/s), io=40.0MiB (41.9MB), run=157368-157590msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146126: Thu Aug 27 08:26:59 2026 read: IOPS=11, BW=45.0KiB/s (46.1kB/s)(5048KiB/112052msec) slat (usec): min=38, max=61153, avg=14567.52, stdev=8707.31 clat (msec): min=19, max=1229, avg=425.25, stdev=132.45 lat (msec): min=26, max=1263, avg=439.82, stdev=133.95 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 268], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 359], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 426], | 70.00th=[ 456], 80.00th=[ 506], 90.00th=[ 584], 95.00th=[ 642], | 99.00th=[ 911], 99.50th=[ 1011], 99.90th=[ 1217], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 119, per=25.14%, avg=44.74, stdev=21.77, samples=221 iops : min= 1, max= 29, avg=10.67, stdev= 5.47, samples=221 write: IOPS=11, BW=46.3KiB/s (47.4kB/s)(5192KiB/112052msec); 0 zone resets slat (usec): min=25, max=12855, avg=180.81, stdev=650.08 clat (msec): min=212, max=2511, avg=952.54, stdev=303.46 lat (msec): min=212, max=2511, avg=952.73, stdev=303.46 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 535], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1045], 80.00th=[ 1150], 90.00th=[ 1334], 95.00th=[ 1552], | 99.00th=[ 1838], 99.50th=[ 2005], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 119, per=24.91%, avg=46.09, stdev=20.62, samples=220 iops : min= 1, max= 29, avg=11.00, stdev= 5.23, samples=220 lat (msec) : 20=0.04%, 50=0.12%, 100=0.16%, 250=1.52%, 500=38.48% lat (msec) : 750=19.69%, 1000=21.56%, 2000=18.20%, >=2000=0.23% cpu : usr=0.10%, sys=0.54%, ctx=4043, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146127: Thu Aug 27 08:26:59 2026 read: IOPS=11, BW=44.2KiB/s (45.3kB/s)(4976KiB/112521msec) slat (usec): min=2490, max=76975, avg=14701.38, stdev=8757.29 clat (msec): min=15, max=1102, avg=424.76, stdev=128.41 lat (msec): min=25, max=1117, avg=439.47, stdev=129.83 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 380], 50.00th=[ 401], 60.00th=[ 422], | 70.00th=[ 456], 80.00th=[ 510], 90.00th=[ 584], 95.00th=[ 659], | 99.00th=[ 844], 99.50th=[ 936], 99.90th=[ 1003], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 111, per=24.97%, avg=44.45, stdev=21.52, samples=219 iops : min= 1, max= 27, avg=10.59, stdev= 5.40, samples=219 write: IOPS=11, BW=46.8KiB/s (47.9kB/s)(5264KiB/112521msec); 0 zone resets slat (usec): min=29, max=13132, avg=208.96, stdev=680.84 clat (msec): min=211, max=2582, avg=949.46, stdev=303.31 lat (msec): min=211, max=2582, avg=949.67, stdev=303.30 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 844], 50.00th=[ 902], 60.00th=[ 969], | 70.00th=[ 1036], 80.00th=[ 1150], 90.00th=[ 1334], 95.00th=[ 1536], | 99.00th=[ 1888], 99.50th=[ 2123], 99.90th=[ 2400], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 103, per=24.96%, avg=46.18, stdev=20.25, samples=223 iops : min= 1, max= 25, avg=11.02, stdev= 5.11, samples=223 lat (msec) : 20=0.08%, 100=0.27%, 250=1.21%, 500=38.32%, 750=19.77% lat (msec) : 1000=22.30%, 2000=17.66%, >=2000=0.39% cpu : usr=0.09%, sys=0.57%, ctx=4508, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146128: Thu Aug 27 08:26:59 2026 read: IOPS=11, BW=44.3KiB/s (45.4kB/s)(4984KiB/112452msec) slat (usec): min=2428, max=68327, avg=14946.01, stdev=8488.19 clat (msec): min=36, max=1212, avg=429.99, stdev=129.17 lat (msec): min=44, max=1253, avg=444.94, stdev=130.20 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 284], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 363], 40.00th=[ 380], 50.00th=[ 401], 60.00th=[ 426], | 70.00th=[ 464], 80.00th=[ 518], 90.00th=[ 592], 95.00th=[ 667], | 99.00th=[ 869], 99.50th=[ 1011], 99.90th=[ 1099], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 103, per=25.11%, avg=44.70, stdev=19.91, samples=219 iops : min= 1, max= 25, avg=10.64, stdev= 5.03, samples=219 write: IOPS=11, BW=46.7KiB/s (47.9kB/s)(5256KiB/112452msec); 0 zone resets slat (usec): min=30, max=26368, avg=218.11, stdev=932.85 clat (msec): min=188, max=2590, avg=943.91, stdev=292.60 lat (msec): min=189, max=2591, avg=944.13, stdev=292.63 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 894], 60.00th=[ 953], | 70.00th=[ 1028], 80.00th=[ 1133], 90.00th=[ 1318], 95.00th=[ 1536], | 99.00th=[ 1854], 99.50th=[ 1989], 99.90th=[ 2534], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 96, per=24.82%, avg=45.91, stdev=20.04, samples=223 iops : min= 1, max= 24, avg=10.96, stdev= 5.07, samples=223 lat (msec) : 50=0.04%, 100=0.20%, 250=0.94%, 500=37.38%, 750=21.41% lat (msec) : 1000=21.95%, 2000=17.85%, >=2000=0.23% cpu : usr=0.10%, sys=0.53%, ctx=3935, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146129: Thu Aug 27 08:26:59 2026 read: IOPS=11, BW=44.0KiB/s (46.1kB/s)(5060KiB/112511msec) slat (msec): min=2, max=374, avg=15.23, stdev=15.76 clat (msec): min=14, max=1100, avg=429.21, stdev=126.39 lat (msec): min=22, max=1117, avg=444.45, stdev=127.24 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 279], 10.00th=[ 305], 20.00th=[ 334], | 30.00th=[ 359], 40.00th=[ 384], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 506], 90.00th=[ 592], 95.00th=[ 667], | 99.00th=[ 894], 99.50th=[ 927], 99.90th=[ 1020], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 127, per=25.04%, avg=44.56, stdev=21.91, samples=222 iops : min= 1, max= 31, avg=10.62, stdev= 5.51, samples=222 write: IOPS=11, BW=46.0KiB/s (47.1kB/s)(5180KiB/112511msec); 0 zone resets slat (usec): min=24, max=17146, avg=227.90, stdev=909.70 clat (msec): min=275, max=2645, avg=952.54, stdev=305.64 lat (msec): min=275, max=2645, avg=952.77, stdev=305.72 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 550], 10.00th=[ 634], 20.00th=[ 735], | 30.00th=[ 785], 40.00th=[ 844], 50.00th=[ 911], 60.00th=[ 969], | 70.00th=[ 1045], 80.00th=[ 1150], 90.00th=[ 1318], 95.00th=[ 1519], | 99.00th=[ 1871], 99.50th=[ 2366], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 104, per=24.63%, avg=45.57, stdev=20.56, samples=221 iops : min= 1, max= 26, avg=10.87, stdev= 5.17, samples=221 lat (msec) : 20=0.08%, 50=0.04%, 100=0.08%, 250=1.05%, 500=38.83% lat (msec) : 750=19.84%, 1000=21.76%, 2000=17.97%, >=2000=0.35% cpu : usr=0.09%, sys=0.58%, ctx=4493, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=178KiB/s (183kB/s), 44.2KiB/s-45.0KiB/s (45.3kB/s-46.1kB/s), io=19.6MiB (20.5MB), run=112052-112521msec WRITE: bw=186KiB/s (190kB/s), 46.0KiB/s-46.8KiB/s (47.1kB/s-47.9kB/s), io=20.4MiB (21.4MB), run=112052-112521msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146150: Thu Aug 27 08:27:10 2026 read: IOPS=1, BW=60.2MiB/s (63.2MB/s)(40.0MiB/664msec) slat (nsec): min=37994k, max=37994k, avg=37994207.00, stdev= 0.00 clat (nsec): min=608080k, max=608080k, avg=608080065.00, stdev= 0.00 lat (nsec): min=646080k, max=646080k, avg=646080409.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 609], 10.00th=[ 609], 20.00th=[ 609], | 30.00th=[ 609], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 609], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.41%, ctx=6, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=60.2MiB/s (63.2MB/s), 60.2MiB/s-60.2MiB/s (63.2MB/s-63.2MB/s), io=40.0MiB (41.9MB), run=664-664msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout