== sanity test 398c: run fio to test AIO ================= 10:51:39 (1788879099) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.204991 s, 205 MB/s osc.lustre-OST0000-osc-ffff939a00b6d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147131: Tue Sep 8 10:52:58 2026 write: IOPS=34, BW=138KiB/s (142kB/s)(10.0MiB/73978msec); 0 zone resets slat (usec): min=22, max=3489, avg=160.29, stdev=241.13 clat (msec): min=131, max=912, avg=461.70, stdev=100.84 lat (msec): min=131, max=912, avg=461.86, stdev=100.85 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 292], 10.00th=[ 342], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 481], | 70.00th=[ 510], 80.00th=[ 542], 90.00th=[ 592], 95.00th=[ 634], | 99.00th=[ 718], 99.50th=[ 751], 99.90th=[ 835], 99.95th=[ 877], | 99.99th=[ 911] bw ( KiB/s): min= 55, max= 256, per=24.93%, avg=137.84, stdev=31.95, samples=147 iops : min= 13, max= 64, avg=34.16, stdev= 8.05, samples=147 lat (msec) : 250=1.17%, 500=66.13%, 750=32.19%, 1000=0.51% cpu : usr=0.07%, sys=0.40%, ctx=2718, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147132: Tue Sep 8 10:52:58 2026 write: IOPS=34, BW=138KiB/s (142kB/s)(10.0MiB/73974msec); 0 zone resets slat (usec): min=21, max=4733, avg=174.88, stdev=290.11 clat (msec): min=154, max=874, avg=461.45, stdev=102.43 lat (msec): min=154, max=874, avg=461.63, stdev=102.44 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 292], 10.00th=[ 342], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 510], 80.00th=[ 542], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 735], 99.50th=[ 768], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 877] bw ( KiB/s): min= 56, max= 248, per=24.92%, avg=137.80, stdev=30.86, samples=147 iops : min= 14, max= 62, avg=34.15, stdev= 7.77, samples=147 lat (msec) : 250=2.07%, 500=63.55%, 750=33.63%, 1000=0.74% cpu : usr=0.09%, sys=0.39%, ctx=3109, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147133: Tue Sep 8 10:52:58 2026 write: IOPS=34, BW=138KiB/s (142kB/s)(10.0MiB/73974msec); 0 zone resets slat (usec): min=21, max=4800, avg=176.90, stdev=308.38 clat (msec): min=154, max=910, avg=461.51, stdev=102.75 lat (msec): min=154, max=910, avg=461.69, stdev=102.78 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 284], 10.00th=[ 342], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 510], 80.00th=[ 542], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 718], 99.50th=[ 751], 99.90th=[ 793], 99.95th=[ 802], | 99.99th=[ 911] bw ( KiB/s): min= 72, max= 256, per=24.92%, avg=137.79, stdev=32.69, samples=147 iops : min= 18, max= 64, avg=34.15, stdev= 8.26, samples=147 lat (msec) : 250=2.62%, 500=63.98%, 750=32.77%, 1000=0.62% cpu : usr=0.09%, sys=0.38%, ctx=2831, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147134: Tue Sep 8 10:52:58 2026 write: IOPS=34, BW=139KiB/s (142kB/s)(10.0MiB/73603msec); 0 zone resets slat (usec): min=24, max=4619, avg=173.60, stdev=292.93 clat (msec): min=138, max=876, avg=459.61, stdev=103.02 lat (msec): min=138, max=876, avg=459.78, stdev=103.02 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 288], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 456], 60.00th=[ 481], | 70.00th=[ 506], 80.00th=[ 535], 90.00th=[ 592], 95.00th=[ 634], | 99.00th=[ 726], 99.50th=[ 760], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 877] bw ( KiB/s): min= 64, max= 256, per=25.09%, avg=138.74, stdev=33.90, samples=146 iops : min= 16, max= 64, avg=34.38, stdev= 8.54, samples=146 lat (msec) : 250=1.95%, 500=65.51%, 750=31.95%, 1000=0.59% cpu : usr=0.07%, sys=0.40%, ctx=2937, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=554KiB/s (567kB/s), 138KiB/s-139KiB/s (142kB/s-142kB/s), io=40.0MiB (41.9MB), run=73603-73978msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147154: Tue Sep 8 10:53:49 2026 read: IOPS=26, BW=108KiB/s (110kB/s)(5048KiB/46819msec) slat (usec): min=29, max=17805, avg=5705.41, stdev=2371.21 clat (msec): min=22, max=480, avg=184.33, stdev=44.66 lat (msec): min=24, max=487, avg=190.04, stdev=45.19 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 117], 10.00th=[ 134], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 220], 90.00th=[ 236], 95.00th=[ 253], | 99.00th=[ 313], 99.50th=[ 334], 99.90th=[ 430], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 39, max= 208, per=25.27%, avg=107.41, stdev=31.43, samples=93 iops : min= 9, max= 52, avg=26.59, stdev= 7.95, samples=93 write: IOPS=27, BW=111KiB/s (114kB/s)(5192KiB/46819msec); 0 zone resets slat (usec): min=30, max=4445, avg=195.53, stdev=306.35 clat (msec): min=83, max=751, avg=391.65, stdev=92.46 lat (msec): min=83, max=751, avg=391.85, stdev=92.47 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 249], 10.00th=[ 279], 20.00th=[ 313], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 435], 80.00th=[ 464], 90.00th=[ 502], 95.00th=[ 535], | 99.00th=[ 659], 99.50th=[ 684], 99.90th=[ 743], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 40, max= 192, per=24.88%, avg=110.23, stdev=28.63, samples=93 iops : min= 10, max= 48, avg=27.30, stdev= 7.19, samples=93 lat (msec) : 50=0.27%, 100=0.86%, 250=48.05%, 500=45.74%, 750=5.04% lat (msec) : 1000=0.04% cpu : usr=0.14%, sys=1.02%, ctx=5093, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147155: Tue Sep 8 10:53:49 2026 read: IOPS=26, BW=106KiB/s (108kB/s)(4976KiB/47123msec) slat (usec): min=1955, max=221970, avg=6380.95, stdev=10562.83 clat (msec): min=3, max=846, avg=181.75, stdev=52.79 lat (msec): min=8, max=863, avg=188.13, stdev=56.29 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 110], 10.00th=[ 132], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 220], 90.00th=[ 236], 95.00th=[ 253], | 99.00th=[ 300], 99.50th=[ 326], 99.90th=[ 600], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 39, max= 208, per=24.21%, avg=102.90, stdev=34.91, samples=93 iops : min= 9, max= 52, avg=25.46, stdev= 8.73, samples=93 write: IOPS=27, BW=112KiB/s (114kB/s)(5264KiB/47123msec); 0 zone resets slat (usec): min=34, max=4902, avg=202.84, stdev=319.00 clat (msec): min=26, max=1082, avg=393.74, stdev=113.17 lat (msec): min=26, max=1082, avg=393.95, stdev=113.18 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 224], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 376], 50.00th=[ 397], 60.00th=[ 418], | 70.00th=[ 439], 80.00th=[ 464], 90.00th=[ 502], 95.00th=[ 542], | 99.00th=[ 709], 99.50th=[ 936], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 24, max= 168, per=24.56%, avg=108.78, stdev=23.95, samples=93 iops : min= 6, max= 42, avg=26.94, stdev= 6.05, samples=93 lat (msec) : 4=0.04%, 10=0.08%, 20=0.31%, 50=1.05%, 100=1.17% lat (msec) : 250=46.99%, 500=44.73%, 750=5.08%, 1000=0.35%, 2000=0.20% cpu : usr=0.14%, sys=1.02%, ctx=5097, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147156: Tue Sep 8 10:53:49 2026 read: IOPS=26, BW=106KiB/s (109kB/s)(4984KiB/46909msec) slat (usec): min=1815, max=18257, avg=5727.06, stdev=2378.24 clat (msec): min=7, max=330, avg=183.94, stdev=43.34 lat (msec): min=16, max=341, avg=189.67, stdev=43.67 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 120], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 239], 95.00th=[ 259], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 39, max= 224, per=25.01%, avg=106.29, stdev=34.01, samples=93 iops : min= 9, max= 56, avg=26.31, stdev= 8.53, samples=93 write: IOPS=28, BW=112KiB/s (115kB/s)(5256KiB/46909msec); 0 zone resets slat (usec): min=34, max=5826, avg=196.80, stdev=317.24 clat (msec): min=86, max=799, avg=390.08, stdev=90.96 lat (msec): min=87, max=799, avg=390.28, stdev=90.95 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 245], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 464], 90.00th=[ 498], 95.00th=[ 531], | 99.00th=[ 651], 99.50th=[ 693], 99.90th=[ 760], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 48, max= 184, per=25.08%, avg=111.09, stdev=25.96, samples=93 iops : min= 12, max= 46, avg=27.51, stdev= 6.59, samples=93 lat (msec) : 10=0.04%, 20=0.12%, 50=0.31%, 100=0.47%, 250=47.50% lat (msec) : 500=46.76%, 750=4.73%, 1000=0.08% cpu : usr=0.14%, sys=1.03%, ctx=5137, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147157: Tue Sep 8 10:53:49 2026 read: IOPS=26, BW=108KiB/s (110kB/s)(5060KiB/46954msec) slat (usec): min=1886, max=19202, avg=5674.89, stdev=2346.19 clat (msec): min=10, max=321, avg=184.23, stdev=41.09 lat (msec): min=15, max=331, avg=189.91, stdev=41.54 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 123], 10.00th=[ 138], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 234], 95.00th=[ 255], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 31, max= 224, per=25.32%, avg=107.61, stdev=35.91, samples=93 iops : min= 7, max= 56, avg=26.65, stdev= 9.08, samples=93 write: IOPS=27, BW=110KiB/s (113kB/s)(5180KiB/46954msec); 0 zone resets slat (usec): min=28, max=7832, avg=202.60, stdev=385.28 clat (msec): min=108, max=799, avg=393.17, stdev=91.51 lat (msec): min=108, max=799, avg=393.37, stdev=91.52 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 255], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 435], 80.00th=[ 464], 90.00th=[ 493], 95.00th=[ 531], | 99.00th=[ 676], 99.50th=[ 718], 99.90th=[ 785], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 55, max= 184, per=24.71%, avg=109.46, stdev=26.86, samples=93 iops : min= 13, max= 46, avg=27.11, stdev= 6.74, samples=93 lat (msec) : 20=0.04%, 50=0.16%, 100=0.47%, 250=48.16%, 500=46.48% lat (msec) : 750=4.57%, 1000=0.12% cpu : usr=0.17%, sys=0.98%, ctx=5256, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=426KiB/s (436kB/s), 106KiB/s-108KiB/s (108kB/s-110kB/s), io=19.6MiB (20.5MB), run=46819-47123msec WRITE: bw=443KiB/s (454kB/s), 110KiB/s-112KiB/s (113kB/s-115kB/s), io=20.4MiB (21.4MB), run=46819-47123msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147167: Tue Sep 8 10:53:53 2026 read: IOPS=5, BW=202MiB/s (212MB/s)(40.0MiB/198msec) slat (nsec): min=19552k, max=19552k, avg=19551856.00, stdev= 0.00 clat (nsec): min=166999k, max=166999k, avg=166999388.00, stdev= 0.00 lat (nsec): min=186556k, max=186556k, avg=186556385.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 167], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.08%, ctx=37, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=202MiB/s (212MB/s), 202MiB/s-202MiB/s (212MB/s-212MB/s), io=40.0MiB (41.9MB), run=198-198msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout