== sanity test 398c: run fio to test AIO ================= 11:22:13 (1781277733) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.359907 s, 117 MB/s osc.lustre-OST0000-osc-ffff97d769278000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145444: Fri Jun 12 11:24:46 2026 write: IOPS=17, BW=71.7KiB/s (73.4kB/s)(10.0MiB/142861msec); 0 zone resets slat (usec): min=35, max=19771, avg=311.63, stdev=874.86 clat (msec): min=169, max=2123, avg=891.28, stdev=303.13 lat (msec): min=169, max=2123, avg=891.59, stdev=303.15 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 477], 10.00th=[ 558], 20.00th=[ 659], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 986], 80.00th=[ 1116], 90.00th=[ 1284], 95.00th=[ 1469], | 99.00th=[ 1804], 99.50th=[ 1921], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 253, per=24.97%, avg=71.42, stdev=31.71, samples=282 iops : min= 1, max= 63, avg=17.33, stdev= 7.94, samples=282 lat (msec) : 250=1.33%, 500=4.57%, 750=26.99%, 1000=38.24%, 2000=28.52% lat (msec) : >=2000=0.35% cpu : usr=0.05%, sys=0.36%, ctx=1955, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145445: Fri Jun 12 11:24:46 2026 write: IOPS=17, BW=71.7KiB/s (73.4kB/s)(10.0MiB/142811msec); 0 zone resets slat (usec): min=35, max=10698, avg=292.23, stdev=698.48 clat (msec): min=90, max=2892, avg=891.18, stdev=314.31 lat (msec): min=91, max=2892, avg=891.47, stdev=314.30 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 472], 10.00th=[ 558], 20.00th=[ 651], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 844], 60.00th=[ 919], | 70.00th=[ 995], 80.00th=[ 1099], 90.00th=[ 1301], 95.00th=[ 1469], | 99.00th=[ 1888], 99.50th=[ 2022], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 253, per=25.34%, avg=72.47, stdev=36.72, samples=278 iops : min= 1, max= 63, avg=17.59, stdev= 9.20, samples=278 lat (msec) : 100=0.04%, 250=1.21%, 500=5.43%, 750=28.32%, 1000=35.66% lat (msec) : 2000=28.83%, >=2000=0.51% cpu : usr=0.06%, sys=0.34%, ctx=1907, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145446: Fri Jun 12 11:24:46 2026 write: IOPS=17, BW=71.7KiB/s (73.4kB/s)(10.0MiB/142853msec); 0 zone resets slat (usec): min=37, max=22802, avg=325.70, stdev=1082.67 clat (msec): min=182, max=2190, avg=891.24, stdev=302.08 lat (msec): min=183, max=2190, avg=891.57, stdev=302.10 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 481], 10.00th=[ 567], 20.00th=[ 659], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1099], 90.00th=[ 1284], 95.00th=[ 1502], | 99.00th=[ 1804], 99.50th=[ 1955], 99.90th=[ 2123], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 253, per=25.34%, avg=72.47, stdev=38.60, samples=278 iops : min= 1, max= 63, avg=17.60, stdev= 9.66, samples=278 lat (msec) : 250=1.33%, 500=4.06%, 750=27.19%, 1000=39.02%, 2000=27.93% lat (msec) : >=2000=0.47% cpu : usr=0.05%, sys=0.34%, ctx=1942, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145447: Fri Jun 12 11:24:46 2026 write: IOPS=17, BW=71.7KiB/s (73.4kB/s)(10.0MiB/142811msec); 0 zone resets slat (usec): min=37, max=14792, avg=374.18, stdev=1038.10 clat (msec): min=193, max=2455, avg=891.57, stdev=313.08 lat (msec): min=193, max=2456, avg=891.95, stdev=313.07 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 485], 10.00th=[ 558], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 986], 80.00th=[ 1116], 90.00th=[ 1301], 95.00th=[ 1485], | 99.00th=[ 1921], 99.50th=[ 1989], 99.90th=[ 2366], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 247, per=25.15%, avg=71.94, stdev=34.97, samples=280 iops : min= 1, max= 61, avg=17.46, stdev= 8.77, samples=280 lat (msec) : 250=0.51%, 500=5.04%, 750=27.70%, 1000=38.12%, 2000=28.20% lat (msec) : >=2000=0.43% cpu : usr=0.07%, sys=0.38%, ctx=2372, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=287KiB/s (294kB/s), 71.7KiB/s-71.7KiB/s (73.4kB/s-73.4kB/s), io=40.0MiB (41.9MB), run=142811-142861msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145477: Fri Jun 12 11:26:39 2026 read: IOPS=12, BW=49.2KiB/s (50.4kB/s)(5048KiB/102572msec) slat (usec): min=3215, max=57487, avg=15492.05, stdev=8183.13 clat (msec): min=15, max=786, avg=405.58, stdev=107.47 lat (msec): min=33, max=804, avg=421.08, stdev=108.88 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 255], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 443], 80.00th=[ 481], 90.00th=[ 550], 95.00th=[ 609], | 99.00th=[ 693], 99.50th=[ 743], 99.90th=[ 776], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 7, max= 119, per=25.38%, avg=49.23, stdev=21.29, samples=201 iops : min= 1, max= 29, avg=11.80, stdev= 5.36, samples=201 write: IOPS=12, BW=50.6KiB/s (51.8kB/s)(5192KiB/102572msec); 0 zone resets slat (usec): min=40, max=20109, avg=441.20, stdev=1125.61 clat (msec): min=213, max=2019, avg=850.89, stdev=234.02 lat (msec): min=214, max=2020, avg=851.34, stdev=234.07 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 510], 10.00th=[ 567], 20.00th=[ 651], | 30.00th=[ 718], 40.00th=[ 776], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1020], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1552], 99.50th=[ 1854], 99.90th=[ 1921], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 6, max= 96, per=24.90%, avg=50.31, stdev=19.48, samples=202 iops : min= 1, max= 24, avg=12.07, stdev= 4.90, samples=202 lat (msec) : 20=0.04%, 50=0.12%, 100=0.16%, 250=2.15%, 500=40.94% lat (msec) : 750=23.55%, 1000=21.76%, 2000=11.25%, >=2000=0.04% cpu : usr=0.12%, sys=0.83%, ctx=4788, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145478: Fri Jun 12 11:26:39 2026 read: IOPS=12, BW=48.3KiB/s (49.5kB/s)(4976KiB/102978msec) slat (usec): min=56, max=82327, avg=15234.39, stdev=8361.22 clat (msec): min=33, max=802, avg=405.31, stdev=108.50 lat (msec): min=44, max=841, avg=420.55, stdev=110.25 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 243], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 401], 60.00th=[ 422], | 70.00th=[ 451], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 600], | 99.00th=[ 726], 99.50th=[ 760], 99.90th=[ 785], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 7, max= 112, per=24.53%, avg=47.59, stdev=23.17, samples=204 iops : min= 1, max= 28, avg=11.39, stdev= 5.84, samples=204 write: IOPS=12, BW=51.1KiB/s (52.3kB/s)(5264KiB/102978msec); 0 zone resets slat (usec): min=42, max=18607, avg=476.93, stdev=1253.75 clat (msec): min=226, max=2047, avg=853.25, stdev=248.60 lat (msec): min=226, max=2047, avg=853.73, stdev=248.68 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 485], 10.00th=[ 542], 20.00th=[ 642], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 911], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1485], 99.50th=[ 1737], 99.90th=[ 1972], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 103, per=25.00%, avg=50.50, stdev=20.07, samples=204 iops : min= 1, max= 25, avg=12.11, stdev= 5.06, samples=204 lat (msec) : 50=0.08%, 100=0.08%, 250=2.89%, 500=40.08%, 750=22.85% lat (msec) : 1000=20.98%, 2000=13.01%, >=2000=0.04% cpu : usr=0.11%, sys=0.82%, ctx=4892, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145479: Fri Jun 12 11:26:39 2026 read: IOPS=12, BW=48.4KiB/s (49.6kB/s)(4984KiB/102899msec) slat (usec): min=3179, max=56410, avg=15559.30, stdev=8246.76 clat (msec): min=20, max=809, avg=403.65, stdev=106.28 lat (msec): min=28, max=842, avg=419.23, stdev=107.63 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 251], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 422], | 70.00th=[ 451], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 592], | 99.00th=[ 718], 99.50th=[ 751], 99.90th=[ 785], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 7, max= 159, per=24.71%, avg=47.95, stdev=22.60, samples=204 iops : min= 1, max= 39, avg=11.47, stdev= 5.69, samples=204 write: IOPS=12, BW=51.1KiB/s (52.3kB/s)(5256KiB/102899msec); 0 zone resets slat (usec): min=40, max=27622, avg=494.61, stdev=1462.65 clat (msec): min=154, max=2090, avg=852.13, stdev=242.23 lat (msec): min=154, max=2090, avg=852.63, stdev=242.29 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 502], 10.00th=[ 550], 20.00th=[ 642], | 30.00th=[ 709], 40.00th=[ 776], 50.00th=[ 844], 60.00th=[ 911], | 70.00th=[ 969], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1536], 99.50th=[ 1754], 99.90th=[ 1989], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 111, per=24.92%, avg=50.33, stdev=21.10, samples=204 iops : min= 1, max= 27, avg=12.07, stdev= 5.32, samples=204 lat (msec) : 50=0.04%, 100=0.16%, 250=2.46%, 500=40.82%, 750=23.24% lat (msec) : 1000=20.70%, 2000=12.54%, >=2000=0.04% cpu : usr=0.10%, sys=0.86%, ctx=4765, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145480: Fri Jun 12 11:26:39 2026 read: IOPS=12, BW=49.3KiB/s (50.5kB/s)(5060KiB/102575msec) slat (usec): min=3441, max=80691, avg=15233.00, stdev=8313.89 clat (msec): min=23, max=768, avg=402.21, stdev=106.31 lat (msec): min=34, max=788, avg=417.45, stdev=107.81 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 245], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 451], 80.00th=[ 489], 90.00th=[ 542], 95.00th=[ 592], | 99.00th=[ 676], 99.50th=[ 735], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 7, max= 125, per=25.33%, avg=49.14, stdev=23.14, samples=202 iops : min= 1, max= 31, avg=11.77, stdev= 5.84, samples=202 write: IOPS=12, BW=50.5KiB/s (51.7kB/s)(5180KiB/102575msec); 0 zone resets slat (usec): min=41, max=13399, avg=396.59, stdev=891.59 clat (msec): min=312, max=2102, avg=855.54, stdev=242.48 lat (msec): min=312, max=2102, avg=855.94, stdev=242.47 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 510], 10.00th=[ 558], 20.00th=[ 651], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1028], 90.00th=[ 1133], 95.00th=[ 1234], | 99.00th=[ 1653], 99.50th=[ 1921], 99.90th=[ 2056], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 103, per=24.68%, avg=49.85, stdev=20.63, samples=203 iops : min= 1, max= 25, avg=11.95, stdev= 5.20, samples=203 lat (msec) : 50=0.16%, 100=0.08%, 250=2.54%, 500=40.47%, 750=23.48% lat (msec) : 1000=21.80%, 2000=11.33%, >=2000=0.16% cpu : usr=0.12%, sys=0.87%, ctx=4721, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=195KiB/s (200kB/s), 48.3KiB/s-49.3KiB/s (49.5kB/s-50.5kB/s), io=19.6MiB (20.5MB), run=102572-102978msec WRITE: bw=203KiB/s (208kB/s), 50.5KiB/s-51.1KiB/s (51.7kB/s-52.3kB/s), io=20.4MiB (21.4MB), run=102572-102978msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145499: Fri Jun 12 11:26:49 2026 read: IOPS=1, BW=72.1MiB/s (75.6MB/s)(40.0MiB/555msec) slat (nsec): min=15854k, max=15854k, avg=15854269.00, stdev= 0.00 clat (nsec): min=512934k, max=512934k, avg=512934136.00, stdev= 0.00 lat (nsec): min=528793k, max=528793k, avg=528792780.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 514], 10.00th=[ 514], 20.00th=[ 514], | 30.00th=[ 514], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] lat (msec) : 750=100.00% cpu : usr=0.18%, sys=2.17%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=72.1MiB/s (75.6MB/s), 72.1MiB/s-72.1MiB/s (75.6MB/s-75.6MB/s), io=40.0MiB (41.9MB), run=555-555msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout