== sanity test 398c: run fio to test AIO ================= 08:19:46 (1787141986) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.568029 s, 73.8 MB/s osc.lustre-OST0000-osc-ffff88fc100bc000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179592: Wed Aug 19 08:20:31 2026 write: IOPS=76, BW=304KiB/s (312kB/s)(10.0MiB/33654msec); 0 zone resets slat (usec): min=38, max=14602, avg=349.61, stdev=1062.16 clat (msec): min=66, max=403, avg=208.91, stdev=54.22 lat (msec): min=67, max=403, avg=209.27, stdev=54.26 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 132], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 234], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 372], 99.90th=[ 397], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 160, max= 510, per=24.79%, avg=301.74, stdev=67.31, samples=66 iops : min= 40, max= 127, avg=75.00, stdev=16.80, samples=66 lat (msec) : 100=0.59%, 250=75.66%, 500=23.75% cpu : usr=0.18%, sys=1.03%, ctx=2302, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179593: Wed Aug 19 08:20:31 2026 write: IOPS=76, BW=305KiB/s (312kB/s)(10.0MiB/33556msec); 0 zone resets slat (usec): min=36, max=14898, avg=297.13, stdev=891.46 clat (msec): min=28, max=397, avg=208.56, stdev=55.48 lat (msec): min=29, max=397, avg=208.86, stdev=55.49 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 128], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 234], 80.00th=[ 255], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 393], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 176, max= 510, per=24.89%, avg=302.95, stdev=69.63, samples=66 iops : min= 44, max= 127, avg=75.32, stdev=17.39, samples=66 lat (msec) : 50=0.04%, 100=1.41%, 250=76.09%, 500=22.46% cpu : usr=0.16%, sys=1.14%, ctx=2471, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179594: Wed Aug 19 08:20:31 2026 write: IOPS=76, BW=307KiB/s (314kB/s)(10.0MiB/33367msec); 0 zone resets slat (usec): min=40, max=15728, avg=329.89, stdev=1017.03 clat (msec): min=57, max=394, avg=208.00, stdev=56.98 lat (msec): min=57, max=394, avg=208.33, stdev=56.99 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 234], 80.00th=[ 255], 90.00th=[ 292], 95.00th=[ 309], | 99.00th=[ 347], 99.50th=[ 388], 99.90th=[ 393], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 160, max= 510, per=25.12%, avg=305.73, stdev=72.81, samples=66 iops : min= 40, max= 127, avg=76.00, stdev=18.13, samples=66 lat (msec) : 100=1.99%, 250=76.33%, 500=21.68% cpu : usr=0.14%, sys=0.98%, ctx=2119, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179595: Wed Aug 19 08:20:31 2026 write: IOPS=76, BW=305KiB/s (313kB/s)(10.0MiB/33532msec); 0 zone resets slat (usec): min=49, max=17425, avg=386.84, stdev=1235.08 clat (msec): min=36, max=392, avg=208.34, stdev=54.70 lat (msec): min=36, max=392, avg=208.74, stdev=54.70 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 253], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 338], 99.50th=[ 384], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 174, max= 510, per=24.97%, avg=303.85, stdev=69.70, samples=66 iops : min= 43, max= 127, avg=75.53, stdev=17.38, samples=66 lat (msec) : 50=0.12%, 100=1.29%, 250=76.76%, 500=21.84% cpu : usr=0.16%, sys=1.11%, ctx=2754, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1217KiB/s (1246kB/s), 304KiB/s-307KiB/s (312kB/s-314kB/s), io=40.0MiB (41.9MB), run=33367-33654msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179610: Wed Aug 19 08:21:12 2026 read: IOPS=39, BW=157KiB/s (160kB/s)(5048KiB/32253msec) slat (usec): min=89, max=56296, avg=14315.80, stdev=8159.89 clat (msec): min=16, max=515, avg=189.17, stdev=54.17 lat (msec): min=20, max=529, avg=203.49, stdev=56.66 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 108], 10.00th=[ 127], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 255], 95.00th=[ 279], | 99.00th=[ 342], 99.50th=[ 388], 99.90th=[ 510], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 47, max= 248, per=25.15%, avg=155.94, stdev=41.17, samples=63 iops : min= 11, max= 62, avg=38.59, stdev=10.36, samples=63 write: IOPS=40, BW=161KiB/s (165kB/s)(5192KiB/32253msec); 0 zone resets slat (usec): min=39, max=15872, avg=287.18, stdev=791.21 clat (msec): min=50, max=475, avg=196.55, stdev=54.23 lat (msec): min=51, max=475, avg=196.84, stdev=54.21 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 115], 10.00th=[ 131], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 275], | 99.00th=[ 409], 99.50th=[ 426], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 84, max= 344, per=24.91%, avg=160.90, stdev=44.80, samples=63 iops : min= 21, max= 86, avg=39.81, stdev=11.24, samples=63 lat (msec) : 20=0.08%, 50=0.12%, 100=2.85%, 250=84.77%, 500=12.11% lat (msec) : 750=0.08% cpu : usr=0.24%, sys=1.92%, ctx=4313, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179611: Wed Aug 19 08:21:12 2026 read: IOPS=38, BW=154KiB/s (158kB/s)(4976KiB/32302msec) slat (usec): min=61, max=64320, avg=14524.43, stdev=8367.77 clat (msec): min=56, max=498, avg=190.36, stdev=54.41 lat (msec): min=59, max=525, avg=204.89, stdev=57.07 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 109], 10.00th=[ 125], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 253], 95.00th=[ 279], | 99.00th=[ 351], 99.50th=[ 397], 99.90th=[ 489], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 79, max= 256, per=24.71%, avg=153.19, stdev=40.73, samples=63 iops : min= 19, max= 64, avg=37.87, stdev=10.22, samples=63 write: IOPS=40, BW=163KiB/s (167kB/s)(5264KiB/32302msec); 0 zone resets slat (usec): min=43, max=23804, avg=335.68, stdev=1109.51 clat (msec): min=72, max=472, avg=198.02, stdev=53.53 lat (msec): min=72, max=472, avg=198.36, stdev=53.51 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 117], 10.00th=[ 136], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 284], | 99.00th=[ 376], 99.50th=[ 422], 99.90th=[ 464], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 71, max= 296, per=25.27%, avg=163.22, stdev=44.59, samples=63 iops : min= 17, max= 74, avg=40.40, stdev=11.25, samples=63 lat (msec) : 100=3.01%, 250=84.84%, 500=12.15% cpu : usr=0.25%, sys=1.80%, ctx=4177, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179612: Wed Aug 19 08:21:12 2026 read: IOPS=38, BW=154KiB/s (158kB/s)(4984KiB/32323msec) slat (usec): min=88, max=86288, avg=14417.36, stdev=8550.82 clat (msec): min=59, max=468, avg=191.09, stdev=53.33 lat (msec): min=60, max=511, avg=205.51, stdev=56.30 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 112], 10.00th=[ 128], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 199], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 255], 95.00th=[ 279], | 99.00th=[ 342], 99.50th=[ 422], 99.90th=[ 451], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 63, max= 269, per=24.76%, avg=153.52, stdev=43.40, samples=64 iops : min= 15, max= 67, avg=37.95, stdev=10.93, samples=64 write: IOPS=40, BW=163KiB/s (167kB/s)(5256KiB/32323msec); 0 zone resets slat (usec): min=51, max=8099, avg=271.46, stdev=623.60 clat (msec): min=67, max=471, avg=195.41, stdev=52.40 lat (msec): min=67, max=471, avg=195.68, stdev=52.38 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 116], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 194], 60.00th=[ 207], | 70.00th=[ 218], 80.00th=[ 230], 90.00th=[ 255], 95.00th=[ 279], | 99.00th=[ 372], 99.50th=[ 405], 99.90th=[ 464], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 79, max= 320, per=24.94%, avg=161.09, stdev=42.75, samples=64 iops : min= 19, max= 80, avg=39.87, stdev=10.72, samples=64 lat (msec) : 100=2.85%, 250=85.43%, 500=11.72% cpu : usr=0.24%, sys=1.96%, ctx=4395, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179613: Wed Aug 19 08:21:12 2026 read: IOPS=39, BW=157KiB/s (160kB/s)(5060KiB/32326msec) slat (usec): min=51, max=53496, avg=14014.86, stdev=7771.61 clat (msec): min=56, max=444, avg=190.01, stdev=51.09 lat (msec): min=56, max=473, avg=204.03, stdev=53.43 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 110], 10.00th=[ 127], 20.00th=[ 148], | 30.00th=[ 165], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 253], 95.00th=[ 268], | 99.00th=[ 338], 99.50th=[ 363], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 71, max= 320, per=25.05%, avg=155.33, stdev=43.06, samples=64 iops : min= 17, max= 80, avg=38.39, stdev=10.87, samples=64 write: IOPS=40, BW=160KiB/s (164kB/s)(5180KiB/32326msec); 0 zone resets slat (usec): min=48, max=12448, avg=275.84, stdev=654.65 clat (msec): min=24, max=506, avg=197.23, stdev=52.59 lat (msec): min=24, max=506, avg=197.50, stdev=52.57 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 111], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 234], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 393], 99.50th=[ 430], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 87, max= 269, per=24.64%, avg=159.16, stdev=37.46, samples=64 iops : min= 21, max= 67, avg=39.36, stdev= 9.41, samples=64 lat (msec) : 50=0.04%, 100=2.77%, 250=86.41%, 500=10.70%, 750=0.08% cpu : usr=0.25%, sys=1.87%, ctx=4235, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=621KiB/s (636kB/s), 154KiB/s-157KiB/s (158kB/s-160kB/s), io=19.6MiB (20.5MB), run=32253-32326msec WRITE: bw=646KiB/s (662kB/s), 160KiB/s-163KiB/s (164kB/s-167kB/s), io=20.4MiB (21.4MB), run=32253-32326msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179623: Wed Aug 19 08:21:23 2026 read: IOPS=1, BW=67.6MiB/s (70.8MB/s)(40.0MiB/592msec) slat (nsec): min=44398k, max=44398k, avg=44398051.00, stdev= 0.00 clat (nsec): min=505780k, max=505780k, avg=505780227.00, stdev= 0.00 lat (nsec): min=550183k, max=550183k, avg=550183375.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.37%, ctx=8, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=67.6MiB/s (70.8MB/s), 67.6MiB/s-67.6MiB/s (70.8MB/s-70.8MB/s), io=40.0MiB (41.9MB), run=592-592msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout