== sanity test 398c: run fio to test AIO ================= 15:56:23 (1778356583) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.248652 s, 169 MB/s osc.lustre-OST0000-osc-ffff9336f133c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=159844: Sat May 9 15:57:09 2026 write: IOPS=60, BW=241KiB/s (246kB/s)(10.0MiB/42550msec); 0 zone resets slat (usec): min=56, max=1755, avg=333.26, stdev=183.31 clat (msec): min=25, max=504, avg=265.45, stdev=76.49 lat (msec): min=26, max=504, avg=265.78, stdev=76.53 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 163], 10.00th=[ 178], 20.00th=[ 192], | 30.00th=[ 211], 40.00th=[ 230], 50.00th=[ 257], 60.00th=[ 284], | 70.00th=[ 309], 80.00th=[ 338], 90.00th=[ 376], 95.00th=[ 401], | 99.00th=[ 439], 99.50th=[ 456], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 506] bw ( KiB/s): min= 128, max= 414, per=25.01%, avg=240.64, stdev=66.27, samples=84 iops : min= 32, max= 103, avg=60.15, stdev=16.55, samples=84 lat (msec) : 50=0.04%, 100=0.16%, 250=47.89%, 500=51.88%, 750=0.04% cpu : usr=0.14%, sys=1.54%, ctx=2793, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159845: Sat May 9 15:57:09 2026 write: IOPS=60, BW=241KiB/s (246kB/s)(10.0MiB/42549msec); 0 zone resets slat (usec): min=58, max=1373, avg=342.90, stdev=187.11 clat (msec): min=119, max=464, avg=265.17, stdev=75.41 lat (msec): min=119, max=465, avg=265.51, stdev=75.44 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 165], 10.00th=[ 176], 20.00th=[ 192], | 30.00th=[ 211], 40.00th=[ 232], 50.00th=[ 253], 60.00th=[ 279], | 70.00th=[ 309], 80.00th=[ 338], 90.00th=[ 376], 95.00th=[ 397], | 99.00th=[ 439], 99.50th=[ 447], 99.90th=[ 460], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 136, max= 392, per=24.93%, avg=239.82, stdev=66.74, samples=84 iops : min= 34, max= 98, avg=59.95, stdev=16.68, samples=84 lat (msec) : 250=48.20%, 500=51.80% cpu : usr=0.11%, sys=1.59%, ctx=2827, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159846: Sat May 9 15:57:09 2026 write: IOPS=60, BW=241KiB/s (246kB/s)(10.0MiB/42549msec); 0 zone resets slat (usec): min=59, max=1477, avg=350.70, stdev=178.50 clat (msec): min=77, max=462, avg=265.17, stdev=76.03 lat (msec): min=78, max=463, avg=265.52, stdev=76.06 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 163], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 209], 40.00th=[ 230], 50.00th=[ 253], 60.00th=[ 279], | 70.00th=[ 309], 80.00th=[ 342], 90.00th=[ 376], 95.00th=[ 401], | 99.00th=[ 443], 99.50th=[ 451], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 464] bw ( KiB/s): min= 144, max= 392, per=24.98%, avg=240.30, stdev=63.92, samples=84 iops : min= 36, max= 98, avg=60.07, stdev=15.98, samples=84 lat (msec) : 100=0.08%, 250=48.67%, 500=51.25% cpu : usr=0.11%, sys=1.67%, ctx=2860, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159847: Sat May 9 15:57:09 2026 write: IOPS=60, BW=241KiB/s (246kB/s)(10.0MiB/42550msec); 0 zone resets slat (usec): min=47, max=2005, avg=356.74, stdev=191.49 clat (msec): min=26, max=557, avg=265.23, stdev=77.47 lat (msec): min=26, max=557, avg=265.58, stdev=77.50 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 163], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 228], 50.00th=[ 253], 60.00th=[ 279], | 70.00th=[ 309], 80.00th=[ 338], 90.00th=[ 376], 95.00th=[ 401], | 99.00th=[ 443], 99.50th=[ 451], 99.90th=[ 542], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 136, max= 400, per=25.03%, avg=240.79, stdev=64.54, samples=84 iops : min= 34, max= 100, avg=60.19, stdev=16.12, samples=84 lat (msec) : 50=0.04%, 100=0.23%, 250=48.28%, 500=51.29%, 750=0.16% cpu : usr=0.15%, sys=1.64%, ctx=2891, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=963KiB/s (986kB/s), 241KiB/s-241KiB/s (246kB/s-246kB/s), io=40.0MiB (41.9MB), run=42549-42550msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=159862: Sat May 9 15:57:36 2026 read: IOPS=54, BW=219KiB/s (225kB/s)(5048KiB/23021msec) slat (usec): min=52, max=8083, avg=2992.11, stdev=1087.75 clat (msec): min=4, max=282, avg=90.71, stdev=20.48 lat (msec): min=7, max=285, avg=93.71, stdev=20.53 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 65], 10.00th=[ 71], 20.00th=[ 79], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 92], 60.00th=[ 95], | 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 110], 95.00th=[ 116], | 99.00th=[ 128], 99.50th=[ 132], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 144, max= 312, per=25.48%, avg=220.62, stdev=37.54, samples=45 iops : min= 36, max= 78, avg=55.16, stdev= 9.39, samples=45 write: IOPS=56, BW=226KiB/s (231kB/s)(5192KiB/23021msec); 0 zone resets slat (usec): min=47, max=1056, avg=217.50, stdev=123.77 clat (msec): min=52, max=412, avg=192.09, stdev=46.07 lat (msec): min=52, max=412, avg=192.31, stdev=46.06 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 123], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 165], 40.00th=[ 178], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 234], 90.00th=[ 251], 95.00th=[ 262], | 99.00th=[ 292], 99.50th=[ 330], 99.90th=[ 405], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 168, max= 304, per=25.10%, avg=226.40, stdev=29.84, samples=45 iops : min= 42, max= 76, avg=56.60, stdev= 7.46, samples=45 lat (msec) : 10=0.08%, 20=0.39%, 50=0.43%, 100=35.39%, 250=58.52% lat (msec) : 500=5.20% cpu : usr=0.15%, sys=2.14%, ctx=5034, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159863: Sat May 9 15:57:36 2026 read: IOPS=53, BW=215KiB/s (220kB/s)(4976KiB/23148msec) slat (usec): min=1070, max=6880, avg=2974.99, stdev=1061.89 clat (msec): min=3, max=127, avg=90.36, stdev=16.27 lat (msec): min=7, max=132, avg=93.33, stdev=16.29 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 67], 10.00th=[ 72], 20.00th=[ 79], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 91], 60.00th=[ 95], | 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 110], 95.00th=[ 115], | 99.00th=[ 123], 99.50th=[ 126], 99.90th=[ 127], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 104, max= 336, per=24.79%, avg=214.64, stdev=52.14, samples=45 iops : min= 26, max= 84, avg=53.64, stdev=13.05, samples=45 write: IOPS=56, BW=227KiB/s (233kB/s)(5264KiB/23148msec); 0 zone resets slat (usec): min=55, max=1082, avg=209.07, stdev=118.10 clat (msec): min=55, max=317, avg=192.17, stdev=44.01 lat (msec): min=56, max=317, avg=192.38, stdev=44.00 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 120], 10.00th=[ 132], 20.00th=[ 153], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 253], 95.00th=[ 264], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 160, max= 288, per=25.07%, avg=226.16, stdev=33.93, samples=45 iops : min= 40, max= 72, avg=56.53, stdev= 8.49, samples=45 lat (msec) : 4=0.08%, 10=0.12%, 20=0.20%, 50=0.12%, 100=34.22% lat (msec) : 250=59.77%, 500=5.51% cpu : usr=0.15%, sys=2.11%, ctx=4917, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159864: Sat May 9 15:57:36 2026 read: IOPS=53, BW=215KiB/s (220kB/s)(4984KiB/23148msec) slat (usec): min=1076, max=9197, avg=3020.21, stdev=1083.66 clat (msec): min=3, max=137, avg=90.58, stdev=15.59 lat (msec): min=6, max=140, avg=93.60, stdev=15.66 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 67], 10.00th=[ 72], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 88], 50.00th=[ 91], 60.00th=[ 95], | 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 111], 95.00th=[ 115], | 99.00th=[ 127], 99.50th=[ 128], 99.90th=[ 136], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 152, max= 336, per=24.90%, avg=215.64, stdev=45.06, samples=45 iops : min= 38, max= 84, avg=53.91, stdev=11.27, samples=45 write: IOPS=56, BW=227KiB/s (233kB/s)(5256KiB/23148msec); 0 zone resets slat (usec): min=59, max=895, avg=209.98, stdev=118.68 clat (msec): min=35, max=332, avg=192.16, stdev=45.65 lat (msec): min=35, max=332, avg=192.37, stdev=45.64 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 118], 10.00th=[ 133], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 234], 90.00th=[ 251], 95.00th=[ 264], | 99.00th=[ 292], 99.50th=[ 296], 99.90th=[ 309], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 160, max= 296, per=24.85%, avg=224.13, stdev=30.67, samples=45 iops : min= 40, max= 74, avg=56.02, stdev= 7.68, samples=45 lat (msec) : 4=0.08%, 20=0.04%, 50=0.39%, 100=36.09%, 250=57.93% lat (msec) : 500=5.47% cpu : usr=0.17%, sys=2.10%, ctx=4903, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159865: Sat May 9 15:57:36 2026 read: IOPS=54, BW=220KiB/s (225kB/s)(5060KiB/23030msec) slat (usec): min=1193, max=8830, avg=2993.09, stdev=1056.67 clat (msec): min=3, max=164, avg=90.67, stdev=15.84 lat (msec): min=5, max=167, avg=93.66, stdev=15.92 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 68], 10.00th=[ 73], 20.00th=[ 79], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 92], 60.00th=[ 95], | 70.00th=[ 100], 80.00th=[ 103], 90.00th=[ 109], 95.00th=[ 115], | 99.00th=[ 126], 99.50th=[ 127], 99.90th=[ 163], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 136, max= 336, per=25.50%, avg=220.87, stdev=43.73, samples=45 iops : min= 34, max= 84, avg=55.20, stdev=10.95, samples=45 write: IOPS=56, BW=225KiB/s (230kB/s)(5180KiB/23030msec); 0 zone resets slat (usec): min=60, max=2948, avg=216.12, stdev=147.48 clat (msec): min=88, max=328, avg=191.81, stdev=43.25 lat (msec): min=88, max=328, avg=192.03, stdev=43.25 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 251], 95.00th=[ 264], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 309], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 152, max= 304, per=24.87%, avg=224.31, stdev=30.88, samples=45 iops : min= 38, max= 76, avg=56.07, stdev= 7.74, samples=45 lat (msec) : 4=0.12%, 10=0.08%, 20=0.04%, 50=0.20%, 100=35.62% lat (msec) : 250=58.87%, 500=5.08% cpu : usr=0.16%, sys=2.14%, ctx=5049, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=867KiB/s (888kB/s), 215KiB/s-220KiB/s (220kB/s-225kB/s), io=19.6MiB (20.5MB), run=23021-23148msec WRITE: bw=903KiB/s (924kB/s), 225KiB/s-227KiB/s (230kB/s-233kB/s), io=20.4MiB (21.4MB), run=23021-23148msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=159872: Sat May 9 15:57:39 2026 read: IOPS=5, BW=233MiB/s (244MB/s)(40.0MiB/172msec) slat (nsec): min=9630.5k, max=9630.5k, avg=9630511.00, stdev= 0.00 clat (nsec): min=153972k, max=153972k, avg=153972080.00, stdev= 0.00 lat (nsec): min=163605k, max=163605k, avg=163604809.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.26%, ctx=35, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=233MiB/s (244MB/s), 233MiB/s-233MiB/s (244MB/s-244MB/s), io=40.0MiB (41.9MB), run=172-172msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout