== sanity test 398c: run fio to test AIO ================= 03:15:27 (1772180127) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.528592 s, 79.3 MB/s osc.lustre-OST0000-osc-ffff923ac8424000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182800: Fri Feb 27 03:16:12 2026 write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31080msec); 0 zone resets slat (usec): min=35, max=15530, avg=230.95, stdev=730.21 clat (msec): min=11, max=379, avg=193.71, stdev=58.16 lat (msec): min=11, max=379, avg=193.94, stdev=58.14 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 112], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 176], 50.00th=[ 190], 60.00th=[ 209], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 279], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 376], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 125, max= 512, per=25.13%, avg=328.93, stdev=93.21, samples=61 iops : min= 31, max= 128, avg=81.84, stdev=23.30, samples=61 lat (msec) : 20=0.04%, 100=1.56%, 250=81.17%, 500=17.23% cpu : usr=0.15%, sys=1.00%, ctx=1959, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182801: Fri Feb 27 03:16:12 2026 write: IOPS=81, BW=327KiB/s (335kB/s)(10.0MiB/31278msec); 0 zone resets slat (usec): min=37, max=12935, avg=238.28, stdev=681.03 clat (msec): min=12, max=396, avg=193.84, stdev=58.31 lat (msec): min=14, max=397, avg=194.08, stdev=58.28 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 114], 10.00th=[ 125], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 171], 50.00th=[ 194], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 351], 99.50th=[ 359], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 397] bw ( KiB/s): min= 151, max= 536, per=24.88%, avg=325.74, stdev=95.43, samples=61 iops : min= 37, max= 134, avg=81.05, stdev=23.88, samples=61 lat (msec) : 20=0.16%, 50=0.12%, 100=1.37%, 250=80.39%, 500=17.97% cpu : usr=0.08%, sys=0.98%, ctx=2520, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182802: Fri Feb 27 03:16:12 2026 write: IOPS=81, BW=327KiB/s (335kB/s)(10.0MiB/31288msec); 0 zone resets slat (usec): min=36, max=16257, avg=238.61, stdev=779.69 clat (msec): min=36, max=403, avg=194.04, stdev=58.58 lat (msec): min=36, max=403, avg=194.28, stdev=58.57 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 116], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 169], 50.00th=[ 192], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 359], 99.50th=[ 380], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 125, max= 520, per=24.74%, avg=323.84, stdev=95.55, samples=61 iops : min= 31, max= 130, avg=80.57, stdev=23.92, samples=61 lat (msec) : 50=0.12%, 100=1.72%, 250=81.25%, 500=16.91% cpu : usr=0.14%, sys=0.95%, ctx=2093, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182803: Fri Feb 27 03:16:12 2026 write: IOPS=81, BW=327KiB/s (335kB/s)(10.0MiB/31285msec); 0 zone resets slat (usec): min=36, max=18062, avg=252.97, stdev=783.57 clat (msec): min=75, max=402, avg=194.00, stdev=58.04 lat (msec): min=76, max=402, avg=194.25, stdev=58.00 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 114], 10.00th=[ 128], 20.00th=[ 138], | 30.00th=[ 153], 40.00th=[ 169], 50.00th=[ 192], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 359], 99.50th=[ 363], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 405] bw ( KiB/s): min= 151, max= 536, per=24.84%, avg=325.15, stdev=92.44, samples=61 iops : min= 37, max= 134, avg=80.90, stdev=23.09, samples=61 lat (msec) : 100=1.56%, 250=82.19%, 500=16.25% cpu : usr=0.12%, sys=1.05%, ctx=2626, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1309KiB/s (1341kB/s), 327KiB/s-329KiB/s (335kB/s-337kB/s), io=40.0MiB (41.9MB), run=31080-31288msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182818: Fri Feb 27 03:16:43 2026 read: IOPS=52, BW=212KiB/s (217kB/s)(5048KiB/23820msec) slat (usec): min=63, max=51334, avg=10609.10, stdev=6069.92 clat (msec): min=10, max=260, avg=140.69, stdev=37.04 lat (msec): min=26, max=267, avg=151.30, stdev=38.54 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 89], 10.00th=[ 95], 20.00th=[ 106], | 30.00th=[ 117], 40.00th=[ 127], 50.00th=[ 138], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 176], 90.00th=[ 190], 95.00th=[ 209], | 99.00th=[ 228], 99.50th=[ 236], 99.90th=[ 251], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 127, max= 359, per=25.24%, avg=211.53, stdev=52.83, samples=47 iops : min= 31, max= 89, avg=52.51, stdev=13.25, samples=47 write: IOPS=54, BW=218KiB/s (223kB/s)(5192KiB/23820msec); 0 zone resets slat (usec): min=53, max=14570, avg=256.56, stdev=649.10 clat (msec): min=67, max=265, avg=145.85, stdev=36.73 lat (msec): min=67, max=265, avg=146.11, stdev=36.71 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 92], 10.00th=[ 100], 20.00th=[ 112], | 30.00th=[ 124], 40.00th=[ 133], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 167], 80.00th=[ 178], 90.00th=[ 197], 95.00th=[ 211], | 99.00th=[ 232], 99.50th=[ 247], 99.90th=[ 249], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 128, max= 327, per=24.98%, avg=217.85, stdev=47.36, samples=47 iops : min= 32, max= 81, avg=54.09, stdev=11.78, samples=47 lat (msec) : 20=0.04%, 100=12.50%, 250=87.34%, 500=0.12% cpu : usr=0.23%, sys=2.09%, ctx=4579, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182819: Fri Feb 27 03:16:43 2026 read: IOPS=51, BW=208KiB/s (213kB/s)(4976KiB/23942msec) slat (usec): min=2425, max=43973, avg=10697.03, stdev=5848.39 clat (msec): min=52, max=289, avg=142.20, stdev=37.35 lat (msec): min=56, max=311, avg=152.90, stdev=38.79 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 88], 10.00th=[ 96], 20.00th=[ 108], | 30.00th=[ 118], 40.00th=[ 129], 50.00th=[ 140], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 174], 90.00th=[ 192], 95.00th=[ 207], | 99.00th=[ 234], 99.50th=[ 245], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 106, max= 336, per=24.62%, avg=206.28, stdev=46.84, samples=47 iops : min= 26, max= 84, avg=51.19, stdev=11.80, samples=47 write: IOPS=54, BW=220KiB/s (225kB/s)(5264KiB/23942msec); 0 zone resets slat (usec): min=36, max=5623, avg=239.73, stdev=464.63 clat (msec): min=36, max=260, avg=144.38, stdev=36.91 lat (msec): min=37, max=260, avg=144.62, stdev=36.92 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 91], 10.00th=[ 97], 20.00th=[ 111], | 30.00th=[ 123], 40.00th=[ 131], 50.00th=[ 140], 60.00th=[ 153], | 70.00th=[ 167], 80.00th=[ 176], 90.00th=[ 197], 95.00th=[ 211], | 99.00th=[ 234], 99.50th=[ 243], 99.90th=[ 253], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 110, max= 391, per=25.19%, avg=219.68, stdev=58.66, samples=47 iops : min= 27, max= 97, avg=54.53, stdev=14.61, samples=47 lat (msec) : 50=0.08%, 100=12.38%, 250=87.23%, 500=0.31% cpu : usr=0.21%, sys=2.16%, ctx=4603, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182820: Fri Feb 27 03:16:43 2026 read: IOPS=52, BW=208KiB/s (213kB/s)(4984KiB/23943msec) slat (usec): min=1783, max=41654, avg=10692.12, stdev=5959.34 clat (msec): min=51, max=249, avg=142.14, stdev=38.55 lat (msec): min=59, max=261, avg=152.84, stdev=40.08 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 86], 10.00th=[ 93], 20.00th=[ 107], | 30.00th=[ 118], 40.00th=[ 129], 50.00th=[ 140], 60.00th=[ 150], | 70.00th=[ 163], 80.00th=[ 178], 90.00th=[ 197], 95.00th=[ 211], | 99.00th=[ 234], 99.50th=[ 241], 99.90th=[ 243], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 135, max= 331, per=24.77%, avg=207.55, stdev=47.48, samples=47 iops : min= 33, max= 82, avg=51.51, stdev=11.88, samples=47 write: IOPS=54, BW=220KiB/s (225kB/s)(5256KiB/23943msec); 0 zone resets slat (usec): min=39, max=6228, avg=238.51, stdev=512.76 clat (msec): min=60, max=274, avg=144.39, stdev=38.12 lat (msec): min=60, max=274, avg=144.63, stdev=38.11 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 90], 10.00th=[ 99], 20.00th=[ 109], | 30.00th=[ 120], 40.00th=[ 130], 50.00th=[ 142], 60.00th=[ 155], | 70.00th=[ 167], 80.00th=[ 180], 90.00th=[ 197], 95.00th=[ 211], | 99.00th=[ 241], 99.50th=[ 249], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 104, max= 359, per=25.03%, avg=218.26, stdev=60.84, samples=47 iops : min= 26, max= 89, avg=54.17, stdev=15.18, samples=47 lat (msec) : 100=13.32%, 250=86.52%, 500=0.16% cpu : usr=0.21%, sys=2.29%, ctx=4331, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182821: Fri Feb 27 03:16:43 2026 read: IOPS=52, BW=211KiB/s (216kB/s)(5060KiB/23943msec) slat (usec): min=357, max=48108, avg=10542.71, stdev=5829.09 clat (msec): min=61, max=269, avg=140.99, stdev=38.31 lat (msec): min=66, max=273, avg=151.55, stdev=39.65 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 108], | 30.00th=[ 118], 40.00th=[ 127], 50.00th=[ 136], 60.00th=[ 146], | 70.00th=[ 159], 80.00th=[ 174], 90.00th=[ 194], 95.00th=[ 215], | 99.00th=[ 249], 99.50th=[ 253], 99.90th=[ 266], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 112, max= 359, per=25.13%, avg=210.55, stdev=48.87, samples=47 iops : min= 28, max= 89, avg=52.26, stdev=12.19, samples=47 write: IOPS=54, BW=216KiB/s (222kB/s)(5180KiB/23943msec); 0 zone resets slat (usec): min=41, max=6386, avg=245.62, stdev=485.78 clat (msec): min=64, max=287, avg=145.75, stdev=38.15 lat (msec): min=64, max=287, avg=146.00, stdev=38.13 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 89], 10.00th=[ 99], 20.00th=[ 111], | 30.00th=[ 123], 40.00th=[ 132], 50.00th=[ 142], 60.00th=[ 153], | 70.00th=[ 167], 80.00th=[ 180], 90.00th=[ 197], 95.00th=[ 215], | 99.00th=[ 251], 99.50th=[ 259], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 120, max= 311, per=24.65%, avg=214.94, stdev=51.04, samples=47 iops : min= 30, max= 77, avg=53.36, stdev=12.78, samples=47 lat (msec) : 100=12.11%, 250=86.95%, 500=0.94% cpu : usr=0.25%, sys=2.21%, ctx=4456, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=838KiB/s (858kB/s), 208KiB/s-212KiB/s (213kB/s-217kB/s), io=19.6MiB (20.5MB), run=23820-23943msec WRITE: bw=873KiB/s (894kB/s), 216KiB/s-220KiB/s (222kB/s-225kB/s), io=20.4MiB (21.4MB), run=23820-23943msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182829: Fri Feb 27 03:16:50 2026 read: IOPS=2, BW=109MiB/s (115MB/s)(40.0MiB/366msec) slat (nsec): min=35185k, max=35185k, avg=35185157.00, stdev= 0.00 clat (nsec): min=313423k, max=313423k, avg=313423213.00, stdev= 0.00 lat (nsec): min=348612k, max=348612k, avg=348612367.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.74%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s), io=40.0MiB (41.9MB), run=366-366msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout