== sanity test 398c: run fio to test AIO ================= 09:46:40 (1770648400) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.154608 s, 271 MB/s osc.lustre-OST0000-osc-ffff902d30f0b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=162111: Mon Feb 9 09:47:12 2026 write: IOPS=90, BW=360KiB/s (369kB/s)(10.0MiB/28431msec); 0 zone resets slat (usec): min=24, max=653, avg=105.97, stdev=56.83 clat (msec): min=35, max=433, avg=177.48, stdev=36.69 lat (msec): min=35, max=433, avg=177.58, stdev=36.70 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 127], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 171], 50.00th=[ 176], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 284], 99.50th=[ 372], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 435] bw ( KiB/s): min= 232, max= 614, per=24.98%, avg=358.96, stdev=53.66, samples=56 iops : min= 58, max= 153, avg=89.73, stdev=13.37, samples=56 lat (msec) : 50=0.62%, 100=1.48%, 250=95.86%, 500=2.03% cpu : usr=0.11%, sys=0.82%, ctx=2828, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=162112: Mon Feb 9 09:47:12 2026 write: IOPS=89, BW=360KiB/s (368kB/s)(10.0MiB/28479msec); 0 zone resets slat (usec): min=24, max=747, avg=104.86, stdev=55.53 clat (msec): min=38, max=433, avg=177.67, stdev=36.84 lat (msec): min=39, max=433, avg=177.78, stdev=36.85 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 125], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 215], 95.00th=[ 228], | 99.00th=[ 300], 99.50th=[ 368], 99.90th=[ 418], 99.95th=[ 422], | 99.99th=[ 435] bw ( KiB/s): min= 232, max= 614, per=24.96%, avg=358.68, stdev=53.71, samples=56 iops : min= 58, max= 153, avg=89.66, stdev=13.39, samples=56 lat (msec) : 50=0.62%, 100=1.99%, 250=95.31%, 500=2.07% cpu : usr=0.09%, sys=0.82%, ctx=2909, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=162113: Mon Feb 9 09:47:12 2026 write: IOPS=89, BW=359KiB/s (368kB/s)(10.0MiB/28488msec); 0 zone resets slat (usec): min=24, max=430, avg=106.23, stdev=52.19 clat (msec): min=41, max=446, avg=177.70, stdev=34.97 lat (msec): min=42, max=446, avg=177.81, stdev=34.98 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 127], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 279], 99.50th=[ 376], 99.90th=[ 418], 99.95th=[ 422], | 99.99th=[ 447] bw ( KiB/s): min= 240, max= 614, per=24.89%, avg=357.68, stdev=54.68, samples=56 iops : min= 60, max= 153, avg=89.41, stdev=13.63, samples=56 lat (msec) : 50=0.04%, 100=1.91%, 250=96.37%, 500=1.68% cpu : usr=0.11%, sys=0.83%, ctx=2812, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=162114: Mon Feb 9 09:47:12 2026 write: IOPS=89, BW=359KiB/s (368kB/s)(10.0MiB/28488msec); 0 zone resets slat (usec): min=28, max=600, avg=106.10, stdev=55.89 clat (msec): min=20, max=408, avg=177.72, stdev=35.84 lat (msec): min=20, max=408, avg=177.83, stdev=35.85 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 182], | 70.00th=[ 188], 80.00th=[ 199], 90.00th=[ 218], 95.00th=[ 232], | 99.00th=[ 288], 99.50th=[ 376], 99.90th=[ 393], 99.95th=[ 405], | 99.99th=[ 409] bw ( KiB/s): min= 208, max= 627, per=24.91%, avg=357.91, stdev=57.33, samples=56 iops : min= 52, max= 156, avg=89.46, stdev=14.27, samples=56 lat (msec) : 50=0.23%, 100=1.99%, 250=95.90%, 500=1.88% cpu : usr=0.10%, sys=0.82%, ctx=2915, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1438KiB/s (1472kB/s), 359KiB/s-360KiB/s (368kB/s-369kB/s), io=40.0MiB (41.9MB), run=28431-28488msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=162127: Mon Feb 9 09:47:36 2026 read: IOPS=59, BW=237KiB/s (243kB/s)(5048KiB/21260msec) slat (usec): min=1084, max=75657, avg=3049.31, stdev=2316.15 clat (msec): min=5, max=256, avg=84.44, stdev=17.66 lat (msec): min=8, max=260, avg=87.49, stdev=17.64 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 88], | 70.00th=[ 91], 80.00th=[ 95], 90.00th=[ 102], 95.00th=[ 106], | 99.00th=[ 122], 99.50th=[ 184], 99.90th=[ 253], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 128, max= 320, per=25.23%, avg=237.64, stdev=44.38, samples=42 iops : min= 32, max= 80, avg=59.40, stdev=11.10, samples=42 write: IOPS=61, BW=244KiB/s (250kB/s)(5192KiB/21260msec); 0 zone resets slat (usec): min=27, max=551, avg=120.12, stdev=65.39 clat (msec): min=77, max=292, avg=175.81, stdev=38.33 lat (msec): min=77, max=293, avg=175.93, stdev=38.33 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 109], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 226], 95.00th=[ 234], | 99.00th=[ 257], 99.50th=[ 259], 99.90th=[ 271], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 176, max= 320, per=24.93%, avg=244.55, stdev=31.76, samples=42 iops : min= 44, max= 80, avg=61.12, stdev= 7.94, samples=42 lat (msec) : 10=0.12%, 20=0.16%, 50=0.12%, 100=44.73%, 250=53.83% lat (msec) : 500=1.05% cpu : usr=0.18%, sys=1.71%, ctx=4757, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=162128: Mon Feb 9 09:47:36 2026 read: IOPS=58, BW=234KiB/s (239kB/s)(4976KiB/21291msec) slat (usec): min=1081, max=8346, avg=2983.14, stdev=1046.15 clat (msec): min=3, max=136, avg=82.98, stdev=15.61 lat (msec): min=6, max=140, avg=85.96, stdev=15.72 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 88], | 70.00th=[ 91], 80.00th=[ 95], 90.00th=[ 101], 95.00th=[ 106], | 99.00th=[ 113], 99.50th=[ 117], 99.90th=[ 131], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 136, max= 352, per=24.68%, avg=232.45, stdev=53.23, samples=42 iops : min= 34, max= 88, avg=58.10, stdev=13.31, samples=42 write: IOPS=61, BW=247KiB/s (253kB/s)(5264KiB/21291msec); 0 zone resets slat (usec): min=33, max=485, avg=121.12, stdev=69.16 clat (msec): min=23, max=272, avg=176.62, stdev=38.63 lat (msec): min=23, max=272, avg=176.74, stdev=38.63 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 123], 20.00th=[ 140], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 228], 95.00th=[ 236], | 99.00th=[ 253], 99.50th=[ 262], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 184, max= 296, per=25.04%, avg=245.69, stdev=25.62, samples=42 iops : min= 46, max= 74, avg=61.40, stdev= 6.41, samples=42 lat (msec) : 4=0.08%, 10=0.16%, 20=0.39%, 50=0.35%, 100=43.95% lat (msec) : 250=54.41%, 500=0.66% cpu : usr=0.21%, sys=1.65%, ctx=4933, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=162129: Mon Feb 9 09:47:36 2026 read: IOPS=58, BW=235KiB/s (240kB/s)(4984KiB/21245msec) slat (usec): min=1134, max=10584, avg=3005.43, stdev=1060.77 clat (msec): min=3, max=240, avg=83.43, stdev=15.21 lat (msec): min=6, max=243, avg=86.44, stdev=15.36 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 84], 60.00th=[ 88], | 70.00th=[ 91], 80.00th=[ 94], 90.00th=[ 100], 95.00th=[ 103], | 99.00th=[ 114], 99.50th=[ 125], 99.90th=[ 215], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 136, max= 368, per=25.00%, avg=235.48, stdev=48.87, samples=42 iops : min= 34, max= 92, avg=58.86, stdev=12.22, samples=42 write: IOPS=61, BW=247KiB/s (253kB/s)(5256KiB/21245msec); 0 zone resets slat (usec): min=34, max=1975, avg=120.76, stdev=84.95 clat (msec): min=72, max=281, avg=175.68, stdev=38.74 lat (msec): min=73, max=281, avg=175.80, stdev=38.74 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 109], 10.00th=[ 122], 20.00th=[ 140], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 226], 95.00th=[ 236], | 99.00th=[ 257], 99.50th=[ 264], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 176, max= 368, per=25.13%, avg=246.57, stdev=35.56, samples=42 iops : min= 44, max= 92, avg=61.64, stdev= 8.89, samples=42 lat (msec) : 4=0.04%, 10=0.04%, 20=0.23%, 50=0.20%, 100=45.27% lat (msec) : 250=53.24%, 500=0.98% cpu : usr=0.16%, sys=1.73%, ctx=4750, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=162130: Mon Feb 9 09:47:36 2026 read: IOPS=59, BW=238KiB/s (244kB/s)(5060KiB/21221msec) slat (usec): min=123, max=90679, avg=2998.34, stdev=2702.14 clat (msec): min=6, max=221, avg=84.30, stdev=16.87 lat (msec): min=8, max=224, avg=87.30, stdev=16.86 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 85], 60.00th=[ 88], | 70.00th=[ 91], 80.00th=[ 95], 90.00th=[ 101], 95.00th=[ 105], | 99.00th=[ 133], 99.50th=[ 171], 99.90th=[ 218], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 120, max= 352, per=25.27%, avg=238.07, stdev=51.76, samples=42 iops : min= 30, max= 88, avg=59.50, stdev=12.93, samples=42 write: IOPS=61, BW=244KiB/s (250kB/s)(5180KiB/21221msec); 0 zone resets slat (usec): min=37, max=1437, avg=120.88, stdev=79.84 clat (msec): min=51, max=284, avg=176.48, stdev=40.23 lat (msec): min=51, max=284, avg=176.60, stdev=40.23 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 111], 10.00th=[ 120], 20.00th=[ 138], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 228], 95.00th=[ 241], | 99.00th=[ 255], 99.50th=[ 262], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 184, max= 352, per=24.81%, avg=243.43, stdev=34.63, samples=42 iops : min= 46, max= 88, avg=60.86, stdev= 8.66, samples=42 lat (msec) : 10=0.20%, 20=0.27%, 50=0.23%, 100=44.92%, 250=53.28% lat (msec) : 500=1.09% cpu : usr=0.16%, sys=1.71%, ctx=4871, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=943KiB/s (965kB/s), 234KiB/s-238KiB/s (239kB/s-244kB/s), io=19.6MiB (20.5MB), run=21221-21291msec WRITE: bw=981KiB/s (1005kB/s), 244KiB/s-247KiB/s (250kB/s-253kB/s), io=20.4MiB (21.4MB), run=21221-21291msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=162137: Mon Feb 9 09:47:40 2026 read: IOPS=6, BW=267MiB/s (280MB/s)(40.0MiB/150msec) slat (nsec): min=12785k, max=12785k, avg=12784807.00, stdev= 0.00 clat (nsec): min=128778k, max=128778k, avg=128778177.00, stdev= 0.00 lat (nsec): min=141565k, max=141565k, avg=141565112.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 129], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.05%, ctx=8, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=40.0MiB (41.9MB), run=150-150msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout