== sanity test 398c: run fio to test AIO ================= 18:08:37 (1776204517) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.494543 s, 84.8 MB/s osc.lustre-OST0000-osc-ffff9558211cb800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148500: Tue Apr 14 18:09:12 2026 write: IOPS=92, BW=370KiB/s (378kB/s)(10.0MiB/27707msec); 0 zone resets slat (usec): min=20, max=17757, avg=244.04, stdev=850.09 clat (msec): min=12, max=378, avg=172.62, stdev=46.86 lat (msec): min=13, max=378, avg=172.88, stdev=46.83 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 101], 10.00th=[ 118], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 253], | 99.00th=[ 279], 99.50th=[ 296], 99.90th=[ 342], 99.95th=[ 368], | 99.99th=[ 380] bw ( KiB/s): min= 199, max= 731, per=24.91%, avg=367.60, stdev=91.51, samples=55 iops : min= 49, max= 182, avg=91.49, stdev=22.87, samples=55 lat (msec) : 20=0.04%, 50=0.51%, 100=4.45%, 250=89.02%, 500=5.98% cpu : usr=0.22%, sys=1.07%, ctx=1276, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148501: Tue Apr 14 18:09:12 2026 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27732msec); 0 zone resets slat (usec): min=25, max=19181, avg=247.02, stdev=1005.42 clat (msec): min=17, max=335, avg=172.77, stdev=45.66 lat (msec): min=19, max=338, avg=173.01, stdev=45.68 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 102], 10.00th=[ 120], 20.00th=[ 136], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 247], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 248, max= 732, per=24.89%, avg=367.33, stdev=85.16, samples=55 iops : min= 62, max= 183, avg=91.42, stdev=21.31, samples=55 lat (msec) : 20=0.04%, 50=0.82%, 100=3.87%, 250=90.70%, 500=4.57% cpu : usr=0.22%, sys=1.09%, ctx=1358, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148502: Tue Apr 14 18:09:12 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27656msec); 0 zone resets slat (usec): min=21, max=18487, avg=264.29, stdev=991.66 clat (msec): min=22, max=379, avg=172.43, stdev=47.35 lat (msec): min=22, max=380, avg=172.69, stdev=47.35 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 105], 10.00th=[ 117], 20.00th=[ 133], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 234], 95.00th=[ 249], | 99.00th=[ 300], 99.50th=[ 321], 99.90th=[ 368], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 224, max= 760, per=25.10%, avg=370.52, stdev=85.99, samples=54 iops : min= 56, max= 190, avg=92.24, stdev=21.52, samples=54 lat (msec) : 50=0.74%, 100=3.75%, 250=90.86%, 500=4.65% cpu : usr=0.17%, sys=0.99%, ctx=1069, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148503: Tue Apr 14 18:09:12 2026 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27715msec); 0 zone resets slat (usec): min=21, max=12787, avg=236.05, stdev=778.26 clat (msec): min=11, max=318, avg=172.57, stdev=46.06 lat (msec): min=13, max=318, avg=172.81, stdev=46.04 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 104], 10.00th=[ 118], 20.00th=[ 134], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 284], 99.50th=[ 296], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 321] bw ( KiB/s): min= 192, max= 617, per=24.91%, avg=367.73, stdev=80.16, samples=55 iops : min= 48, max= 154, avg=91.53, stdev=20.02, samples=55 lat (msec) : 20=0.12%, 50=0.35%, 100=3.55%, 250=90.59%, 500=5.39% cpu : usr=0.18%, sys=1.05%, ctx=1465, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1477KiB/s (1512kB/s), 369KiB/s-370KiB/s (378kB/s-379kB/s), io=40.0MiB (41.9MB), run=27656-27732msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148517: Tue Apr 14 18:09:45 2026 read: IOPS=50, BW=202KiB/s (207kB/s)(5048KiB/24939msec) slat (usec): min=2331, max=46665, avg=10588.55, stdev=6301.63 clat (msec): min=9, max=309, avg=146.82, stdev=34.80 lat (msec): min=31, max=314, avg=157.42, stdev=35.73 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 99], 10.00th=[ 107], 20.00th=[ 117], | 30.00th=[ 126], 40.00th=[ 134], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 194], 95.00th=[ 207], | 99.00th=[ 241], 99.50th=[ 251], 99.90th=[ 275], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 135, max= 319, per=25.04%, avg=201.33, stdev=37.59, samples=49 iops : min= 33, max= 79, avg=49.86, stdev= 9.34, samples=49 write: IOPS=52, BW=208KiB/s (213kB/s)(5192KiB/24939msec); 0 zone resets slat (usec): min=26, max=11761, avg=228.02, stdev=718.87 clat (msec): min=51, max=297, avg=153.02, stdev=32.95 lat (msec): min=51, max=297, avg=153.25, stdev=32.96 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 108], 10.00th=[ 113], 20.00th=[ 125], | 30.00th=[ 133], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 197], 95.00th=[ 211], | 99.00th=[ 243], 99.50th=[ 251], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 135, max= 287, per=24.80%, avg=207.57, stdev=38.03, samples=49 iops : min= 33, max= 71, avg=51.43, stdev= 9.52, samples=49 lat (msec) : 10=0.04%, 50=0.08%, 100=4.10%, 250=95.20%, 500=0.59% cpu : usr=0.37%, sys=2.33%, ctx=2743, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148518: Tue Apr 14 18:09:45 2026 read: IOPS=50, BW=201KiB/s (205kB/s)(4976KiB/24803msec) slat (usec): min=38, max=57492, avg=10567.13, stdev=6041.01 clat (msec): min=5, max=297, avg=146.21, stdev=36.35 lat (msec): min=5, max=304, avg=156.78, stdev=37.55 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 96], 10.00th=[ 106], 20.00th=[ 116], | 30.00th=[ 125], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 163], 80.00th=[ 176], 90.00th=[ 192], 95.00th=[ 213], | 99.00th=[ 262], 99.50th=[ 271], 99.90th=[ 288], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 127, max= 311, per=24.86%, avg=199.84, stdev=43.70, samples=49 iops : min= 31, max= 77, avg=49.49, stdev=10.87, samples=49 write: IOPS=53, BW=212KiB/s (217kB/s)(5264KiB/24803msec); 0 zone resets slat (usec): min=27, max=14311, avg=269.33, stdev=923.25 clat (msec): min=31, max=297, avg=152.69, stdev=34.67 lat (msec): min=32, max=297, avg=152.97, stdev=34.72 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 107], 10.00th=[ 114], 20.00th=[ 124], | 30.00th=[ 132], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 199], 95.00th=[ 215], | 99.00th=[ 249], 99.50th=[ 257], 99.90th=[ 284], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 144, max= 280, per=25.36%, avg=212.24, stdev=33.92, samples=49 iops : min= 36, max= 70, avg=52.59, stdev= 8.51, samples=49 lat (msec) : 10=0.04%, 50=0.20%, 100=4.73%, 250=94.02%, 500=1.02% cpu : usr=0.28%, sys=2.40%, ctx=3027, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148519: Tue Apr 14 18:09:45 2026 read: IOPS=50, BW=200KiB/s (205kB/s)(4984KiB/24900msec) slat (usec): min=2258, max=58353, avg=10603.59, stdev=6059.84 clat (msec): min=16, max=286, avg=147.67, stdev=34.77 lat (msec): min=33, max=289, avg=158.28, stdev=35.71 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 99], 10.00th=[ 106], 20.00th=[ 118], | 30.00th=[ 128], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 176], 90.00th=[ 194], 95.00th=[ 211], | 99.00th=[ 243], 99.50th=[ 257], 99.90th=[ 279], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 142, max= 279, per=24.92%, avg=200.39, stdev=35.30, samples=49 iops : min= 35, max= 69, avg=49.63, stdev= 8.78, samples=49 write: IOPS=52, BW=211KiB/s (216kB/s)(5256KiB/24900msec); 0 zone resets slat (usec): min=27, max=22254, avg=296.91, stdev=1043.79 clat (msec): min=43, max=294, avg=151.85, stdev=33.00 lat (msec): min=44, max=298, avg=152.15, stdev=33.01 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 107], 10.00th=[ 114], 20.00th=[ 125], | 30.00th=[ 133], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 194], 95.00th=[ 215], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 279], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 140, max= 310, per=25.15%, avg=210.47, stdev=43.08, samples=49 iops : min= 35, max= 77, avg=52.16, stdev=10.80, samples=49 lat (msec) : 20=0.04%, 50=0.12%, 100=4.26%, 250=94.65%, 500=0.94% cpu : usr=0.34%, sys=2.22%, ctx=2858, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148520: Tue Apr 14 18:09:45 2026 read: IOPS=50, BW=203KiB/s (208kB/s)(5060KiB/24913msec) slat (usec): min=1550, max=62313, avg=10668.78, stdev=6273.91 clat (msec): min=46, max=288, avg=147.07, stdev=34.31 lat (msec): min=54, max=302, avg=157.74, stdev=35.29 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 99], 10.00th=[ 107], 20.00th=[ 118], | 30.00th=[ 127], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 194], 95.00th=[ 211], | 99.00th=[ 241], 99.50th=[ 249], 99.90th=[ 271], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 128, max= 286, per=25.26%, avg=203.08, stdev=38.39, samples=49 iops : min= 32, max= 71, avg=50.31, stdev= 9.55, samples=49 write: IOPS=51, BW=208KiB/s (213kB/s)(5180KiB/24913msec); 0 zone resets slat (usec): min=26, max=10333, avg=237.76, stdev=667.20 clat (msec): min=57, max=273, avg=152.49, stdev=32.68 lat (msec): min=57, max=273, avg=152.73, stdev=32.66 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 108], 10.00th=[ 114], 20.00th=[ 124], | 30.00th=[ 133], 40.00th=[ 140], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 167], 80.00th=[ 178], 90.00th=[ 197], 95.00th=[ 215], | 99.00th=[ 239], 99.50th=[ 245], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 117, max= 319, per=24.75%, avg=207.12, stdev=39.21, samples=49 iops : min= 29, max= 79, avg=51.33, stdev= 9.79, samples=49 lat (msec) : 50=0.04%, 100=3.87%, 250=95.74%, 500=0.35% cpu : usr=0.30%, sys=2.13%, ctx=2873, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=805KiB/s (824kB/s), 200KiB/s-203KiB/s (205kB/s-208kB/s), io=19.6MiB (20.5MB), run=24803-24939msec WRITE: bw=838KiB/s (858kB/s), 208KiB/s-212KiB/s (213kB/s-217kB/s), io=20.4MiB (21.4MB), run=24803-24939msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148527: Tue Apr 14 18:09:54 2026 read: IOPS=2, BW=111MiB/s (117MB/s)(40.0MiB/359msec) slat (nsec): min=34653k, max=34653k, avg=34653300.00, stdev= 0.00 clat (nsec): min=310793k, max=310793k, avg=310793449.00, stdev= 0.00 lat (nsec): min=345451k, max=345451k, avg=345451071.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.63%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=111MiB/s (117MB/s), 111MiB/s-111MiB/s (117MB/s-117MB/s), io=40.0MiB (41.9MB), run=359-359msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck