== sanity test 398c: run fio to test AIO ================= 20:34:08 (1777422848) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.131551 s, 319 MB/s osc.lustre-OST0000-osc-ffff96f289566800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=157209: Tue Apr 28 20:34:39 2026 write: IOPS=93, BW=375KiB/s (384kB/s)(10.0MiB/27308msec); 0 zone resets slat (usec): min=36, max=1600, avg=216.59, stdev=109.81 clat (msec): min=48, max=283, avg=170.35, stdev=34.71 lat (msec): min=48, max=283, avg=170.57, stdev=34.72 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 115], 10.00th=[ 131], 20.00th=[ 146], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 188], 80.00th=[ 199], 90.00th=[ 213], 95.00th=[ 224], | 99.00th=[ 249], 99.50th=[ 266], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 272, max= 719, per=24.98%, avg=373.74, stdev=63.73, samples=54 iops : min= 68, max= 179, avg=93.41, stdev=15.85, samples=54 lat (msec) : 50=0.62%, 100=3.09%, 250=95.35%, 500=0.94% cpu : usr=0.18%, sys=1.71%, ctx=2586, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=157210: Tue Apr 28 20:34:39 2026 write: IOPS=93, BW=374KiB/s (383kB/s)(10.0MiB/27348msec); 0 zone resets slat (usec): min=54, max=1407, avg=230.33, stdev=106.50 clat (msec): min=18, max=278, avg=170.52, stdev=33.94 lat (msec): min=18, max=278, avg=170.75, stdev=33.94 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 116], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 180], | 70.00th=[ 188], 80.00th=[ 197], 90.00th=[ 213], 95.00th=[ 224], | 99.00th=[ 251], 99.50th=[ 255], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 279] bw ( KiB/s): min= 256, max= 674, per=24.93%, avg=372.91, stdev=57.93, samples=54 iops : min= 64, max= 168, avg=93.20, stdev=14.43, samples=54 lat (msec) : 20=0.04%, 50=0.16%, 100=3.05%, 250=95.78%, 500=0.98% cpu : usr=0.23%, sys=1.76%, ctx=2786, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=157211: Tue Apr 28 20:34:39 2026 write: IOPS=93, BW=374KiB/s (383kB/s)(10.0MiB/27348msec); 0 zone resets slat (usec): min=36, max=1032, avg=228.05, stdev=100.32 clat (msec): min=43, max=271, avg=170.50, stdev=33.79 lat (msec): min=43, max=271, avg=170.72, stdev=33.80 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 115], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 180], | 70.00th=[ 188], 80.00th=[ 197], 90.00th=[ 213], 95.00th=[ 224], | 99.00th=[ 255], 99.50th=[ 264], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 256, max= 629, per=24.93%, avg=372.96, stdev=54.68, samples=54 iops : min= 64, max= 157, avg=93.22, stdev=13.64, samples=54 lat (msec) : 50=0.08%, 100=3.01%, 250=95.74%, 500=1.17% cpu : usr=0.18%, sys=1.83%, ctx=2659, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=157212: Tue Apr 28 20:34:39 2026 write: IOPS=93, BW=374KiB/s (383kB/s)(10.0MiB/27366msec); 0 zone resets slat (usec): min=48, max=1047, avg=227.93, stdev=101.87 clat (msec): min=36, max=283, avg=170.61, stdev=33.52 lat (msec): min=37, max=283, avg=170.84, stdev=33.53 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 117], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 188], 80.00th=[ 197], 90.00th=[ 211], 95.00th=[ 224], | 99.00th=[ 253], 99.50th=[ 262], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 284] bw ( KiB/s): min= 256, max= 622, per=24.90%, avg=372.54, stdev=55.73, samples=54 iops : min= 64, max= 155, avg=93.11, stdev=13.88, samples=54 lat (msec) : 50=0.04%, 100=2.89%, 250=95.94%, 500=1.13% cpu : usr=0.17%, sys=1.80%, ctx=2755, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1497KiB/s (1533kB/s), 374KiB/s-375KiB/s (383kB/s-384kB/s), io=40.0MiB (41.9MB), run=27308-27366msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=157225: Tue Apr 28 20:35:02 2026 read: IOPS=63, BW=253KiB/s (260kB/s)(5048KiB/19918msec) slat (usec): min=1187, max=6856, avg=2632.02, stdev=814.31 clat (msec): min=3, max=145, avg=78.96, stdev=15.13 lat (msec): min=6, max=147, avg=81.59, stdev=15.16 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 96], 95.00th=[ 102], | 99.00th=[ 117], 99.50th=[ 124], 99.90th=[ 131], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 152, max= 400, per=25.19%, avg=253.64, stdev=56.03, samples=39 iops : min= 38, max= 100, avg=63.41, stdev=14.01, samples=39 write: IOPS=65, BW=261KiB/s (267kB/s)(5192KiB/19918msec); 0 zone resets slat (usec): min=57, max=763, avg=208.15, stdev=95.45 clat (msec): min=51, max=274, avg=165.51, stdev=38.02 lat (msec): min=51, max=274, avg=165.72, stdev=38.02 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 104], 10.00th=[ 114], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 215], 95.00th=[ 228], | 99.00th=[ 253], 99.50th=[ 259], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 200, max= 328, per=24.84%, avg=260.33, stdev=29.89, samples=39 iops : min= 50, max= 82, avg=65.08, stdev= 7.47, samples=39 lat (msec) : 4=0.12%, 10=0.16%, 20=0.12%, 50=0.51%, 100=47.54% lat (msec) : 250=50.90%, 500=0.66% cpu : usr=0.27%, sys=2.64%, ctx=4921, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157226: Tue Apr 28 20:35:02 2026 read: IOPS=62, BW=250KiB/s (256kB/s)(4976KiB/19918msec) slat (usec): min=1099, max=6708, avg=2620.93, stdev=804.09 clat (msec): min=3, max=143, avg=79.14, stdev=14.93 lat (msec): min=5, max=145, avg=81.76, stdev=14.99 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 59], 10.00th=[ 63], 20.00th=[ 67], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 99], 95.00th=[ 103], | 99.00th=[ 115], 99.50th=[ 121], 99.90th=[ 142], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 152, max= 400, per=24.72%, avg=248.92, stdev=59.10, samples=39 iops : min= 38, max= 100, avg=62.23, stdev=14.77, samples=39 write: IOPS=66, BW=264KiB/s (271kB/s)(5264KiB/19918msec); 0 zone resets slat (usec): min=40, max=741, avg=205.20, stdev=98.13 clat (msec): min=50, max=281, avg=164.16, stdev=37.93 lat (msec): min=50, max=281, avg=164.37, stdev=37.93 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 103], 10.00th=[ 115], 20.00th=[ 132], | 30.00th=[ 142], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 213], 95.00th=[ 226], | 99.00th=[ 253], 99.50th=[ 257], 99.90th=[ 271], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 184, max= 344, per=25.19%, avg=264.03, stdev=40.47, samples=39 iops : min= 46, max= 86, avg=66.00, stdev=10.12, samples=39 lat (msec) : 4=0.08%, 10=0.12%, 20=0.16%, 50=0.35%, 100=46.45% lat (msec) : 250=52.30%, 500=0.55% cpu : usr=0.21%, sys=2.67%, ctx=4889, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157227: Tue Apr 28 20:35:02 2026 read: IOPS=62, BW=251KiB/s (257kB/s)(4984KiB/19868msec) slat (usec): min=62, max=7160, avg=2649.40, stdev=830.03 clat (msec): min=4, max=158, avg=78.57, stdev=14.97 lat (msec): min=7, max=160, avg=81.22, stdev=15.09 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 67], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 97], 95.00th=[ 102], | 99.00th=[ 112], 99.50th=[ 115], 99.90th=[ 129], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 160, max= 360, per=25.04%, avg=252.13, stdev=49.72, samples=39 iops : min= 40, max= 90, avg=63.03, stdev=12.42, samples=39 write: IOPS=66, BW=265KiB/s (271kB/s)(5256KiB/19868msec); 0 zone resets slat (usec): min=55, max=855, avg=208.34, stdev=97.25 clat (msec): min=32, max=274, avg=164.51, stdev=39.15 lat (msec): min=33, max=275, avg=164.72, stdev=39.16 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 101], 10.00th=[ 114], 20.00th=[ 131], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 213], 95.00th=[ 228], | 99.00th=[ 247], 99.50th=[ 255], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 192, max= 338, per=25.10%, avg=263.03, stdev=36.31, samples=39 iops : min= 48, max= 84, avg=65.74, stdev= 9.05, samples=39 lat (msec) : 10=0.20%, 20=0.31%, 50=0.94%, 100=46.76%, 250=51.41% lat (msec) : 500=0.39% cpu : usr=0.28%, sys=2.64%, ctx=4833, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157228: Tue Apr 28 20:35:02 2026 read: IOPS=63, BW=254KiB/s (260kB/s)(5060KiB/19918msec) slat (usec): min=1165, max=6687, avg=2593.84, stdev=801.64 clat (msec): min=5, max=195, avg=79.28, stdev=16.11 lat (msec): min=7, max=198, avg=81.88, stdev=16.18 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 58], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 91], 90.00th=[ 97], 95.00th=[ 103], | 99.00th=[ 121], 99.50th=[ 130], 99.90th=[ 192], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 136, max= 352, per=25.43%, avg=256.05, stdev=51.41, samples=39 iops : min= 34, max= 88, avg=64.00, stdev=12.85, samples=39 write: IOPS=65, BW=260KiB/s (266kB/s)(5180KiB/19918msec); 0 zone resets slat (usec): min=58, max=807, avg=207.63, stdev=101.31 clat (msec): min=28, max=280, avg=165.42, stdev=37.66 lat (msec): min=29, max=280, avg=165.63, stdev=37.66 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 104], 10.00th=[ 114], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 213], 95.00th=[ 228], | 99.00th=[ 255], 99.50th=[ 268], 99.90th=[ 271], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 176, max= 328, per=24.75%, avg=259.33, stdev=31.80, samples=39 iops : min= 44, max= 82, avg=64.82, stdev= 7.94, samples=39 lat (msec) : 10=0.08%, 20=0.31%, 50=0.86%, 100=46.37%, 250=51.80% lat (msec) : 500=0.59% cpu : usr=0.24%, sys=2.64%, ctx=4920, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1008KiB/s (1032kB/s), 250KiB/s-254KiB/s (256kB/s-260kB/s), io=19.6MiB (20.5MB), run=19868-19918msec WRITE: bw=1049KiB/s (1074kB/s), 260KiB/s-265KiB/s (266kB/s-271kB/s), io=20.4MiB (21.4MB), run=19868-19918msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=157235: Tue Apr 28 20:35:05 2026 read: IOPS=7, BW=305MiB/s (320MB/s)(40.0MiB/131msec) slat (nsec): min=14037k, max=14037k, avg=14037102.00, stdev= 0.00 clat (nsec): min=111772k, max=111772k, avg=111772214.00, stdev= 0.00 lat (nsec): min=125840k, max=125840k, avg=125840323.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 112], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.46%, ctx=43, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=305MiB/s (320MB/s), 305MiB/s-305MiB/s (320MB/s-320MB/s), io=40.0MiB (41.9MB), run=131-131msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout