== sanity test 398c: run fio to test AIO ================= 11:20:16 (1775834416) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.162402 s, 258 MB/s osc.lustre-OST0000-osc-ffff91722c1a2800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=150393: Fri Apr 10 11:20:52 2026 write: IOPS=78, BW=313KiB/s (321kB/s)(10.0MiB/32667msec); 0 zone resets slat (usec): min=35, max=2647, avg=202.02, stdev=110.43 clat (msec): min=37, max=463, avg=203.85, stdev=45.33 lat (msec): min=37, max=463, avg=204.05, stdev=45.34 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 138], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 215], | 70.00th=[ 224], 80.00th=[ 234], 90.00th=[ 249], 95.00th=[ 266], | 99.00th=[ 347], 99.50th=[ 393], 99.90th=[ 430], 99.95th=[ 439], | 99.99th=[ 464] bw ( KiB/s): min= 176, max= 620, per=25.04%, avg=313.52, stdev=59.39, samples=64 iops : min= 44, max= 155, avg=78.30, stdev=14.88, samples=64 lat (msec) : 50=0.62%, 100=1.25%, 250=88.91%, 500=9.22% cpu : usr=0.16%, sys=1.25%, ctx=2821, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150394: Fri Apr 10 11:20:52 2026 write: IOPS=78, BW=313KiB/s (321kB/s)(10.0MiB/32712msec); 0 zone resets slat (usec): min=37, max=1531, avg=213.66, stdev=102.29 clat (msec): min=57, max=410, avg=204.03, stdev=43.23 lat (msec): min=57, max=410, avg=204.25, stdev=43.24 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 213], | 70.00th=[ 224], 80.00th=[ 234], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 342], 99.50th=[ 355], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 409] bw ( KiB/s): min= 176, max= 536, per=24.85%, avg=311.09, stdev=52.42, samples=65 iops : min= 44, max= 134, avg=77.69, stdev=13.16, samples=65 lat (msec) : 100=1.88%, 250=87.97%, 500=10.16% cpu : usr=0.15%, sys=1.31%, ctx=2900, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150395: Fri Apr 10 11:20:52 2026 write: IOPS=78, BW=314KiB/s (321kB/s)(10.0MiB/32655msec); 0 zone resets slat (usec): min=55, max=2340, avg=213.97, stdev=116.27 clat (msec): min=18, max=420, avg=203.69, stdev=44.22 lat (msec): min=18, max=420, avg=203.91, stdev=44.22 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 213], | 70.00th=[ 224], 80.00th=[ 234], 90.00th=[ 253], 95.00th=[ 271], | 99.00th=[ 338], 99.50th=[ 355], 99.90th=[ 405], 99.95th=[ 418], | 99.99th=[ 422] bw ( KiB/s): min= 192, max= 588, per=25.05%, avg=313.64, stdev=54.47, samples=64 iops : min= 48, max= 147, avg=78.33, stdev=13.66, samples=64 lat (msec) : 20=0.04%, 50=0.31%, 100=1.72%, 250=87.15%, 500=10.78% cpu : usr=0.19%, sys=1.30%, ctx=2905, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150396: Fri Apr 10 11:20:52 2026 write: IOPS=78, BW=313KiB/s (321kB/s)(10.0MiB/32713msec); 0 zone resets slat (usec): min=49, max=1749, avg=214.10, stdev=111.84 clat (msec): min=59, max=415, avg=204.03, stdev=43.55 lat (msec): min=59, max=415, avg=204.25, stdev=43.55 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 215], | 70.00th=[ 226], 80.00th=[ 234], 90.00th=[ 249], 95.00th=[ 268], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 405], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 168, max= 529, per=24.85%, avg=311.11, stdev=51.49, samples=65 iops : min= 42, max= 132, avg=77.69, stdev=12.90, samples=65 lat (msec) : 100=1.91%, 250=88.55%, 500=9.53% cpu : usr=0.17%, sys=1.29%, ctx=2998, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1252KiB/s (1282kB/s), 313KiB/s-314KiB/s (321kB/s-321kB/s), io=40.0MiB (41.9MB), run=32655-32713msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=150410: Fri Apr 10 11:21:20 2026 read: IOPS=51, BW=206KiB/s (211kB/s)(5048KiB/24504msec) slat (usec): min=1197, max=11495, avg=3126.87, stdev=1332.43 clat (msec): min=3, max=222, avg=97.00, stdev=25.87 lat (msec): min=5, max=226, avg=100.13, stdev=26.20 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 65], 10.00th=[ 72], 20.00th=[ 80], | 30.00th=[ 85], 40.00th=[ 89], 50.00th=[ 95], 60.00th=[ 100], | 70.00th=[ 105], 80.00th=[ 114], 90.00th=[ 125], 95.00th=[ 140], | 99.00th=[ 203], 99.50th=[ 213], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 95, max= 336, per=25.13%, avg=205.56, stdev=55.48, samples=48 iops : min= 23, max= 84, avg=51.27, stdev=14.00, samples=48 write: IOPS=52, BW=212KiB/s (217kB/s)(5192KiB/24504msec); 0 zone resets slat (usec): min=56, max=2386, avg=204.75, stdev=137.21 clat (msec): min=66, max=437, avg=203.82, stdev=57.88 lat (msec): min=66, max=438, avg=204.03, stdev=57.89 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 121], 10.00th=[ 136], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 247], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 368], 99.50th=[ 401], 99.90th=[ 430], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 120, max= 280, per=24.77%, avg=211.00, stdev=37.42, samples=48 iops : min= 30, max= 70, avg=52.65, stdev= 9.43, samples=48 lat (msec) : 4=0.08%, 10=0.04%, 20=0.27%, 50=0.23%, 100=29.73% lat (msec) : 250=60.39%, 500=9.26% cpu : usr=0.20%, sys=1.99%, ctx=4784, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150411: Fri Apr 10 11:21:20 2026 read: IOPS=50, BW=204KiB/s (208kB/s)(4976KiB/24448msec) slat (usec): min=56, max=11777, avg=3097.30, stdev=1293.58 clat (msec): min=3, max=237, avg=97.27, stdev=24.79 lat (msec): min=6, max=240, avg=100.37, stdev=25.19 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 67], 10.00th=[ 72], 20.00th=[ 79], | 30.00th=[ 85], 40.00th=[ 90], 50.00th=[ 95], 60.00th=[ 100], | 70.00th=[ 106], 80.00th=[ 113], 90.00th=[ 128], 95.00th=[ 140], | 99.00th=[ 171], 99.50th=[ 184], 99.90th=[ 234], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 104, max= 320, per=24.82%, avg=203.00, stdev=58.00, samples=48 iops : min= 26, max= 80, avg=50.62, stdev=14.54, samples=48 write: IOPS=53, BW=215KiB/s (220kB/s)(5264KiB/24448msec); 0 zone resets slat (usec): min=44, max=4251, avg=197.45, stdev=151.58 clat (msec): min=38, max=421, avg=201.92, stdev=57.17 lat (msec): min=38, max=421, avg=202.12, stdev=57.19 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 118], 10.00th=[ 134], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 397], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 119, max= 312, per=25.25%, avg=215.15, stdev=45.16, samples=48 iops : min= 29, max= 78, avg=53.67, stdev=11.39, samples=48 lat (msec) : 4=0.04%, 10=0.04%, 20=0.23%, 50=0.31%, 100=29.41% lat (msec) : 250=60.78%, 500=9.18% cpu : usr=0.23%, sys=1.92%, ctx=4610, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150412: Fri Apr 10 11:21:20 2026 read: IOPS=50, BW=203KiB/s (208kB/s)(4984KiB/24506msec) slat (usec): min=1240, max=11634, avg=3134.21, stdev=1338.85 clat (msec): min=3, max=214, avg=97.46, stdev=23.72 lat (msec): min=5, max=219, avg=100.60, stdev=24.03 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 66], 10.00th=[ 72], 20.00th=[ 80], | 30.00th=[ 85], 40.00th=[ 90], 50.00th=[ 96], 60.00th=[ 101], | 70.00th=[ 107], 80.00th=[ 116], 90.00th=[ 127], 95.00th=[ 138], | 99.00th=[ 163], 99.50th=[ 180], 99.90th=[ 209], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 112, max= 336, per=24.88%, avg=203.52, stdev=60.53, samples=48 iops : min= 28, max= 84, avg=50.77, stdev=15.21, samples=48 write: IOPS=53, BW=214KiB/s (220kB/s)(5256KiB/24506msec); 0 zone resets slat (usec): min=48, max=1052, avg=196.50, stdev=105.01 clat (msec): min=54, max=410, avg=202.31, stdev=59.26 lat (msec): min=54, max=410, avg=202.50, stdev=59.26 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 194], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 245], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 376], 99.50th=[ 401], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 127, max= 296, per=24.94%, avg=212.48, stdev=41.46, samples=48 iops : min= 31, max= 74, avg=53.00, stdev=10.44, samples=48 lat (msec) : 4=0.04%, 10=0.12%, 20=0.20%, 50=0.16%, 100=28.98% lat (msec) : 250=61.56%, 500=8.95% cpu : usr=0.22%, sys=1.95%, ctx=4740, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150413: Fri Apr 10 11:21:20 2026 read: IOPS=51, BW=207KiB/s (212kB/s)(5060KiB/24447msec) slat (usec): min=1223, max=11333, avg=3133.56, stdev=1354.98 clat (msec): min=7, max=215, avg=97.84, stdev=25.41 lat (msec): min=9, max=223, avg=100.98, stdev=25.77 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 64], 10.00th=[ 71], 20.00th=[ 79], | 30.00th=[ 85], 40.00th=[ 90], 50.00th=[ 95], 60.00th=[ 102], | 70.00th=[ 108], 80.00th=[ 117], 90.00th=[ 130], 95.00th=[ 140], | 99.00th=[ 174], 99.50th=[ 194], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 103, max= 320, per=25.48%, avg=208.40, stdev=52.64, samples=48 iops : min= 25, max= 80, avg=51.98, stdev=13.28, samples=48 write: IOPS=52, BW=212KiB/s (217kB/s)(5180KiB/24447msec); 0 zone resets slat (usec): min=55, max=2220, avg=197.60, stdev=135.81 clat (msec): min=48, max=409, avg=202.59, stdev=58.72 lat (msec): min=48, max=409, avg=202.79, stdev=58.72 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 118], 10.00th=[ 132], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 247], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 363], 99.50th=[ 401], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 128, max= 304, per=24.79%, avg=211.25, stdev=41.38, samples=48 iops : min= 32, max= 76, avg=52.71, stdev=10.40, samples=48 lat (msec) : 10=0.08%, 20=0.35%, 50=0.35%, 100=28.40%, 250=61.37% lat (msec) : 500=9.45% cpu : usr=0.20%, sys=2.01%, ctx=4760, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=819KiB/s (839kB/s), 203KiB/s-207KiB/s (208kB/s-212kB/s), io=19.6MiB (20.5MB), run=24447-24506msec WRITE: bw=853KiB/s (873kB/s), 212KiB/s-215KiB/s (217kB/s-220kB/s), io=20.4MiB (21.4MB), run=24447-24506msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=150420: Fri Apr 10 11:21:24 2026 read: IOPS=8, BW=351MiB/s (368MB/s)(40.0MiB/114msec) slat (nsec): min=9776.3k, max=9776.3k, avg=9776301.00, stdev= 0.00 clat (nsec): min=99073k, max=99073k, avg=99073181.00, stdev= 0.00 lat (nsec): min=108855k, max=108855k, avg=108854992.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[99091], 5.00th=[99091], 10.00th=[99091], 20.00th=[99091], | 30.00th=[99091], 40.00th=[99091], 50.00th=[99091], 60.00th=[99091], | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.96%, ctx=40, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=40.0MiB (41.9MB), run=114-114msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout