== sanity test 398c: run fio to test AIO ================= 03:21:34 (1759821694) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.136655 s, 307 MB/s osc.lustre-OST0000-osc-ffff9d4b1043a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154449: Tue Oct 7 03:22:08 2025 write: IOPS=83, BW=336KiB/s (344kB/s)(10.0MiB/30508msec); 0 zone resets slat (usec): min=21, max=1368, avg=124.59, stdev=61.55 clat (msec): min=67, max=311, avg=190.35, stdev=37.13 lat (msec): min=67, max=312, avg=190.48, stdev=37.14 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 129], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 224], 90.00th=[ 234], 95.00th=[ 247], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 313] bw ( KiB/s): min= 256, max= 617, per=24.89%, avg=334.00, stdev=54.62, samples=60 iops : min= 64, max= 154, avg=83.48, stdev=13.64, samples=60 lat (msec) : 100=2.50%, 250=93.40%, 500=4.10% cpu : usr=0.14%, sys=0.93%, ctx=2751, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154450: Tue Oct 7 03:22:08 2025 write: IOPS=83, BW=336KiB/s (344kB/s)(10.0MiB/30498msec); 0 zone resets slat (usec): min=28, max=461, avg=119.75, stdev=54.99 clat (msec): min=57, max=337, avg=190.30, stdev=37.32 lat (msec): min=57, max=337, avg=190.42, stdev=37.33 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 129], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 222], 90.00th=[ 234], 95.00th=[ 247], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 256, max= 617, per=24.97%, avg=335.07, stdev=55.94, samples=60 iops : min= 64, max= 154, avg=83.75, stdev=13.97, samples=60 lat (msec) : 100=2.54%, 250=93.20%, 500=4.26% cpu : usr=0.18%, sys=0.87%, ctx=2754, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154451: Tue Oct 7 03:22:08 2025 write: IOPS=84, BW=336KiB/s (344kB/s)(10.0MiB/30453msec); 0 zone resets slat (usec): min=20, max=491, avg=119.62, stdev=56.19 clat (msec): min=37, max=389, avg=190.08, stdev=39.01 lat (msec): min=38, max=389, avg=190.21, stdev=39.02 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 128], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 222], 90.00th=[ 236], 95.00th=[ 249], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 351], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 256, max= 616, per=25.03%, avg=335.85, stdev=58.72, samples=60 iops : min= 64, max= 154, avg=83.95, stdev=14.67, samples=60 lat (msec) : 50=0.62%, 100=1.99%, 250=93.28%, 500=4.10% cpu : usr=0.17%, sys=0.88%, ctx=2798, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154452: Tue Oct 7 03:22:08 2025 write: IOPS=83, BW=336KiB/s (344kB/s)(10.0MiB/30487msec); 0 zone resets slat (usec): min=26, max=571, avg=121.58, stdev=55.41 clat (msec): min=28, max=382, avg=190.25, stdev=37.96 lat (msec): min=28, max=382, avg=190.37, stdev=37.97 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 130], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 222], 90.00th=[ 234], 95.00th=[ 247], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 342], 99.95th=[ 355], | 99.99th=[ 384] bw ( KiB/s): min= 256, max= 643, per=24.97%, avg=335.10, stdev=57.68, samples=60 iops : min= 64, max= 160, avg=83.75, stdev=14.35, samples=60 lat (msec) : 50=0.23%, 100=2.54%, 250=93.20%, 500=4.02% cpu : usr=0.17%, sys=0.89%, ctx=2768, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1343KiB/s (1375kB/s), 336KiB/s-336KiB/s (344kB/s-344kB/s), io=40.0MiB (41.9MB), run=30453-30508msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154466: Tue Oct 7 03:22:34 2025 read: IOPS=54, BW=219KiB/s (224kB/s)(5048KiB/23094msec) slat (usec): min=22, max=8856, avg=2871.95, stdev=1102.39 clat (msec): min=4, max=168, avg=95.84, stdev=19.09 lat (msec): min=6, max=171, avg=98.72, stdev=19.29 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 68], 10.00th=[ 75], 20.00th=[ 82], | 30.00th=[ 88], 40.00th=[ 92], 50.00th=[ 96], 60.00th=[ 101], | 70.00th=[ 106], 80.00th=[ 112], 90.00th=[ 120], 95.00th=[ 125], | 99.00th=[ 138], 99.50th=[ 146], 99.90th=[ 163], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 112, max= 320, per=25.18%, avg=218.53, stdev=54.05, samples=45 iops : min= 28, max= 80, avg=54.62, stdev=13.50, samples=45 write: IOPS=56, BW=225KiB/s (230kB/s)(5192KiB/23094msec); 0 zone resets slat (usec): min=24, max=3465, avg=113.06, stdev=116.97 clat (msec): min=26, max=324, avg=188.38, stdev=45.55 lat (msec): min=26, max=324, avg=188.50, stdev=45.56 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 122], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 249], 95.00th=[ 268], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 160, max= 288, per=24.88%, avg=224.96, stdev=30.02, samples=45 iops : min= 40, max= 72, avg=56.22, stdev= 7.49, samples=45 lat (msec) : 10=0.12%, 20=0.16%, 50=0.86%, 100=28.32%, 250=65.74% lat (msec) : 500=4.80% cpu : usr=0.17%, sys=1.45%, ctx=4840, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154467: Tue Oct 7 03:22:34 2025 read: IOPS=53, BW=216KiB/s (221kB/s)(4976KiB/23053msec) slat (usec): min=1100, max=9094, avg=2905.25, stdev=1052.68 clat (msec): min=3, max=259, avg=95.38, stdev=20.64 lat (msec): min=6, max=261, avg=98.28, stdev=20.76 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 81], | 30.00th=[ 87], 40.00th=[ 91], 50.00th=[ 95], 60.00th=[ 101], | 70.00th=[ 106], 80.00th=[ 110], 90.00th=[ 118], 95.00th=[ 124], | 99.00th=[ 140], 99.50th=[ 182], 99.90th=[ 243], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 112, max= 312, per=24.91%, avg=216.18, stdev=48.56, samples=45 iops : min= 28, max= 78, avg=54.04, stdev=12.14, samples=45 write: IOPS=57, BW=228KiB/s (234kB/s)(5264KiB/23053msec); 0 zone resets slat (usec): min=26, max=2712, avg=112.16, stdev=91.42 clat (msec): min=36, max=385, avg=186.85, stdev=46.31 lat (msec): min=36, max=385, avg=186.96, stdev=46.32 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 118], 10.00th=[ 133], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 197], | 70.00th=[ 209], 80.00th=[ 228], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 300], 99.50th=[ 317], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 128, max= 328, per=25.37%, avg=229.38, stdev=44.35, samples=45 iops : min= 32, max= 82, avg=57.33, stdev=11.08, samples=45 lat (msec) : 4=0.04%, 10=0.16%, 20=0.23%, 50=0.27%, 100=29.18% lat (msec) : 250=65.35%, 500=4.77% cpu : usr=0.20%, sys=1.45%, ctx=4847, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154468: Tue Oct 7 03:22:34 2025 read: IOPS=53, BW=216KiB/s (221kB/s)(4984KiB/23107msec) slat (usec): min=1074, max=9096, avg=2987.84, stdev=1051.91 clat (msec): min=3, max=161, avg=94.66, stdev=18.19 lat (msec): min=5, max=165, avg=97.65, stdev=18.34 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 81], | 30.00th=[ 86], 40.00th=[ 90], 50.00th=[ 95], 60.00th=[ 101], | 70.00th=[ 105], 80.00th=[ 110], 90.00th=[ 117], 95.00th=[ 124], | 99.00th=[ 132], 99.50th=[ 142], 99.90th=[ 148], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 96, max= 384, per=24.93%, avg=216.38, stdev=58.92, samples=45 iops : min= 24, max= 96, avg=54.09, stdev=14.73, samples=45 write: IOPS=56, BW=227KiB/s (233kB/s)(5256KiB/23107msec); 0 zone resets slat (usec): min=27, max=2433, avg=118.19, stdev=87.39 clat (msec): min=44, max=317, avg=188.19, stdev=45.26 lat (msec): min=44, max=317, avg=188.31, stdev=45.26 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 121], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 197], | 70.00th=[ 211], 80.00th=[ 230], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 160, max= 328, per=24.96%, avg=225.60, stdev=32.57, samples=45 iops : min= 40, max= 82, avg=56.40, stdev= 8.14, samples=45 lat (msec) : 4=0.12%, 10=0.04%, 20=0.16%, 50=0.16%, 100=29.84% lat (msec) : 250=64.80%, 500=4.88% cpu : usr=0.19%, sys=1.49%, ctx=4843, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154469: Tue Oct 7 03:22:34 2025 read: IOPS=54, BW=219KiB/s (225kB/s)(5060KiB/23054msec) slat (usec): min=1004, max=13789, avg=2943.88, stdev=1122.16 clat (msec): min=3, max=215, avg=95.07, stdev=19.11 lat (msec): min=6, max=221, avg=98.02, stdev=19.29 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 68], 10.00th=[ 73], 20.00th=[ 82], | 30.00th=[ 86], 40.00th=[ 91], 50.00th=[ 94], 60.00th=[ 100], | 70.00th=[ 105], 80.00th=[ 110], 90.00th=[ 118], 95.00th=[ 125], | 99.00th=[ 138], 99.50th=[ 146], 99.90th=[ 186], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 112, max= 344, per=25.49%, avg=221.22, stdev=46.44, samples=45 iops : min= 28, max= 86, avg=55.29, stdev=11.61, samples=45 write: IOPS=56, BW=225KiB/s (230kB/s)(5180KiB/23054msec); 0 zone resets slat (usec): min=38, max=437, avg=108.14, stdev=52.97 clat (msec): min=65, max=388, avg=188.39, stdev=46.03 lat (msec): min=65, max=388, avg=188.50, stdev=46.03 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 124], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 186], 60.00th=[ 197], | 70.00th=[ 209], 80.00th=[ 226], 90.00th=[ 251], 95.00th=[ 275], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 144, max= 312, per=24.90%, avg=225.07, stdev=35.60, samples=45 iops : min= 36, max= 78, avg=56.27, stdev= 8.90, samples=45 lat (msec) : 4=0.08%, 10=0.04%, 20=0.23%, 50=0.20%, 100=30.31% lat (msec) : 250=63.95%, 500=5.20% cpu : usr=0.16%, sys=1.48%, ctx=4805, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=868KiB/s (889kB/s), 216KiB/s-219KiB/s (221kB/s-225kB/s), io=19.6MiB (20.5MB), run=23053-23107msec WRITE: bw=904KiB/s (926kB/s), 225KiB/s-228KiB/s (230kB/s-234kB/s), io=20.4MiB (21.4MB), run=23053-23107msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154476: Tue Oct 7 03:22:37 2025 read: IOPS=9, BW=360MiB/s (378MB/s)(40.0MiB/111msec) slat (nsec): min=9318.6k, max=9318.6k, avg=9318595.00, stdev= 0.00 clat (nsec): min=96433k, max=96433k, avg=96432753.00, stdev= 0.00 lat (nsec): min=105755k, max=105755k, avg=105754705.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[95945], 5.00th=[95945], 10.00th=[95945], 20.00th=[95945], | 30.00th=[95945], 40.00th=[95945], 50.00th=[95945], 60.00th=[95945], | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.27%, ctx=41, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=360MiB/s (378MB/s), 360MiB/s-360MiB/s (378MB/s-378MB/s), io=40.0MiB (41.9MB), run=111-111msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout