== sanity test 398c: run fio to test AIO ================= 08:52:53 (1785761573) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.388347 s, 108 MB/s osc.lustre-OST0000-osc-ffff95d5508e0800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180490: Mon Aug 3 08:53:31 2026 write: IOPS=90, BW=362KiB/s (371kB/s)(10.0MiB/28273msec); 0 zone resets slat (usec): min=49, max=11112, avg=308.84, stdev=755.70 clat (msec): min=66, max=348, avg=175.69, stdev=40.51 lat (msec): min=69, max=348, avg=176.01, stdev=40.49 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 122], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 226], 95.00th=[ 243], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 351] bw ( KiB/s): min= 238, max= 512, per=24.92%, avg=360.79, stdev=70.91, samples=56 iops : min= 59, max= 128, avg=89.89, stdev=17.72, samples=56 lat (msec) : 100=0.82%, 250=95.59%, 500=3.59% cpu : usr=0.16%, sys=1.14%, ctx=2685, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180491: Mon Aug 3 08:53:31 2026 write: IOPS=90, BW=362KiB/s (371kB/s)(10.0MiB/28268msec); 0 zone resets slat (usec): min=51, max=12608, avg=274.96, stdev=707.47 clat (msec): min=31, max=312, avg=175.79, stdev=39.86 lat (msec): min=31, max=313, avg=176.07, stdev=39.88 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 116], 10.00th=[ 127], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 241], | 99.00th=[ 284], 99.50th=[ 305], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 251, max= 512, per=24.90%, avg=360.59, stdev=67.79, samples=56 iops : min= 62, max= 128, avg=89.84, stdev=16.97, samples=56 lat (msec) : 50=0.12%, 100=0.35%, 250=96.76%, 500=2.77% cpu : usr=0.16%, sys=1.09%, ctx=1830, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180492: Mon Aug 3 08:53:31 2026 write: IOPS=90, BW=363KiB/s (372kB/s)(10.0MiB/28198msec); 0 zone resets slat (usec): min=51, max=17946, avg=359.12, stdev=896.88 clat (msec): min=24, max=339, avg=175.27, stdev=41.71 lat (msec): min=25, max=341, avg=175.63, stdev=41.70 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 138], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 243], | 99.00th=[ 292], 99.50th=[ 317], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 342] bw ( KiB/s): min= 251, max= 512, per=24.95%, avg=361.35, stdev=74.95, samples=55 iops : min= 62, max= 128, avg=90.04, stdev=18.73, samples=55 lat (msec) : 50=0.43%, 100=0.51%, 250=95.55%, 500=3.52% cpu : usr=0.20%, sys=1.35%, ctx=2687, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180493: Mon Aug 3 08:53:31 2026 write: IOPS=90, BW=364KiB/s (372kB/s)(10.0MiB/28162msec); 0 zone resets slat (usec): min=52, max=13222, avg=301.00, stdev=716.13 clat (msec): min=34, max=340, avg=175.55, stdev=40.26 lat (msec): min=34, max=340, avg=175.86, stdev=40.26 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 117], 10.00th=[ 127], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 207], 90.00th=[ 228], 95.00th=[ 243], | 99.00th=[ 288], 99.50th=[ 305], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 342] bw ( KiB/s): min= 251, max= 512, per=25.12%, avg=363.78, stdev=71.37, samples=55 iops : min= 62, max= 128, avg=90.60, stdev=17.85, samples=55 lat (msec) : 50=0.16%, 100=1.41%, 250=95.12%, 500=3.32% cpu : usr=0.18%, sys=1.32%, ctx=2701, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1449KiB/s (1484kB/s), 362KiB/s-364KiB/s (371kB/s-372kB/s), io=40.0MiB (41.9MB), run=28162-28273msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180507: Mon Aug 3 08:54:14 2026 read: IOPS=36, BW=147KiB/s (151kB/s)(5048KiB/34292msec) slat (usec): min=3374, max=69419, avg=14676.43, stdev=8066.97 clat (msec): min=21, max=382, avg=201.28, stdev=48.39 lat (msec): min=32, max=401, avg=215.96, stdev=50.32 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 132], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 264], 95.00th=[ 284], | 99.00th=[ 342], 99.50th=[ 355], 99.90th=[ 368], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 69, max= 231, per=24.98%, avg=146.15, stdev=36.68, samples=68 iops : min= 17, max= 57, avg=36.01, stdev= 9.21, samples=68 write: IOPS=37, BW=151KiB/s (155kB/s)(5192KiB/34292msec); 0 zone resets slat (usec): min=52, max=28168, avg=315.26, stdev=1074.65 clat (msec): min=34, max=407, avg=208.69, stdev=48.39 lat (msec): min=34, max=407, avg=209.01, stdev=48.38 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 401], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 55, max= 239, per=24.76%, avg=150.78, stdev=40.09, samples=68 iops : min= 13, max= 59, avg=37.16, stdev=10.03, samples=68 lat (msec) : 50=0.08%, 100=0.16%, 250=83.20%, 500=16.56% cpu : usr=0.19%, sys=1.67%, ctx=4362, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180508: Mon Aug 3 08:54:14 2026 read: IOPS=36, BW=146KiB/s (149kB/s)(4976KiB/34103msec) slat (usec): min=50, max=66147, avg=14917.59, stdev=8632.15 clat (msec): min=24, max=407, avg=202.65, stdev=51.05 lat (msec): min=51, max=427, avg=217.57, stdev=53.28 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 132], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 266], 95.00th=[ 292], | 99.00th=[ 359], 99.50th=[ 384], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 71, max= 223, per=24.77%, avg=144.90, stdev=34.81, samples=67 iops : min= 17, max= 55, avg=35.69, stdev= 8.75, samples=67 write: IOPS=38, BW=154KiB/s (158kB/s)(5264KiB/34103msec); 0 zone resets slat (usec): min=50, max=10149, avg=281.71, stdev=687.21 clat (msec): min=55, max=425, avg=208.23, stdev=50.89 lat (msec): min=56, max=426, avg=208.51, stdev=50.87 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 140], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 372], 99.50th=[ 388], 99.90th=[ 422], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 71, max= 247, per=25.30%, avg=154.07, stdev=39.45, samples=67 iops : min= 17, max= 61, avg=37.99, stdev= 9.85, samples=67 lat (msec) : 50=0.08%, 100=0.27%, 250=82.34%, 500=17.30% cpu : usr=0.21%, sys=1.66%, ctx=4045, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180509: Mon Aug 3 08:54:14 2026 read: IOPS=36, BW=145KiB/s (149kB/s)(4984KiB/34293msec) slat (usec): min=3139, max=73450, avg=15201.17, stdev=8609.14 clat (msec): min=58, max=383, avg=204.68, stdev=51.04 lat (msec): min=70, max=408, avg=219.89, stdev=52.84 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 132], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 380], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 55, max= 231, per=24.74%, avg=144.74, stdev=35.53, samples=68 iops : min= 13, max= 57, avg=35.65, stdev= 8.93, samples=68 write: IOPS=38, BW=153KiB/s (157kB/s)(5256KiB/34293msec); 0 zone resets slat (usec): min=54, max=17377, avg=296.78, stdev=878.13 clat (msec): min=26, max=411, avg=206.47, stdev=50.87 lat (msec): min=26, max=411, avg=206.76, stdev=50.91 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 271], 95.00th=[ 300], | 99.00th=[ 359], 99.50th=[ 388], 99.90th=[ 409], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 63, max= 240, per=24.98%, avg=152.10, stdev=40.91, samples=68 iops : min= 15, max= 60, avg=37.50, stdev=10.25, samples=68 lat (msec) : 50=0.08%, 100=0.27%, 250=82.27%, 500=17.38% cpu : usr=0.18%, sys=1.66%, ctx=4482, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180510: Mon Aug 3 08:54:14 2026 read: IOPS=36, BW=148KiB/s (151kB/s)(5060KiB/34213msec) slat (usec): min=3128, max=55079, avg=14833.57, stdev=8077.80 clat (msec): min=7, max=392, avg=202.18, stdev=49.45 lat (msec): min=26, max=419, avg=217.02, stdev=51.44 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 134], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 266], 95.00th=[ 292], | 99.00th=[ 351], 99.50th=[ 372], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 87, max= 223, per=25.15%, avg=147.13, stdev=33.72, samples=67 iops : min= 21, max= 55, avg=36.24, stdev= 8.49, samples=67 write: IOPS=37, BW=151KiB/s (155kB/s)(5180KiB/34213msec); 0 zone resets slat (usec): min=55, max=7739, avg=279.03, stdev=626.32 clat (msec): min=96, max=388, avg=207.78, stdev=47.42 lat (msec): min=97, max=388, avg=208.06, stdev=47.44 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 384], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 71, max= 247, per=24.70%, avg=150.43, stdev=38.29, samples=67 iops : min= 17, max= 61, avg=37.07, stdev= 9.58, samples=67 lat (msec) : 10=0.04%, 100=0.12%, 250=83.01%, 500=16.84% cpu : usr=0.18%, sys=1.63%, ctx=4150, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=585KiB/s (599kB/s), 145KiB/s-148KiB/s (149kB/s-151kB/s), io=19.6MiB (20.5MB), run=34103-34293msec WRITE: bw=609KiB/s (624kB/s), 151KiB/s-154KiB/s (155kB/s-158kB/s), io=20.4MiB (21.4MB), run=34103-34293msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180522: Mon Aug 3 08:54:25 2026 read: IOPS=2, BW=83.2MiB/s (87.2MB/s)(40.0MiB/481msec) slat (nsec): min=31413k, max=31413k, avg=31413067.00, stdev= 0.00 clat (nsec): min=432415k, max=432415k, avg=432414737.00, stdev= 0.00 lat (nsec): min=463912k, max=463912k, avg=463912057.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.29%, ctx=9, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.2MiB/s (87.2MB/s), 83.2MiB/s-83.2MiB/s (87.2MB/s-87.2MB/s), io=40.0MiB (41.9MB), run=481-481msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout