== sanity test 398c: run fio to test AIO ================= 00:17:06 (1782533826) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.163471 s, 257 MB/s osc.lustre-OST0000-osc-ffff8c73d17b3000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146346: Sat Jun 27 00:18:19 2026 write: IOPS=37, BW=149KiB/s (153kB/s)(10.0MiB/68703msec); 0 zone resets slat (usec): min=35, max=4306, avg=229.17, stdev=296.61 clat (msec): min=116, max=742, avg=428.96, stdev=104.38 lat (msec): min=116, max=742, avg=429.19, stdev=104.41 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 268], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 481], 80.00th=[ 514], 90.00th=[ 567], 95.00th=[ 600], | 99.00th=[ 667], 99.50th=[ 693], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 88, max= 414, per=24.94%, avg=148.37, stdev=39.72, samples=136 iops : min= 22, max= 103, avg=36.81, stdev= 9.96, samples=136 lat (msec) : 250=4.53%, 500=71.80%, 750=23.67% cpu : usr=0.08%, sys=0.57%, ctx=2976, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146347: Sat Jun 27 00:18:19 2026 write: IOPS=37, BW=149KiB/s (152kB/s)(10.0MiB/68788msec); 0 zone resets slat (usec): min=36, max=9636, avg=237.23, stdev=331.82 clat (msec): min=56, max=748, avg=429.35, stdev=102.16 lat (msec): min=56, max=748, avg=429.59, stdev=102.18 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 268], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 456], | 70.00th=[ 481], 80.00th=[ 510], 90.00th=[ 558], 95.00th=[ 600], | 99.00th=[ 667], 99.50th=[ 684], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 751] bw ( KiB/s): min= 79, max= 310, per=24.82%, avg=147.70, stdev=37.55, samples=137 iops : min= 19, max= 77, avg=36.64, stdev= 9.47, samples=137 lat (msec) : 100=0.04%, 250=3.67%, 500=73.24%, 750=23.05% cpu : usr=0.06%, sys=0.59%, ctx=3054, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146348: Sat Jun 27 00:18:19 2026 write: IOPS=37, BW=149KiB/s (153kB/s)(10.0MiB/68745msec); 0 zone resets slat (usec): min=37, max=3602, avg=236.21, stdev=274.64 clat (msec): min=125, max=749, avg=429.03, stdev=102.98 lat (msec): min=125, max=749, avg=429.26, stdev=102.99 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 271], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 451], | 70.00th=[ 481], 80.00th=[ 510], 90.00th=[ 567], 95.00th=[ 600], | 99.00th=[ 676], 99.50th=[ 701], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 751] bw ( KiB/s): min= 64, max= 311, per=24.94%, avg=148.38, stdev=36.67, samples=136 iops : min= 16, max= 77, avg=36.81, stdev= 9.23, samples=136 lat (msec) : 250=3.71%, 500=72.42%, 750=23.87% cpu : usr=0.07%, sys=0.58%, ctx=3060, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146349: Sat Jun 27 00:18:19 2026 write: IOPS=37, BW=149KiB/s (153kB/s)(10.0MiB/68702msec); 0 zone resets slat (usec): min=42, max=5797, avg=244.77, stdev=306.24 clat (msec): min=53, max=742, avg=428.80, stdev=105.60 lat (msec): min=53, max=742, avg=429.05, stdev=105.60 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 264], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 477], 80.00th=[ 514], 90.00th=[ 567], 95.00th=[ 600], | 99.00th=[ 676], 99.50th=[ 693], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 87, max= 382, per=24.94%, avg=148.38, stdev=38.91, samples=136 iops : min= 21, max= 95, avg=36.81, stdev= 9.79, samples=136 lat (msec) : 100=0.27%, 250=3.75%, 500=72.54%, 750=23.44% cpu : usr=0.06%, sys=0.59%, ctx=3052, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=595KiB/s (610kB/s), 149KiB/s-149KiB/s (152kB/s-153kB/s), io=40.0MiB (41.9MB), run=68702-68788msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146368: Sat Jun 27 00:18:59 2026 read: IOPS=34, BW=140KiB/s (143kB/s)(5048KiB/36176msec) slat (usec): min=1517, max=13279, avg=4395.57, stdev=1702.78 clat (msec): min=4, max=300, avg=145.03, stdev=38.00 lat (msec): min=9, max=306, avg=149.43, stdev=38.43 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 116], | 30.00th=[ 125], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 153], | 70.00th=[ 163], 80.00th=[ 176], 90.00th=[ 188], 95.00th=[ 207], | 99.00th=[ 262], 99.50th=[ 271], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 55, max= 264, per=25.10%, avg=139.06, stdev=41.18, samples=71 iops : min= 13, max= 66, avg=34.56, stdev=10.34, samples=71 write: IOPS=35, BW=144KiB/s (147kB/s)(5192KiB/36176msec); 0 zone resets slat (usec): min=41, max=3923, avg=236.75, stdev=221.24 clat (msec): min=41, max=589, avg=299.67, stdev=90.99 lat (msec): min=42, max=589, avg=299.91, stdev=91.00 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 159], 10.00th=[ 186], 20.00th=[ 220], | 30.00th=[ 249], 40.00th=[ 275], 50.00th=[ 296], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 376], 90.00th=[ 426], 95.00th=[ 460], | 99.00th=[ 527], 99.50th=[ 542], 99.90th=[ 575], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 79, max= 224, per=24.72%, avg=142.66, stdev=35.36, samples=71 iops : min= 19, max= 56, avg=35.46, stdev= 8.96, samples=71 lat (msec) : 10=0.12%, 20=0.12%, 50=0.43%, 100=3.32%, 250=60.51% lat (msec) : 500=34.26%, 750=1.25% cpu : usr=0.19%, sys=1.40%, ctx=5183, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146369: Sat Jun 27 00:18:59 2026 read: IOPS=34, BW=138KiB/s (141kB/s)(4976KiB/36175msec) slat (usec): min=36, max=14062, avg=4370.79, stdev=1769.12 clat (msec): min=5, max=299, avg=143.94, stdev=37.66 lat (msec): min=8, max=303, avg=148.32, stdev=38.24 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 92], 10.00th=[ 103], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 188], 95.00th=[ 209], | 99.00th=[ 259], 99.50th=[ 275], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 48, max= 256, per=24.70%, avg=136.85, stdev=43.50, samples=71 iops : min= 12, max= 64, avg=34.00, stdev=10.91, samples=71 write: IOPS=36, BW=146KiB/s (149kB/s)(5264KiB/36175msec); 0 zone resets slat (usec): min=57, max=4855, avg=245.70, stdev=288.49 clat (msec): min=40, max=556, avg=299.10, stdev=91.08 lat (msec): min=41, max=556, avg=299.35, stdev=91.11 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 159], 10.00th=[ 188], 20.00th=[ 222], | 30.00th=[ 249], 40.00th=[ 271], 50.00th=[ 292], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 380], 90.00th=[ 422], 95.00th=[ 460], | 99.00th=[ 523], 99.50th=[ 542], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 80, max= 296, per=25.15%, avg=145.10, stdev=36.03, samples=71 iops : min= 20, max= 74, avg=36.07, stdev= 9.11, samples=71 lat (msec) : 10=0.16%, 20=0.08%, 50=0.35%, 100=4.02%, 250=59.06% lat (msec) : 500=35.12%, 750=1.21% cpu : usr=0.17%, sys=1.46%, ctx=5218, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146370: Sat Jun 27 00:18:59 2026 read: IOPS=34, BW=138KiB/s (141kB/s)(4984KiB/36175msec) slat (usec): min=1747, max=14790, avg=4333.48, stdev=1731.06 clat (msec): min=7, max=302, avg=143.91, stdev=36.79 lat (msec): min=10, max=306, avg=148.25, stdev=37.20 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 94], 10.00th=[ 103], 20.00th=[ 114], | 30.00th=[ 125], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 188], 95.00th=[ 203], | 99.00th=[ 268], 99.50th=[ 284], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 56, max= 248, per=24.95%, avg=138.24, stdev=41.52, samples=71 iops : min= 14, max= 62, avg=34.35, stdev=10.44, samples=71 write: IOPS=36, BW=145KiB/s (149kB/s)(5256KiB/36175msec); 0 zone resets slat (usec): min=61, max=2933, avg=234.38, stdev=211.65 clat (msec): min=39, max=590, avg=298.93, stdev=93.17 lat (msec): min=40, max=590, avg=299.17, stdev=93.17 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 161], 10.00th=[ 182], 20.00th=[ 218], | 30.00th=[ 243], 40.00th=[ 268], 50.00th=[ 292], 60.00th=[ 317], | 70.00th=[ 347], 80.00th=[ 380], 90.00th=[ 422], 95.00th=[ 464], | 99.00th=[ 542], 99.50th=[ 558], 99.90th=[ 575], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 63, max= 248, per=24.95%, avg=143.94, stdev=38.13, samples=71 iops : min= 15, max= 62, avg=35.77, stdev= 9.62, samples=71 lat (msec) : 10=0.08%, 20=0.08%, 50=0.31%, 100=3.59%, 250=60.62% lat (msec) : 500=33.87%, 750=1.45% cpu : usr=0.17%, sys=1.43%, ctx=5208, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146371: Sat Jun 27 00:18:59 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(5060KiB/36012msec) slat (usec): min=1443, max=15695, avg=4364.91, stdev=1779.01 clat (msec): min=5, max=357, avg=144.47, stdev=38.12 lat (msec): min=8, max=361, avg=148.84, stdev=38.60 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 188], 95.00th=[ 205], | 99.00th=[ 259], 99.50th=[ 288], 99.90th=[ 355], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 48, max= 248, per=25.41%, avg=140.75, stdev=39.01, samples=71 iops : min= 12, max= 62, avg=34.99, stdev= 9.83, samples=71 write: IOPS=35, BW=144KiB/s (147kB/s)(5180KiB/36012msec); 0 zone resets slat (usec): min=57, max=6431, avg=238.73, stdev=296.15 clat (msec): min=51, max=593, avg=298.54, stdev=91.10 lat (msec): min=51, max=594, avg=298.78, stdev=91.11 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 161], 10.00th=[ 182], 20.00th=[ 222], | 30.00th=[ 247], 40.00th=[ 268], 50.00th=[ 292], 60.00th=[ 313], | 70.00th=[ 342], 80.00th=[ 372], 90.00th=[ 426], 95.00th=[ 464], | 99.00th=[ 542], 99.50th=[ 550], 99.90th=[ 575], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 79, max= 224, per=24.82%, avg=143.24, stdev=37.29, samples=71 iops : min= 19, max= 56, avg=35.61, stdev= 9.40, samples=71 lat (msec) : 10=0.04%, 20=0.12%, 50=0.39%, 100=3.59%, 250=61.09% lat (msec) : 500=33.63%, 750=1.13% cpu : usr=0.18%, sys=1.45%, ctx=5190, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=555KiB/s (568kB/s), 138KiB/s-141KiB/s (141kB/s-144kB/s), io=19.6MiB (20.5MB), run=36012-36176msec WRITE: bw=578KiB/s (591kB/s), 144KiB/s-146KiB/s (147kB/s-149kB/s), io=20.4MiB (21.4MB), run=36012-36176msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146380: Sat Jun 27 00:19:03 2026 read: IOPS=5, BW=220MiB/s (230MB/s)(40.0MiB/182msec) slat (nsec): min=12342k, max=12342k, avg=12341685.00, stdev= 0.00 clat (nsec): min=162498k, max=162498k, avg=162497646.00, stdev= 0.00 lat (nsec): min=174843k, max=174843k, avg=174843379.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 163], 50.00th=[ 163], 60.00th=[ 163], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.52%, ctx=43, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=220MiB/s (230MB/s), 220MiB/s-220MiB/s (230MB/s-230MB/s), io=40.0MiB (41.9MB), run=182-182msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout