== sanity test 398c: run fio to test AIO ================= 07:45:14 (1789386314) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.63688 s, 65.9 MB/s osc.lustre-OST0000-osc-ffff996c73242800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180816: Mon Sep 14 07:45:53 2026 write: IOPS=96, BW=385KiB/s (394kB/s)(10.0MiB/26609msec); 0 zone resets slat (usec): min=27, max=18120, avg=278.06, stdev=968.58 clat (msec): min=27, max=362, avg=165.06, stdev=38.32 lat (msec): min=27, max=362, avg=165.34, stdev=38.32 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 114], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 194], 90.00th=[ 211], 95.00th=[ 228], | 99.00th=[ 257], 99.50th=[ 359], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 174, max= 504, per=25.05%, avg=384.94, stdev=70.16, samples=52 iops : min= 43, max= 126, avg=95.83, stdev=17.63, samples=52 lat (msec) : 50=0.78%, 100=0.47%, 250=96.48%, 500=2.27% cpu : usr=0.20%, sys=1.01%, ctx=2425, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180817: Mon Sep 14 07:45:53 2026 write: IOPS=96, BW=384KiB/s (394kB/s)(10.0MiB/26639msec); 0 zone resets slat (usec): min=28, max=15001, avg=280.05, stdev=884.74 clat (msec): min=8, max=370, avg=165.02, stdev=37.97 lat (msec): min=9, max=370, avg=165.31, stdev=37.96 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 114], 10.00th=[ 121], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 213], 95.00th=[ 232], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 355], 99.95th=[ 363], | 99.99th=[ 372] bw ( KiB/s): min= 175, max= 512, per=25.05%, avg=385.04, stdev=70.32, samples=52 iops : min= 43, max= 128, avg=95.83, stdev=17.68, samples=52 lat (msec) : 10=0.04%, 50=0.23%, 100=1.17%, 250=96.80%, 500=1.76% cpu : usr=0.21%, sys=1.15%, ctx=2449, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180818: Mon Sep 14 07:45:53 2026 write: IOPS=96, BW=386KiB/s (395kB/s)(10.0MiB/26535msec); 0 zone resets slat (usec): min=30, max=16630, avg=282.83, stdev=913.20 clat (msec): min=21, max=381, avg=165.19, stdev=38.16 lat (msec): min=21, max=381, avg=165.48, stdev=38.17 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 117], 10.00th=[ 124], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 209], 95.00th=[ 234], | 99.00th=[ 271], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 251, max= 508, per=25.15%, avg=386.48, stdev=70.95, samples=52 iops : min= 62, max= 127, avg=96.21, stdev=17.82, samples=52 lat (msec) : 50=0.23%, 100=0.90%, 250=96.05%, 500=2.81% cpu : usr=0.17%, sys=1.06%, ctx=2291, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180819: Mon Sep 14 07:45:53 2026 write: IOPS=96, BW=384KiB/s (394kB/s)(10.0MiB/26640msec); 0 zone resets slat (usec): min=23, max=20114, avg=282.28, stdev=971.15 clat (msec): min=23, max=364, avg=165.18, stdev=37.59 lat (msec): min=23, max=364, avg=165.46, stdev=37.59 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 115], 10.00th=[ 123], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 178], 80.00th=[ 192], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 259], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 175, max= 502, per=24.97%, avg=383.77, stdev=69.94, samples=52 iops : min= 43, max= 125, avg=95.48, stdev=17.55, samples=52 lat (msec) : 50=0.08%, 100=0.27%, 250=97.62%, 500=2.03% cpu : usr=0.19%, sys=1.23%, ctx=2509, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1538KiB/s (1574kB/s), 384KiB/s-386KiB/s (394kB/s-395kB/s), io=40.0MiB (41.9MB), run=26535-26640msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180833: Mon Sep 14 07:46:36 2026 read: IOPS=36, BW=145KiB/s (149kB/s)(5048KiB/34798msec) slat (msec): min=2, max=271, avg=15.45, stdev=11.15 clat (msec): min=32, max=488, avg=205.85, stdev=51.71 lat (msec): min=54, max=508, avg=221.30, stdev=53.65 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 130], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 372], 99.50th=[ 397], 99.90th=[ 472], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 73, max= 246, per=25.03%, avg=143.93, stdev=35.66, samples=68 iops : min= 18, max= 61, avg=35.46, stdev= 8.95, samples=68 write: IOPS=37, BW=149KiB/s (153kB/s)(5192KiB/34798msec); 0 zone resets slat (usec): min=33, max=14967, avg=261.07, stdev=830.39 clat (msec): min=105, max=495, avg=210.47, stdev=47.98 lat (msec): min=105, max=495, avg=210.73, stdev=47.93 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 220], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 334], 99.50th=[ 351], 99.90th=[ 489], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 53, max= 248, per=24.91%, avg=148.94, stdev=37.69, samples=68 iops : min= 13, max= 62, avg=36.69, stdev= 9.44, samples=68 lat (msec) : 50=0.04%, 100=0.16%, 250=82.19%, 500=17.62% cpu : usr=0.32%, sys=1.73%, ctx=4452, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180834: Mon Sep 14 07:46:36 2026 read: IOPS=35, BW=143KiB/s (146kB/s)(4976KiB/34899msec) slat (usec): min=35, max=152212, avg=15800.01, stdev=10106.99 clat (msec): min=14, max=395, avg=206.40, stdev=48.78 lat (msec): min=31, max=409, avg=222.21, stdev=50.69 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 167], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 338], 99.50th=[ 355], 99.90th=[ 388], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 79, max= 222, per=24.61%, avg=141.54, stdev=31.75, samples=69 iops : min= 19, max= 55, avg=34.84, stdev= 7.97, samples=69 write: IOPS=37, BW=151KiB/s (154kB/s)(5264KiB/34899msec); 0 zone resets slat (usec): min=32, max=20426, avg=292.21, stdev=1220.10 clat (msec): min=97, max=427, avg=213.56, stdev=50.30 lat (msec): min=98, max=427, avg=213.85, stdev=50.25 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 140], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 222], | 70.00th=[ 234], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 363], 99.50th=[ 401], 99.90th=[ 422], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 95, max= 269, per=25.12%, avg=150.22, stdev=36.56, samples=69 iops : min= 23, max= 67, avg=37.04, stdev= 9.22, samples=69 lat (msec) : 20=0.04%, 50=0.23%, 100=0.16%, 250=81.48%, 500=18.09% cpu : usr=0.31%, sys=1.65%, ctx=4261, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180835: Mon Sep 14 07:46:36 2026 read: IOPS=35, BW=143KiB/s (146kB/s)(4984KiB/34891msec) slat (usec): min=3271, max=93890, avg=15594.38, stdev=8947.26 clat (msec): min=93, max=405, avg=204.45, stdev=48.66 lat (msec): min=107, max=418, avg=220.05, stdev=51.17 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 134], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 338], 99.50th=[ 347], 99.90th=[ 397], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 60, max= 277, per=24.80%, avg=142.62, stdev=37.62, samples=69 iops : min= 15, max= 69, avg=35.13, stdev= 9.43, samples=69 write: IOPS=37, BW=151KiB/s (154kB/s)(5256KiB/34891msec); 0 zone resets slat (usec): min=32, max=14355, avg=243.38, stdev=752.49 clat (msec): min=98, max=392, avg=213.09, stdev=48.16 lat (msec): min=98, max=392, avg=213.34, stdev=48.14 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 234], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 67, max= 248, per=25.03%, avg=149.65, stdev=35.85, samples=69 iops : min= 16, max= 62, avg=36.88, stdev= 9.02, samples=69 lat (msec) : 100=0.20%, 250=82.38%, 500=17.42% cpu : usr=0.26%, sys=1.79%, ctx=4264, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180836: Mon Sep 14 07:46:36 2026 read: IOPS=36, BW=145KiB/s (148kB/s)(5060KiB/34899msec) slat (usec): min=2864, max=77786, avg=15402.20, stdev=8173.80 clat (msec): min=32, max=373, avg=206.74, stdev=48.87 lat (msec): min=56, max=399, avg=222.15, stdev=50.28 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 342], 99.50th=[ 351], 99.90th=[ 368], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 79, max= 245, per=25.09%, avg=144.26, stdev=36.48, samples=69 iops : min= 19, max= 61, avg=35.55, stdev= 9.20, samples=69 write: IOPS=37, BW=148KiB/s (152kB/s)(5180KiB/34899msec); 0 zone resets slat (usec): min=36, max=24717, avg=308.22, stdev=1136.91 clat (msec): min=33, max=378, avg=211.73, stdev=45.74 lat (msec): min=33, max=380, avg=212.04, stdev=45.68 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 220], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 372], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 72, max= 222, per=24.73%, avg=147.90, stdev=34.43, samples=69 iops : min= 18, max= 55, avg=36.45, stdev= 8.59, samples=69 lat (msec) : 50=0.12%, 100=0.04%, 250=82.11%, 500=17.73% cpu : usr=0.31%, sys=1.76%, ctx=4683, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=575KiB/s (589kB/s), 143KiB/s-145KiB/s (146kB/s-149kB/s), io=19.6MiB (20.5MB), run=34798-34899msec WRITE: bw=599KiB/s (613kB/s), 148KiB/s-151KiB/s (152kB/s-154kB/s), io=20.4MiB (21.4MB), run=34798-34899msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180846: Mon Sep 14 07:46:48 2026 read: IOPS=1, BW=62.3MiB/s (65.3MB/s)(40.0MiB/642msec) slat (nsec): min=40209k, max=40209k, avg=40209182.00, stdev= 0.00 clat (nsec): min=564175k, max=564175k, avg=564174709.00, stdev= 0.00 lat (nsec): min=604389k, max=604389k, avg=604389497.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 750=100.00% cpu : usr=0.16%, sys=2.03%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.3MiB/s (65.3MB/s), 62.3MiB/s-62.3MiB/s (65.3MB/s-65.3MB/s), io=40.0MiB (41.9MB), run=642-642msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout