== sanity test 398c: run fio to test AIO ================= 19:36:01 (1768264561) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.556472 s, 75.4 MB/s osc.lustre-OST0000-osc-ffff8a0582fb1000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182748: Mon Jan 12 19:36:41 2026 write: IOPS=90, BW=364KiB/s (373kB/s)(10.0MiB/28143msec); 0 zone resets slat (usec): min=22, max=14936, avg=200.29, stdev=751.66 clat (msec): min=17, max=301, avg=175.26, stdev=39.48 lat (msec): min=18, max=301, avg=175.46, stdev=39.50 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 123], 10.00th=[ 129], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 228], 95.00th=[ 243], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 255, max= 532, per=25.04%, avg=364.02, stdev=63.76, samples=55 iops : min= 63, max= 133, avg=90.64, stdev=15.95, samples=55 lat (msec) : 20=0.04%, 50=0.23%, 100=0.70%, 250=94.61%, 500=4.41% cpu : usr=0.19%, sys=1.02%, ctx=2553, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182749: Mon Jan 12 19:36:41 2026 write: IOPS=90, BW=364KiB/s (372kB/s)(10.0MiB/28170msec); 0 zone resets slat (usec): min=23, max=10246, avg=121.61, stdev=518.57 clat (msec): min=31, max=287, avg=174.83, stdev=39.31 lat (msec): min=31, max=287, avg=174.95, stdev=39.31 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 122], 10.00th=[ 129], 20.00th=[ 140], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 232], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 251, max= 512, per=24.87%, avg=361.57, stdev=73.55, samples=56 iops : min= 62, max= 128, avg=90.04, stdev=18.44, samples=56 lat (msec) : 50=0.12%, 250=96.02%, 500=3.87% cpu : usr=0.13%, sys=0.60%, ctx=589, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182750: Mon Jan 12 19:36:41 2026 write: IOPS=91, BW=365KiB/s (373kB/s)(10.0MiB/28089msec); 0 zone resets slat (usec): min=25, max=31991, avg=207.60, stdev=1004.77 clat (msec): min=24, max=322, avg=174.99, stdev=39.87 lat (msec): min=25, max=322, avg=175.19, stdev=39.84 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 120], 10.00th=[ 129], 20.00th=[ 140], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 207], 90.00th=[ 228], 95.00th=[ 245], | 99.00th=[ 284], 99.50th=[ 296], 99.90th=[ 305], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 254, max= 509, per=25.09%, avg=364.80, stdev=70.14, samples=55 iops : min= 63, max= 127, avg=90.84, stdev=17.59, samples=55 lat (msec) : 50=0.12%, 100=0.62%, 250=95.23%, 500=4.02% cpu : usr=0.19%, sys=0.92%, ctx=2136, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182751: Mon Jan 12 19:36:41 2026 write: IOPS=90, BW=364KiB/s (372kB/s)(10.0MiB/28165msec); 0 zone resets slat (usec): min=24, max=23978, avg=238.05, stdev=937.59 clat (msec): min=19, max=299, avg=174.81, stdev=40.25 lat (msec): min=19, max=299, avg=175.05, stdev=40.20 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 121], 10.00th=[ 128], 20.00th=[ 140], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 228], 95.00th=[ 251], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 255, max= 513, per=24.87%, avg=361.64, stdev=71.84, samples=56 iops : min= 63, max= 128, avg=90.02, stdev=18.00, samples=56 lat (msec) : 20=0.04%, 50=0.23%, 100=0.70%, 250=93.91%, 500=5.12% cpu : usr=0.25%, sys=1.00%, ctx=2786, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1454KiB/s (1489kB/s), 364KiB/s-365KiB/s (372kB/s-373kB/s), io=40.0MiB (41.9MB), run=28089-28170msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182766: Mon Jan 12 19:37:25 2026 read: IOPS=36, BW=147KiB/s (151kB/s)(5048KiB/34313msec) slat (usec): min=2825, max=84810, avg=16434.54, stdev=9714.04 clat (msec): min=55, max=468, avg=200.33, stdev=53.07 lat (msec): min=64, max=490, avg=216.77, stdev=55.63 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 226], 80.00th=[ 241], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 342], 99.50th=[ 368], 99.90th=[ 460], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 55, max= 212, per=25.01%, avg=146.06, stdev=37.16, samples=67 iops : min= 13, max= 53, avg=36.03, stdev= 9.34, samples=67 write: IOPS=37, BW=151KiB/s (155kB/s)(5192KiB/34313msec); 0 zone resets slat (usec): min=26, max=19926, avg=209.35, stdev=758.59 clat (msec): min=96, max=471, avg=208.40, stdev=53.41 lat (msec): min=96, max=471, avg=208.61, stdev=53.39 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 355], 99.50th=[ 376], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 53, max= 237, per=24.84%, avg=151.00, stdev=42.14, samples=67 iops : min= 13, max= 59, avg=37.28, stdev=10.57, samples=67 lat (msec) : 100=0.31%, 250=81.64%, 500=18.05% cpu : usr=0.29%, sys=1.91%, ctx=4068, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182767: Mon Jan 12 19:37:25 2026 read: IOPS=36, BW=145KiB/s (148kB/s)(4976KiB/34318msec) slat (msec): min=2, max=156, avg=16.26, stdev=10.07 clat (msec): min=85, max=450, avg=202.28, stdev=55.53 lat (msec): min=97, max=469, avg=218.55, stdev=57.18 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 123], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 355], 99.50th=[ 380], 99.90th=[ 422], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 33, max= 220, per=24.69%, avg=144.16, stdev=38.28, samples=67 iops : min= 8, max= 55, avg=35.55, stdev= 9.63, samples=67 write: IOPS=38, BW=153KiB/s (157kB/s)(5264KiB/34318msec); 0 zone resets slat (usec): min=29, max=20590, avg=256.01, stdev=1071.20 clat (msec): min=79, max=412, avg=206.56, stdev=49.66 lat (msec): min=79, max=412, avg=206.82, stdev=49.70 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 129], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 245], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 334], 99.50th=[ 359], 99.90th=[ 409], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 71, max= 277, per=25.24%, avg=153.45, stdev=41.88, samples=67 iops : min= 17, max= 69, avg=37.90, stdev=10.51, samples=67 lat (msec) : 100=0.51%, 250=81.37%, 500=18.12% cpu : usr=0.24%, sys=1.73%, ctx=4161, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182768: Mon Jan 12 19:37:25 2026 read: IOPS=36, BW=145KiB/s (149kB/s)(4984KiB/34315msec) slat (usec): min=3124, max=73396, avg=16532.01, stdev=9905.57 clat (msec): min=75, max=372, avg=202.91, stdev=52.27 lat (msec): min=89, max=431, avg=219.45, stdev=54.47 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 128], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 351], 99.50th=[ 363], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 47, max= 231, per=24.87%, avg=145.25, stdev=37.13, samples=67 iops : min= 11, max= 57, avg=35.82, stdev= 9.31, samples=67 write: IOPS=38, BW=153KiB/s (157kB/s)(5256KiB/34315msec); 0 zone resets slat (usec): min=32, max=11632, avg=231.48, stdev=776.62 clat (msec): min=92, max=438, avg=206.10, stdev=51.62 lat (msec): min=92, max=438, avg=206.33, stdev=51.65 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 130], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 224], 80.00th=[ 243], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 363], 99.50th=[ 380], 99.90th=[ 422], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 72, max= 254, per=25.06%, avg=152.34, stdev=41.82, samples=67 iops : min= 18, max= 63, avg=37.63, stdev=10.49, samples=67 lat (msec) : 100=0.43%, 250=81.91%, 500=17.66% cpu : usr=0.20%, sys=1.84%, ctx=4052, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182769: Mon Jan 12 19:37:25 2026 read: IOPS=37, BW=149KiB/s (152kB/s)(5060KiB/34008msec) slat (usec): min=49, max=70424, avg=15971.77, stdev=9386.08 clat (msec): min=16, max=487, avg=199.99, stdev=55.90 lat (msec): min=17, max=510, avg=215.97, stdev=58.20 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 122], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 182], 50.00th=[ 194], 60.00th=[ 209], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 342], 99.50th=[ 363], 99.90th=[ 397], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 94, max= 230, per=25.40%, avg=148.34, stdev=31.94, samples=67 iops : min= 23, max= 57, avg=36.61, stdev= 8.00, samples=67 write: IOPS=38, BW=152KiB/s (156kB/s)(5180KiB/34008msec); 0 zone resets slat (usec): min=32, max=37845, avg=260.00, stdev=1360.72 clat (msec): min=82, max=472, avg=208.09, stdev=52.96 lat (msec): min=82, max=472, avg=208.36, stdev=52.99 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 134], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 363], 99.50th=[ 393], 99.90th=[ 426], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 70, max= 248, per=25.01%, avg=152.06, stdev=39.41, samples=67 iops : min= 17, max= 62, avg=37.54, stdev= 9.92, samples=67 lat (msec) : 20=0.04%, 50=0.16%, 100=0.90%, 250=80.16%, 500=18.75% cpu : usr=0.28%, sys=1.69%, ctx=3930, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=585KiB/s (599kB/s), 145KiB/s-149KiB/s (148kB/s-152kB/s), io=19.6MiB (20.5MB), run=34008-34318msec WRITE: bw=609KiB/s (623kB/s), 151KiB/s-153KiB/s (155kB/s-157kB/s), io=20.4MiB (21.4MB), run=34008-34318msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182778: Mon Jan 12 19:37:40 2026 read: IOPS=1, BW=63.9MiB/s (67.0MB/s)(40.0MiB/626msec) slat (nsec): min=42165k, max=42165k, avg=42164698.00, stdev= 0.00 clat (nsec): min=574650k, max=574650k, avg=574649714.00, stdev= 0.00 lat (nsec): min=616847k, max=616847k, avg=616847281.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.24%, ctx=6, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.9MiB/s (67.0MB/s), 63.9MiB/s-63.9MiB/s (67.0MB/s-67.0MB/s), io=40.0MiB (41.9MB), run=626-626msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout