== sanity test 398c: run fio to test AIO ================= 20:50:58 (1770688258) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.196681 s, 213 MB/s osc.lustre-OST0000-osc-ffff9a3166b8f000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147553: Mon Feb 9 20:52:05 2026 write: IOPS=40, BW=162KiB/s (166kB/s)(10.0MiB/63189msec); 0 zone resets slat (usec): min=22, max=3997, avg=137.93, stdev=207.51 clat (msec): min=93, max=603, avg=394.53, stdev=74.32 lat (msec): min=93, max=604, avg=394.67, stdev=74.32 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 275], 10.00th=[ 317], 20.00th=[ 347], | 30.00th=[ 363], 40.00th=[ 380], 50.00th=[ 397], 60.00th=[ 414], | 70.00th=[ 430], 80.00th=[ 451], 90.00th=[ 489], 95.00th=[ 510], | 99.00th=[ 558], 99.50th=[ 575], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 600] bw ( KiB/s): min= 104, max= 310, per=24.91%, avg=161.40, stdev=32.51, samples=125 iops : min= 26, max= 77, avg=40.18, stdev= 8.14, samples=125 lat (msec) : 100=0.35%, 250=3.79%, 500=89.34%, 750=6.52% cpu : usr=0.08%, sys=0.41%, ctx=3098, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147554: Mon Feb 9 20:52:05 2026 write: IOPS=40, BW=162KiB/s (166kB/s)(10.0MiB/63108msec); 0 zone resets slat (usec): min=22, max=30299, avg=143.71, stdev=624.53 clat (msec): min=84, max=646, avg=394.08, stdev=76.14 lat (msec): min=84, max=646, avg=394.23, stdev=76.09 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 363], 40.00th=[ 380], 50.00th=[ 397], 60.00th=[ 414], | 70.00th=[ 430], 80.00th=[ 451], 90.00th=[ 489], 95.00th=[ 514], | 99.00th=[ 558], 99.50th=[ 584], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 651] bw ( KiB/s): min= 103, max= 344, per=24.95%, avg=161.66, stdev=35.47, samples=125 iops : min= 25, max= 86, avg=40.24, stdev= 8.89, samples=125 lat (msec) : 100=0.47%, 250=3.75%, 500=88.75%, 750=7.03% cpu : usr=0.07%, sys=0.46%, ctx=2912, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147555: Mon Feb 9 20:52:05 2026 write: IOPS=40, BW=162KiB/s (166kB/s)(10.0MiB/63149msec); 0 zone resets slat (usec): min=24, max=4817, avg=129.92, stdev=187.59 clat (msec): min=93, max=636, avg=394.17, stdev=77.63 lat (msec): min=93, max=637, avg=394.30, stdev=77.64 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 363], 40.00th=[ 376], 50.00th=[ 393], 60.00th=[ 409], | 70.00th=[ 430], 80.00th=[ 456], 90.00th=[ 489], 95.00th=[ 518], | 99.00th=[ 575], 99.50th=[ 592], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 634] bw ( KiB/s): min= 88, max= 384, per=24.93%, avg=161.54, stdev=38.21, samples=125 iops : min= 22, max= 96, avg=40.21, stdev= 9.57, samples=125 lat (msec) : 100=0.62%, 250=3.52%, 500=88.36%, 750=7.50% cpu : usr=0.07%, sys=0.41%, ctx=2944, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147556: Mon Feb 9 20:52:05 2026 write: IOPS=40, BW=162KiB/s (166kB/s)(10.0MiB/63188msec); 0 zone resets slat (usec): min=22, max=3972, avg=128.27, stdev=174.68 clat (msec): min=109, max=637, avg=394.42, stdev=73.21 lat (msec): min=109, max=637, avg=394.55, stdev=73.22 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 275], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 363], 40.00th=[ 380], 50.00th=[ 397], 60.00th=[ 409], | 70.00th=[ 426], 80.00th=[ 451], 90.00th=[ 485], 95.00th=[ 514], | 99.00th=[ 558], 99.50th=[ 575], 99.90th=[ 592], 99.95th=[ 600], | 99.99th=[ 642] bw ( KiB/s): min= 103, max= 311, per=24.83%, avg=160.90, stdev=34.32, samples=125 iops : min= 25, max= 77, avg=40.05, stdev= 8.56, samples=125 lat (msec) : 250=4.14%, 500=88.83%, 750=7.03% cpu : usr=0.08%, sys=0.40%, ctx=2858, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=648KiB/s (664kB/s), 162KiB/s-162KiB/s (166kB/s-166kB/s), io=40.0MiB (41.9MB), run=63108-63189msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147575: Mon Feb 9 20:52:47 2026 read: IOPS=32, BW=132KiB/s (135kB/s)(5048KiB/38280msec) slat (usec): min=1267, max=27472, avg=4056.87, stdev=2082.73 clat (msec): min=3, max=281, avg=150.37, stdev=33.99 lat (msec): min=7, max=285, avg=154.43, stdev=34.32 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 104], 10.00th=[ 114], 20.00th=[ 127], | 30.00th=[ 134], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 192], 95.00th=[ 209], | 99.00th=[ 247], 99.50th=[ 253], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 48, max= 240, per=25.14%, avg=131.72, stdev=39.39, samples=76 iops : min= 12, max= 60, avg=32.75, stdev= 9.90, samples=76 write: IOPS=33, BW=136KiB/s (139kB/s)(5192KiB/38280msec); 0 zone resets slat (usec): min=35, max=4609, avg=138.83, stdev=172.61 clat (msec): min=153, max=578, avg=320.49, stdev=69.15 lat (msec): min=153, max=578, avg=320.63, stdev=69.16 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 218], 10.00th=[ 239], 20.00th=[ 262], | 30.00th=[ 279], 40.00th=[ 300], 50.00th=[ 317], 60.00th=[ 338], | 70.00th=[ 355], 80.00th=[ 376], 90.00th=[ 401], 95.00th=[ 447], | 99.00th=[ 523], 99.50th=[ 542], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 72, max= 192, per=24.77%, avg=134.97, stdev=24.80, samples=76 iops : min= 18, max= 48, avg=33.57, stdev= 6.18, samples=76 lat (msec) : 4=0.04%, 10=0.04%, 20=0.08%, 50=0.20%, 100=1.60% lat (msec) : 250=55.00%, 500=42.11%, 750=0.94% cpu : usr=0.15%, sys=1.05%, ctx=5075, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147576: Mon Feb 9 20:52:47 2026 read: IOPS=32, BW=130KiB/s (133kB/s)(4976KiB/38237msec) slat (usec): min=32, max=20436, avg=4057.26, stdev=2163.00 clat (msec): min=4, max=375, avg=150.61, stdev=37.72 lat (msec): min=6, max=377, avg=154.67, stdev=38.09 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 100], 10.00th=[ 112], 20.00th=[ 126], | 30.00th=[ 133], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 197], 95.00th=[ 213], | 99.00th=[ 249], 99.50th=[ 264], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 48, max= 239, per=24.71%, avg=129.46, stdev=37.49, samples=76 iops : min= 12, max= 59, avg=32.18, stdev= 9.37, samples=76 write: IOPS=34, BW=138KiB/s (141kB/s)(5264KiB/38237msec); 0 zone resets slat (usec): min=34, max=4828, avg=138.73, stdev=189.88 clat (msec): min=53, max=583, avg=318.18, stdev=72.87 lat (msec): min=53, max=583, avg=318.32, stdev=72.88 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 215], 10.00th=[ 239], 20.00th=[ 259], | 30.00th=[ 279], 40.00th=[ 300], 50.00th=[ 321], 60.00th=[ 334], | 70.00th=[ 351], 80.00th=[ 368], 90.00th=[ 405], 95.00th=[ 443], | 99.00th=[ 518], 99.50th=[ 550], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 80, max= 200, per=25.11%, avg=136.86, stdev=24.42, samples=76 iops : min= 20, max= 50, avg=34.03, stdev= 6.12, samples=76 lat (msec) : 10=0.12%, 20=0.31%, 50=0.27%, 100=2.38%, 250=53.40% lat (msec) : 500=42.73%, 750=0.78% cpu : usr=0.16%, sys=1.04%, ctx=5029, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147577: Mon Feb 9 20:52:47 2026 read: IOPS=32, BW=130KiB/s (133kB/s)(4984KiB/38281msec) slat (usec): min=1241, max=27219, avg=4027.61, stdev=2095.96 clat (msec): min=4, max=270, avg=149.49, stdev=33.74 lat (msec): min=6, max=276, avg=153.52, stdev=34.05 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 102], 10.00th=[ 114], 20.00th=[ 126], | 30.00th=[ 133], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 192], 95.00th=[ 207], | 99.00th=[ 245], 99.50th=[ 249], 99.90th=[ 262], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 56, max= 248, per=24.84%, avg=130.17, stdev=38.86, samples=76 iops : min= 14, max= 62, avg=32.36, stdev= 9.76, samples=76 write: IOPS=34, BW=137KiB/s (141kB/s)(5256KiB/38281msec); 0 zone resets slat (usec): min=27, max=3390, avg=147.11, stdev=176.11 clat (msec): min=94, max=575, avg=319.62, stdev=68.80 lat (msec): min=94, max=575, avg=319.77, stdev=68.81 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 218], 10.00th=[ 241], 20.00th=[ 262], | 30.00th=[ 279], 40.00th=[ 300], 50.00th=[ 321], 60.00th=[ 338], | 70.00th=[ 351], 80.00th=[ 372], 90.00th=[ 401], 95.00th=[ 443], | 99.00th=[ 523], 99.50th=[ 542], 99.90th=[ 550], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 80, max= 183, per=25.03%, avg=136.42, stdev=26.14, samples=76 iops : min= 20, max= 45, avg=33.92, stdev= 6.53, samples=76 lat (msec) : 10=0.08%, 20=0.08%, 50=0.35%, 100=1.80%, 250=53.40% lat (msec) : 500=43.55%, 750=0.74% cpu : usr=0.14%, sys=1.07%, ctx=5040, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147578: Mon Feb 9 20:52:47 2026 read: IOPS=33, BW=132KiB/s (136kB/s)(5060KiB/38195msec) slat (usec): min=1320, max=27523, avg=4086.52, stdev=2074.94 clat (msec): min=4, max=333, avg=149.67, stdev=33.74 lat (msec): min=7, max=336, avg=153.76, stdev=33.99 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 104], 10.00th=[ 112], 20.00th=[ 124], | 30.00th=[ 133], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 155], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 194], 95.00th=[ 209], | 99.00th=[ 241], 99.50th=[ 253], 99.90th=[ 284], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 48, max= 224, per=25.42%, avg=133.21, stdev=36.90, samples=75 iops : min= 12, max= 56, avg=33.13, stdev= 9.25, samples=75 write: IOPS=33, BW=136KiB/s (139kB/s)(5180KiB/38195msec); 0 zone resets slat (usec): min=35, max=1771, avg=131.45, stdev=109.22 clat (msec): min=109, max=582, avg=320.63, stdev=70.04 lat (msec): min=110, max=582, avg=320.77, stdev=70.05 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 220], 10.00th=[ 236], 20.00th=[ 259], | 30.00th=[ 284], 40.00th=[ 305], 50.00th=[ 317], 60.00th=[ 334], | 70.00th=[ 351], 80.00th=[ 372], 90.00th=[ 405], 95.00th=[ 447], | 99.00th=[ 523], 99.50th=[ 542], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 80, max= 192, per=24.78%, avg=135.05, stdev=25.06, samples=75 iops : min= 20, max= 48, avg=33.59, stdev= 6.26, samples=75 lat (msec) : 10=0.08%, 20=0.12%, 50=0.12%, 100=1.45%, 250=55.23% lat (msec) : 500=42.23%, 750=0.78% cpu : usr=0.19%, sys=1.01%, ctx=5165, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=524KiB/s (537kB/s), 130KiB/s-132KiB/s (133kB/s-136kB/s), io=19.6MiB (20.5MB), run=38195-38281msec WRITE: bw=546KiB/s (559kB/s), 136KiB/s-138KiB/s (139kB/s-141kB/s), io=20.4MiB (21.4MB), run=38195-38281msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147586: Mon Feb 9 20:52:51 2026 read: IOPS=7, BW=292MiB/s (306MB/s)(40.0MiB/137msec) slat (nsec): min=11109k, max=11109k, avg=11108869.00, stdev= 0.00 clat (nsec): min=117800k, max=117800k, avg=117799593.00, stdev= 0.00 lat (nsec): min=128913k, max=128913k, avg=128913181.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 118], 50.00th=[ 118], 60.00th=[ 118], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.09%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=40.0MiB (41.9MB), run=137-137msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout