== sanity test 398c: run fio to test AIO ================= 19:44:54 (1786405494) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.479209 s, 87.5 MB/s osc.lustre-OST0000-osc-ffff9aba0847b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=178493: Mon Aug 10 19:45:28 2026 write: IOPS=104, BW=418KiB/s (428kB/s)(10.0MiB/24500msec); 0 zone resets slat (usec): min=51, max=73756, avg=387.42, stdev=1878.12 clat (msec): min=12, max=756, avg=151.57, stdev=78.61 lat (msec): min=13, max=757, avg=151.96, stdev=78.80 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 106], | 30.00th=[ 116], 40.00th=[ 127], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 157], 80.00th=[ 176], 90.00th=[ 218], 95.00th=[ 266], | 99.00th=[ 558], 99.50th=[ 743], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 95, max= 661, per=25.16%, avg=420.23, stdev=131.11, samples=47 iops : min= 23, max= 165, avg=104.70, stdev=32.83, samples=47 lat (msec) : 20=0.12%, 50=0.27%, 100=14.77%, 250=78.16%, 500=5.55% lat (msec) : 750=0.86%, 1000=0.27% cpu : usr=0.16%, sys=1.39%, ctx=2396, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178494: Mon Aug 10 19:45:28 2026 write: IOPS=105, BW=422KiB/s (432kB/s)(10.0MiB/24277msec); 0 zone resets slat (usec): min=43, max=25141, avg=402.75, stdev=1219.32 clat (msec): min=10, max=771, avg=151.11, stdev=79.11 lat (msec): min=10, max=771, avg=151.52, stdev=79.17 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 87], 10.00th=[ 95], 20.00th=[ 106], | 30.00th=[ 115], 40.00th=[ 125], 50.00th=[ 134], 60.00th=[ 146], | 70.00th=[ 157], 80.00th=[ 174], 90.00th=[ 211], 95.00th=[ 264], | 99.00th=[ 523], 99.50th=[ 751], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 776] bw ( KiB/s): min= 79, max= 653, per=25.37%, avg=423.72, stdev=126.78, samples=47 iops : min= 19, max= 163, avg=105.64, stdev=31.73, samples=47 lat (msec) : 20=0.08%, 50=0.08%, 100=15.31%, 250=78.67%, 500=4.38% lat (msec) : 750=1.17%, 1000=0.31% cpu : usr=0.24%, sys=1.64%, ctx=2846, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178495: Mon Aug 10 19:45:28 2026 write: IOPS=104, BW=418KiB/s (428kB/s)(10.0MiB/24497msec); 0 zone resets slat (usec): min=51, max=116928, avg=444.43, stdev=2797.12 clat (msec): min=21, max=769, avg=151.68, stdev=78.98 lat (msec): min=22, max=769, avg=152.12, stdev=79.30 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 86], 10.00th=[ 95], 20.00th=[ 107], | 30.00th=[ 118], 40.00th=[ 128], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 157], 80.00th=[ 174], 90.00th=[ 215], 95.00th=[ 271], | 99.00th=[ 625], 99.50th=[ 735], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 23, max= 637, per=25.15%, avg=419.98, stdev=129.62, samples=47 iops : min= 5, max= 159, avg=104.68, stdev=32.45, samples=47 lat (msec) : 50=0.04%, 100=15.20%, 250=77.70%, 500=5.82%, 750=0.94% lat (msec) : 1000=0.31% cpu : usr=0.24%, sys=1.78%, ctx=2519, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178496: Mon Aug 10 19:45:28 2026 write: IOPS=104, BW=418KiB/s (428kB/s)(10.0MiB/24519msec); 0 zone resets slat (usec): min=52, max=24295, avg=385.52, stdev=1242.95 clat (msec): min=10, max=768, avg=151.59, stdev=81.85 lat (msec): min=10, max=768, avg=151.98, stdev=81.97 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 87], 10.00th=[ 95], 20.00th=[ 105], | 30.00th=[ 114], 40.00th=[ 125], 50.00th=[ 136], 60.00th=[ 146], | 70.00th=[ 159], 80.00th=[ 176], 90.00th=[ 211], 95.00th=[ 262], | 99.00th=[ 609], 99.50th=[ 726], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 99, max= 661, per=24.96%, avg=416.91, stdev=124.06, samples=47 iops : min= 24, max= 165, avg=103.94, stdev=31.07, samples=47 lat (msec) : 20=0.12%, 50=0.20%, 100=14.73%, 250=78.67%, 500=4.73% lat (msec) : 750=1.25%, 1000=0.31% cpu : usr=0.18%, sys=1.66%, ctx=2772, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1671KiB/s (1711kB/s), 418KiB/s-422KiB/s (428kB/s-432kB/s), io=40.0MiB (41.9MB), run=24277-24519msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=178509: Mon Aug 10 19:46:12 2026 read: IOPS=34, BW=136KiB/s (139kB/s)(5048KiB/37095msec) slat (msec): min=3, max=137, avg=17.52, stdev=11.87 clat (msec): min=44, max=879, avg=217.30, stdev=78.38 lat (msec): min=77, max=915, avg=234.83, stdev=80.44 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 122], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 209], 60.00th=[ 226], | 70.00th=[ 243], 80.00th=[ 266], 90.00th=[ 309], 95.00th=[ 342], | 99.00th=[ 447], 99.50th=[ 485], 99.90th=[ 869], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 13, max= 215, per=25.41%, avg=136.44, stdev=40.06, samples=72 iops : min= 3, max= 53, avg=33.67, stdev=10.03, samples=72 write: IOPS=34, BW=140KiB/s (143kB/s)(5192KiB/37095msec); 0 zone resets slat (usec): min=43, max=10551, avg=369.07, stdev=929.14 clat (msec): min=16, max=913, avg=225.14, stdev=88.19 lat (msec): min=16, max=917, avg=225.51, stdev=88.19 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 125], 10.00th=[ 142], 20.00th=[ 163], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 213], 60.00th=[ 232], | 70.00th=[ 247], 80.00th=[ 271], 90.00th=[ 313], 95.00th=[ 363], | 99.00th=[ 464], 99.50th=[ 844], 99.90th=[ 894], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 32, max= 285, per=25.11%, avg=140.35, stdev=43.85, samples=72 iops : min= 8, max= 71, avg=34.60, stdev=11.00, samples=72 lat (msec) : 20=0.04%, 50=0.04%, 100=1.25%, 250=71.52%, 500=26.45% lat (msec) : 750=0.16%, 1000=0.55% cpu : usr=0.16%, sys=1.64%, ctx=4185, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178510: Mon Aug 10 19:46:12 2026 read: IOPS=33, BW=134KiB/s (137kB/s)(4976KiB/37229msec) slat (msec): min=3, max=151, avg=17.69, stdev=13.02 clat (msec): min=25, max=869, avg=220.07, stdev=80.07 lat (msec): min=41, max=902, avg=237.77, stdev=82.34 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 118], 10.00th=[ 136], 20.00th=[ 161], | 30.00th=[ 180], 40.00th=[ 197], 50.00th=[ 213], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 268], 90.00th=[ 305], 95.00th=[ 347], | 99.00th=[ 443], 99.50th=[ 558], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 26, max= 221, per=24.90%, avg=133.71, stdev=39.08, samples=72 iops : min= 6, max= 55, avg=32.93, stdev= 9.79, samples=72 write: IOPS=35, BW=141KiB/s (145kB/s)(5264KiB/37229msec); 0 zone resets slat (usec): min=54, max=31271, avg=450.35, stdev=1651.85 clat (msec): min=15, max=913, avg=224.64, stdev=87.10 lat (msec): min=15, max=913, avg=225.09, stdev=87.21 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 124], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 180], 40.00th=[ 197], 50.00th=[ 213], 60.00th=[ 230], | 70.00th=[ 249], 80.00th=[ 275], 90.00th=[ 313], 95.00th=[ 368], | 99.00th=[ 464], 99.50th=[ 818], 99.90th=[ 852], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 39, max= 285, per=25.38%, avg=141.87, stdev=46.97, samples=72 iops : min= 9, max= 71, avg=34.99, stdev=11.76, samples=72 lat (msec) : 20=0.04%, 50=0.12%, 100=1.64%, 250=69.73%, 500=27.81% lat (msec) : 750=0.12%, 1000=0.55% cpu : usr=0.17%, sys=1.72%, ctx=4152, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178511: Mon Aug 10 19:46:12 2026 read: IOPS=33, BW=134KiB/s (137kB/s)(4984KiB/37182msec) slat (usec): min=95, max=130310, avg=17158.06, stdev=12045.51 clat (msec): min=42, max=889, avg=220.04, stdev=85.93 lat (msec): min=66, max=902, avg=237.21, stdev=88.13 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 120], 10.00th=[ 134], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 224], | 70.00th=[ 247], 80.00th=[ 271], 90.00th=[ 313], 95.00th=[ 355], | 99.00th=[ 460], 99.50th=[ 768], 99.90th=[ 877], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 26, max= 246, per=25.05%, avg=134.50, stdev=37.90, samples=72 iops : min= 6, max= 61, avg=33.15, stdev= 9.48, samples=72 write: IOPS=35, BW=141KiB/s (145kB/s)(5256KiB/37182msec); 0 zone resets slat (usec): min=42, max=41765, avg=387.91, stdev=1534.27 clat (msec): min=39, max=817, avg=226.97, stdev=82.67 lat (msec): min=39, max=818, avg=227.36, stdev=82.70 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 163], | 30.00th=[ 182], 40.00th=[ 199], 50.00th=[ 215], 60.00th=[ 234], | 70.00th=[ 253], 80.00th=[ 279], 90.00th=[ 326], 95.00th=[ 363], | 99.00th=[ 456], 99.50th=[ 768], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 39, max= 253, per=25.25%, avg=141.17, stdev=48.28, samples=72 iops : min= 9, max= 63, avg=34.82, stdev=12.14, samples=72 lat (msec) : 50=0.12%, 100=1.17%, 250=69.02%, 500=28.98%, 750=0.12% lat (msec) : 1000=0.59% cpu : usr=0.19%, sys=1.72%, ctx=4335, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178512: Mon Aug 10 19:46:12 2026 read: IOPS=33, BW=136KiB/s (139kB/s)(5060KiB/37341msec) slat (usec): min=139, max=150366, avg=17602.96, stdev=12877.09 clat (msec): min=23, max=888, avg=218.47, stdev=88.26 lat (msec): min=31, max=901, avg=236.08, stdev=90.39 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 116], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 190], 50.00th=[ 207], 60.00th=[ 226], | 70.00th=[ 243], 80.00th=[ 264], 90.00th=[ 300], 95.00th=[ 342], | 99.00th=[ 584], 99.50th=[ 835], 99.90th=[ 869], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 6, max= 238, per=25.14%, avg=135.01, stdev=43.15, samples=73 iops : min= 1, max= 59, avg=33.27, stdev=10.82, samples=73 write: IOPS=34, BW=139KiB/s (142kB/s)(5180KiB/37341msec); 0 zone resets slat (usec): min=55, max=20456, avg=412.96, stdev=1315.60 clat (msec): min=42, max=873, avg=226.49, stdev=80.70 lat (msec): min=42, max=873, avg=226.90, stdev=80.67 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 184], 40.00th=[ 199], 50.00th=[ 218], 60.00th=[ 232], | 70.00th=[ 251], 80.00th=[ 275], 90.00th=[ 313], 95.00th=[ 368], | 99.00th=[ 481], 99.50th=[ 567], 99.90th=[ 844], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 39, max= 285, per=24.70%, avg=138.05, stdev=40.47, samples=73 iops : min= 9, max= 71, avg=34.03, stdev=10.11, samples=73 lat (msec) : 50=0.31%, 100=0.94%, 250=70.43%, 500=27.38%, 750=0.39% lat (msec) : 1000=0.55% cpu : usr=0.18%, sys=1.72%, ctx=4391, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=537KiB/s (550kB/s), 134KiB/s-136KiB/s (137kB/s-139kB/s), io=19.6MiB (20.5MB), run=37095-37341msec WRITE: bw=559KiB/s (573kB/s), 139KiB/s-141KiB/s (142kB/s-145kB/s), io=20.4MiB (21.4MB), run=37095-37341msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=178522: Mon Aug 10 19:46:20 2026 read: IOPS=1, BW=58.7MiB/s (61.6MB/s)(40.0MiB/681msec) slat (nsec): min=20924k, max=20924k, avg=20923577.00, stdev= 0.00 clat (nsec): min=586599k, max=586599k, avg=586599478.00, stdev= 0.00 lat (nsec): min=607527k, max=607527k, avg=607527447.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.32%, ctx=3, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=58.7MiB/s (61.6MB/s), 58.7MiB/s-58.7MiB/s (61.6MB/s-61.6MB/s), io=40.0MiB (41.9MB), run=681-681msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout