== sanity test 398c: run fio to test AIO ================= 20:59:01 (1789433941) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.505121 s, 83.0 MB/s osc.lustre-OST0000-osc-ffff95d242cde800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180779: Mon Sep 14 20:59:33 2026 write: IOPS=114, BW=458KiB/s (469kB/s)(10.0MiB/22355msec); 0 zone resets slat (usec): min=22, max=12744, avg=139.63, stdev=512.30 clat (msec): min=32, max=302, avg=138.68, stdev=34.66 lat (msec): min=32, max=302, avg=138.83, stdev=34.67 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 96], 10.00th=[ 104], 20.00th=[ 113], | 30.00th=[ 120], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 161], 90.00th=[ 188], 95.00th=[ 207], | 99.00th=[ 251], 99.50th=[ 300], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 246, max= 622, per=24.98%, avg=457.34, stdev=85.73, samples=44 iops : min= 61, max= 155, avg=114.07, stdev=21.38, samples=44 lat (msec) : 50=0.08%, 100=6.95%, 250=91.88%, 500=1.09% cpu : usr=0.18%, sys=0.86%, ctx=1781, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180780: Mon Sep 14 20:59:33 2026 write: IOPS=114, BW=458KiB/s (469kB/s)(10.0MiB/22365msec); 0 zone resets slat (usec): min=22, max=22921, avg=149.90, stdev=761.86 clat (msec): min=57, max=303, avg=138.90, stdev=34.08 lat (msec): min=57, max=303, avg=139.05, stdev=34.08 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 99], 10.00th=[ 104], 20.00th=[ 113], | 30.00th=[ 120], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 163], 90.00th=[ 192], 95.00th=[ 207], | 99.00th=[ 251], 99.50th=[ 257], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 280, max= 638, per=24.92%, avg=456.27, stdev=80.45, samples=44 iops : min= 70, max= 159, avg=113.77, stdev=20.11, samples=44 lat (msec) : 100=7.81%, 250=90.94%, 500=1.25% cpu : usr=0.15%, sys=1.07%, ctx=2548, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180781: Mon Sep 14 20:59:33 2026 write: IOPS=114, BW=459KiB/s (470kB/s)(10.0MiB/22307msec); 0 zone resets slat (usec): min=22, max=11661, avg=124.60, stdev=454.92 clat (msec): min=67, max=303, avg=139.06, stdev=34.40 lat (msec): min=67, max=303, avg=139.19, stdev=34.39 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 99], 10.00th=[ 106], 20.00th=[ 114], | 30.00th=[ 120], 40.00th=[ 125], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 165], 90.00th=[ 190], 95.00th=[ 205], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 292], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 256, max= 606, per=25.00%, avg=457.68, stdev=87.72, samples=44 iops : min= 64, max= 151, avg=114.14, stdev=21.91, samples=44 lat (msec) : 100=5.78%, 250=92.85%, 500=1.37% cpu : usr=0.16%, sys=1.01%, ctx=2253, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180782: Mon Sep 14 20:59:33 2026 write: IOPS=114, BW=459KiB/s (470kB/s)(10.0MiB/22293msec); 0 zone resets slat (usec): min=24, max=19539, avg=185.68, stdev=727.40 clat (msec): min=12, max=303, avg=138.54, stdev=35.36 lat (msec): min=12, max=303, avg=138.73, stdev=35.35 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 96], 10.00th=[ 104], 20.00th=[ 114], | 30.00th=[ 121], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 163], 90.00th=[ 188], 95.00th=[ 207], | 99.00th=[ 259], 99.50th=[ 279], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 305] bw ( KiB/s): min= 256, max= 598, per=25.05%, avg=458.70, stdev=81.48, samples=44 iops : min= 64, max= 149, avg=114.39, stdev=20.34, samples=44 lat (msec) : 20=0.04%, 50=0.39%, 100=7.07%, 250=91.37%, 500=1.13% cpu : usr=0.28%, sys=1.19%, ctx=3267, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1831KiB/s (1875kB/s), 458KiB/s-459KiB/s (469kB/s-470kB/s), io=40.0MiB (41.9MB), run=22293-22365msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180795: Mon Sep 14 21:00:11 2026 read: IOPS=41, BW=167KiB/s (171kB/s)(5048KiB/30197msec) slat (usec): min=2779, max=59649, avg=13312.41, stdev=7556.80 clat (msec): min=37, max=336, avg=177.32, stdev=47.89 lat (msec): min=48, max=343, avg=190.64, stdev=50.01 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 107], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 186], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 247], 95.00th=[ 259], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 321], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 95, max= 317, per=25.20%, avg=167.36, stdev=41.81, samples=59 iops : min= 23, max= 79, avg=41.32, stdev=10.55, samples=59 write: IOPS=42, BW=172KiB/s (176kB/s)(5192KiB/30197msec); 0 zone resets slat (usec): min=24, max=17987, avg=221.21, stdev=845.42 clat (msec): min=17, max=331, avg=183.94, stdev=48.39 lat (msec): min=18, max=331, avg=184.16, stdev=48.37 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 115], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 192], | 70.00th=[ 209], 80.00th=[ 224], 90.00th=[ 249], 95.00th=[ 279], | 99.00th=[ 309], 99.50th=[ 313], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 80, max= 280, per=24.91%, avg=172.10, stdev=42.22, samples=59 iops : min= 20, max= 70, avg=42.54, stdev=10.54, samples=59 lat (msec) : 20=0.04%, 50=0.08%, 100=2.30%, 250=88.91%, 500=8.67% cpu : usr=0.26%, sys=1.97%, ctx=4192, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180796: Mon Sep 14 21:00:11 2026 read: IOPS=41, BW=165KiB/s (169kB/s)(4976KiB/30203msec) slat (usec): min=2792, max=62649, avg=13723.96, stdev=8358.46 clat (msec): min=33, max=365, avg=177.29, stdev=48.58 lat (msec): min=42, max=387, avg=191.02, stdev=50.77 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 106], 10.00th=[ 117], 20.00th=[ 136], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 186], | 70.00th=[ 201], 80.00th=[ 222], 90.00th=[ 245], 95.00th=[ 262], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 363], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 87, max= 293, per=24.69%, avg=163.93, stdev=42.61, samples=59 iops : min= 21, max= 73, avg=40.46, stdev=10.67, samples=59 write: IOPS=43, BW=174KiB/s (178kB/s)(5264KiB/30203msec); 0 zone resets slat (usec): min=31, max=10254, avg=197.15, stdev=617.93 clat (msec): min=51, max=351, avg=184.14, stdev=48.89 lat (msec): min=51, max=351, avg=184.34, stdev=48.92 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 228], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 338], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 96, max= 320, per=25.17%, avg=173.95, stdev=41.31, samples=59 iops : min= 24, max= 80, avg=43.00, stdev=10.40, samples=59 lat (msec) : 50=0.04%, 100=2.42%, 250=88.05%, 500=9.49% cpu : usr=0.30%, sys=1.93%, ctx=4674, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180797: Mon Sep 14 21:00:11 2026 read: IOPS=41, BW=165KiB/s (169kB/s)(4984KiB/30176msec) slat (usec): min=36, max=51363, avg=13418.36, stdev=7877.64 clat (msec): min=21, max=405, avg=178.41, stdev=49.73 lat (msec): min=29, max=415, avg=191.84, stdev=51.21 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 110], 10.00th=[ 118], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 186], | 70.00th=[ 201], 80.00th=[ 220], 90.00th=[ 247], 95.00th=[ 264], | 99.00th=[ 317], 99.50th=[ 338], 99.90th=[ 384], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 64, max= 261, per=24.93%, avg=165.54, stdev=41.98, samples=59 iops : min= 16, max= 65, avg=40.88, stdev=10.55, samples=59 write: IOPS=43, BW=174KiB/s (178kB/s)(5256KiB/30176msec); 0 zone resets slat (usec): min=31, max=19250, avg=244.19, stdev=962.83 clat (msec): min=67, max=404, avg=184.51, stdev=49.68 lat (msec): min=67, max=405, avg=184.75, stdev=49.70 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 116], 10.00th=[ 127], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 178], 60.00th=[ 194], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 309], 99.50th=[ 330], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 96, max= 301, per=25.10%, avg=173.46, stdev=44.73, samples=59 iops : min= 24, max= 75, avg=42.88, stdev=11.20, samples=59 lat (msec) : 50=0.08%, 100=2.11%, 250=88.20%, 500=9.61% cpu : usr=0.29%, sys=1.93%, ctx=4367, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180798: Mon Sep 14 21:00:11 2026 read: IOPS=41, BW=168KiB/s (172kB/s)(5060KiB/30153msec) slat (usec): min=2602, max=57132, avg=13573.32, stdev=7515.47 clat (msec): min=36, max=396, avg=177.64, stdev=50.14 lat (msec): min=59, max=416, avg=191.23, stdev=52.11 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 106], 10.00th=[ 118], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 186], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 313], 99.50th=[ 334], 99.90th=[ 384], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 64, max= 248, per=25.38%, avg=168.53, stdev=39.52, samples=59 iops : min= 16, max= 62, avg=41.61, stdev= 9.91, samples=59 write: IOPS=42, BW=172KiB/s (176kB/s)(5180KiB/30153msec); 0 zone resets slat (usec): min=33, max=8345, avg=173.02, stdev=455.57 clat (msec): min=52, max=433, avg=183.47, stdev=49.86 lat (msec): min=52, max=433, avg=183.65, stdev=49.87 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 116], 10.00th=[ 127], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 205], 80.00th=[ 226], 90.00th=[ 247], 95.00th=[ 271], | 99.00th=[ 317], 99.50th=[ 338], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 72, max= 341, per=24.85%, avg=171.73, stdev=44.27, samples=59 iops : min= 18, max= 85, avg=42.44, stdev=11.12, samples=59 lat (msec) : 50=0.04%, 100=2.38%, 250=88.98%, 500=8.59% cpu : usr=0.23%, sys=1.86%, ctx=4240, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=664KiB/s (680kB/s), 165KiB/s-168KiB/s (169kB/s-172kB/s), io=19.6MiB (20.5MB), run=30153-30203msec WRITE: bw=692KiB/s (708kB/s), 172KiB/s-174KiB/s (176kB/s-178kB/s), io=20.4MiB (21.4MB), run=30153-30203msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180808: Mon Sep 14 21:00:23 2026 read: IOPS=1, BW=47.3MiB/s (49.6MB/s)(40.0MiB/846msec) slat (nsec): min=61174k, max=61174k, avg=61173856.00, stdev= 0.00 clat (nsec): min=729991k, max=729991k, avg=729990764.00, stdev= 0.00 lat (nsec): min=791192k, max=791192k, avg=791191800.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 735], 5.00th=[ 735], 10.00th=[ 735], 20.00th=[ 735], | 30.00th=[ 735], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 735], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.66%, ctx=8, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=47.3MiB/s (49.6MB/s), 47.3MiB/s-47.3MiB/s (49.6MB/s-49.6MB/s), io=40.0MiB (41.9MB), run=846-846msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout