== sanity test 398c: run fio to test AIO ================= 18:41:32 (1789080092) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.392474 s, 107 MB/s osc.lustre-OST0000-osc-ffff9b0faba07800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180035: Thu Sep 10 18:42:04 2026 write: IOPS=107, BW=430KiB/s (441kB/s)(10.0MiB/23803msec); 0 zone resets slat (usec): min=28, max=16290, avg=180.64, stdev=573.67 clat (msec): min=14, max=285, avg=148.40, stdev=33.91 lat (msec): min=14, max=285, avg=148.59, stdev=33.89 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 104], 10.00th=[ 110], 20.00th=[ 121], | 30.00th=[ 132], 40.00th=[ 140], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 192], 95.00th=[ 203], | 99.00th=[ 249], 99.50th=[ 264], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 288] bw ( KiB/s): min= 286, max= 640, per=25.25%, avg=429.79, stdev=71.57, samples=47 iops : min= 71, max= 160, avg=107.15, stdev=17.91, samples=47 lat (msec) : 20=0.08%, 50=1.21%, 100=2.97%, 250=94.96%, 500=0.78% cpu : usr=0.19%, sys=1.04%, ctx=2654, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180036: Thu Sep 10 18:42:04 2026 write: IOPS=106, BW=426KiB/s (436kB/s)(10.0MiB/24037msec); 0 zone resets slat (usec): min=27, max=12716, avg=149.81, stdev=476.90 clat (msec): min=48, max=307, avg=149.49, stdev=30.88 lat (msec): min=48, max=307, avg=149.64, stdev=30.90 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 105], 10.00th=[ 113], 20.00th=[ 123], | 30.00th=[ 131], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 157], | 70.00th=[ 167], 80.00th=[ 174], 90.00th=[ 188], 95.00th=[ 203], | 99.00th=[ 243], 99.50th=[ 259], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 309] bw ( KiB/s): min= 264, max= 638, per=24.97%, avg=425.02, stdev=76.25, samples=47 iops : min= 66, max= 159, avg=105.98, stdev=19.07, samples=47 lat (msec) : 50=0.08%, 100=1.91%, 250=97.19%, 500=0.82% cpu : usr=0.17%, sys=0.80%, ctx=1748, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180037: Thu Sep 10 18:42:04 2026 write: IOPS=106, BW=427KiB/s (437kB/s)(10.0MiB/24005msec); 0 zone resets slat (usec): min=30, max=8366, avg=192.13, stdev=564.50 clat (msec): min=8, max=288, avg=149.25, stdev=32.49 lat (msec): min=9, max=288, avg=149.44, stdev=32.46 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 103], 10.00th=[ 111], 20.00th=[ 122], | 30.00th=[ 132], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 190], 95.00th=[ 205], | 99.00th=[ 243], 99.50th=[ 255], 99.90th=[ 262], 99.95th=[ 264], | 99.99th=[ 288] bw ( KiB/s): min= 271, max= 640, per=25.09%, avg=427.04, stdev=71.55, samples=47 iops : min= 67, max= 160, avg=106.47, stdev=17.91, samples=47 lat (msec) : 10=0.04%, 20=0.04%, 50=0.31%, 100=3.79%, 250=95.04% lat (msec) : 500=0.78% cpu : usr=0.19%, sys=1.12%, ctx=2782, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180038: Thu Sep 10 18:42:04 2026 write: IOPS=106, BW=426KiB/s (436kB/s)(10.0MiB/24061msec); 0 zone resets slat (usec): min=28, max=10654, avg=179.35, stdev=546.64 clat (msec): min=18, max=260, avg=149.58, stdev=31.62 lat (msec): min=18, max=260, avg=149.76, stdev=31.63 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 104], 10.00th=[ 110], 20.00th=[ 123], | 30.00th=[ 132], 40.00th=[ 140], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 192], 95.00th=[ 201], | 99.00th=[ 245], 99.50th=[ 257], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 256, max= 632, per=24.92%, avg=424.17, stdev=78.98, samples=47 iops : min= 64, max= 158, avg=105.77, stdev=19.76, samples=47 lat (msec) : 20=0.04%, 50=0.27%, 100=2.54%, 250=96.33%, 500=0.82% cpu : usr=0.17%, sys=0.94%, ctx=2098, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1702KiB/s (1743kB/s), 426KiB/s-430KiB/s (436kB/s-441kB/s), io=40.0MiB (41.9MB), run=23803-24061msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180052: Thu Sep 10 18:42:41 2026 read: IOPS=40, BW=163KiB/s (167kB/s)(5048KiB/30996msec) slat (usec): min=29, max=109133, avg=13547.09, stdev=8492.26 clat (msec): min=29, max=353, avg=183.16, stdev=48.72 lat (msec): min=35, max=369, avg=196.71, stdev=50.90 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 112], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 224], 90.00th=[ 247], 95.00th=[ 268], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 342], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 94, max= 263, per=24.84%, avg=160.72, stdev=33.00, samples=61 iops : min= 23, max= 65, avg=39.72, stdev= 8.21, samples=61 write: IOPS=41, BW=168KiB/s (172kB/s)(5192KiB/30996msec); 0 zone resets slat (usec): min=31, max=13832, avg=205.11, stdev=734.18 clat (msec): min=35, max=341, avg=188.18, stdev=46.02 lat (msec): min=35, max=341, avg=188.39, stdev=46.07 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 120], 10.00th=[ 134], 20.00th=[ 153], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 197], | 70.00th=[ 209], 80.00th=[ 228], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 309], 99.50th=[ 313], 99.90th=[ 330], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 77, max= 255, per=24.62%, avg=165.92, stdev=39.37, samples=61 iops : min= 19, max= 63, avg=41.02, stdev= 9.81, samples=61 lat (msec) : 50=0.59%, 100=1.68%, 250=88.75%, 500=8.98% cpu : usr=0.25%, sys=1.59%, ctx=4139, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180053: Thu Sep 10 18:42:41 2026 read: IOPS=40, BW=161KiB/s (165kB/s)(4976KiB/30924msec) slat (usec): min=3070, max=70192, avg=13381.79, stdev=7817.62 clat (msec): min=62, max=325, avg=181.88, stdev=46.21 lat (msec): min=69, max=363, avg=195.27, stdev=48.13 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 112], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 222], 90.00th=[ 245], 95.00th=[ 266], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 317], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 71, max= 280, per=24.73%, avg=160.02, stdev=43.28, samples=61 iops : min= 17, max= 70, avg=39.52, stdev=10.79, samples=61 write: IOPS=42, BW=170KiB/s (174kB/s)(5264KiB/30924msec); 0 zone resets slat (usec): min=33, max=12732, avg=235.02, stdev=800.13 clat (msec): min=54, max=331, avg=189.45, stdev=45.31 lat (msec): min=54, max=331, avg=189.69, stdev=45.35 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 120], 10.00th=[ 134], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 226], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 330], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 88, max= 256, per=25.17%, avg=169.64, stdev=36.44, samples=61 iops : min= 22, max= 64, avg=41.95, stdev= 9.13, samples=61 lat (msec) : 100=1.95%, 250=89.18%, 500=8.87% cpu : usr=0.19%, sys=1.65%, ctx=4262, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180054: Thu Sep 10 18:42:41 2026 read: IOPS=40, BW=161KiB/s (165kB/s)(4984KiB/30936msec) slat (usec): min=2487, max=71174, avg=13727.65, stdev=8167.25 clat (msec): min=19, max=357, avg=182.99, stdev=47.43 lat (msec): min=28, max=369, avg=196.72, stdev=49.68 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 111], 10.00th=[ 127], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 222], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 305], 99.50th=[ 309], 99.90th=[ 330], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 92, max= 248, per=24.84%, avg=160.74, stdev=37.79, samples=61 iops : min= 23, max= 62, avg=39.72, stdev= 9.42, samples=61 write: IOPS=42, BW=170KiB/s (174kB/s)(5256KiB/30936msec); 0 zone resets slat (usec): min=28, max=12935, avg=188.35, stdev=579.41 clat (msec): min=28, max=344, avg=187.97, stdev=44.85 lat (msec): min=28, max=344, avg=188.16, stdev=44.85 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 121], 10.00th=[ 134], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 207], 80.00th=[ 222], 90.00th=[ 247], 95.00th=[ 271], | 99.00th=[ 317], 99.50th=[ 321], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 87, max= 312, per=25.02%, avg=168.61, stdev=43.21, samples=61 iops : min= 21, max= 78, avg=41.67, stdev=10.81, samples=61 lat (msec) : 20=0.04%, 50=0.08%, 100=1.95%, 250=88.95%, 500=8.98% cpu : usr=0.22%, sys=1.49%, ctx=4038, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180055: Thu Sep 10 18:42:41 2026 read: IOPS=41, BW=164KiB/s (168kB/s)(5060KiB/30782msec) slat (usec): min=53, max=66182, avg=13356.21, stdev=7951.68 clat (msec): min=19, max=348, avg=183.29, stdev=49.15 lat (msec): min=19, max=363, avg=196.65, stdev=51.20 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 111], 10.00th=[ 123], 20.00th=[ 142], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 79, max= 271, per=25.32%, avg=163.80, stdev=39.88, samples=60 iops : min= 19, max= 67, avg=40.47, stdev= 9.96, samples=60 write: IOPS=42, BW=168KiB/s (172kB/s)(5180KiB/30782msec); 0 zone resets slat (usec): min=33, max=16092, avg=203.89, stdev=731.93 clat (msec): min=77, max=329, avg=187.58, stdev=45.59 lat (msec): min=77, max=329, avg=187.79, stdev=45.60 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 207], 80.00th=[ 226], 90.00th=[ 253], 95.00th=[ 271], | 99.00th=[ 313], 99.50th=[ 317], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 79, max= 272, per=24.83%, avg=167.38, stdev=39.79, samples=60 iops : min= 19, max= 68, avg=41.38, stdev= 9.95, samples=60 lat (msec) : 20=0.04%, 50=0.12%, 100=1.48%, 250=87.89%, 500=10.47% cpu : usr=0.21%, sys=1.45%, ctx=4396, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=647KiB/s (663kB/s), 161KiB/s-164KiB/s (165kB/s-168kB/s), io=19.6MiB (20.5MB), run=30782-30996msec WRITE: bw=674KiB/s (690kB/s), 168KiB/s-170KiB/s (172kB/s-174kB/s), io=20.4MiB (21.4MB), run=30782-30996msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180065: Thu Sep 10 18:42:50 2026 read: IOPS=2, BW=98.5MiB/s (103MB/s)(40.0MiB/406msec) slat (nsec): min=39069k, max=39069k, avg=39069425.00, stdev= 0.00 clat (nsec): min=329878k, max=329878k, avg=329877648.00, stdev= 0.00 lat (nsec): min=368975k, max=368975k, avg=368974977.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.22%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=98.5MiB/s (103MB/s), 98.5MiB/s-98.5MiB/s (103MB/s-103MB/s), io=40.0MiB (41.9MB), run=406-406msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout