== sanity test 398c: run fio to test AIO ================= 01:21:07 (1788672067) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.477448 s, 87.8 MB/s osc.lustre-OST0000-osc-ffff9ddb0aa1c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180476: Sun Sep 6 01:21:36 2026 write: IOPS=119, BW=478KiB/s (489kB/s)(10.0MiB/21426msec); 0 zone resets slat (usec): min=23, max=10423, avg=184.24, stdev=553.27 clat (msec): min=14, max=300, avg=132.76, stdev=35.22 lat (msec): min=14, max=300, avg=132.95, stdev=35.21 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 86], 10.00th=[ 96], 20.00th=[ 109], | 30.00th=[ 117], 40.00th=[ 125], 50.00th=[ 130], 60.00th=[ 136], | 70.00th=[ 142], 80.00th=[ 148], 90.00th=[ 176], 95.00th=[ 199], | 99.00th=[ 257], 99.50th=[ 279], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 300] bw ( KiB/s): min= 256, max= 688, per=24.98%, avg=476.17, stdev=88.96, samples=42 iops : min= 64, max= 172, avg=118.74, stdev=22.25, samples=42 lat (msec) : 20=0.08%, 50=0.39%, 100=12.23%, 250=85.86%, 500=1.45% cpu : usr=0.22%, sys=1.40%, ctx=3046, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180477: Sun Sep 6 01:21:36 2026 write: IOPS=119, BW=477KiB/s (488kB/s)(10.0MiB/21480msec); 0 zone resets slat (usec): min=21, max=21657, avg=172.02, stdev=688.91 clat (msec): min=19, max=298, avg=132.94, stdev=35.00 lat (msec): min=19, max=298, avg=133.12, stdev=35.00 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 86], 10.00th=[ 96], 20.00th=[ 110], | 30.00th=[ 118], 40.00th=[ 123], 50.00th=[ 128], 60.00th=[ 136], | 70.00th=[ 142], 80.00th=[ 153], 90.00th=[ 176], 95.00th=[ 199], | 99.00th=[ 251], 99.50th=[ 296], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 256, max= 640, per=24.90%, avg=474.55, stdev=92.88, samples=42 iops : min= 64, max= 160, avg=118.33, stdev=23.25, samples=42 lat (msec) : 20=0.04%, 50=0.04%, 100=12.50%, 250=86.13%, 500=1.29% cpu : usr=0.26%, sys=1.20%, ctx=2221, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180478: Sun Sep 6 01:21:36 2026 write: IOPS=119, BW=480KiB/s (491kB/s)(10.0MiB/21344msec); 0 zone resets slat (usec): min=21, max=12716, avg=111.21, stdev=373.67 clat (msec): min=54, max=294, avg=133.09, stdev=35.15 lat (msec): min=54, max=294, avg=133.21, stdev=35.14 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 86], 10.00th=[ 99], 20.00th=[ 110], | 30.00th=[ 118], 40.00th=[ 124], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 142], 80.00th=[ 150], 90.00th=[ 176], 95.00th=[ 194], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 296] bw ( KiB/s): min= 253, max= 704, per=25.08%, avg=478.12, stdev=97.23, samples=42 iops : min= 63, max= 176, avg=119.24, stdev=24.34, samples=42 lat (msec) : 100=10.74%, 250=87.34%, 500=1.91% cpu : usr=0.13%, sys=0.80%, ctx=1159, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180479: Sun Sep 6 01:21:36 2026 write: IOPS=119, BW=477KiB/s (488kB/s)(10.0MiB/21474msec); 0 zone resets slat (usec): min=21, max=44063, avg=179.99, stdev=1037.76 clat (msec): min=11, max=295, avg=133.14, stdev=35.66 lat (msec): min=12, max=296, avg=133.32, stdev=35.63 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 84], 10.00th=[ 95], 20.00th=[ 110], | 30.00th=[ 118], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 142], 80.00th=[ 153], 90.00th=[ 176], 95.00th=[ 203], | 99.00th=[ 251], 99.50th=[ 292], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 256, max= 648, per=24.99%, avg=476.38, stdev=90.03, samples=42 iops : min= 64, max= 162, avg=118.81, stdev=22.51, samples=42 lat (msec) : 20=0.04%, 50=0.74%, 100=10.62%, 250=87.66%, 500=0.94% cpu : usr=0.32%, sys=1.18%, ctx=2920, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1907KiB/s (1953kB/s), 477KiB/s-480KiB/s (488kB/s-491kB/s), io=40.0MiB (41.9MB), run=21344-21480msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180492: Sun Sep 6 01:22:14 2026 read: IOPS=44, BW=177KiB/s (181kB/s)(5048KiB/28562msec) slat (usec): min=32, max=217204, avg=12919.57, stdev=10423.48 clat (msec): min=23, max=391, avg=168.19, stdev=44.97 lat (msec): min=27, max=406, avg=181.11, stdev=46.64 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 106], 10.00th=[ 114], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 241], | 99.00th=[ 321], 99.50th=[ 347], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 96, max= 286, per=24.91%, avg=174.86, stdev=39.70, samples=56 iops : min= 24, max= 71, avg=43.30, stdev= 9.90, samples=56 write: IOPS=45, BW=182KiB/s (186kB/s)(5192KiB/28562msec); 0 zone resets slat (usec): min=23, max=60316, avg=287.86, stdev=1860.58 clat (msec): min=22, max=419, avg=175.49, stdev=44.03 lat (msec): min=23, max=419, avg=175.78, stdev=43.98 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 115], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 228], 95.00th=[ 247], | 99.00th=[ 326], 99.50th=[ 368], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 71, max= 275, per=24.72%, avg=180.70, stdev=40.72, samples=56 iops : min= 17, max= 68, avg=44.75, stdev=10.19, samples=56 lat (msec) : 50=0.27%, 100=2.19%, 250=93.59%, 500=3.95% cpu : usr=0.30%, sys=2.00%, ctx=4298, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180493: Sun Sep 6 01:22:14 2026 read: IOPS=43, BW=176KiB/s (180kB/s)(4976KiB/28335msec) slat (usec): min=2616, max=57642, avg=12860.37, stdev=7464.67 clat (msec): min=15, max=339, avg=167.98, stdev=41.24 lat (msec): min=19, max=354, avg=180.84, stdev=42.75 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 108], 10.00th=[ 118], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 199], 90.00th=[ 220], 95.00th=[ 243], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 88, max= 286, per=24.83%, avg=174.32, stdev=38.01, samples=56 iops : min= 22, max= 71, avg=43.18, stdev= 9.47, samples=56 write: IOPS=46, BW=186KiB/s (190kB/s)(5264KiB/28335msec); 0 zone resets slat (usec): min=27, max=18560, avg=302.25, stdev=1141.43 clat (msec): min=34, max=336, avg=171.80, stdev=38.42 lat (msec): min=34, max=336, avg=172.11, stdev=38.42 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 140], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 203], 90.00th=[ 220], 95.00th=[ 239], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 309], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 100, max= 311, per=25.37%, avg=185.48, stdev=42.77, samples=56 iops : min= 25, max= 77, avg=45.96, stdev=10.69, samples=56 lat (msec) : 20=0.08%, 50=0.04%, 100=1.88%, 250=94.84%, 500=3.16% cpu : usr=0.30%, sys=2.36%, ctx=4583, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180494: Sun Sep 6 01:22:14 2026 read: IOPS=43, BW=175KiB/s (179kB/s)(4984KiB/28467msec) slat (usec): min=2431, max=59078, avg=13010.05, stdev=7267.08 clat (msec): min=28, max=344, avg=169.12, stdev=42.33 lat (msec): min=46, max=367, avg=182.14, stdev=43.56 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 108], 10.00th=[ 120], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 205], 90.00th=[ 226], 95.00th=[ 245], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 75, max= 302, per=24.97%, avg=175.30, stdev=42.48, samples=56 iops : min= 18, max= 75, avg=43.37, stdev=10.62, samples=56 write: IOPS=46, BW=185KiB/s (189kB/s)(5256KiB/28467msec); 0 zone resets slat (usec): min=29, max=14427, avg=243.29, stdev=791.88 clat (msec): min=80, max=322, avg=171.74, stdev=37.66 lat (msec): min=80, max=322, avg=171.99, stdev=37.70 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 116], 10.00th=[ 128], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 222], 95.00th=[ 239], | 99.00th=[ 271], 99.50th=[ 292], 99.90th=[ 313], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 81, max= 287, per=25.18%, avg=184.07, stdev=48.00, samples=56 iops : min= 20, max= 71, avg=45.59, stdev=11.99, samples=56 lat (msec) : 50=0.08%, 100=2.23%, 250=94.34%, 500=3.36% cpu : usr=0.33%, sys=2.14%, ctx=4518, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180495: Sun Sep 6 01:22:14 2026 read: IOPS=44, BW=177KiB/s (181kB/s)(5060KiB/28558msec) slat (usec): min=2429, max=57357, avg=12792.70, stdev=7125.32 clat (msec): min=61, max=396, avg=168.02, stdev=42.21 lat (msec): min=70, max=420, avg=180.83, stdev=43.75 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 105], 10.00th=[ 115], 20.00th=[ 133], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 243], | 99.00th=[ 284], 99.50th=[ 296], 99.90th=[ 351], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 54, max= 306, per=25.18%, avg=176.77, stdev=47.14, samples=56 iops : min= 13, max= 76, avg=43.77, stdev=11.79, samples=56 write: IOPS=45, BW=181KiB/s (186kB/s)(5180KiB/28558msec); 0 zone resets slat (usec): min=27, max=22750, avg=290.85, stdev=1202.71 clat (msec): min=87, max=342, avg=173.47, stdev=39.05 lat (msec): min=87, max=342, avg=173.76, stdev=39.02 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 117], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 205], 90.00th=[ 226], 95.00th=[ 245], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 82, max= 279, per=24.69%, avg=180.52, stdev=41.81, samples=56 iops : min= 20, max= 69, avg=44.73, stdev=10.45, samples=56 lat (msec) : 100=2.34%, 250=93.79%, 500=3.87% cpu : usr=0.29%, sys=2.07%, ctx=4312, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=703KiB/s (719kB/s), 175KiB/s-177KiB/s (179kB/s-181kB/s), io=19.6MiB (20.5MB), run=28335-28562msec WRITE: bw=731KiB/s (749kB/s), 181KiB/s-186KiB/s (186kB/s-190kB/s), io=20.4MiB (21.4MB), run=28335-28562msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180503: Sun Sep 6 01:22:25 2026 read: IOPS=1, BW=58.3MiB/s (61.1MB/s)(40.0MiB/686msec) slat (nsec): min=27168k, max=27168k, avg=27167510.00, stdev= 0.00 clat (nsec): min=640215k, max=640215k, avg=640214984.00, stdev= 0.00 lat (nsec): min=667388k, max=667388k, avg=667387517.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.75%, ctx=5, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=58.3MiB/s (61.1MB/s), 58.3MiB/s-58.3MiB/s (61.1MB/s-61.1MB/s), io=40.0MiB (41.9MB), run=686-686msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout