== sanity test 398c: run fio to test AIO ================= 10:55:49 (1790952949) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.570619 s, 73.5 MB/s osc.lustre-OST0000-osc-ffff905958165800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180192: Fri Oct 2 10:56:26 2026 write: IOPS=92, BW=371KiB/s (380kB/s)(10.0MiB/27620msec); 0 zone resets slat (usec): min=22, max=10600, avg=103.84, stdev=377.16 clat (msec): min=55, max=310, avg=171.70, stdev=44.74 lat (msec): min=55, max=311, avg=171.81, stdev=44.74 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 109], 10.00th=[ 122], 20.00th=[ 133], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 207], 90.00th=[ 234], 95.00th=[ 255], | 99.00th=[ 288], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 159, max= 512, per=24.78%, avg=367.28, stdev=92.95, samples=54 iops : min= 39, max= 128, avg=91.48, stdev=23.24, samples=54 lat (msec) : 100=3.16%, 250=89.34%, 500=7.50% cpu : usr=0.10%, sys=0.50%, ctx=402, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180193: Fri Oct 2 10:56:26 2026 write: IOPS=92, BW=371KiB/s (380kB/s)(10.0MiB/27582msec); 0 zone resets slat (usec): min=23, max=10728, avg=148.20, stdev=484.54 clat (msec): min=27, max=317, avg=171.48, stdev=45.16 lat (msec): min=27, max=317, avg=171.63, stdev=45.16 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 112], 10.00th=[ 118], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 176], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 241], 95.00th=[ 259], | 99.00th=[ 288], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 151, max= 518, per=24.85%, avg=368.26, stdev=87.03, samples=54 iops : min= 37, max= 129, avg=91.69, stdev=21.82, samples=54 lat (msec) : 50=0.16%, 100=2.34%, 250=89.77%, 500=7.73% cpu : usr=0.12%, sys=0.81%, ctx=2106, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180194: Fri Oct 2 10:56:26 2026 write: IOPS=92, BW=371KiB/s (380kB/s)(10.0MiB/27585msec); 0 zone resets slat (usec): min=28, max=15619, avg=195.62, stdev=654.58 clat (msec): min=24, max=336, avg=172.03, stdev=46.31 lat (msec): min=24, max=336, avg=172.23, stdev=46.31 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 110], 10.00th=[ 120], 20.00th=[ 132], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 178], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 239], 95.00th=[ 262], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 338] bw ( KiB/s): min= 159, max= 526, per=24.95%, avg=369.78, stdev=82.27, samples=54 iops : min= 39, max= 131, avg=92.07, stdev=20.61, samples=54 lat (msec) : 50=0.20%, 100=2.38%, 250=89.92%, 500=7.50% cpu : usr=0.29%, sys=1.31%, ctx=3174, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180195: Fri Oct 2 10:56:26 2026 write: IOPS=92, BW=371KiB/s (380kB/s)(10.0MiB/27595msec); 0 zone resets slat (usec): min=23, max=16905, avg=168.31, stdev=722.16 clat (msec): min=28, max=329, avg=171.80, stdev=45.19 lat (msec): min=29, max=329, avg=171.97, stdev=45.22 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 112], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 190], 80.00th=[ 209], 90.00th=[ 236], 95.00th=[ 264], | 99.00th=[ 296], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 247, max= 512, per=24.83%, avg=367.98, stdev=78.97, samples=54 iops : min= 61, max= 128, avg=91.63, stdev=19.78, samples=54 lat (msec) : 50=0.08%, 100=1.45%, 250=92.27%, 500=6.21% cpu : usr=0.19%, sys=0.72%, ctx=1802, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1483KiB/s (1519kB/s), 371KiB/s-371KiB/s (380kB/s-380kB/s), io=40.0MiB (41.9MB), run=27582-27620msec 0% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180209: Fri Oct 2 10:57:11 2026 read: IOPS=35, BW=142KiB/s (146kB/s)(5048KiB/35506msec) slat (usec): min=3349, max=72700, avg=15793.43, stdev=9236.53 clat (msec): min=80, max=541, avg=209.41, stdev=56.45 lat (msec): min=84, max=553, avg=225.21, stdev=58.79 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 133], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 247], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 388], 99.50th=[ 405], 99.90th=[ 531], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 71, max= 256, per=25.15%, avg=141.59, stdev=34.59, samples=70 iops : min= 17, max= 64, avg=34.93, stdev= 8.70, samples=70 write: IOPS=36, BW=146KiB/s (150kB/s)(5192KiB/35506msec); 0 zone resets slat (usec): min=33, max=16988, avg=221.71, stdev=818.54 clat (msec): min=102, max=430, avg=215.19, stdev=52.37 lat (msec): min=102, max=430, avg=215.41, stdev=52.37 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 138], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 182], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 363], 99.50th=[ 380], 99.90th=[ 409], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 39, max= 263, per=24.89%, avg=146.09, stdev=41.67, samples=70 iops : min= 9, max= 65, avg=36.03, stdev=10.46, samples=70 lat (msec) : 100=0.20%, 250=78.32%, 500=21.37%, 750=0.12% cpu : usr=0.23%, sys=1.79%, ctx=4387, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180210: Fri Oct 2 10:57:11 2026 read: IOPS=34, BW=140KiB/s (143kB/s)(4976KiB/35587msec) slat (usec): min=2653, max=71447, avg=15902.03, stdev=9375.79 clat (msec): min=16, max=465, avg=210.38, stdev=56.67 lat (msec): min=26, max=507, avg=226.30, stdev=58.95 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 134], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 380], 99.50th=[ 430], 99.90th=[ 451], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 34, max= 231, per=24.74%, avg=139.27, stdev=38.89, samples=70 iops : min= 8, max= 57, avg=34.33, stdev= 9.74, samples=70 write: IOPS=36, BW=148KiB/s (151kB/s)(5264KiB/35587msec); 0 zone resets slat (usec): min=26, max=14231, avg=255.30, stdev=903.30 clat (msec): min=85, max=432, avg=215.41, stdev=55.62 lat (msec): min=86, max=432, avg=215.66, stdev=55.61 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 142], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 255], 90.00th=[ 292], 95.00th=[ 321], | 99.00th=[ 388], 99.50th=[ 414], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 63, max= 295, per=25.17%, avg=147.77, stdev=43.28, samples=70 iops : min= 15, max= 73, avg=36.49, stdev=10.87, samples=70 lat (msec) : 20=0.08%, 50=0.04%, 100=0.43%, 250=77.62%, 500=21.84% cpu : usr=0.34%, sys=1.73%, ctx=4587, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180211: Fri Oct 2 10:57:11 2026 read: IOPS=35, BW=141KiB/s (145kB/s)(4984KiB/35268msec) slat (usec): min=41, max=66113, avg=15599.89, stdev=8930.39 clat (msec): min=24, max=571, avg=206.54, stdev=55.10 lat (msec): min=24, max=594, avg=222.15, stdev=57.23 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 228], 80.00th=[ 245], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 363], 99.50th=[ 414], 99.90th=[ 518], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 48, max= 247, per=25.01%, avg=140.78, stdev=41.26, samples=69 iops : min= 12, max= 61, avg=34.74, stdev=10.33, samples=69 write: IOPS=37, BW=149KiB/s (153kB/s)(5256KiB/35268msec); 0 zone resets slat (usec): min=32, max=11395, avg=220.29, stdev=723.30 clat (msec): min=105, max=477, avg=217.92, stdev=54.07 lat (msec): min=105, max=477, avg=218.14, stdev=54.05 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 372], 99.50th=[ 397], 99.90th=[ 460], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 56, max= 222, per=25.22%, avg=148.04, stdev=34.60, samples=69 iops : min= 14, max= 55, avg=36.54, stdev= 8.70, samples=69 lat (msec) : 50=0.12%, 100=0.43%, 250=77.93%, 500=21.45%, 750=0.08% cpu : usr=0.24%, sys=1.73%, ctx=4594, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180212: Fri Oct 2 10:57:11 2026 read: IOPS=35, BW=142KiB/s (146kB/s)(5060KiB/35577msec) slat (usec): min=2982, max=68838, avg=15531.76, stdev=8873.73 clat (msec): min=49, max=496, avg=206.29, stdev=52.15 lat (msec): min=71, max=525, avg=221.82, stdev=54.60 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 228], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 363], 99.50th=[ 380], 99.90th=[ 489], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 62, max= 245, per=25.22%, avg=142.01, stdev=42.27, samples=70 iops : min= 15, max= 61, avg=35.04, stdev=10.60, samples=70 write: IOPS=36, BW=146KiB/s (149kB/s)(5180KiB/35577msec); 0 zone resets slat (usec): min=26, max=12164, avg=213.77, stdev=826.68 clat (msec): min=59, max=456, avg=219.66, stdev=57.39 lat (msec): min=59, max=456, avg=219.87, stdev=57.45 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 176], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 243], 80.00th=[ 262], 90.00th=[ 296], 95.00th=[ 326], | 99.00th=[ 409], 99.50th=[ 435], 99.90th=[ 451], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 68, max= 216, per=24.70%, avg=145.01, stdev=34.40, samples=70 iops : min= 17, max= 54, avg=35.79, stdev= 8.62, samples=70 lat (msec) : 50=0.04%, 100=0.43%, 250=77.50%, 500=22.03% cpu : usr=0.26%, sys=1.48%, ctx=4104, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=564KiB/s (577kB/s), 140KiB/s-142KiB/s (143kB/s-146kB/s), io=19.6MiB (20.5MB), run=35268-35587msec WRITE: bw=587KiB/s (601kB/s), 146KiB/s-149KiB/s (149kB/s-153kB/s), io=20.4MiB (21.4MB), run=35268-35587msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180222: Fri Oct 2 10:57:24 2026 read: IOPS=2, BW=80.5MiB/s (84.4MB/s)(40.0MiB/497msec) slat (nsec): min=23942k, max=23942k, avg=23942123.00, stdev= 0.00 clat (nsec): min=458236k, max=458236k, avg=458235789.00, stdev= 0.00 lat (nsec): min=482184k, max=482184k, avg=482184156.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.62%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=80.5MiB/s (84.4MB/s), 80.5MiB/s-80.5MiB/s (84.4MB/s-84.4MB/s), io=40.0MiB (41.9MB), run=497-497msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout