== sanity test 398c: run fio to test AIO ================= 06:27:03 (1789122423) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.484938 s, 86.5 MB/s osc.lustre-OST0000-osc-ffff8e790eee8000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180002: Fri Sep 11 06:27:41 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27666msec); 0 zone resets slat (usec): min=29, max=18890, avg=292.87, stdev=1036.86 clat (msec): min=15, max=296, avg=171.95, stdev=37.80 lat (msec): min=15, max=296, avg=172.25, stdev=37.74 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 241], | 99.00th=[ 271], 99.50th=[ 275], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 254, max= 586, per=24.96%, avg=369.11, stdev=62.40, samples=54 iops : min= 63, max= 146, avg=91.89, stdev=15.56, samples=54 lat (msec) : 20=0.04%, 50=0.27%, 100=2.19%, 250=94.61%, 500=2.89% cpu : usr=0.22%, sys=1.29%, ctx=2911, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180003: Fri Sep 11 06:27:41 2026 write: IOPS=92, BW=371KiB/s (380kB/s)(10.0MiB/27601msec); 0 zone resets slat (usec): min=24, max=17442, avg=217.41, stdev=745.34 clat (msec): min=68, max=344, avg=172.09, stdev=37.27 lat (msec): min=68, max=344, avg=172.31, stdev=37.26 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 116], 10.00th=[ 132], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 180], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 271], 99.50th=[ 275], 99.90th=[ 309], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 254, max= 609, per=25.07%, avg=370.76, stdev=68.08, samples=54 iops : min= 63, max= 152, avg=92.28, stdev=16.99, samples=54 lat (msec) : 100=2.19%, 250=94.69%, 500=3.12% cpu : usr=0.15%, sys=0.93%, ctx=1981, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180004: Fri Sep 11 06:27:41 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27684msec); 0 zone resets slat (usec): min=27, max=16220, avg=201.23, stdev=696.35 clat (msec): min=40, max=295, avg=172.02, stdev=36.32 lat (msec): min=43, max=295, avg=172.23, stdev=36.33 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 118], 10.00th=[ 132], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 226], 95.00th=[ 241], | 99.00th=[ 268], 99.50th=[ 271], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 253, max= 625, per=24.91%, avg=368.46, stdev=71.42, samples=54 iops : min= 63, max= 156, avg=91.72, stdev=17.84, samples=54 lat (msec) : 50=0.04%, 100=1.17%, 250=96.56%, 500=2.23% cpu : usr=0.15%, sys=0.82%, ctx=1472, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180005: Fri Sep 11 06:27:41 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27685msec); 0 zone resets slat (usec): min=30, max=17916, avg=275.38, stdev=977.33 clat (msec): min=18, max=284, avg=171.94, stdev=36.77 lat (msec): min=18, max=284, avg=172.22, stdev=36.77 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 268], 99.50th=[ 271], 99.90th=[ 275], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 255, max= 522, per=24.91%, avg=368.39, stdev=66.93, samples=54 iops : min= 63, max= 130, avg=91.69, stdev=16.70, samples=54 lat (msec) : 20=0.08%, 50=0.12%, 100=1.48%, 250=95.43%, 500=2.89% cpu : usr=0.20%, sys=1.02%, ctx=3087, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1480KiB/s (1515kB/s), 370KiB/s-371KiB/s (379kB/s-380kB/s), io=40.0MiB (41.9MB), run=27601-27685msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180019: Fri Sep 11 06:28:25 2026 read: IOPS=35, BW=140KiB/s (143kB/s)(5048KiB/36031msec) slat (usec): min=35, max=104748, avg=15512.82, stdev=8843.58 clat (msec): min=91, max=486, avg=213.23, stdev=53.43 lat (msec): min=99, max=502, avg=228.75, stdev=55.55 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 140], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 372], 99.50th=[ 393], 99.90th=[ 477], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 47, max= 214, per=25.32%, avg=139.76, stdev=32.59, samples=71 iops : min= 11, max= 53, avg=34.39, stdev= 8.17, samples=71 write: IOPS=36, BW=144KiB/s (148kB/s)(5192KiB/36031msec); 0 zone resets slat (usec): min=23, max=15145, avg=253.09, stdev=862.53 clat (msec): min=24, max=524, avg=220.86, stdev=57.27 lat (msec): min=25, max=524, avg=221.11, stdev=57.26 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 188], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 264], 90.00th=[ 288], 95.00th=[ 321], | 99.00th=[ 405], 99.50th=[ 418], 99.90th=[ 485], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 71, max= 255, per=25.06%, avg=144.08, stdev=34.54, samples=71 iops : min= 17, max= 63, avg=35.51, stdev= 8.67, samples=71 lat (msec) : 50=0.08%, 100=0.35%, 250=75.43%, 500=24.10%, 750=0.04% cpu : usr=0.20%, sys=1.55%, ctx=3898, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180020: Fri Sep 11 06:28:25 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(4976KiB/36290msec) slat (msec): min=3, max=103, avg=15.80, stdev= 8.91 clat (msec): min=63, max=434, avg=214.19, stdev=54.33 lat (msec): min=102, max=456, avg=230.00, stdev=56.23 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 144], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 220], | 70.00th=[ 236], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 376], 99.50th=[ 401], 99.90th=[ 430], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 71, max= 231, per=24.65%, avg=136.07, stdev=34.70, samples=72 iops : min= 17, max= 57, avg=33.50, stdev= 8.68, samples=72 write: IOPS=36, BW=145KiB/s (149kB/s)(5264KiB/36290msec); 0 zone resets slat (usec): min=32, max=29052, avg=296.88, stdev=1289.39 clat (msec): min=43, max=423, avg=221.52, stdev=52.77 lat (msec): min=43, max=423, avg=221.82, stdev=52.82 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 148], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 247], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 359], 99.50th=[ 409], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 79, max= 240, per=25.09%, avg=144.26, stdev=37.97, samples=72 iops : min= 19, max= 60, avg=35.54, stdev= 9.52, samples=72 lat (msec) : 50=0.04%, 100=0.31%, 250=74.26%, 500=25.39% cpu : usr=0.24%, sys=1.58%, ctx=3885, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180021: Fri Sep 11 06:28:25 2026 read: IOPS=34, BW=137KiB/s (141kB/s)(4984KiB/36310msec) slat (usec): min=3010, max=96837, avg=16069.05, stdev=8940.50 clat (msec): min=50, max=453, avg=214.21, stdev=55.10 lat (msec): min=70, max=470, avg=230.28, stdev=56.76 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 142], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 317], | 99.00th=[ 393], 99.50th=[ 401], 99.90th=[ 430], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 48, max= 240, per=24.80%, avg=136.90, stdev=40.55, samples=72 iops : min= 12, max= 60, avg=33.69, stdev=10.14, samples=72 write: IOPS=36, BW=145KiB/s (148kB/s)(5256KiB/36310msec); 0 zone resets slat (usec): min=33, max=17660, avg=263.91, stdev=923.49 clat (msec): min=106, max=433, avg=221.58, stdev=53.51 lat (msec): min=106, max=433, avg=221.85, stdev=53.51 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 148], 10.00th=[ 161], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 215], 60.00th=[ 232], | 70.00th=[ 247], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 321], | 99.00th=[ 384], 99.50th=[ 414], 99.90th=[ 430], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 71, max= 223, per=24.93%, avg=143.38, stdev=35.68, samples=72 iops : min= 17, max= 55, avg=35.32, stdev= 8.93, samples=72 lat (msec) : 100=0.12%, 250=74.26%, 500=25.62% cpu : usr=0.22%, sys=1.56%, ctx=3931, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180022: Fri Sep 11 06:28:25 2026 read: IOPS=34, BW=139KiB/s (143kB/s)(5060KiB/36329msec) slat (usec): min=3150, max=70285, avg=15872.66, stdev=9041.46 clat (msec): min=68, max=505, avg=214.66, stdev=57.05 lat (msec): min=75, max=527, avg=230.54, stdev=58.86 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 140], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 239], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 321], | 99.00th=[ 393], 99.50th=[ 409], 99.90th=[ 485], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 63, max= 208, per=25.09%, avg=138.47, stdev=35.57, samples=72 iops : min= 15, max= 52, avg=34.10, stdev= 8.91, samples=72 write: IOPS=35, BW=143KiB/s (146kB/s)(5180KiB/36329msec); 0 zone resets slat (usec): min=33, max=14149, avg=260.90, stdev=836.76 clat (msec): min=75, max=417, avg=220.91, stdev=51.30 lat (msec): min=75, max=417, avg=221.17, stdev=51.30 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 148], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 190], 40.00th=[ 205], 50.00th=[ 215], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 409], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 62, max= 231, per=24.66%, avg=141.81, stdev=37.11, samples=72 iops : min= 15, max= 57, avg=34.93, stdev= 9.28, samples=72 lat (msec) : 100=0.27%, 250=74.38%, 500=25.31%, 750=0.04% cpu : usr=0.26%, sys=1.62%, ctx=4468, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=552KiB/s (566kB/s), 137KiB/s-140KiB/s (140kB/s-143kB/s), io=19.6MiB (20.5MB), run=36031-36329msec WRITE: bw=575KiB/s (589kB/s), 143KiB/s-145KiB/s (146kB/s-149kB/s), io=20.4MiB (21.4MB), run=36031-36329msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180031: Fri Sep 11 06:28:33 2026 read: IOPS=1, BW=79.1MiB/s (82.9MB/s)(40.0MiB/506msec) slat (nsec): min=17254k, max=17254k, avg=17254086.00, stdev= 0.00 clat (nsec): min=464470k, max=464470k, avg=464469761.00, stdev= 0.00 lat (nsec): min=481728k, max=481728k, avg=481728299.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.57%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=79.1MiB/s (82.9MB/s), 79.1MiB/s-79.1MiB/s (82.9MB/s-82.9MB/s), io=40.0MiB (41.9MB), run=506-506msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout