== sanity test 398c: run fio to test AIO ================= 11:48:40 (1787845720) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.752751 s, 55.7 MB/s osc.lustre-OST0000-osc-ffff98034819f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179626: Thu Aug 27 11:49:26 2026 write: IOPS=76, BW=307KiB/s (314kB/s)(10.0MiB/33375msec); 0 zone resets slat (usec): min=27, max=19992, avg=202.85, stdev=733.94 clat (msec): min=15, max=368, avg=207.36, stdev=43.18 lat (msec): min=16, max=369, avg=207.56, stdev=43.18 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 148], 10.00th=[ 163], 20.00th=[ 178], | 30.00th=[ 182], 40.00th=[ 190], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 266], 95.00th=[ 288], | 99.00th=[ 338], 99.50th=[ 347], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 151, max= 384, per=24.94%, avg=306.05, stdev=56.59, samples=66 iops : min= 37, max= 96, avg=76.12, stdev=14.21, samples=66 lat (msec) : 20=0.08%, 50=0.16%, 100=0.08%, 250=83.59%, 500=16.09% cpu : usr=0.15%, sys=0.79%, ctx=2282, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179627: Thu Aug 27 11:49:26 2026 write: IOPS=77, BW=308KiB/s (316kB/s)(10.0MiB/33198msec); 0 zone resets slat (usec): min=27, max=15835, avg=137.66, stdev=584.57 clat (msec): min=24, max=366, avg=206.96, stdev=42.57 lat (msec): min=24, max=366, avg=207.10, stdev=42.58 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 153], 10.00th=[ 161], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 215], | 70.00th=[ 226], 80.00th=[ 241], 90.00th=[ 262], 95.00th=[ 275], | 99.00th=[ 338], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 252, max= 492, per=25.16%, avg=308.77, stdev=65.28, samples=65 iops : min= 63, max= 123, avg=76.80, stdev=16.39, samples=65 lat (msec) : 50=0.04%, 100=1.25%, 250=82.89%, 500=15.82% cpu : usr=0.09%, sys=0.53%, ctx=660, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179628: Thu Aug 27 11:49:26 2026 write: IOPS=76, BW=307KiB/s (314kB/s)(10.0MiB/33356msec); 0 zone resets slat (usec): min=28, max=12737, avg=172.16, stdev=630.88 clat (msec): min=43, max=385, avg=207.34, stdev=41.48 lat (msec): min=43, max=385, avg=207.52, stdev=41.48 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 148], 10.00th=[ 159], 20.00th=[ 174], | 30.00th=[ 182], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 264], 95.00th=[ 279], | 99.00th=[ 309], 99.50th=[ 334], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 384] bw ( KiB/s): min= 246, max= 501, per=24.99%, avg=306.59, stdev=56.62, samples=66 iops : min= 61, max= 125, avg=76.26, stdev=14.19, samples=66 lat (msec) : 50=0.04%, 100=0.62%, 250=82.15%, 500=17.19% cpu : usr=0.12%, sys=0.69%, ctx=1691, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179629: Thu Aug 27 11:49:26 2026 write: IOPS=76, BW=307KiB/s (314kB/s)(10.0MiB/33353msec); 0 zone resets slat (usec): min=27, max=15957, avg=215.69, stdev=763.56 clat (msec): min=71, max=375, avg=207.47, stdev=42.08 lat (msec): min=72, max=375, avg=207.68, stdev=42.08 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 148], 10.00th=[ 161], 20.00th=[ 176], | 30.00th=[ 184], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 262], 95.00th=[ 284], | 99.00th=[ 313], 99.50th=[ 351], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 223, max= 429, per=24.98%, avg=306.56, stdev=46.51, samples=66 iops : min= 55, max= 107, avg=76.26, stdev=11.70, samples=66 lat (msec) : 100=0.62%, 250=82.54%, 500=16.84% cpu : usr=0.19%, sys=0.79%, ctx=2369, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1227KiB/s (1257kB/s), 307KiB/s-308KiB/s (314kB/s-316kB/s), io=40.0MiB (41.9MB), run=33198-33375msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179644: Thu Aug 27 11:50:12 2026 read: IOPS=34, BW=139KiB/s (142kB/s)(5048KiB/36367msec) slat (usec): min=43, max=136234, avg=15615.88, stdev=9432.28 clat (msec): min=21, max=401, avg=213.79, stdev=50.47 lat (msec): min=42, max=424, avg=229.41, stdev=52.23 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 342], 99.50th=[ 376], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 55, max= 239, per=25.03%, avg=137.89, stdev=35.73, samples=72 iops : min= 13, max= 59, avg=33.96, stdev= 8.99, samples=72 write: IOPS=35, BW=143KiB/s (146kB/s)(5192KiB/36367msec); 0 zone resets slat (usec): min=28, max=13785, avg=254.20, stdev=859.58 clat (msec): min=82, max=390, avg=222.13, stdev=46.87 lat (msec): min=88, max=390, avg=222.39, stdev=46.88 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 153], 10.00th=[ 167], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 243], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 363], 99.90th=[ 380], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 79, max= 213, per=24.81%, avg=142.39, stdev=30.08, samples=72 iops : min= 19, max= 53, avg=35.11, stdev= 7.63, samples=72 lat (msec) : 50=0.04%, 100=0.27%, 250=76.68%, 500=23.01% cpu : usr=0.23%, sys=1.44%, ctx=4147, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179645: Thu Aug 27 11:50:12 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(4976KiB/36367msec) slat (usec): min=2981, max=59931, avg=15632.72, stdev=8559.03 clat (msec): min=40, max=410, avg=213.89, stdev=49.83 lat (msec): min=51, max=428, avg=229.54, stdev=51.67 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 146], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 368], 99.50th=[ 380], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 23, max= 247, per=24.62%, avg=135.65, stdev=38.30, samples=72 iops : min= 5, max= 61, avg=33.42, stdev= 9.66, samples=72 write: IOPS=36, BW=145KiB/s (148kB/s)(5264KiB/36367msec); 0 zone resets slat (usec): min=31, max=12650, avg=245.34, stdev=810.40 clat (msec): min=58, max=441, avg=221.91, stdev=47.22 lat (msec): min=58, max=443, avg=222.16, stdev=47.22 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 153], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 342], 99.50th=[ 351], 99.90th=[ 418], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 70, max= 222, per=25.16%, avg=144.42, stdev=31.89, samples=72 iops : min= 17, max= 55, avg=35.60, stdev= 8.05, samples=72 lat (msec) : 50=0.04%, 100=0.12%, 250=76.95%, 500=22.89% cpu : usr=0.26%, sys=1.50%, ctx=4149, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179646: Thu Aug 27 11:50:12 2026 read: IOPS=34, BW=138KiB/s (142kB/s)(4984KiB/36059msec) slat (usec): min=31, max=60268, avg=15473.53, stdev=8434.68 clat (msec): min=29, max=408, avg=213.67, stdev=50.29 lat (msec): min=35, max=439, avg=229.15, stdev=52.27 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 234], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 347], 99.50th=[ 372], 99.90th=[ 388], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 71, max= 230, per=25.08%, avg=138.17, stdev=35.30, samples=71 iops : min= 17, max= 57, avg=34.01, stdev= 8.92, samples=71 write: IOPS=36, BW=146KiB/s (149kB/s)(5256KiB/36059msec); 0 zone resets slat (usec): min=30, max=21790, avg=242.93, stdev=900.79 clat (msec): min=38, max=379, avg=220.95, stdev=47.81 lat (msec): min=38, max=379, avg=221.20, stdev=47.82 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 148], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 243], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 62, max= 232, per=25.27%, avg=145.07, stdev=36.89, samples=71 iops : min= 15, max= 58, avg=35.76, stdev= 9.30, samples=71 lat (msec) : 50=0.12%, 100=0.55%, 250=75.70%, 500=23.63% cpu : usr=0.24%, sys=1.55%, ctx=4302, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179647: Thu Aug 27 11:50:12 2026 read: IOPS=34, BW=139KiB/s (143kB/s)(5060KiB/36356msec) slat (usec): min=51, max=61061, avg=15604.24, stdev=8739.12 clat (msec): min=26, max=434, avg=215.27, stdev=50.71 lat (msec): min=44, max=465, avg=230.89, stdev=52.47 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 359], 99.50th=[ 376], 99.90th=[ 426], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 71, max= 215, per=25.15%, avg=138.57, stdev=29.45, samples=72 iops : min= 17, max= 53, avg=34.14, stdev= 7.42, samples=72 write: IOPS=35, BW=142KiB/s (146kB/s)(5180KiB/36356msec); 0 zone resets slat (usec): min=28, max=9105, avg=240.13, stdev=704.89 clat (msec): min=77, max=413, avg=221.00, stdev=48.74 lat (msec): min=80, max=413, avg=221.24, stdev=48.71 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 146], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 243], 80.00th=[ 259], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 372], 99.50th=[ 384], 99.90th=[ 397], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 71, max= 215, per=24.73%, avg=141.94, stdev=32.51, samples=72 iops : min= 17, max= 53, avg=34.99, stdev= 8.25, samples=72 lat (msec) : 50=0.16%, 100=0.31%, 250=75.86%, 500=23.67% cpu : usr=0.21%, sys=1.50%, ctx=3942, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=552KiB/s (565kB/s), 137KiB/s-139KiB/s (140kB/s-143kB/s), io=19.6MiB (20.5MB), run=36059-36367msec WRITE: bw=574KiB/s (588kB/s), 142KiB/s-146KiB/s (146kB/s-149kB/s), io=20.4MiB (21.4MB), run=36059-36367msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179657: Thu Aug 27 11:50:23 2026 read: IOPS=2, BW=90.7MiB/s (95.1MB/s)(40.0MiB/441msec) slat (nsec): min=21632k, max=21632k, avg=21631872.00, stdev= 0.00 clat (nsec): min=384679k, max=384679k, avg=384679344.00, stdev= 0.00 lat (nsec): min=406317k, max=406317k, avg=406316672.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.95%, ctx=3, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=90.7MiB/s (95.1MB/s), 90.7MiB/s-90.7MiB/s (95.1MB/s-95.1MB/s), io=40.0MiB (41.9MB), run=441-441msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout