== sanity test 398c: run fio to test AIO ================= 19:26:56 (1755041216) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.138894 s, 302 MB/s osc.lustre-OST0000-osc-ffff9e5082bea800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154609: Tue Aug 12 19:27:30 2025 write: IOPS=84, BW=340KiB/s (348kB/s)(10.0MiB/30131msec); 0 zone resets slat (usec): min=27, max=1225, avg=133.87, stdev=72.92 clat (msec): min=65, max=314, avg=187.93, stdev=35.78 lat (msec): min=65, max=314, avg=188.07, stdev=35.78 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 134], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 236], 95.00th=[ 251], | 99.00th=[ 271], 99.50th=[ 288], 99.90th=[ 305], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 256, max= 463, per=24.95%, avg=339.10, stdev=48.64, samples=59 iops : min= 64, max= 115, avg=84.76, stdev=12.13, samples=59 lat (msec) : 100=0.82%, 250=93.48%, 500=5.70% cpu : usr=0.19%, sys=1.07%, ctx=1853, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154610: Tue Aug 12 19:27:30 2025 write: IOPS=85, BW=341KiB/s (349kB/s)(10.0MiB/30029msec); 0 zone resets slat (usec): min=27, max=1183, avg=138.43, stdev=70.74 clat (msec): min=33, max=307, avg=187.43, stdev=36.74 lat (msec): min=33, max=307, avg=187.57, stdev=36.74 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 132], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 188], 60.00th=[ 197], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 234], 95.00th=[ 251], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 309] bw ( KiB/s): min= 264, max= 541, per=25.02%, avg=340.02, stdev=53.08, samples=59 iops : min= 66, max= 135, avg=85.00, stdev=13.25, samples=59 lat (msec) : 50=0.20%, 100=1.52%, 250=93.20%, 500=5.08% cpu : usr=0.21%, sys=1.13%, ctx=2007, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154611: Tue Aug 12 19:27:30 2025 write: IOPS=84, BW=340KiB/s (348kB/s)(10.0MiB/30132msec); 0 zone resets slat (usec): min=30, max=886, avg=132.34, stdev=70.40 clat (msec): min=69, max=307, avg=187.95, stdev=35.51 lat (msec): min=69, max=307, avg=188.08, stdev=35.52 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 136], 10.00th=[ 146], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 188], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 234], 95.00th=[ 251], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 309] bw ( KiB/s): min= 256, max= 496, per=24.89%, avg=338.32, stdev=48.99, samples=59 iops : min= 64, max= 124, avg=84.58, stdev=12.24, samples=59 lat (msec) : 100=0.74%, 250=94.10%, 500=5.16% cpu : usr=0.19%, sys=1.05%, ctx=1828, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154612: Tue Aug 12 19:27:30 2025 write: IOPS=84, BW=340KiB/s (348kB/s)(10.0MiB/30132msec); 0 zone resets slat (usec): min=30, max=1225, avg=137.01, stdev=71.21 clat (msec): min=65, max=299, avg=187.98, stdev=35.52 lat (msec): min=65, max=299, avg=188.12, stdev=35.53 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 134], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 188], 60.00th=[ 197], | 70.00th=[ 207], 80.00th=[ 218], 90.00th=[ 234], 95.00th=[ 249], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 256, max= 476, per=24.89%, avg=338.24, stdev=49.10, samples=59 iops : min= 64, max= 119, avg=84.56, stdev=12.27, samples=59 lat (msec) : 100=0.90%, 250=94.34%, 500=4.77% cpu : usr=0.19%, sys=1.10%, ctx=1916, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1359KiB/s (1392kB/s), 340KiB/s-341KiB/s (348kB/s-349kB/s), io=40.0MiB (41.9MB), run=30029-30132msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154625: Tue Aug 12 19:27:54 2025 read: IOPS=60, BW=243KiB/s (249kB/s)(5048KiB/20762msec) slat (usec): min=74, max=6669, avg=2551.83, stdev=857.80 clat (msec): min=4, max=227, avg=85.34, stdev=17.69 lat (msec): min=6, max=229, avg=87.90, stdev=17.88 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 79], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 89], | 70.00th=[ 93], 80.00th=[ 96], 90.00th=[ 104], 95.00th=[ 112], | 99.00th=[ 130], 99.50th=[ 136], 99.90th=[ 215], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 136, max= 336, per=25.37%, avg=243.85, stdev=45.21, samples=41 iops : min= 34, max= 84, avg=60.95, stdev=11.30, samples=41 write: IOPS=62, BW=250KiB/s (256kB/s)(5192KiB/20762msec); 0 zone resets slat (usec): min=34, max=607, avg=116.61, stdev=59.89 clat (msec): min=28, max=404, avg=170.08, stdev=39.89 lat (msec): min=28, max=404, avg=170.20, stdev=39.88 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 111], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 220], 95.00th=[ 234], | 99.00th=[ 266], 99.50th=[ 292], 99.90th=[ 380], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 168, max= 304, per=24.93%, avg=249.27, stdev=31.60, samples=41 iops : min= 42, max= 76, avg=62.32, stdev= 7.90, samples=41 lat (msec) : 10=0.20%, 20=0.23%, 50=0.59%, 100=42.30%, 250=55.59% lat (msec) : 500=1.09% cpu : usr=0.31%, sys=1.88%, ctx=2751, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154626: Tue Aug 12 19:27:54 2025 read: IOPS=59, BW=238KiB/s (244kB/s)(4976KiB/20872msec) slat (usec): min=1109, max=7788, avg=2594.93, stdev=868.52 clat (msec): min=3, max=138, avg=85.68, stdev=16.39 lat (msec): min=5, max=143, avg=88.28, stdev=16.52 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 90], | 70.00th=[ 94], 80.00th=[ 97], 90.00th=[ 107], 95.00th=[ 112], | 99.00th=[ 124], 99.50th=[ 133], 99.90th=[ 136], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 144, max= 344, per=24.66%, avg=236.95, stdev=47.50, samples=41 iops : min= 36, max= 86, avg=59.22, stdev=11.89, samples=41 write: IOPS=63, BW=252KiB/s (258kB/s)(5264KiB/20872msec); 0 zone resets slat (usec): min=37, max=2711, avg=120.37, stdev=96.60 clat (msec): min=53, max=283, avg=169.67, stdev=36.32 lat (msec): min=53, max=283, avg=169.79, stdev=36.31 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 114], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 220], 95.00th=[ 232], | 99.00th=[ 249], 99.50th=[ 255], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 176, max= 360, per=25.06%, avg=250.63, stdev=36.87, samples=41 iops : min= 44, max= 90, avg=62.66, stdev= 9.22, samples=41 lat (msec) : 4=0.04%, 10=0.12%, 20=0.31%, 50=0.23%, 100=41.21% lat (msec) : 250=57.62%, 500=0.47% cpu : usr=0.27%, sys=1.94%, ctx=2793, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154627: Tue Aug 12 19:27:54 2025 read: IOPS=59, BW=239KiB/s (245kB/s)(4984KiB/20872msec) slat (usec): min=988, max=7726, avg=2617.62, stdev=873.69 clat (msec): min=11, max=133, avg=86.23, stdev=14.52 lat (msec): min=13, max=138, avg=88.85, stdev=14.64 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 90], | 70.00th=[ 93], 80.00th=[ 97], 90.00th=[ 104], 95.00th=[ 110], | 99.00th=[ 126], 99.50th=[ 129], 99.90th=[ 133], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 128, max= 368, per=24.91%, avg=239.41, stdev=54.83, samples=41 iops : min= 32, max= 92, avg=59.85, stdev=13.71, samples=41 write: IOPS=62, BW=252KiB/s (258kB/s)(5256KiB/20872msec); 0 zone resets slat (usec): min=37, max=2318, avg=120.00, stdev=86.12 clat (msec): min=61, max=284, avg=169.27, stdev=36.12 lat (msec): min=61, max=284, avg=169.39, stdev=36.12 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 113], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 218], 95.00th=[ 230], | 99.00th=[ 253], 99.50th=[ 266], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 168, max= 320, per=24.94%, avg=249.41, stdev=31.93, samples=41 iops : min= 42, max= 80, avg=62.34, stdev= 7.99, samples=41 lat (msec) : 20=0.04%, 50=0.27%, 100=41.68%, 250=57.03%, 500=0.98% cpu : usr=0.26%, sys=1.94%, ctx=2801, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154628: Tue Aug 12 19:27:54 2025 read: IOPS=60, BW=243KiB/s (249kB/s)(5060KiB/20838msec) slat (usec): min=1135, max=6275, avg=2611.72, stdev=857.26 clat (msec): min=3, max=174, avg=85.69, stdev=16.14 lat (msec): min=6, max=177, avg=88.31, stdev=16.26 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 86], 60.00th=[ 89], | 70.00th=[ 93], 80.00th=[ 97], 90.00th=[ 105], 95.00th=[ 113], | 99.00th=[ 128], 99.50th=[ 130], 99.90th=[ 174], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 144, max= 328, per=25.39%, avg=244.02, stdev=45.09, samples=41 iops : min= 36, max= 82, avg=61.00, stdev=11.28, samples=41 write: IOPS=62, BW=249KiB/s (255kB/s)(5180KiB/20838msec); 0 zone resets slat (usec): min=29, max=702, avg=115.83, stdev=57.96 clat (msec): min=52, max=275, avg=170.40, stdev=35.11 lat (msec): min=52, max=275, avg=170.52, stdev=35.11 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 112], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 218], 95.00th=[ 232], | 99.00th=[ 253], 99.50th=[ 257], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 176, max= 328, per=24.78%, avg=247.80, stdev=35.89, samples=41 iops : min= 44, max= 82, avg=61.95, stdev= 8.97, samples=41 lat (msec) : 4=0.04%, 20=0.31%, 50=0.20%, 100=42.27%, 250=56.56% lat (msec) : 500=0.62% cpu : usr=0.19%, sys=2.01%, ctx=2809, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=961KiB/s (985kB/s), 238KiB/s-243KiB/s (244kB/s-249kB/s), io=19.6MiB (20.5MB), run=20762-20872msec WRITE: bw=1001KiB/s (1025kB/s), 249KiB/s-252KiB/s (255kB/s-258kB/s), io=20.4MiB (21.4MB), run=20762-20872msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154635: Tue Aug 12 19:27:57 2025 read: IOPS=10, BW=404MiB/s (424MB/s)(40.0MiB/99msec) slat (nsec): min=8821.1k, max=8821.1k, avg=8821085.00, stdev= 0.00 clat (nsec): min=85220k, max=85220k, avg=85219951.00, stdev= 0.00 lat (nsec): min=94044k, max=94044k, avg=94044466.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[85459], 5.00th=[85459], 10.00th=[85459], 20.00th=[85459], | 30.00th=[85459], 40.00th=[85459], 50.00th=[85459], 60.00th=[85459], | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.18%, ctx=2, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=404MiB/s (424MB/s), 404MiB/s-404MiB/s (424MB/s-424MB/s), io=40.0MiB (41.9MB), run=99-99msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout