== sanity test 398c: run fio to test AIO ================= 20:28:46 (1787790526) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.273483 s, 153 MB/s osc.lustre-OST0000-osc-ffff8e608918a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146069: Wed Aug 26 20:30:56 2026 write: IOPS=20, BW=82.7KiB/s (84.7kB/s)(10.0MiB/123790msec); 0 zone resets slat (usec): min=22, max=17503, avg=206.73, stdev=646.57 clat (msec): min=111, max=1208, avg=773.01, stdev=179.56 lat (msec): min=111, max=1208, avg=773.22, stdev=179.56 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 481], 10.00th=[ 567], 20.00th=[ 634], | 30.00th=[ 701], 40.00th=[ 751], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 869], 80.00th=[ 919], 90.00th=[ 978], 95.00th=[ 1036], | 99.00th=[ 1133], 99.50th=[ 1150], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 8, max= 315, per=25.22%, avg=83.22, stdev=35.00, samples=242 iops : min= 2, max= 78, avg=20.37, stdev= 8.73, samples=242 lat (msec) : 250=1.25%, 500=4.06%, 750=35.04%, 1000=52.11%, 2000=7.54% cpu : usr=0.04%, sys=0.27%, ctx=1869, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146070: Wed Aug 26 20:30:56 2026 write: IOPS=20, BW=82.6KiB/s (84.6kB/s)(10.0MiB/123904msec); 0 zone resets slat (usec): min=20, max=15991, avg=201.30, stdev=670.61 clat (msec): min=121, max=1245, avg=773.78, stdev=178.17 lat (msec): min=121, max=1245, avg=773.99, stdev=178.18 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 477], 10.00th=[ 567], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 751], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 869], 80.00th=[ 911], 90.00th=[ 969], 95.00th=[ 1036], | 99.00th=[ 1133], 99.50th=[ 1183], 99.90th=[ 1217], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 24, max= 313, per=24.91%, avg=82.19, stdev=30.01, samples=245 iops : min= 6, max= 78, avg=20.11, stdev= 7.50, samples=245 lat (msec) : 250=1.41%, 500=3.79%, 750=34.41%, 1000=53.24%, 2000=7.15% cpu : usr=0.05%, sys=0.26%, ctx=2066, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146071: Wed Aug 26 20:30:56 2026 write: IOPS=20, BW=82.9KiB/s (84.9kB/s)(10.0MiB/123521msec); 0 zone resets slat (usec): min=22, max=12949, avg=217.65, stdev=702.70 clat (msec): min=54, max=1351, avg=771.46, stdev=186.23 lat (msec): min=54, max=1351, avg=771.67, stdev=186.21 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 481], 10.00th=[ 558], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 827], | 70.00th=[ 869], 80.00th=[ 927], 90.00th=[ 995], 95.00th=[ 1045], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1301], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 15, max= 320, per=25.11%, avg=82.88, stdev=32.76, samples=243 iops : min= 3, max= 80, avg=20.29, stdev= 8.22, samples=243 lat (msec) : 100=0.39%, 250=1.80%, 500=3.71%, 750=36.13%, 1000=48.87% lat (msec) : 2000=9.10% cpu : usr=0.05%, sys=0.27%, ctx=1879, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146072: Wed Aug 26 20:30:56 2026 write: IOPS=20, BW=82.6KiB/s (84.6kB/s)(10.0MiB/123901msec); 0 zone resets slat (usec): min=26, max=9518, avg=241.72, stdev=712.29 clat (msec): min=115, max=1209, avg=773.75, stdev=179.15 lat (msec): min=115, max=1209, avg=774.00, stdev=179.15 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 472], 10.00th=[ 558], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 827], | 70.00th=[ 869], 80.00th=[ 919], 90.00th=[ 978], 95.00th=[ 1045], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 24, max= 249, per=24.91%, avg=82.20, stdev=27.85, samples=245 iops : min= 6, max= 62, avg=20.12, stdev= 6.98, samples=245 lat (msec) : 250=1.64%, 500=4.18%, 750=36.52%, 1000=49.61%, 2000=8.05% cpu : usr=0.05%, sys=0.29%, ctx=2175, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=331KiB/s (339kB/s), 82.6KiB/s-82.9KiB/s (84.6kB/s-84.9kB/s), io=40.0MiB (41.9MB), run=123521-123904msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146108: Wed Aug 26 20:32:28 2026 read: IOPS=14, BW=59.6KiB/s (61.1kB/s)(5048KiB/84655msec) slat (usec): min=2611, max=47377, avg=11213.65, stdev=5950.57 clat (msec): min=19, max=784, avg=328.10, stdev=78.34 lat (msec): min=28, max=790, avg=339.32, stdev=79.21 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 222], 10.00th=[ 241], 20.00th=[ 268], | 30.00th=[ 288], 40.00th=[ 309], 50.00th=[ 326], 60.00th=[ 342], | 70.00th=[ 363], 80.00th=[ 388], 90.00th=[ 430], 95.00th=[ 456], | 99.00th=[ 523], 99.50th=[ 584], 99.90th=[ 659], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 15, max= 136, per=25.08%, avg=59.18, stdev=25.36, samples=168 iops : min= 3, max= 34, avg=14.40, stdev= 6.38, samples=168 write: IOPS=15, BW=61.3KiB/s (62.8kB/s)(5192KiB/84655msec); 0 zone resets slat (usec): min=28, max=8439, avg=241.85, stdev=600.95 clat (msec): min=196, max=1385, avg=711.28, stdev=173.80 lat (msec): min=196, max=1385, avg=711.52, stdev=173.82 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 443], 10.00th=[ 498], 20.00th=[ 558], | 30.00th=[ 617], 40.00th=[ 667], 50.00th=[ 709], 60.00th=[ 743], | 70.00th=[ 793], 80.00th=[ 852], 90.00th=[ 927], 95.00th=[ 1003], | 99.00th=[ 1183], 99.50th=[ 1301], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 7, max= 143, per=24.71%, avg=60.79, stdev=21.61, samples=168 iops : min= 1, max= 35, avg=14.79, stdev= 5.46, samples=168 lat (msec) : 20=0.04%, 50=0.20%, 100=0.20%, 250=5.86%, 500=47.19% lat (msec) : 750=26.48%, 1000=17.46%, 2000=2.58% cpu : usr=0.12%, sys=0.71%, ctx=4212, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146109: Wed Aug 26 20:32:28 2026 read: IOPS=14, BW=58.8KiB/s (60.2kB/s)(4976KiB/84606msec) slat (usec): min=55, max=49083, avg=11117.11, stdev=5492.63 clat (msec): min=12, max=733, avg=330.59, stdev=83.12 lat (msec): min=13, max=740, avg=341.72, stdev=84.23 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 215], 10.00th=[ 241], 20.00th=[ 268], | 30.00th=[ 288], 40.00th=[ 309], 50.00th=[ 326], 60.00th=[ 342], | 70.00th=[ 363], 80.00th=[ 393], 90.00th=[ 439], 95.00th=[ 464], | 99.00th=[ 567], 99.50th=[ 600], 99.90th=[ 676], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 7, max= 127, per=24.70%, avg=58.29, stdev=25.43, samples=168 iops : min= 1, max= 31, avg=14.17, stdev= 6.42, samples=168 write: IOPS=15, BW=62.2KiB/s (63.7kB/s)(5264KiB/84606msec); 0 zone resets slat (usec): min=27, max=10564, avg=237.32, stdev=598.85 clat (msec): min=88, max=1392, avg=704.69, stdev=177.65 lat (msec): min=89, max=1392, avg=704.93, stdev=177.65 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 435], 10.00th=[ 485], 20.00th=[ 558], | 30.00th=[ 617], 40.00th=[ 676], 50.00th=[ 701], 60.00th=[ 743], | 70.00th=[ 785], 80.00th=[ 844], 90.00th=[ 919], 95.00th=[ 1003], | 99.00th=[ 1150], 99.50th=[ 1301], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 16, max= 120, per=25.06%, avg=61.64, stdev=20.94, samples=168 iops : min= 4, max= 30, avg=15.01, stdev= 5.27, samples=168 lat (msec) : 20=0.08%, 50=0.35%, 100=0.16%, 250=6.05%, 500=46.84% lat (msec) : 750=26.52%, 1000=17.66%, 2000=2.34% cpu : usr=0.11%, sys=0.70%, ctx=4722, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146110: Wed Aug 26 20:32:28 2026 read: IOPS=14, BW=58.7KiB/s (60.1kB/s)(4984KiB/84863msec) slat (msec): min=2, max=372, avg=11.12, stdev=11.73 clat (msec): min=13, max=624, avg=329.61, stdev=82.15 lat (msec): min=24, max=645, avg=340.74, stdev=82.55 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 218], 10.00th=[ 241], 20.00th=[ 266], | 30.00th=[ 284], 40.00th=[ 309], 50.00th=[ 326], 60.00th=[ 347], | 70.00th=[ 363], 80.00th=[ 388], 90.00th=[ 435], 95.00th=[ 468], | 99.00th=[ 567], 99.50th=[ 592], 99.90th=[ 600], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 7, max= 128, per=24.75%, avg=58.41, stdev=24.96, samples=167 iops : min= 1, max= 32, avg=14.20, stdev= 6.29, samples=167 write: IOPS=15, BW=61.9KiB/s (63.4kB/s)(5256KiB/84863msec); 0 zone resets slat (usec): min=27, max=8270, avg=218.71, stdev=493.51 clat (msec): min=73, max=1384, avg=708.37, stdev=180.91 lat (msec): min=75, max=1384, avg=708.60, stdev=180.92 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 435], 10.00th=[ 485], 20.00th=[ 567], | 30.00th=[ 625], 40.00th=[ 667], 50.00th=[ 709], 60.00th=[ 751], | 70.00th=[ 785], 80.00th=[ 852], 90.00th=[ 927], 95.00th=[ 1011], | 99.00th=[ 1200], 99.50th=[ 1284], 99.90th=[ 1318], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 16, max= 104, per=24.69%, avg=60.73, stdev=17.60, samples=168 iops : min= 4, max= 26, avg=14.78, stdev= 4.39, samples=168 lat (msec) : 20=0.08%, 50=0.20%, 100=0.35%, 250=6.56%, 500=45.98% lat (msec) : 750=26.68%, 1000=17.46%, 2000=2.70% cpu : usr=0.10%, sys=0.66%, ctx=4050, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146111: Wed Aug 26 20:32:28 2026 read: IOPS=14, BW=59.9KiB/s (61.3kB/s)(5060KiB/84528msec) slat (usec): min=3035, max=39040, avg=11109.97, stdev=5521.87 clat (msec): min=12, max=931, avg=330.34, stdev=84.37 lat (msec): min=21, max=956, avg=341.46, stdev=85.33 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 220], 10.00th=[ 247], 20.00th=[ 271], | 30.00th=[ 288], 40.00th=[ 309], 50.00th=[ 321], 60.00th=[ 342], | 70.00th=[ 363], 80.00th=[ 388], 90.00th=[ 430], 95.00th=[ 464], | 99.00th=[ 575], 99.50th=[ 625], 99.90th=[ 885], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 152, per=25.50%, avg=60.18, stdev=25.54, samples=166 iops : min= 1, max= 38, avg=14.64, stdev= 6.43, samples=166 write: IOPS=15, BW=61.3KiB/s (62.8kB/s)(5180KiB/84528msec); 0 zone resets slat (usec): min=32, max=6990, avg=209.87, stdev=455.44 clat (msec): min=202, max=1492, avg=708.15, stdev=177.29 lat (msec): min=202, max=1493, avg=708.36, stdev=177.32 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 447], 10.00th=[ 493], 20.00th=[ 550], | 30.00th=[ 609], 40.00th=[ 667], 50.00th=[ 709], 60.00th=[ 751], | 70.00th=[ 793], 80.00th=[ 844], 90.00th=[ 919], 95.00th=[ 995], | 99.00th=[ 1284], 99.50th=[ 1368], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 7, max= 127, per=24.76%, avg=60.92, stdev=21.46, samples=167 iops : min= 1, max= 31, avg=14.83, stdev= 5.38, samples=167 lat (msec) : 20=0.12%, 50=0.12%, 100=0.27%, 250=5.12%, 500=48.44% lat (msec) : 750=25.70%, 1000=17.93%, 2000=2.30% cpu : usr=0.09%, sys=0.67%, ctx=4226, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=236KiB/s (242kB/s), 58.7KiB/s-59.9KiB/s (60.1kB/s-61.3kB/s), io=19.6MiB (20.5MB), run=84528-84863msec WRITE: bw=246KiB/s (252kB/s), 61.3KiB/s-62.2KiB/s (62.8kB/s-63.7kB/s), io=20.4MiB (21.4MB), run=84528-84863msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146127: Wed Aug 26 20:32:36 2026 read: IOPS=2, BW=104MiB/s (109MB/s)(40.0MiB/384msec) slat (nsec): min=36690k, max=36690k, avg=36689731.00, stdev= 0.00 clat (nsec): min=324912k, max=324912k, avg=324912163.00, stdev= 0.00 lat (nsec): min=361606k, max=361606k, avg=361605775.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.39%, ctx=43, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=40.0MiB (41.9MB), run=384-384msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout