== sanity test 127f: OST IO latency histograms by size === 07:45:13 (1777031113) osd-ldiskfs.lustre-OST0000.read_cache_enable=0 osd-ldiskfs.lustre-OST0000.writethrough_cache_enable=0 osd-ldiskfs.lustre-OST0000.io_latency_stats=clear osd-ldiskfs.lustre-OST0000.stats=clear osd-ldiskfs.lustre-OST0001.stats=clear Generating I/O with size 4K 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.142622 s, 287 kB/s 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0730712 s, 561 kB/s Generating I/O with size 64K 10+0 records in 10+0 records out 655360 bytes (655 kB, 640 KiB) copied, 0.199891 s, 3.3 MB/s 10+0 records in 10+0 records out 655360 bytes (655 kB, 640 KiB) copied, 0.151155 s, 4.3 MB/s Generating I/O with size 1024K 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.492731 s, 21.3 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.337207 s, 31.1 MB/s osd-*.lustre-OST0000.io_latency_stats stats after I/O: osd-ldiskfs.lustre-OST0000.io_latency_stats= io_latency_by_size: snapshot_time: 1777031125.703522807 start_time: 1777031117.836734563 elapsed_time: 7.866788244 rd_64K: { 128us: 1, 256us: 3, 512us: 4, 1024us: 1, 2048us: 1, } rd_1024K: { 512us: 5, 1024us: 4, 4096us: 1, } wr_64K: { 256us: 5, 512us: 3, 1024us: 1, 2048us: 1, } wr_1024K: { 1024us: 7, 2048us: 2, 4096us: 1, } osc.lustre-OST0000-osc-ffff99da8143b000.io_latency_stats= io_latency_by_size: snapshot_time: 1777031125.262250352 start_time: 1777031103.950589409 elapsed_time: 21.311660943 rd_4K: { 2048us: 10, 4096us: 10, } rd_64K: { 2048us: 3, 4096us: 9, 8192us: 7, 16384us: 1, } rd_1024K: { 4096us: 2, 8192us: 16, 16384us: 2, } wr_4K: { 4096us: 3, 8192us: 14, 16384us: 3, } wr_64K: { 4096us: 1, 8192us: 16, 16384us: 3, } wr_1024K: { 8192us: 8, 16384us: 12, } osc.lustre-OST0001-osc-ffff99da8143b000.io_latency_stats= io_latency_by_size: snapshot_time: 1777031125.262317915 start_time: 1777028831.081756626 elapsed_time: 2294.180561289 rd_4K: { 2048us: 1, } rd_2048K: { 65536us: 1, } rd_4096K: { 32768us: 1, 65536us: 5, 131072us: 7, } wr_512K: { 16384us: 1, } wr_2048K: { 32768us: 2, } wr_4096K: { 32768us: 2, 65536us: 8, 131072us: 4, } osc.lustre-OST0000-osc-ffff99da8143b000.rpc_stats= snapshot_time: 1777031125.263546741 secs.nsecs start_time: 1777031097.250392243 secs.nsecs elapsed_time: 28.013154498 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 20 31 31 | 20 31 31 2: 0 0 31 | 0 0 31 4: 0 0 31 | 0 0 31 8: 0 0 31 | 0 0 31 16: 20 31 62 | 20 31 63 32: 0 0 62 | 0 0 63 64: 0 0 62 | 0 0 63 128: 0 0 62 | 0 0 63 256: 21 32 95 | 20 31 95 512: 1 1 96 | 1 1 96 1024: 2 3 100 | 2 3 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 63 98 98 | 63 100 100 2: 1 1 100 | 0 0 100 read write offset rpcs % cum % | rpcs % cum % 0: 7 10 10 | 7 11 11 1: 2 3 14 | 2 3 14 2: 4 6 20 | 4 6 20 4: 8 12 32 | 8 12 33 8: 4 6 39 | 4 6 39 16: 2 3 42 | 2 3 42 32: 4 6 48 | 4 6 49 64: 8 12 60 | 8 12 61 128: 4 6 67 | 4 6 68 256: 3 4 71 | 2 3 71 512: 4 6 78 | 4 6 77 1024: 9 14 92 | 9 14 92 2048: 5 7 100 | 5 7 100 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 13 20 20 | 0 0 0 4096: 21 32 53 | 4 6 6 8192: 23 35 89 | 38 60 66 16384: 5 7 96 | 19 30 96 32768: 1 1 98 | 2 3 100 65536: 1 1 100 | 0 0 100 osc.lustre-OST0001-osc-ffff99da8143b000.rpc_stats= snapshot_time: 1777031125.263658435 secs.nsecs start_time: 1777031097.250669355 secs.nsecs elapsed_time: 28.012989080 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 1 100 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 0 0 0 | 1 100 100 read write offset rpcs % cum % | rpcs % cum % 0: 0 0 0 | 1 100 100 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 0 0 0 | 0 0 0 4096: 0 0 0 | 0 0 0 8192: 0 0 0 | 0 0 0 16384: 0 0 0 | 1 100 100 osd-ldiskfs.lustre-MDT0000.brw_stats= snapshot_time: 1777031126.224985047 secs.nsecs start_time: 1777023524.752186228 secs.nsecs elapsed_time: 7601.472798819 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 32: 0 0 0 | 10 100 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 1: 0 0 0 | 10 100 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 1: 0 0 0 | 10 100 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 0 0 0 | 10 100 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 0 0 0 | 10 100 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 0 0 0 | 10 100 100 read | write disk I/O size ios % cum % | ios % cum % 128K: 0 0 0 | 10 100 100 read | write block maps msec maps % cum % | maps % cum % 1: 0 0 0 | 10 100 100 osd-ldiskfs.lustre-MDT0001.brw_stats= snapshot_time: 1777031126.225215523 secs.nsecs start_time: 1777023537.410165305 secs.nsecs elapsed_time: 7588.815050218 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % read | write discontiguous pages rpcs % cum % | rpcs % cum % read | write discontiguous blocks rpcs % cum % | rpcs % cum % read | write disk fragmented I/Os ios % cum % | ios % cum % read | write disk I/Os in flight ios % cum % | ios % cum % read | write I/O time (1/1000s) ios % cum % | ios % cum % read | write disk I/O size ios % cum % | ios % cum % read | write block maps msec maps % cum % | maps % cum % osd-ldiskfs.lustre-OST0000.brw_stats= snapshot_time: 1777031126.225353142 secs.nsecs start_time: 1777023990.372175321 secs.nsecs elapsed_time: 7135.853177821 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 5 0 0 | 90 11 11 2: 0 0 0 | 26 3 14 4: 2 0 0 | 38 4 19 8: 4 0 0 | 40 4 24 16: 3419 98 98 | 52 6 30 32: 0 0 98 | 26 3 33 64: 0 0 98 | 2 0 34 128: 0 0 98 | 51 6 40 256: 10 0 99 | 265 33 73 512: 2 0 99 | 17 2 75 1K: 3 0 99 | 176 21 97 2K: 10 0 99 | 10 1 98 4K: 10 0 100 | 10 1 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 0 0 0 | 1 0 0 1: 3465 100 100 | 799 99 99 2: 0 0 100 | 3 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 0 0 0 | 1 0 0 1: 3465 100 100 | 778 96 97 2: 0 0 100 | 24 2 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 3434 99 99 | 380 64 64 2: 2 0 99 | 188 31 96 3: 0 0 99 | 1 0 96 4: 10 0 99 | 10 1 98 5: 0 0 99 | 0 0 98 6: 0 0 99 | 0 0 98 7: 0 0 99 | 0 0 98 8: 10 0 100 | 10 1 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 3456 97 97 | 581 66 66 2: 22 0 97 | 209 23 89 3: 20 0 98 | 25 2 92 4: 20 0 98 | 24 2 95 5: 10 0 99 | 10 1 96 6: 10 0 99 | 10 1 97 7: 10 0 99 | 10 1 98 8: 10 0 100 | 10 1 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 3436 99 99 | 272 46 46 2: 15 0 99 | 98 16 62 4: 4 0 99 | 135 22 85 8: 1 0 100 | 56 9 95 16: 0 0 100 | 25 4 99 32: 0 0 100 | 3 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 0 0 0 | 15 1 1 8K: 0 0 0 | 0 0 1 16K: 1 0 0 | 0 0 1 32K: 3 0 0 | 0 0 1 64K: 3419 96 96 | 24 2 4 128K: 0 0 96 | 28 3 7 256K: 0 0 96 | 2 0 7 512K: 0 0 96 | 52 5 13 1M: 10 0 96 | 278 31 45 2M: 125 3 100 | 480 54 100 read | write block maps msec maps % cum % | maps % cum % 1: 3631 100 100 | 576 99 99 2: 0 0 100 | 0 0 99 4: 0 0 100 | 0 0 99 8: 0 0 100 | 1 0 100 osd-ldiskfs.lustre-OST0001.brw_stats= snapshot_time: 1777031126.225486092 secs.nsecs start_time: 1777023561.338106121 secs.nsecs elapsed_time: 7564.887379971 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 1 0 0 | 82 9 9 2: 16 1 1 | 29 3 12 4: 18 1 3 | 50 5 18 8: 16 1 5 | 54 6 24 16: 805 87 92 | 28 3 27 32: 6 0 93 | 30 3 31 64: 6 0 93 | 1 0 31 128: 9 0 94 | 53 6 37 256: 37 4 98 | 259 29 67 512: 5 0 99 | 13 1 68 1K: 6 0 100 | 271 31 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 1: 925 100 100 | 869 99 99 2: 0 0 100 | 1 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 1: 923 99 99 | 839 96 96 2: 0 0 99 | 31 3 100 3: 0 0 99 | 0 0 100 4: 0 0 99 | 0 0 100 5: 0 0 99 | 0 0 100 6: 0 0 99 | 0 0 100 7: 0 0 99 | 0 0 100 8: 0 0 99 | 0 0 100 9: 0 0 99 | 0 0 100 10: 0 0 99 | 0 0 100 11: 0 0 99 | 0 0 100 12: 0 0 99 | 0 0 100 13: 0 0 99 | 0 0 100 14: 0 0 99 | 0 0 100 15: 0 0 99 | 0 0 100 16: 0 0 99 | 0 0 100 17: 0 0 99 | 0 0 100 18: 0 0 99 | 0 0 100 19: 0 0 99 | 0 0 100 20: 0 0 99 | 0 0 100 21: 0 0 99 | 0 0 100 22: 0 0 99 | 0 0 100 23: 0 0 99 | 0 0 100 24: 0 0 99 | 0 0 100 25: 0 0 99 | 0 0 100 26: 0 0 99 | 0 0 100 27: 0 0 99 | 0 0 100 28: 0 0 99 | 0 0 100 29: 0 0 99 | 0 0 100 30: 0 0 99 | 0 0 100 31: 2 0 100 | 0 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 912 99 99 | 347 54 54 2: 5 0 100 | 291 45 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 914 99 99 | 591 63 63 2: 5 0 99 | 255 27 91 3: 3 0 100 | 44 4 95 4: 0 0 100 | 39 4 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 917 100 100 | 239 37 37 2: 0 0 100 | 140 21 59 4: 0 0 100 | 181 28 87 8: 0 0 100 | 70 10 98 16: 0 0 100 | 8 1 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 0 0 0 | 20 2 2 8K: 16 1 1 | 0 0 2 16K: 16 1 3 | 0 0 2 32K: 16 1 5 | 0 0 2 64K: 805 87 92 | 2 0 2 128K: 6 0 93 | 32 3 5 256K: 6 0 93 | 1 0 5 512K: 9 0 94 | 54 5 11 1M: 38 4 98 | 272 29 41 2M: 10 1 100 | 548 58 100 read | write block maps msec maps % cum % | maps % cum % 1: 2029 100 100 | 480 100 100 osd-ldiskfs.lustre-MDT0000.io_latency_stats= io_latency_by_size: snapshot_time: 1777031126.227681293 start_time: 1777023524.753638186 elapsed_time: 7601.474043107 wr_128K: { 256us: 8, 512us: 2, } osd-ldiskfs.lustre-MDT0001.io_latency_stats= io_latency_by_size: snapshot_time: 1777031126.227888881 start_time: 1777023537.411013620 elapsed_time: 7588.816875261 osd-ldiskfs.lustre-OST0000.io_latency_stats= io_latency_by_size: snapshot_time: 1777031126.228204736 start_time: 1777031117.836734563 elapsed_time: 8.391470173 rd_64K: { 128us: 1, 256us: 3, 512us: 4, 1024us: 1, 2048us: 1, } rd_1024K: { 512us: 5, 1024us: 4, 4096us: 1, } wr_64K: { 256us: 5, 512us: 3, 1024us: 1, 2048us: 1, } wr_1024K: { 1024us: 7, 2048us: 2, 4096us: 1, } osd-ldiskfs.lustre-OST0001.io_latency_stats= io_latency_by_size: snapshot_time: 1777031126.228399704 start_time: 1777023561.339913607 elapsed_time: 7564.888486097 rd_8K: { 128us: 8, 256us: 8, } rd_16K: { 128us: 9, 256us: 7, } rd_32K: { 128us: 12, 256us: 4, } rd_64K: { 128us: 479, 256us: 344, 512us: 1, } rd_128K: { 256us: 6, } rd_256K: { 256us: 6, } rd_512K: { 512us: 38, 1024us: 1, } rd_1024K: { 512us: 6, } rd_2048K: { 1024us: 4, } rd_4096K: { 1024us: 3, } wr_4K: { 128us: 1, 2048us: 1, } wr_64K: { 256us: 8, 512us: 9, 1024us: 7, 2048us: 1, } wr_128K: { 256us: 2, 512us: 4, 1024us: 1, 2048us: 1, } wr_256K: { 256us: 1, 512us: 43, 1024us: 6, 2048us: 1, } wr_512K: { 512us: 7, 1024us: 70, 2048us: 26, 4096us: 2, } wr_1024K: { 512us: 4, 1024us: 92, 2048us: 67, 4096us: 11, 8192us: 2, } wr_2048K: { 1024us: 2, 2048us: 1, 4096us: 1, 8192us: 1, } wr_4096K: { 1024us: 3, 2048us: 158, 4096us: 105, 8192us: 10, } osd-ldiskfs.lustre-OST0000.stats= snapshot_time 1777031126.733823585 secs.nsecs start_time 1777031118.349141404 secs.nsecs elapsed_time 8.384682181 secs.nsecs get_page 60 samples [usecs] 0 9 86 400 cache_access 30 samples [pages] 1 256 2730 cache_hit 10 samples [pages] 1 1 10 cache_miss 20 samples [pages] 16 256 2720 delay_writes 10 samples [pages] 1 1 10 page_writes 30 samples [pages] 1 256 2730 osd-ldiskfs.lustre-OST0001.stats= snapshot_time 1777031126.734009285 secs.nsecs start_time 1777031118.349214362 secs.nsecs elapsed_time 8.384794923 secs.nsecs sanity test_127f: @@@@@@ FAIL: No stats found for size 4K Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7402:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:17407:test_127_io_latency_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:17442:test_127f() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7778:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7841:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7641:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:17444:main() Dumping lctl log to /tmp/testlogs//sanity.test_127f.*.1777031127.log osd-ldiskfs.lustre-OST0000.read_cache_enable=1 osd-ldiskfs.lustre-OST0000.writethrough_cache_enable=1 rsync: chown "/tmp/testlogs/.sanity.test_127f.debug_log.oleg204-server.1777031127.log.ku7AEK" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_127f.dmesg.oleg204-server.1777031127.log.YBzv9U" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]