== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:24:25 (1786771465) Stopping clients: oleg443-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg443-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg443-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg443-server oleg443-server: oleg443-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg443-server: detected 4 online CPUs by sysfs oleg443-server: MODOPTS_LIBCFS= oleg443-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg443-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg443-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg443-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 719 (time 1786771666.11 total 10.01 last 71.85) - open/close 1413 (time 1786771676.13 total 20.03 last 69.26) - open/close 2166 (time 1786771686.13 total 30.03 last 75.27) - open/close 2856 (time 1786771696.14 total 40.03 last 68.97) - open/close 3469 (time 1786771706.16 total 50.06 last 61.13) - open/close 4025 (time 1786771716.19 total 60.09 last 55.43) total: 4500 open/close in 67.31 seconds: 66.86 ops/second - /unlink 1366 (time 1786771738.45 total 10.00 last 136.56) - /unlink 2581 (time 1786771748.46 total 20.00 last 121.48) - /unlink 3826 (time 1786771758.46 total 30.01 last 124.43) total: 4500 /unlink in 35.61 seconds: 126.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 672 (time 1786771779.20 total 10.06 last 66.79) - open/close 1261 (time 1786771789.21 total 20.08 last 58.81) - open/close 1869 (time 1786771799.23 total 30.10 last 60.68) - open/close 2286 (time 1786771809.26 total 40.13 last 41.57) - open/close 2791 (time 1786771819.27 total 50.14 last 50.44) - open/close 3241 (time 1786771829.29 total 60.15 last 44.95) - open/close 3670 (time 1786771839.29 total 70.15 last 42.90) - open/close 4234 (time 1786771849.30 total 80.17 last 56.29) total: 4500 open/close in 83.69 seconds: 53.77 ops/second - /unlink 1136 (time 1786771869.16 total 10.00 last 113.57) - /unlink 2295 (time 1786771879.16 total 20.00 last 115.90) - /unlink 3509 (time 1786771889.16 total 30.00 last 121.39) - /unlink 4467 (time 1786771899.16 total 40.01 last 95.78) total: 4500 /unlink in 40.36 seconds: 111.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 565 (time 1786771915.48 total 10.02 last 56.42) - open/close 1182 (time 1786771925.48 total 20.02 last 61.70) - open/close 1820 (time 1786771935.49 total 30.03 last 63.71) - open/close 2443 (time 1786771945.50 total 40.04 last 62.26) - open/close 3145 (time 1786771955.50 total 50.04 last 70.19) - open/close 3786 (time 1786771965.50 total 60.04 last 64.09) - open/close 4375 (time 1786771975.52 total 70.06 last 58.77) total: 4500 open/close in 72.14 seconds: 62.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1273 (time 1786771992.84 total 10.00 last 127.24) - /unlink 2472 (time 1786772002.84 total 20.01 last 119.81) - /unlink 3653 (time 1786772012.85 total 30.01 last 118.08) total: 4500 /unlink in 37.32 seconds: 120.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 634 (time 1786772035.88 total 10.01 last 63.37) - open/close 1127 (time 1786772045.89 total 20.02 last 49.22) - open/close 1577 (time 1786772055.90 total 30.03 last 44.96) - open/close 2041 (time 1786772065.91 total 40.04 last 46.35) - open/close 2495 (time 1786772075.92 total 50.05 last 45.37) - open/close 3049 (time 1786772085.93 total 60.06 last 55.37) - open/close 3635 (time 1786772095.93 total 70.06 last 58.56) - open/close 4360 (time 1786772105.95 total 80.08 last 72.37) total: 4500 open/close in 82.19 seconds: 54.75 ops/second - /unlink 1163 (time 1786772122.26 total 10.01 last 116.22) - /unlink 2379 (time 1786772132.26 total 20.01 last 121.59) - /unlink 3615 (time 1786772142.26 total 30.01 last 123.59) total: 4500 /unlink in 36.55 seconds: 123.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 688 (time 1786772162.54 total 10.01 last 68.76) - open/close 1303 (time 1786772172.57 total 20.03 last 61.32) - open/close 1878 (time 1786772182.58 total 30.04 last 57.48) - open/close 2500 (time 1786772192.60 total 40.06 last 62.08) - open/close 3132 (time 1786772202.60 total 50.06 last 63.17) - open/close 3792 (time 1786772212.62 total 60.08 last 65.90) - open/close 4443 (time 1786772222.64 total 70.10 last 64.96) total: 4500 open/close in 70.93 seconds: 63.44 ops/second - /unlink 1275 (time 1786772238.12 total 10.00 last 127.47) - /unlink 2403 (time 1786772248.12 total 20.01 last 112.75) - /unlink 3592 (time 1786772258.13 total 30.02 last 118.78) total: 4500 /unlink in 38.22 seconds: 117.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 714 (time 1786772281.12 total 10.00 last 71.37) - open/close 1390 (time 1786772291.14 total 20.03 last 67.45) - open/close 2071 (time 1786772301.15 total 30.04 last 68.03) - open/close 2698 (time 1786772311.15 total 40.04 last 62.69) - open/close 3426 (time 1786772321.16 total 50.04 last 72.75) - open/close 4062 (time 1786772331.18 total 60.06 last 63.49) total: 4500 open/close in 66.05 seconds: 68.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1150 (time 1786772352.74 total 10.02 last 114.80) - /unlink 2290 (time 1786772362.74 total 20.02 last 113.98) - /unlink 3567 (time 1786772372.75 total 30.02 last 127.66) total: 4500 /unlink in 37.47 seconds: 120.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 690 (time 1786772395.16 total 10.01 last 68.96) - open/close 1399 (time 1786772405.17 total 20.02 last 70.83) - open/close 2108 (time 1786772415.19 total 30.03 last 70.80) - open/close 2820 (time 1786772425.19 total 40.04 last 71.16) - open/close 3490 (time 1786772435.19 total 50.04 last 66.98) - open/close 4163 (time 1786772445.20 total 60.05 last 67.24) total: 4500 open/close in 65.10 seconds: 69.13 ops/second - /unlink 1240 (time 1786772465.03 total 10.01 last 123.84) - /unlink 2353 (time 1786772475.03 total 20.01 last 111.29) - /unlink 3596 (time 1786772485.03 total 30.01 last 124.29) total: 4500 /unlink in 37.27 seconds: 120.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 680 (time 1786772507.17 total 10.01 last 67.94) - open/close 1387 (time 1786772517.19 total 20.02 last 70.61) - open/close 2048 (time 1786772527.19 total 30.03 last 66.07) - open/close 2683 (time 1786772537.19 total 40.03 last 63.49) - open/close 3386 (time 1786772547.21 total 50.04 last 70.19) - open/close 4080 (time 1786772557.21 total 60.05 last 69.39) total: 4500 open/close in 66.16 seconds: 68.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1212 (time 1786772578.74 total 10.01 last 121.11) - /unlink 2443 (time 1786772588.75 total 20.01 last 123.03) - /unlink 3601 (time 1786772598.76 total 30.02 last 115.69) total: 4500 /unlink in 39.00 seconds: 115.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 683 (time 1786772622.68 total 10.01 last 68.22) - open/close 1306 (time 1786772632.69 total 20.02 last 62.25) - open/close 1944 (time 1786772642.71 total 30.03 last 63.72) - open/close 2601 (time 1786772652.72 total 40.05 last 65.60) - open/close 3246 (time 1786772662.73 total 50.06 last 64.43) - open/close 3935 (time 1786772672.74 total 60.06 last 68.86) - open/close 4442 (time 1786772682.74 total 70.07 last 50.66) total: 4500 open/close in 71.19 seconds: 63.21 ops/second - /unlink 1229 (time 1786772699.23 total 10.00 last 122.85) - /unlink 2465 (time 1786772709.24 total 20.01 last 123.50) - /unlink 3624 (time 1786772719.24 total 30.02 last 115.81) total: 4500 /unlink in 36.90 seconds: 121.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 736 (time 1786772741.25 total 10.00 last 73.59) - open/close 1389 (time 1786772751.26 total 20.01 last 65.25) - open/close 2019 (time 1786772761.28 total 30.02 last 62.91) - open/close 2661 (time 1786772771.28 total 40.03 last 64.19) - open/close 3276 (time 1786772781.29 total 50.04 last 61.43) - open/close 3938 (time 1786772791.29 total 60.04 last 66.17) total: 4500 open/close in 67.83 seconds: 66.34 ops/second - /unlink 1251 (time 1786772814.13 total 10.00 last 125.08) - /unlink 2438 (time 1786772824.15 total 20.02 last 118.49) - /unlink 3683 (time 1786772834.15 total 30.02 last 124.48) total: 4500 /unlink in 37.54 seconds: 119.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 716 (time 1786772856.96 total 10.01 last 71.51) - open/close 1356 (time 1786772866.97 total 20.03 last 63.91) - open/close 2036 (time 1786772876.99 total 30.05 last 67.86) - open/close 2713 (time 1786772887.00 total 40.05 last 67.66) - open/close 3372 (time 1786772897.01 total 50.06 last 65.82) - open/close 4029 (time 1786772907.01 total 60.07 last 65.69) total: 4500 open/close in 67.25 seconds: 66.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1226 (time 1786772929.58 total 10.01 last 122.50) - /unlink 2518 (time 1786772939.58 total 20.01 last 129.19) - /unlink 3843 (time 1786772949.58 total 30.01 last 132.45) total: 4500 /unlink in 34.57 seconds: 130.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 684 (time 1786772969.13 total 10.02 last 68.29) - open/close 1403 (time 1786772979.13 total 20.02 last 71.87) - open/close 2111 (time 1786772989.14 total 30.02 last 70.78) - open/close 2771 (time 1786772999.14 total 40.03 last 65.96) - open/close 3423 (time 1786773009.14 total 50.03 last 65.20) - open/close 4099 (time 1786773019.16 total 60.04 last 67.50) total: 4500 open/close in 66.13 seconds: 68.05 ops/second - /unlink 1122 (time 1786773039.25 total 10.00 last 112.17) - /unlink 2160 (time 1786773049.26 total 20.01 last 103.75) - /unlink 3298 (time 1786773059.26 total 30.01 last 113.72) - /unlink 4289 (time 1786773069.26 total 40.02 last 99.10) total: 4500 /unlink in 41.70 seconds: 107.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 551 (time 1786773086.99 total 10.01 last 55.05) - open/close 1166 (time 1786773096.99 total 20.02 last 61.45) - open/close 1765 (time 1786773107.02 total 30.04 last 59.76) - open/close 2446 (time 1786773117.02 total 40.04 last 68.08) - open/close 3158 (time 1786773127.02 total 50.04 last 71.20) - open/close 3860 (time 1786773137.02 total 60.04 last 70.20) total: 4500 open/close in 69.10 seconds: 65.13 ops/second - /unlink 1240 (time 1786773159.47 total 10.02 last 123.81) - /unlink 2490 (time 1786773169.47 total 20.02 last 124.97) - /unlink 3862 (time 1786773179.47 total 30.02 last 137.18) total: 4500 /unlink in 34.95 seconds: 128.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 717 (time 1786773198.24 total 10.00 last 71.68) - open/close 1379 (time 1786773208.25 total 20.01 last 66.14) - open/close 2068 (time 1786773218.25 total 30.02 last 68.88) - open/close 2765 (time 1786773228.29 total 40.05 last 69.44) - open/close 3542 (time 1786773238.30 total 50.07 last 77.59) - open/close 4300 (time 1786773248.30 total 60.07 last 75.79) total: 4500 open/close in 62.85 seconds: 71.60 ops/second - /unlink 1317 (time 1786773265.34 total 10.02 last 131.48) - /unlink 2577 (time 1786773275.34 total 20.02 last 125.99) - /unlink 3881 (time 1786773285.34 total 30.02 last 130.34) total: 4500 /unlink in 34.92 seconds: 128.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 693 (time 1786773304.47 total 10.01 last 69.23) - open/close 1379 (time 1786773314.48 total 20.02 last 68.55) - open/close 2085 (time 1786773324.49 total 30.02 last 70.56) - open/close 2788 (time 1786773334.49 total 40.03 last 70.28) - open/close 3454 (time 1786773344.50 total 50.03 last 66.55) - open/close 4185 (time 1786773354.51 total 60.04 last 73.05) total: 4500 open/close in 64.47 seconds: 69.80 ops/second - /unlink 1297 (time 1786773373.22 total 10.00 last 129.67) - /unlink 2654 (time 1786773383.23 total 20.01 last 135.64) - /unlink 4039 (time 1786773393.23 total 30.01 last 138.46) total: 4500 /unlink in 33.47 seconds: 134.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 709 (time 1786773411.32 total 10.01 last 70.85) - open/close 1405 (time 1786773421.33 total 20.02 last 69.53) - open/close 2061 (time 1786773431.33 total 30.02 last 65.60) - open/close 2821 (time 1786773441.34 total 40.03 last 75.92) - open/close 3558 (time 1786773451.34 total 50.03 last 73.67) - open/close 4302 (time 1786773461.35 total 60.04 last 74.34) total: 4500 open/close in 62.94 seconds: 71.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1346 (time 1786773478.64 total 10.00 last 134.55) - /unlink 2669 (time 1786773488.64 total 20.00 last 132.29) - /unlink 3995 (time 1786773498.65 total 30.01 last 132.47) total: 4500 /unlink in 34.10 seconds: 131.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 530 (time 1786773517.70 total 10.01 last 52.94) - open/close 1292 (time 1786773527.72 total 20.03 last 76.05) - open/close 1980 (time 1786773537.74 total 30.05 last 68.70) - open/close 2691 (time 1786773547.75 total 40.05 last 71.05) - open/close 3424 (time 1786773557.75 total 50.06 last 73.26) - open/close 4133 (time 1786773567.77 total 60.08 last 70.77) total: 4500 open/close in 65.73 seconds: 68.46 ops/second - /unlink 973 (time 1786773589.10 total 10.00 last 97.28) - /unlink 2156 (time 1786773599.10 total 20.01 last 118.25) - /unlink 3524 (time 1786773609.10 total 30.01 last 136.78) total: 4500 /unlink in 38.01 seconds: 118.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 633 (time 1786773632.52 total 10.01 last 63.24) - open/close 1185 (time 1786773642.54 total 20.03 last 55.09) - open/close 1718 (time 1786773652.55 total 30.04 last 53.26) - open/close 2331 (time 1786773662.56 total 40.05 last 61.24) - open/close 2971 (time 1786773672.56 total 50.05 last 63.98) - open/close 3582 (time 1786773682.57 total 60.06 last 61.06) - open/close 4251 (time 1786773692.57 total 70.06 last 66.87) total: 4500 open/close in 74.06 seconds: 60.76 ops/second - /unlink 1300 (time 1786773712.58 total 10.01 last 129.91) - /unlink 2445 (time 1786773722.58 total 20.01 last 114.49) - /unlink 3713 (time 1786773732.58 total 30.01 last 126.79) total: 4500 /unlink in 36.42 seconds: 123.56 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95304 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg443-server unloading modules via unload_modules_local on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local oleg443-server: modules unloaded.