== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:21:19 (1787602879) Stopping clients: oleg603-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg603-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg603-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg603-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg603-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg603-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg603-server oleg603-server: oleg603-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg603-server' oleg603-server: oleg603-server.virtnet: executing load_modules_local oleg603-server: Loading modules from /home/green/git/lustre-release/lustre oleg603-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg603-server: detected 4 online CPUs by sysfs oleg603-server: MODOPTS_LIBCFS= oleg603-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg603-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg603-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg603-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 602 (time 1787603073.86 total 10.01 last 60.11) - open/close 1312 (time 1787603083.86 total 20.02 last 71.00) - open/close 2004 (time 1787603093.89 total 30.04 last 69.04) - open/close 2723 (time 1787603103.90 total 40.05 last 71.78) - open/close 3522 (time 1787603113.91 total 50.06 last 79.83) - open/close 4305 (time 1787603123.93 total 60.08 last 78.17) total: 4500 open/close in 62.62 seconds: 71.86 ops/second - /unlink 1312 (time 1787603141.07 total 10.00 last 131.18) - /unlink 2646 (time 1787603151.07 total 20.01 last 133.34) - /unlink 4096 (time 1787603161.07 total 30.01 last 144.94) total: 4500 /unlink in 32.79 seconds: 137.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 701 (time 1787603178.81 total 10.02 last 69.96) - open/close 1402 (time 1787603188.82 total 20.03 last 70.04) - open/close 2133 (time 1787603198.83 total 30.04 last 72.99) - open/close 2740 (time 1787603208.84 total 40.05 last 60.68) - open/close 3484 (time 1787603218.85 total 50.06 last 74.33) - open/close 4227 (time 1787603228.86 total 60.07 last 74.19) total: 4500 open/close in 64.20 seconds: 70.10 ops/second - /unlink 1520 (time 1787603247.68 total 10.01 last 151.87) - /unlink 2874 (time 1787603257.69 total 20.01 last 135.37) - /unlink 4200 (time 1787603267.69 total 30.01 last 132.59) total: 4500 /unlink in 32.47 seconds: 138.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 674 (time 1787603284.45 total 10.02 last 67.29) - open/close 1240 (time 1787603294.46 total 20.03 last 56.54) - open/close 1900 (time 1787603304.48 total 30.05 last 65.86) - open/close 2554 (time 1787603314.48 total 40.05 last 65.39) - open/close 3276 (time 1787603324.50 total 50.07 last 72.05) - open/close 4067 (time 1787603334.51 total 60.08 last 79.03) total: 4500 open/close in 66.25 seconds: 67.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1337 (time 1787603355.44 total 10.00 last 133.66) - /unlink 2575 (time 1787603365.44 total 20.00 last 123.78) - /unlink 3788 (time 1787603375.45 total 30.01 last 121.26) total: 4500 /unlink in 35.03 seconds: 128.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 692 (time 1787603395.88 total 10.03 last 68.96) - open/close 1432 (time 1787603405.89 total 20.05 last 73.89) - open/close 2057 (time 1787603415.89 total 30.05 last 62.48) - open/close 2757 (time 1787603425.91 total 40.07 last 69.86) - open/close 3427 (time 1787603435.92 total 50.08 last 66.97) - open/close 4168 (time 1787603445.93 total 60.09 last 73.99) total: 4500 open/close in 64.83 seconds: 69.41 ops/second - /unlink 1252 (time 1787603465.56 total 10.02 last 124.96) - /unlink 2460 (time 1787603475.59 total 20.04 last 120.51) - /unlink 3668 (time 1787603485.59 total 30.05 last 120.75) total: 4500 /unlink in 36.36 seconds: 123.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 611 (time 1787603506.66 total 10.01 last 61.06) - open/close 1292 (time 1787603516.66 total 20.01 last 68.09) - open/close 2006 (time 1787603526.68 total 30.02 last 71.29) - open/close 2719 (time 1787603536.69 total 40.03 last 71.25) - open/close 3436 (time 1787603546.70 total 50.04 last 71.62) - open/close 4169 (time 1787603556.70 total 60.04 last 73.29) total: 4500 open/close in 65.44 seconds: 68.77 ops/second - /unlink 1355 (time 1787603577.01 total 10.00 last 135.44) - /unlink 2717 (time 1787603587.01 total 20.00 last 136.19) - /unlink 4089 (time 1787603597.02 total 30.01 last 137.08) total: 4500 /unlink in 32.78 seconds: 137.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 775 (time 1787603613.87 total 10.01 last 77.46) - open/close 1533 (time 1787603623.89 total 20.02 last 75.71) - open/close 2232 (time 1787603633.90 total 30.03 last 69.82) - open/close 2963 (time 1787603643.90 total 40.03 last 73.06) - open/close 3673 (time 1787603653.91 total 50.04 last 70.93) - open/close 4387 (time 1787603663.93 total 60.06 last 71.25) total: 4500 open/close in 61.70 seconds: 72.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1347 (time 1787603680.50 total 10.01 last 134.54) - /unlink 2660 (time 1787603690.51 total 20.02 last 131.16) - /unlink 4146 (time 1787603700.52 total 30.03 last 148.56) total: 4500 /unlink in 32.22 seconds: 139.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 620 (time 1787603717.34 total 10.01 last 61.91) - open/close 1229 (time 1787603727.35 total 20.02 last 60.87) - open/close 1875 (time 1787603737.35 total 30.02 last 64.59) - open/close 2506 (time 1787603747.36 total 40.03 last 63.04) - open/close 3184 (time 1787603757.37 total 50.04 last 67.76) - open/close 3891 (time 1787603767.38 total 60.05 last 70.61) total: 4500 open/close in 68.14 seconds: 66.04 ops/second - /unlink 1343 (time 1787603790.22 total 10.00 last 134.24) - /unlink 2814 (time 1787603800.23 total 20.01 last 146.98) - /unlink 4406 (time 1787603810.23 total 30.01 last 159.17) total: 4500 /unlink in 30.69 seconds: 146.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 812 (time 1787603825.50 total 10.00 last 81.16) - open/close 1586 (time 1787603835.51 total 20.01 last 77.36) - open/close 2317 (time 1787603845.51 total 30.01 last 73.09) - open/close 3140 (time 1787603855.52 total 40.02 last 82.20) - open/close 3893 (time 1787603865.52 total 50.03 last 75.27) total: 4500 open/close in 57.38 seconds: 78.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1406 (time 1787603886.62 total 10.00 last 140.59) - /unlink 2549 (time 1787603896.62 total 20.00 last 114.25) - /unlink 3960 (time 1787603906.62 total 30.01 last 141.09) total: 4500 /unlink in 33.66 seconds: 133.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 795 (time 1787603925.37 total 10.01 last 79.44) - open/close 1512 (time 1787603935.38 total 20.01 last 71.68) - open/close 2184 (time 1787603945.40 total 30.03 last 67.04) - open/close 2738 (time 1787603955.41 total 40.04 last 55.35) - open/close 3407 (time 1787603965.41 total 50.05 last 66.86) - open/close 4051 (time 1787603975.43 total 60.07 last 64.28) total: 4500 open/close in 67.50 seconds: 66.67 ops/second - /unlink 1405 (time 1787603998.76 total 10.00 last 140.45) - /unlink 2872 (time 1787604008.76 total 20.01 last 146.64) - /unlink 4299 (time 1787604018.76 total 30.01 last 142.65) total: 4500 /unlink in 31.57 seconds: 142.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 722 (time 1787604035.39 total 10.01 last 72.11) - open/close 1458 (time 1787604045.40 total 20.02 last 73.53) - open/close 2243 (time 1787604055.41 total 30.03 last 78.41) - open/close 3058 (time 1787604065.42 total 40.04 last 81.44) - open/close 3891 (time 1787604075.43 total 50.05 last 83.21) total: 4500 open/close in 57.76 seconds: 77.90 ops/second - /unlink 1554 (time 1787604097.66 total 10.00 last 155.35) - /unlink 2976 (time 1787604107.67 total 20.01 last 142.13) total: 4500 /unlink in 30.01 seconds: 149.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 749 (time 1787604132.18 total 10.00 last 74.90) - open/close 1489 (time 1787604142.18 total 20.01 last 73.96) - open/close 2233 (time 1787604152.20 total 30.02 last 74.26) - open/close 2989 (time 1787604162.20 total 40.03 last 75.60) - open/close 3794 (time 1787604172.21 total 50.03 last 80.45) total: 4500 open/close in 58.80 seconds: 76.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1534 (time 1787604195.23 total 10.00 last 153.39) - /unlink 2860 (time 1787604205.24 total 20.01 last 132.43) - /unlink 4224 (time 1787604215.24 total 30.01 last 136.38) total: 4500 /unlink in 31.75 seconds: 141.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 739 (time 1787604232.09 total 10.02 last 73.78) - open/close 1456 (time 1787604242.10 total 20.03 last 71.62) - open/close 2183 (time 1787604252.11 total 30.04 last 72.63) - open/close 2813 (time 1787604262.11 total 40.04 last 62.98) - open/close 3603 (time 1787604272.13 total 50.06 last 78.88) - open/close 4328 (time 1787604282.14 total 60.07 last 72.41) total: 4500 open/close in 61.89 seconds: 72.71 ops/second - /unlink 1549 (time 1787604298.73 total 10.00 last 154.86) - /unlink 3013 (time 1787604308.73 total 20.01 last 146.34) - /unlink 4464 (time 1787604318.74 total 30.01 last 145.06) total: 4500 /unlink in 30.32 seconds: 148.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 752 (time 1787604333.56 total 10.02 last 75.04) - open/close 1529 (time 1787604343.56 total 20.02 last 77.68) - open/close 2394 (time 1787604353.56 total 30.03 last 86.44) - open/close 3217 (time 1787604363.57 total 40.04 last 82.24) - open/close 3959 (time 1787604373.58 total 50.05 last 74.11) total: 4500 open/close in 56.76 seconds: 79.29 ops/second - /unlink 1606 (time 1787604394.18 total 10.01 last 160.45) - /unlink 3182 (time 1787604404.18 total 20.01 last 157.59) total: 4500 /unlink in 28.57 seconds: 157.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 845 (time 1787604427.80 total 10.01 last 84.40) - open/close 1696 (time 1787604437.80 total 20.01 last 85.08) - open/close 2508 (time 1787604447.81 total 30.02 last 81.16) - open/close 3310 (time 1787604457.81 total 40.02 last 80.15) - open/close 4135 (time 1787604467.84 total 50.05 last 82.27) total: 4500 open/close in 54.25 seconds: 82.95 ops/second - /unlink 1657 (time 1787604486.27 total 10.00 last 165.62) - /unlink 3238 (time 1787604496.27 total 20.01 last 158.04) total: 4500 /unlink in 28.75 seconds: 156.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 769 (time 1787604520.37 total 10.02 last 76.73) - open/close 1622 (time 1787604530.38 total 20.03 last 85.24) - open/close 2344 (time 1787604540.38 total 30.03 last 72.19) - open/close 3226 (time 1787604550.39 total 40.05 last 88.06) - open/close 4022 (time 1787604560.41 total 50.06 last 79.48) total: 4500 open/close in 55.39 seconds: 81.24 ops/second - /unlink 1420 (time 1787604580.21 total 10.01 last 141.87) - /unlink 2928 (time 1787604590.22 total 20.02 last 150.61) - /unlink 4422 (time 1787604600.23 total 30.03 last 149.30) total: 4500 /unlink in 30.62 seconds: 146.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 858 (time 1787604614.88 total 10.01 last 85.69) - open/close 1679 (time 1787604624.89 total 20.02 last 82.05) - open/close 2429 (time 1787604634.92 total 30.05 last 74.73) - open/close 3072 (time 1787604644.94 total 40.07 last 64.18) - open/close 3692 (time 1787604654.97 total 50.10 last 61.83) - open/close 4454 (time 1787604664.97 total 60.10 last 76.19) total: 4500 open/close in 60.51 seconds: 74.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1521 (time 1787604679.81 total 10.00 last 152.04) - /unlink 3002 (time 1787604689.82 total 20.01 last 148.01) - /unlink 4191 (time 1787604699.82 total 30.01 last 118.88) total: 4500 /unlink in 32.13 seconds: 140.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 757 (time 1787604718.23 total 10.01 last 75.65) - open/close 1558 (time 1787604728.23 total 20.01 last 80.05) - open/close 2411 (time 1787604738.24 total 30.02 last 85.27) - open/close 3161 (time 1787604748.25 total 40.03 last 74.88) - open/close 3980 (time 1787604758.27 total 50.05 last 81.80) total: 4500 open/close in 56.24 seconds: 80.01 ops/second - /unlink 1526 (time 1787604778.63 total 10.00 last 152.56) - /unlink 3340 (time 1787604788.63 total 20.01 last 181.35) total: 4500 /unlink in 27.43 seconds: 164.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 886 (time 1787604809.73 total 10.01 last 88.56) - open/close 1759 (time 1787604819.74 total 20.01 last 87.27) - open/close 2641 (time 1787604829.74 total 30.01 last 88.18) - open/close 3571 (time 1787604839.74 total 40.01 last 92.98) - open/close 4358 (time 1787604849.74 total 50.01 last 78.70) total: 4500 open/close in 51.72 seconds: 87.00 ops/second - /unlink 1400 (time 1787604866.15 total 10.01 last 139.81) - /unlink 3049 (time 1787604876.16 total 20.02 last 164.86) total: 4500 /unlink in 29.02 seconds: 155.09 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95230 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg603-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg603-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg603-server stop mds service on oleg603-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg603-server stop mds service on oleg603-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg603-server unloading modules via unload_modules_local on: 'oleg603-server' oleg603-server: oleg603-server.virtnet: executing unload_modules_local oleg603-server: modules unloaded.