== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:32:13 (1787578333) Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg216-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg216-server oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: detected 4 online CPUs by sysfs oleg216-server: MODOPTS_LIBCFS= oleg216-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg216-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg216-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg216-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 566 (time 1787578575.49 total 10.00 last 56.60) - open/close 1127 (time 1787578585.50 total 20.02 last 56.02) - open/close 1681 (time 1787578595.51 total 30.02 last 55.36) - open/close 2164 (time 1787578605.51 total 40.02 last 48.29) - open/close 2732 (time 1787578615.51 total 50.03 last 56.77) - open/close 3363 (time 1787578625.52 total 60.04 last 63.05) - open/close 4024 (time 1787578635.53 total 70.05 last 66.05) total: 4500 open/close in 76.93 seconds: 58.49 ops/second - /unlink 1313 (time 1787578656.99 total 10.00 last 131.26) - /unlink 2459 (time 1787578667.00 total 20.01 last 114.51) - /unlink 3585 (time 1787578677.00 total 30.01 last 112.55) total: 4500 /unlink in 37.70 seconds: 119.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 665 (time 1787578699.34 total 10.00 last 66.48) - open/close 1310 (time 1787578709.35 total 20.01 last 64.45) - open/close 1855 (time 1787578719.36 total 30.02 last 54.48) - open/close 2442 (time 1787578729.41 total 40.06 last 58.42) - open/close 2831 (time 1787578739.41 total 50.07 last 38.88) - open/close 3268 (time 1787578749.43 total 60.09 last 43.61) - open/close 4009 (time 1787578759.46 total 70.12 last 73.91) total: 4500 open/close in 75.89 seconds: 59.30 ops/second - /unlink 1249 (time 1787578779.88 total 10.00 last 124.88) - /unlink 2550 (time 1787578789.89 total 20.01 last 130.03) - /unlink 3900 (time 1787578799.90 total 30.02 last 134.89) total: 4500 /unlink in 34.84 seconds: 129.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 702 (time 1787578819.94 total 10.01 last 70.16) - open/close 1292 (time 1787578829.97 total 20.04 last 58.82) - open/close 1720 (time 1787578840.01 total 30.08 last 42.62) - open/close 2384 (time 1787578850.02 total 40.08 last 66.39) - open/close 3011 (time 1787578860.02 total 50.08 last 62.69) - open/close 3625 (time 1787578870.02 total 60.08 last 61.39) - open/close 4306 (time 1787578880.04 total 70.10 last 67.96) total: 4500 open/close in 73.28 seconds: 61.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1376 (time 1787578899.42 total 10.00 last 137.55) - /unlink 3181 (time 1787578909.42 total 20.01 last 180.44) total: 4500 /unlink in 28.95 seconds: 155.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 686 (time 1787578933.76 total 10.01 last 68.53) - open/close 1372 (time 1787578943.77 total 20.02 last 68.52) - open/close 1992 (time 1787578953.79 total 30.03 last 61.93) - open/close 2591 (time 1787578963.81 total 40.06 last 59.76) - open/close 3153 (time 1787578973.82 total 50.06 last 56.16) - open/close 3760 (time 1787578983.82 total 60.07 last 60.68) - open/close 4382 (time 1787578993.85 total 70.09 last 62.03) total: 4500 open/close in 71.93 seconds: 62.56 ops/second - /unlink 981 (time 1787579011.34 total 10.00 last 98.09) - /unlink 2184 (time 1787579021.34 total 20.00 last 120.27) - /unlink 3596 (time 1787579031.34 total 30.01 last 141.17) - /unlink 4437 (time 1787579041.36 total 40.02 last 84.00) total: 4500 /unlink in 40.74 seconds: 110.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 694 (time 1787579057.95 total 10.00 last 69.38) - open/close 1335 (time 1787579067.96 total 20.01 last 64.05) - open/close 2061 (time 1787579077.97 total 30.01 last 72.58) - open/close 2838 (time 1787579087.97 total 40.02 last 77.65) - open/close 3441 (time 1787579097.98 total 50.03 last 60.23) - open/close 4030 (time 1787579108.03 total 60.08 last 58.63) total: 4500 open/close in 68.32 seconds: 65.87 ops/second - /unlink 1173 (time 1787579131.79 total 10.00 last 117.29) - /unlink 2407 (time 1787579141.80 total 20.01 last 123.30) - /unlink 3329 (time 1787579151.80 total 30.01 last 92.18) - /unlink 4204 (time 1787579161.82 total 40.02 last 87.39) total: 4500 /unlink in 43.46 seconds: 103.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 758 (time 1787579181.99 total 10.01 last 75.75) - open/close 1550 (time 1787579192.00 total 20.02 last 79.12) - open/close 2331 (time 1787579202.03 total 30.05 last 77.84) - open/close 2918 (time 1787579212.06 total 40.08 last 58.53) - open/close 3518 (time 1787579222.08 total 50.10 last 59.90) - open/close 4100 (time 1787579232.08 total 60.10 last 58.18) total: 4500 open/close in 65.30 seconds: 68.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1506 (time 1787579251.98 total 10.00 last 150.55) - /unlink 2954 (time 1787579261.98 total 20.01 last 144.76) - /unlink 4220 (time 1787579271.98 total 30.01 last 126.58) total: 4500 /unlink in 32.81 seconds: 137.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 543 (time 1787579292.16 total 10.01 last 54.24) - open/close 1159 (time 1787579302.16 total 20.01 last 61.59) - open/close 1623 (time 1787579312.17 total 30.02 last 46.34) - open/close 2082 (time 1787579322.18 total 40.03 last 45.88) - open/close 2788 (time 1787579332.18 total 50.03 last 70.57) - open/close 3637 (time 1787579342.19 total 60.05 last 84.79) - open/close 4064 (time 1787579352.22 total 70.07 last 42.60) total: 4500 open/close in 75.42 seconds: 59.66 ops/second - /unlink 1514 (time 1787579372.56 total 10.01 last 151.26) - /unlink 2919 (time 1787579382.56 total 20.01 last 140.47) - /unlink 4401 (time 1787579392.56 total 30.01 last 148.18) total: 4500 /unlink in 30.76 seconds: 146.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 784 (time 1787579409.19 total 10.00 last 78.38) - open/close 1409 (time 1787579419.20 total 20.01 last 62.44) - open/close 1957 (time 1787579429.23 total 30.04 last 54.66) - open/close 2470 (time 1787579439.23 total 40.04 last 51.27) - open/close 2974 (time 1787579449.24 total 50.05 last 50.36) - open/close 3470 (time 1787579459.24 total 60.05 last 49.60) - open/close 3955 (time 1787579469.25 total 70.06 last 48.47) total: 4500 open/close in 78.99 seconds: 56.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1290 (time 1787579493.28 total 10.00 last 129.00) - /unlink 2714 (time 1787579503.28 total 20.00 last 142.36) - /unlink 4001 (time 1787579513.28 total 30.01 last 128.63) total: 4500 /unlink in 35.15 seconds: 128.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 610 (time 1787579534.37 total 10.00 last 60.97) - open/close 1303 (time 1787579544.38 total 20.01 last 69.26) - open/close 2031 (time 1787579554.41 total 30.04 last 72.61) - open/close 2684 (time 1787579564.41 total 40.05 last 65.24) - open/close 3314 (time 1787579574.42 total 50.05 last 62.97) - open/close 4013 (time 1787579584.45 total 60.08 last 69.70) total: 4500 open/close in 67.53 seconds: 66.63 ops/second - /unlink 1373 (time 1787579606.83 total 10.01 last 137.22) - /unlink 2786 (time 1787579616.83 total 20.01 last 141.29) - /unlink 4095 (time 1787579626.84 total 30.01 last 130.88) total: 4500 /unlink in 33.63 seconds: 133.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 633 (time 1787579646.52 total 10.01 last 63.25) - open/close 1318 (time 1787579656.54 total 20.03 last 68.32) - open/close 1915 (time 1787579666.56 total 30.05 last 59.62) - open/close 2407 (time 1787579676.58 total 40.07 last 49.07) - open/close 2954 (time 1787579686.60 total 50.09 last 54.61) - open/close 3467 (time 1787579696.61 total 60.10 last 51.24) - open/close 3931 (time 1787579706.62 total 70.11 last 46.35) - open/close 4303 (time 1787579716.64 total 80.13 last 37.15) total: 4500 open/close in 83.16 seconds: 54.11 ops/second - /unlink 1369 (time 1787579735.28 total 10.00 last 136.85) - /unlink 2730 (time 1787579745.29 total 20.01 last 136.05) - /unlink 3827 (time 1787579755.29 total 30.01 last 109.66) total: 4500 /unlink in 36.34 seconds: 123.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 656 (time 1787579777.69 total 10.00 last 65.58) - open/close 1276 (time 1787579787.70 total 20.01 last 61.97) - open/close 2038 (time 1787579797.70 total 30.01 last 76.16) - open/close 2806 (time 1787579807.70 total 40.01 last 76.79) - open/close 3626 (time 1787579817.71 total 50.02 last 81.97) - open/close 4291 (time 1787579827.72 total 60.03 last 66.42) total: 4500 open/close in 63.47 seconds: 70.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1217 (time 1787579847.83 total 10.00 last 121.68) - /unlink 2423 (time 1787579857.83 total 20.00 last 120.57) - /unlink 3363 (time 1787579867.83 total 30.01 last 93.96) total: 4500 /unlink in 40.00 seconds: 112.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 438 (time 1787579895.34 total 10.01 last 43.78) - open/close 1144 (time 1787579905.36 total 20.02 last 70.49) - open/close 1936 (time 1787579915.37 total 30.03 last 79.13) - open/close 2709 (time 1787579925.38 total 40.04 last 77.22) - open/close 3364 (time 1787579935.39 total 50.05 last 65.43) - open/close 4086 (time 1787579945.39 total 60.05 last 72.20) total: 4500 open/close in 65.54 seconds: 68.66 ops/second - /unlink 1462 (time 1787579965.67 total 10.01 last 146.11) - /unlink 3115 (time 1787579975.67 total 20.01 last 165.24) - /unlink 4373 (time 1787579985.67 total 30.01 last 125.77) total: 4500 /unlink in 31.14 seconds: 144.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 460 (time 1787580003.81 total 10.02 last 45.89) - open/close 1104 (time 1787580013.81 total 20.03 last 64.37) - open/close 1873 (time 1787580023.82 total 30.04 last 76.84) - open/close 2622 (time 1787580033.83 total 40.05 last 74.81) - open/close 3433 (time 1787580043.85 total 50.06 last 80.99) - open/close 4308 (time 1787580053.85 total 60.06 last 87.50) total: 4500 open/close in 62.04 seconds: 72.53 ops/second - /unlink 1507 (time 1787580070.02 total 10.00 last 150.65) - /unlink 2997 (time 1787580080.03 total 20.00 last 148.99) - /unlink 4136 (time 1787580090.03 total 30.01 last 113.89) total: 4500 /unlink in 32.65 seconds: 137.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 619 (time 1787580109.75 total 10.01 last 61.87) - open/close 1259 (time 1787580119.75 total 20.01 last 63.97) - open/close 2078 (time 1787580129.76 total 30.02 last 81.85) - open/close 2898 (time 1787580139.76 total 40.02 last 81.97) - open/close 3780 (time 1787580149.77 total 50.03 last 88.15) - open/close 4437 (time 1787580159.79 total 60.04 last 65.59) total: 4500 open/close in 60.90 seconds: 73.89 ops/second - /unlink 1377 (time 1787580175.87 total 10.01 last 137.58) - /unlink 2517 (time 1787580185.88 total 20.02 last 113.92) - /unlink 3770 (time 1787580195.88 total 30.02 last 125.26) total: 4500 /unlink in 35.20 seconds: 127.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 683 (time 1787580216.41 total 10.00 last 68.28) - open/close 1427 (time 1787580226.42 total 20.01 last 74.34) - open/close 2190 (time 1787580236.44 total 30.03 last 76.19) - open/close 2866 (time 1787580246.44 total 40.03 last 67.57) - open/close 3521 (time 1787580256.44 total 50.03 last 65.48) - open/close 4201 (time 1787580266.45 total 60.04 last 67.97) total: 4500 open/close in 63.29 seconds: 71.10 ops/second - /unlink 1341 (time 1787580283.73 total 10.01 last 133.98) - /unlink 2651 (time 1787580293.73 total 20.01 last 130.94) - /unlink 3808 (time 1787580303.74 total 30.02 last 115.62) total: 4500 /unlink in 35.91 seconds: 125.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 702 (time 1787580325.26 total 10.01 last 70.11) - open/close 1322 (time 1787580335.27 total 20.03 last 61.92) - open/close 1961 (time 1787580345.28 total 30.03 last 63.90) - open/close 2711 (time 1787580355.29 total 40.04 last 74.89) - open/close 3381 (time 1787580365.29 total 50.05 last 66.97) - open/close 4047 (time 1787580375.29 total 60.05 last 66.60) total: 4500 open/close in 67.70 seconds: 66.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1141 (time 1787580398.24 total 10.01 last 114.03) - /unlink 2262 (time 1787580408.24 total 20.01 last 112.08) - /unlink 3523 (time 1787580418.25 total 30.01 last 126.03) total: 4500 /unlink in 37.63 seconds: 119.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 600 (time 1787580442.83 total 10.01 last 59.95) - open/close 1335 (time 1787580452.84 total 20.01 last 73.48) - open/close 2154 (time 1787580462.84 total 30.01 last 81.88) - open/close 2960 (time 1787580472.85 total 40.03 last 80.51) - open/close 3742 (time 1787580482.85 total 50.03 last 78.18) - open/close 4483 (time 1787580492.86 total 60.04 last 74.03) total: 4500 open/close in 60.17 seconds: 74.79 ops/second - /unlink 1321 (time 1787580507.64 total 10.00 last 132.09) - /unlink 2705 (time 1787580517.64 total 20.00 last 138.40) - /unlink 4003 (time 1787580527.64 total 30.00 last 129.78) total: 4500 /unlink in 33.46 seconds: 134.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 838 (time 1787580546.38 total 10.01 last 83.74) - open/close 1659 (time 1787580556.39 total 20.01 last 82.06) - open/close 2532 (time 1787580566.40 total 30.03 last 87.18) - open/close 3365 (time 1787580576.41 total 40.03 last 83.26) - open/close 4203 (time 1787580586.42 total 50.04 last 83.73) total: 4500 open/close in 53.83 seconds: 83.60 ops/second - /unlink 1649 (time 1787580604.68 total 10.00 last 164.83) - /unlink 3228 (time 1787580614.69 total 20.01 last 157.85) total: 4500 /unlink in 28.18 seconds: 159.67 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95325 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server unloading modules via unload_modules_local on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing unload_modules_local oleg216-server: modules unloaded.