== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:49:29 (1763347769) Stopping clients: oleg144-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg144-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg144-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg144-server oleg144-server: oleg144-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg144-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg144-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg144-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 599 (time 1763347956.75 total 10.01 last 59.83) - open/close 1218 (time 1763347966.75 total 20.02 last 61.86) - open/close 1903 (time 1763347976.77 total 30.03 last 68.39) - open/close 2585 (time 1763347986.79 total 40.05 last 68.05) - open/close 3261 (time 1763347996.80 total 50.06 last 67.56) - open/close 3992 (time 1763348006.80 total 60.07 last 73.06) total: 4500 open/close in 66.92 seconds: 67.25 ops/second - /unlink 1284 (time 1763348027.60 total 10.01 last 128.30) - /unlink 2578 (time 1763348037.60 total 20.01 last 129.39) - /unlink 3852 (time 1763348047.60 total 30.01 last 127.39) total: 4500 /unlink in 35.49 seconds: 126.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 652 (time 1763348067.89 total 10.00 last 65.18) - open/close 1372 (time 1763348077.89 total 20.01 last 71.97) - open/close 2094 (time 1763348087.90 total 30.02 last 72.14) - open/close 2754 (time 1763348097.91 total 40.02 last 65.94) - open/close 3452 (time 1763348107.93 total 50.04 last 69.66) - open/close 4186 (time 1763348117.93 total 60.05 last 73.39) total: 4500 open/close in 64.44 seconds: 69.83 ops/second - /unlink 1196 (time 1763348136.27 total 10.01 last 119.52) - /unlink 2437 (time 1763348146.27 total 20.01 last 124.09) - /unlink 3757 (time 1763348156.28 total 30.01 last 131.97) total: 4500 /unlink in 35.75 seconds: 125.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 720 (time 1763348176.51 total 10.00 last 71.97) - open/close 1442 (time 1763348186.52 total 20.02 last 72.11) - open/close 2205 (time 1763348196.54 total 30.03 last 76.18) - open/close 2922 (time 1763348206.54 total 40.04 last 71.66) - open/close 3621 (time 1763348216.55 total 50.04 last 69.86) - open/close 4284 (time 1763348226.55 total 60.05 last 66.27) total: 4500 open/close in 63.26 seconds: 71.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1105 (time 1763348243.47 total 10.00 last 110.50) - /unlink 2305 (time 1763348253.48 total 20.00 last 119.96) - /unlink 3626 (time 1763348263.48 total 30.00 last 132.08) total: 4500 /unlink in 35.85 seconds: 125.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 716 (time 1763348283.64 total 10.01 last 71.51) - open/close 1331 (time 1763348293.64 total 20.02 last 61.47) - open/close 2014 (time 1763348303.65 total 30.03 last 68.24) - open/close 2667 (time 1763348313.66 total 40.03 last 65.26) - open/close 3367 (time 1763348323.68 total 50.06 last 69.82) - open/close 4042 (time 1763348333.70 total 60.07 last 67.40) total: 4500 open/close in 67.46 seconds: 66.71 ops/second - /unlink 1171 (time 1763348355.87 total 10.00 last 117.08) - /unlink 2333 (time 1763348365.88 total 20.01 last 116.13) - /unlink 3589 (time 1763348375.88 total 30.02 last 125.49) total: 4500 /unlink in 36.89 seconds: 121.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 658 (time 1763348397.84 total 10.03 last 65.63) - open/close 1397 (time 1763348407.84 total 20.03 last 73.87) - open/close 2059 (time 1763348417.86 total 30.05 last 66.08) - open/close 2740 (time 1763348427.89 total 40.08 last 67.88) - open/close 3457 (time 1763348437.90 total 50.09 last 71.64) - open/close 4175 (time 1763348447.93 total 60.11 last 71.61) total: 4500 open/close in 64.29 seconds: 69.99 ops/second - /unlink 1244 (time 1763348465.54 total 10.00 last 124.37) - /unlink 2508 (time 1763348475.54 total 20.00 last 126.39) - /unlink 3815 (time 1763348485.54 total 30.00 last 130.70) total: 4500 /unlink in 35.10 seconds: 128.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 748 (time 1763348505.06 total 10.02 last 74.65) - open/close 1470 (time 1763348515.07 total 20.03 last 72.16) - open/close 2202 (time 1763348525.07 total 30.03 last 73.19) - open/close 2916 (time 1763348535.08 total 40.03 last 71.37) - open/close 3613 (time 1763348545.08 total 50.03 last 69.69) - open/close 4325 (time 1763348555.10 total 60.05 last 71.05) total: 4500 open/close in 63.10 seconds: 71.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1240 (time 1763348573.26 total 10.00 last 124.00) - /unlink 2508 (time 1763348583.27 total 20.01 last 126.71) - /unlink 3787 (time 1763348593.27 total 30.01 last 127.86) total: 4500 /unlink in 35.68 seconds: 126.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 733 (time 1763348613.02 total 10.00 last 73.30) - open/close 1480 (time 1763348623.04 total 20.02 last 74.57) - open/close 2235 (time 1763348633.05 total 30.02 last 75.46) - open/close 2943 (time 1763348643.05 total 40.03 last 70.75) - open/close 3598 (time 1763348653.07 total 50.04 last 65.41) - open/close 4261 (time 1763348663.07 total 60.04 last 66.29) total: 4500 open/close in 63.25 seconds: 71.14 ops/second - /unlink 1342 (time 1763348679.86 total 10.01 last 134.07) - /unlink 2703 (time 1763348689.86 total 20.01 last 136.09) - /unlink 3988 (time 1763348699.86 total 30.01 last 128.49) total: 4500 /unlink in 34.14 seconds: 131.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 736 (time 1763348718.67 total 10.00 last 73.59) - open/close 1454 (time 1763348728.67 total 20.00 last 71.78) - open/close 2179 (time 1763348738.67 total 30.01 last 72.49) - open/close 2945 (time 1763348748.68 total 40.01 last 76.55) - open/close 3704 (time 1763348758.68 total 50.01 last 75.89) - open/close 4418 (time 1763348768.69 total 60.02 last 71.38) total: 4500 open/close in 61.13 seconds: 73.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1398 (time 1763348783.94 total 10.00 last 139.77) - /unlink 2723 (time 1763348793.94 total 20.00 last 132.47) - /unlink 3994 (time 1763348803.95 total 30.01 last 127.02) total: 4500 /unlink in 33.72 seconds: 133.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 716 (time 1763348822.75 total 10.00 last 71.60) - open/close 1436 (time 1763348832.76 total 20.01 last 71.96) - open/close 2057 (time 1763348842.76 total 30.01 last 62.07) - open/close 2773 (time 1763348852.77 total 40.02 last 71.56) - open/close 3449 (time 1763348862.77 total 50.02 last 67.56) - open/close 4166 (time 1763348872.78 total 60.03 last 71.62) total: 4500 open/close in 65.32 seconds: 68.90 ops/second - /unlink 1230 (time 1763348892.36 total 10.00 last 122.98) - /unlink 2475 (time 1763348902.36 total 20.00 last 124.49) - /unlink 3715 (time 1763348912.38 total 30.03 last 123.70) total: 4500 /unlink in 36.51 seconds: 123.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 662 (time 1763348933.06 total 10.00 last 66.17) - open/close 1338 (time 1763348943.06 total 20.01 last 67.55) - open/close 2030 (time 1763348953.07 total 30.02 last 69.12) - open/close 2739 (time 1763348963.08 total 40.03 last 70.87) - open/close 3491 (time 1763348973.09 total 50.04 last 75.08) - open/close 4214 (time 1763348983.10 total 60.04 last 72.29) total: 4500 open/close in 63.99 seconds: 70.32 ops/second - /unlink 1160 (time 1763349001.83 total 10.01 last 115.94) - /unlink 2474 (time 1763349011.84 total 20.01 last 131.37) - /unlink 3866 (time 1763349021.84 total 30.01 last 139.15) total: 4500 /unlink in 34.46 seconds: 130.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 725 (time 1763349040.80 total 10.01 last 72.45) - open/close 1404 (time 1763349050.81 total 20.02 last 67.81) - open/close 2056 (time 1763349060.82 total 30.03 last 65.13) - open/close 2815 (time 1763349070.84 total 40.05 last 75.74) - open/close 3572 (time 1763349080.86 total 50.07 last 75.57) - open/close 4353 (time 1763349090.87 total 60.08 last 78.03) total: 4500 open/close in 62.06 seconds: 72.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1339 (time 1763349106.31 total 10.00 last 133.90) - /unlink 2660 (time 1763349116.31 total 20.00 last 132.06) - /unlink 4090 (time 1763349126.31 total 30.01 last 142.96) total: 4500 /unlink in 32.78 seconds: 137.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 709 (time 1763349142.73 total 10.01 last 70.84) - open/close 1391 (time 1763349152.73 total 20.01 last 68.18) - open/close 2092 (time 1763349162.74 total 30.01 last 70.08) - open/close 2830 (time 1763349172.74 total 40.02 last 73.78) - open/close 3576 (time 1763349182.74 total 50.02 last 74.57) - open/close 4319 (time 1763349192.76 total 60.04 last 74.17) total: 4500 open/close in 62.46 seconds: 72.05 ops/second - /unlink 1417 (time 1763349210.22 total 10.00 last 141.68) - /unlink 2599 (time 1763349220.22 total 20.01 last 118.15) - /unlink 3830 (time 1763349230.23 total 30.02 last 122.95) total: 4500 /unlink in 35.78 seconds: 125.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 652 (time 1763349250.25 total 10.00 last 65.20) - open/close 1397 (time 1763349260.26 total 20.01 last 74.44) - open/close 2142 (time 1763349270.27 total 30.02 last 74.41) - open/close 2856 (time 1763349280.27 total 40.02 last 71.40) - open/close 3596 (time 1763349290.28 total 50.04 last 73.90) - open/close 4337 (time 1763349300.29 total 60.05 last 74.02) total: 4500 open/close in 62.05 seconds: 72.53 ops/second - /unlink 1222 (time 1763349315.90 total 10.00 last 122.14) - /unlink 2409 (time 1763349325.90 total 20.01 last 118.65) - /unlink 3684 (time 1763349335.91 total 30.01 last 127.44) total: 4500 /unlink in 36.96 seconds: 121.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 668 (time 1763349356.87 total 10.01 last 66.76) - open/close 1340 (time 1763349366.87 total 20.01 last 67.17) - open/close 2027 (time 1763349376.87 total 30.01 last 68.69) - open/close 2682 (time 1763349386.88 total 40.02 last 65.47) - open/close 3346 (time 1763349396.93 total 50.07 last 66.07) - open/close 4062 (time 1763349406.93 total 60.07 last 71.56) total: 4500 open/close in 66.04 seconds: 68.14 ops/second - /unlink 1309 (time 1763349426.55 total 10.01 last 130.80) - /unlink 2443 (time 1763349436.57 total 20.02 last 113.25) - /unlink 3728 (time 1763349446.57 total 30.02 last 128.47) total: 4500 /unlink in 36.67 seconds: 122.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 627 (time 1763349467.90 total 10.01 last 62.62) - open/close 1318 (time 1763349477.92 total 20.03 last 69.01) - open/close 2050 (time 1763349487.93 total 30.04 last 73.12) - open/close 2707 (time 1763349497.96 total 40.06 last 65.52) - open/close 3386 (time 1763349508.01 total 50.12 last 67.55) - open/close 4058 (time 1763349518.01 total 60.12 last 67.19) total: 4500 open/close in 66.58 seconds: 67.59 ops/second - /unlink 1148 (time 1763349538.68 total 10.00 last 114.75) - /unlink 2456 (time 1763349548.69 total 20.01 last 130.76) - /unlink 3797 (time 1763349558.69 total 30.01 last 134.07) total: 4500 /unlink in 35.97 seconds: 125.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 675 (time 1763349578.72 total 10.01 last 67.40) - open/close 1357 (time 1763349588.72 total 20.02 last 68.19) - open/close 2055 (time 1763349598.72 total 30.02 last 69.79) - open/close 2749 (time 1763349608.73 total 40.02 last 69.36) - open/close 3468 (time 1763349618.74 total 50.04 last 71.81) - open/close 4149 (time 1763349628.74 total 60.04 last 68.10) total: 4500 open/close in 65.07 seconds: 69.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1241 (time 1763349647.96 total 10.00 last 124.07) - /unlink 2430 (time 1763349657.96 total 20.01 last 118.80) - /unlink 3752 (time 1763349667.97 total 30.01 last 132.16) total: 4500 /unlink in 35.95 seconds: 125.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 679 (time 1763349687.43 total 10.01 last 67.85) - open/close 1426 (time 1763349697.43 total 20.01 last 74.67) - open/close 2172 (time 1763349707.44 total 30.02 last 74.56) - open/close 2960 (time 1763349717.44 total 40.02 last 78.76) - open/close 3666 (time 1763349727.44 total 50.02 last 70.58) - open/close 4411 (time 1763349737.45 total 60.03 last 74.46) total: 4500 open/close in 61.20 seconds: 73.53 ops/second - /unlink 1359 (time 1763349752.39 total 10.00 last 135.87) - /unlink 2857 (time 1763349762.40 total 20.01 last 149.68) - /unlink 4313 (time 1763349772.41 total 30.02 last 145.50) total: 4500 /unlink in 31.79 seconds: 141.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 687 (time 1763349788.47 total 10.02 last 68.53) - open/close 1452 (time 1763349798.47 total 20.02 last 76.50) - open/close 2187 (time 1763349808.48 total 30.03 last 73.46) - open/close 2941 (time 1763349818.49 total 40.04 last 75.33) - open/close 3722 (time 1763349828.50 total 50.05 last 77.99) total: 4500 open/close in 59.63 seconds: 75.47 ops/second - /unlink 1205 (time 1763349851.82 total 10.02 last 120.27) - /unlink 2324 (time 1763349861.83 total 20.03 last 111.81) - /unlink 3521 (time 1763349871.83 total 30.03 last 119.68) total: 4500 /unlink in 37.75 seconds: 119.19 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 93990 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg144-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local oleg144-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg144-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.