== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:40:02 (1761248402) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg339-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg339-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 894 (time 1761248563.51 total 10.01 last 89.35) - open/close 1777 (time 1761248573.53 total 20.03 last 88.12) - open/close 2647 (time 1761248583.53 total 30.03 last 87.00) - open/close 3571 (time 1761248593.54 total 40.03 last 92.37) total: 4500 open/close in 49.37 seconds: 91.15 ops/second - /unlink 1694 (time 1761248616.15 total 10.02 last 169.07) - /unlink 3383 (time 1761248626.15 total 20.02 last 168.89) total: 4500 /unlink in 27.18 seconds: 165.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 834 (time 1761248647.34 total 10.01 last 83.32) - open/close 1655 (time 1761248657.34 total 20.01 last 82.09) - open/close 2490 (time 1761248667.35 total 30.02 last 83.40) - open/close 3376 (time 1761248677.36 total 40.03 last 88.51) - open/close 4219 (time 1761248687.39 total 50.06 last 84.07) total: 4500 open/close in 53.48 seconds: 84.14 ops/second - /unlink 1613 (time 1761248703.92 total 10.01 last 161.16) - /unlink 3036 (time 1761248713.93 total 20.01 last 142.25) total: 4500 /unlink in 29.76 seconds: 151.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 854 (time 1761248737.21 total 10.00 last 85.39) - open/close 1723 (time 1761248747.21 total 20.00 last 86.90) - open/close 2652 (time 1761248757.21 total 30.00 last 92.89) - open/close 3579 (time 1761248767.22 total 40.01 last 92.65) - open/close 4401 (time 1761248777.22 total 50.01 last 82.16) total: 4500 open/close in 51.09 seconds: 88.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1645 (time 1761248791.85 total 10.00 last 164.45) - /unlink 3311 (time 1761248801.85 total 20.01 last 166.53) total: 4500 /unlink in 27.28 seconds: 164.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 912 (time 1761248822.46 total 10.02 last 91.06) - open/close 1813 (time 1761248832.47 total 20.03 last 90.00) - open/close 2746 (time 1761248842.48 total 30.03 last 93.27) - open/close 3645 (time 1761248852.48 total 40.04 last 89.83) total: 4500 open/close in 49.36 seconds: 91.17 ops/second - /unlink 1681 (time 1761248875.16 total 10.01 last 167.93) - /unlink 3247 (time 1761248885.16 total 20.01 last 156.58) total: 4500 /unlink in 27.74 seconds: 162.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 831 (time 1761248906.48 total 10.01 last 83.05) - open/close 1630 (time 1761248916.49 total 20.01 last 79.86) - open/close 2516 (time 1761248926.49 total 30.02 last 88.57) - open/close 3438 (time 1761248936.50 total 40.02 last 92.14) - open/close 4352 (time 1761248946.51 total 50.03 last 91.30) total: 4500 open/close in 51.55 seconds: 87.30 ops/second - /unlink 1885 (time 1761248960.64 total 10.00 last 188.50) - /unlink 3519 (time 1761248970.65 total 20.00 last 163.36) total: 4500 /unlink in 25.64 seconds: 175.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 841 (time 1761248989.97 total 10.00 last 84.06) - open/close 1737 (time 1761248999.97 total 20.01 last 89.59) - open/close 2611 (time 1761249009.97 total 30.01 last 87.39) - open/close 3467 (time 1761249019.98 total 40.02 last 85.53) - open/close 4304 (time 1761249030.00 total 50.03 last 83.57) total: 4500 open/close in 52.58 seconds: 85.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1540 (time 1761249046.64 total 10.00 last 153.94) - /unlink 3163 (time 1761249056.64 total 20.00 last 162.30) total: 4500 /unlink in 28.45 seconds: 158.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 854 (time 1761249079.20 total 10.01 last 85.34) - open/close 1751 (time 1761249089.20 total 20.01 last 89.70) - open/close 2641 (time 1761249099.20 total 30.02 last 88.93) - open/close 3567 (time 1761249109.21 total 40.02 last 92.53) - open/close 4490 (time 1761249119.23 total 50.04 last 92.13) total: 4500 open/close in 50.16 seconds: 89.72 ops/second - /unlink 1671 (time 1761249132.80 total 10.00 last 167.08) - /unlink 3398 (time 1761249142.80 total 20.00 last 172.67) total: 4500 /unlink in 27.16 seconds: 165.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 928 (time 1761249163.19 total 10.00 last 92.77) - open/close 1832 (time 1761249173.19 total 20.01 last 90.35) - open/close 2693 (time 1761249183.20 total 30.01 last 86.08) - open/close 3589 (time 1761249193.20 total 40.02 last 89.54) total: 4500 open/close in 49.76 seconds: 90.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1718 (time 1761249215.76 total 10.00 last 171.75) - /unlink 3260 (time 1761249225.76 total 20.01 last 154.16) total: 4500 /unlink in 28.37 seconds: 158.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 823 (time 1761249248.13 total 10.01 last 82.26) - open/close 1682 (time 1761249258.14 total 20.01 last 85.84) - open/close 2443 (time 1761249268.16 total 30.03 last 75.99) - open/close 3187 (time 1761249278.18 total 40.05 last 74.23) - open/close 4122 (time 1761249288.19 total 50.06 last 93.44) total: 4500 open/close in 54.13 seconds: 83.13 ops/second - /unlink 1673 (time 1761249305.73 total 10.00 last 167.28) - /unlink 3202 (time 1761249315.74 total 20.01 last 152.72) total: 4500 /unlink in 28.61 seconds: 157.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 885 (time 1761249337.38 total 10.00 last 88.49) - open/close 1767 (time 1761249347.39 total 20.01 last 88.16) - open/close 2658 (time 1761249357.39 total 30.01 last 89.09) - open/close 3530 (time 1761249367.39 total 40.01 last 87.17) - open/close 4420 (time 1761249377.42 total 50.04 last 88.75) total: 4500 open/close in 50.94 seconds: 88.34 ops/second - /unlink 1576 (time 1761249391.59 total 10.00 last 157.53) - /unlink 3140 (time 1761249401.60 total 20.01 last 156.39) total: 4500 /unlink in 28.74 seconds: 156.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 902 (time 1761249423.31 total 10.00 last 90.19) - open/close 1675 (time 1761249433.33 total 20.03 last 77.11) - open/close 2411 (time 1761249443.34 total 30.03 last 73.55) - open/close 3187 (time 1761249453.34 total 40.04 last 77.58) - open/close 4040 (time 1761249463.36 total 50.05 last 85.19) total: 4500 open/close in 55.02 seconds: 81.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1629 (time 1761249481.43 total 10.00 last 162.86) - /unlink 3315 (time 1761249491.43 total 20.01 last 168.54) total: 4500 /unlink in 26.95 seconds: 166.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 881 (time 1761249511.42 total 10.00 last 88.08) - open/close 1754 (time 1761249521.42 total 20.00 last 87.28) - open/close 2607 (time 1761249531.43 total 30.01 last 85.22) - open/close 3498 (time 1761249541.44 total 40.03 last 88.99) - open/close 4343 (time 1761249551.46 total 50.04 last 84.41) total: 4500 open/close in 51.89 seconds: 86.72 ops/second - /unlink 1580 (time 1761249567.40 total 10.00 last 157.92) - /unlink 3149 (time 1761249577.40 total 20.01 last 156.84) total: 4500 /unlink in 28.62 seconds: 157.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 828 (time 1761249600.21 total 10.00 last 82.79) - open/close 1651 (time 1761249610.22 total 20.00 last 82.28) - open/close 2521 (time 1761249620.22 total 30.00 last 87.00) - open/close 3371 (time 1761249630.22 total 40.01 last 84.97) - open/close 4206 (time 1761249640.24 total 50.03 last 83.35) total: 4500 open/close in 53.30 seconds: 84.43 ops/second - /unlink 1475 (time 1761249657.60 total 10.00 last 147.45) - /unlink 2868 (time 1761249667.60 total 20.01 last 139.28) - /unlink 4203 (time 1761249677.60 total 30.01 last 133.46) total: 4500 /unlink in 32.19 seconds: 139.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 717 (time 1761249693.42 total 10.01 last 71.64) - open/close 1438 (time 1761249703.42 total 20.01 last 72.07) - open/close 2102 (time 1761249713.42 total 30.01 last 66.39) - open/close 2915 (time 1761249723.43 total 40.02 last 81.26) - open/close 3735 (time 1761249733.44 total 50.02 last 81.95) total: 4500 open/close in 59.19 seconds: 76.03 ops/second - /unlink 1568 (time 1761249756.25 total 10.00 last 156.74) - /unlink 2899 (time 1761249766.26 total 20.01 last 133.08) - /unlink 4285 (time 1761249776.26 total 30.01 last 138.50) total: 4500 /unlink in 31.64 seconds: 142.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 891 (time 1761249792.05 total 10.00 last 89.08) - open/close 1737 (time 1761249802.05 total 20.00 last 84.59) - open/close 2564 (time 1761249812.05 total 30.01 last 82.66) - open/close 3355 (time 1761249822.06 total 40.02 last 79.05) - open/close 4127 (time 1761249832.07 total 50.02 last 77.14) total: 4500 open/close in 54.77 seconds: 82.16 ops/second - /unlink 1492 (time 1761249851.12 total 10.01 last 149.09) - /unlink 2837 (time 1761249861.12 total 20.01 last 134.45) - /unlink 4254 (time 1761249871.13 total 30.02 last 141.64) total: 4500 /unlink in 31.84 seconds: 141.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 765 (time 1761249887.55 total 10.00 last 76.47) - open/close 1622 (time 1761249897.56 total 20.01 last 85.67) - open/close 2492 (time 1761249907.56 total 30.01 last 86.98) - open/close 3330 (time 1761249917.57 total 40.02 last 83.75) - open/close 4186 (time 1761249927.57 total 50.02 last 85.60) total: 4500 open/close in 53.77 seconds: 83.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1546 (time 1761249944.90 total 10.00 last 154.56) - /unlink 3160 (time 1761249954.90 total 20.01 last 161.32) total: 4500 /unlink in 28.60 seconds: 157.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 829 (time 1761249977.62 total 10.00 last 82.87) - open/close 1713 (time 1761249987.63 total 20.01 last 88.38) - open/close 2537 (time 1761249997.63 total 30.01 last 82.34) - open/close 3326 (time 1761250007.64 total 40.02 last 78.85) - open/close 4176 (time 1761250017.65 total 50.03 last 84.94) total: 4500 open/close in 53.57 seconds: 84.00 ops/second - /unlink 1550 (time 1761250034.98 total 10.00 last 154.99) - /unlink 3079 (time 1761250044.98 total 20.00 last 152.89) - /unlink 4486 (time 1761250054.99 total 30.01 last 140.63) total: 4500 /unlink in 30.12 seconds: 149.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 812 (time 1761250068.95 total 10.01 last 81.14) - open/close 1643 (time 1761250078.95 total 20.01 last 83.07) - open/close 2464 (time 1761250088.96 total 30.02 last 82.01) - open/close 3256 (time 1761250098.96 total 40.02 last 79.20) - open/close 4143 (time 1761250108.97 total 50.03 last 88.67) total: 4500 open/close in 54.17 seconds: 83.08 ops/second - /unlink 1680 (time 1761250126.53 total 10.00 last 167.97) - /unlink 3252 (time 1761250136.53 total 20.00 last 157.20) total: 4500 /unlink in 27.90 seconds: 161.26 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11102: 93957 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg339-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.