== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:56:10 (1787867770) Stopping clients: oleg158-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg158-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg158-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg158-server oleg158-server: oleg158-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: detected 4 online CPUs by sysfs oleg158-server: MODOPTS_LIBCFS= oleg158-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg158-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg158-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 837 (time 1787867975.77 total 10.01 last 83.60) - open/close 1635 (time 1787867985.77 total 20.01 last 79.79) - open/close 2431 (time 1787867995.78 total 30.02 last 79.58) - open/close 3272 (time 1787868005.79 total 40.02 last 84.03) - open/close 4020 (time 1787868015.79 total 50.03 last 74.75) total: 4500 open/close in 56.74 seconds: 79.31 ops/second - /unlink 1426 (time 1787868036.94 total 10.00 last 142.54) - /unlink 2919 (time 1787868046.94 total 20.00 last 149.30) total: 4500 /unlink in 29.83 seconds: 150.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 527 (time 1787868071.52 total 10.01 last 52.65) - open/close 996 (time 1787868081.54 total 20.02 last 46.83) - open/close 1590 (time 1787868091.55 total 30.03 last 59.36) - open/close 2253 (time 1787868101.56 total 40.05 last 66.19) - open/close 2896 (time 1787868111.57 total 50.06 last 64.26) - open/close 3658 (time 1787868121.58 total 60.07 last 76.11) - open/close 4379 (time 1787868131.59 total 70.08 last 72.03) total: 4500 open/close in 71.63 seconds: 62.82 ops/second - /unlink 1672 (time 1787868147.80 total 10.00 last 167.13) - /unlink 3164 (time 1787868157.81 total 20.02 last 148.99) total: 4500 /unlink in 29.83 seconds: 150.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 719 (time 1787868182.54 total 10.01 last 71.84) - open/close 1355 (time 1787868192.54 total 20.01 last 63.56) - open/close 2114 (time 1787868202.58 total 30.05 last 75.65) - open/close 2812 (time 1787868212.58 total 40.05 last 69.77) - open/close 3595 (time 1787868222.58 total 50.06 last 78.26) - open/close 4351 (time 1787868232.59 total 60.06 last 75.56) total: 4500 open/close in 61.70 seconds: 72.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1089 (time 1787868248.82 total 10.00 last 108.89) - /unlink 1849 (time 1787868258.84 total 20.02 last 75.85) - /unlink 2675 (time 1787868268.84 total 30.03 last 82.56) - /unlink 3672 (time 1787868278.84 total 40.03 last 99.70) total: 4500 /unlink in 47.12 seconds: 95.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 721 (time 1787868301.01 total 10.01 last 72.00) - open/close 1441 (time 1787868311.01 total 20.02 last 71.99) - open/close 1876 (time 1787868321.03 total 30.03 last 43.44) - open/close 2311 (time 1787868331.03 total 40.03 last 43.48) - open/close 2805 (time 1787868341.04 total 50.04 last 49.38) - open/close 3444 (time 1787868351.05 total 60.05 last 63.82) - open/close 4130 (time 1787868361.06 total 70.06 last 68.54) total: 4500 open/close in 75.76 seconds: 59.40 ops/second - /unlink 1344 (time 1787868381.76 total 10.00 last 134.39) - /unlink 2433 (time 1787868391.76 total 20.00 last 108.89) - /unlink 3804 (time 1787868401.76 total 30.01 last 137.02) total: 4500 /unlink in 35.93 seconds: 125.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 874 (time 1787868421.83 total 10.01 last 87.34) - open/close 1716 (time 1787868431.84 total 20.01 last 84.18) - open/close 2559 (time 1787868441.84 total 30.02 last 84.25) - open/close 3427 (time 1787868451.85 total 40.02 last 86.76) - open/close 4275 (time 1787868461.85 total 50.03 last 84.75) total: 4500 open/close in 52.63 seconds: 85.51 ops/second - /unlink 1662 (time 1787868478.43 total 10.00 last 166.15) - /unlink 3040 (time 1787868488.43 total 20.00 last 137.78) - /unlink 4418 (time 1787868498.44 total 30.01 last 137.75) total: 4500 /unlink in 30.77 seconds: 146.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 728 (time 1787868513.87 total 10.03 last 72.61) - open/close 1463 (time 1787868523.87 total 20.03 last 73.50) - open/close 2214 (time 1787868533.88 total 30.03 last 75.08) - open/close 3063 (time 1787868543.89 total 40.04 last 84.83) - open/close 3821 (time 1787868553.91 total 50.06 last 75.63) total: 4500 open/close in 59.35 seconds: 75.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1463 (time 1787868577.62 total 10.00 last 146.25) - /unlink 3103 (time 1787868587.63 total 20.01 last 163.95) total: 4500 /unlink in 28.06 seconds: 160.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 856 (time 1787868610.33 total 10.01 last 85.53) - open/close 1522 (time 1787868620.33 total 20.01 last 66.57) - open/close 2139 (time 1787868630.34 total 30.02 last 61.68) - open/close 2791 (time 1787868640.34 total 40.02 last 65.15) - open/close 3553 (time 1787868650.36 total 50.04 last 76.10) - open/close 4258 (time 1787868660.36 total 60.04 last 70.47) total: 4500 open/close in 63.36 seconds: 71.03 ops/second - /unlink 1420 (time 1787868678.36 total 10.00 last 141.99) - /unlink 2744 (time 1787868688.37 total 20.00 last 132.38) - /unlink 4110 (time 1787868698.37 total 30.01 last 136.52) total: 4500 /unlink in 33.02 seconds: 136.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 782 (time 1787868716.31 total 10.01 last 78.13) - open/close 1618 (time 1787868726.34 total 20.04 last 83.33) - open/close 2340 (time 1787868736.34 total 30.04 last 72.18) - open/close 3088 (time 1787868746.36 total 40.06 last 74.69) - open/close 3851 (time 1787868756.36 total 50.06 last 76.30) total: 4500 open/close in 60.00 seconds: 75.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1071 (time 1787868781.54 total 10.00 last 107.05) - /unlink 1949 (time 1787868791.55 total 20.01 last 87.76) - /unlink 3548 (time 1787868801.55 total 30.02 last 159.79) total: 4500 /unlink in 38.08 seconds: 118.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 849 (time 1787868825.12 total 10.00 last 84.87) - open/close 1562 (time 1787868835.12 total 20.01 last 71.27) - open/close 2126 (time 1787868845.13 total 30.02 last 56.33) - open/close 2736 (time 1787868855.15 total 40.04 last 60.88) - open/close 3349 (time 1787868865.15 total 50.04 last 61.30) - open/close 3898 (time 1787868875.17 total 60.06 last 54.80) total: 4500 open/close in 69.61 seconds: 64.65 ops/second - /unlink 1487 (time 1787868900.33 total 10.00 last 148.67) - /unlink 3141 (time 1787868910.33 total 20.00 last 165.37) total: 4500 /unlink in 29.89 seconds: 150.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 721 (time 1787868935.31 total 10.02 last 71.98) - open/close 1606 (time 1787868945.32 total 20.02 last 88.44) - open/close 2403 (time 1787868955.33 total 30.04 last 79.59) - open/close 3134 (time 1787868965.34 total 40.04 last 73.05) - open/close 3985 (time 1787868975.35 total 50.06 last 85.01) total: 4500 open/close in 56.80 seconds: 79.22 ops/second - /unlink 1391 (time 1787868996.65 total 10.00 last 139.07) - /unlink 2724 (time 1787869006.65 total 20.00 last 133.26) - /unlink 4252 (time 1787869016.66 total 30.02 last 152.64) total: 4500 /unlink in 31.78 seconds: 141.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 827 (time 1787869033.25 total 10.00 last 82.67) - open/close 1686 (time 1787869043.27 total 20.02 last 85.74) - open/close 2515 (time 1787869053.30 total 30.05 last 82.68) - open/close 3315 (time 1787869063.31 total 40.06 last 79.92) - open/close 4075 (time 1787869073.32 total 50.07 last 75.91) total: 4500 open/close in 56.00 seconds: 80.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1536 (time 1787869094.26 total 10.01 last 153.50) - /unlink 3011 (time 1787869104.26 total 20.01 last 147.45) total: 4500 /unlink in 29.46 seconds: 152.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 898 (time 1787869129.06 total 10.01 last 89.72) - open/close 1738 (time 1787869139.07 total 20.02 last 83.93) - open/close 2653 (time 1787869149.07 total 30.02 last 91.47) - open/close 3454 (time 1787869159.08 total 40.03 last 80.03) - open/close 4174 (time 1787869169.08 total 50.03 last 71.99) total: 4500 open/close in 55.01 seconds: 81.81 ops/second - /unlink 1747 (time 1787869189.93 total 10.00 last 174.67) - /unlink 3371 (time 1787869199.93 total 20.01 last 162.35) total: 4500 /unlink in 27.82 seconds: 161.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 637 (time 1787869223.16 total 10.01 last 63.64) - open/close 1372 (time 1787869233.17 total 20.01 last 73.46) - open/close 2214 (time 1787869243.18 total 30.03 last 84.07) - open/close 2995 (time 1787869253.19 total 40.03 last 78.07) - open/close 3841 (time 1787869263.20 total 50.04 last 84.53) total: 4500 open/close in 57.35 seconds: 78.47 ops/second - /unlink 1628 (time 1787869285.69 total 10.00 last 162.77) - /unlink 3318 (time 1787869295.70 total 20.01 last 168.91) total: 4500 /unlink in 27.54 seconds: 163.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 974 (time 1787869316.94 total 10.01 last 97.34) - open/close 1893 (time 1787869326.94 total 20.01 last 91.90) - open/close 2697 (time 1787869336.94 total 30.01 last 80.38) - open/close 3467 (time 1787869346.94 total 40.01 last 76.97) - open/close 4267 (time 1787869356.94 total 50.01 last 79.98) total: 4500 open/close in 52.72 seconds: 85.35 ops/second - /unlink 1943 (time 1787869373.62 total 10.01 last 194.20) - /unlink 3753 (time 1787869383.63 total 20.01 last 180.92) total: 4500 /unlink in 25.38 seconds: 177.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 838 (time 1787869403.50 total 10.01 last 83.76) - open/close 1714 (time 1787869413.51 total 20.02 last 87.47) - open/close 2614 (time 1787869423.52 total 30.03 last 89.94) - open/close 3477 (time 1787869433.52 total 40.03 last 86.27) - open/close 4357 (time 1787869443.53 total 50.04 last 87.92) total: 4500 open/close in 52.09 seconds: 86.40 ops/second - /unlink 1631 (time 1787869460.18 total 10.00 last 163.10) - /unlink 3328 (time 1787869470.19 total 20.00 last 169.62) total: 4500 /unlink in 27.03 seconds: 166.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 907 (time 1787869491.41 total 10.00 last 90.68) - open/close 1712 (time 1787869501.42 total 20.01 last 80.42) - open/close 2446 (time 1787869511.42 total 30.02 last 73.37) - open/close 3392 (time 1787869521.44 total 40.04 last 94.41) - open/close 4195 (time 1787869531.44 total 50.04 last 80.29) total: 4500 open/close in 53.52 seconds: 84.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1825 (time 1787869549.33 total 10.00 last 182.46) - /unlink 3435 (time 1787869559.33 total 20.00 last 160.99) total: 4500 /unlink in 26.57 seconds: 169.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 993 (time 1787869580.32 total 10.01 last 99.24) - open/close 1866 (time 1787869590.32 total 20.01 last 87.26) - open/close 2844 (time 1787869600.33 total 30.02 last 97.68) - open/close 3591 (time 1787869610.34 total 40.03 last 74.66) total: 4500 open/close in 49.85 seconds: 90.26 ops/second - /unlink 1603 (time 1787869634.59 total 10.00 last 160.28) - /unlink 3237 (time 1787869644.60 total 20.01 last 163.33) total: 4500 /unlink in 27.75 seconds: 162.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 932 (time 1787869667.15 total 10.00 last 93.17) - open/close 1838 (time 1787869677.17 total 20.02 last 90.47) - open/close 2549 (time 1787869687.18 total 30.03 last 71.04) - open/close 3499 (time 1787869697.19 total 40.04 last 94.85) - open/close 4430 (time 1787869707.19 total 50.05 last 93.07) total: 4500 open/close in 50.75 seconds: 88.68 ops/second - /unlink 1382 (time 1787869722.10 total 10.01 last 138.07) - /unlink 2872 (time 1787869732.11 total 20.01 last 148.98) total: 4500 /unlink in 29.13 seconds: 154.50 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11375: 95390 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server unloading modules via unload_modules_local on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: modules unloaded.