== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:50:59 (1787827859) Stopping clients: oleg413-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg413-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg413-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg413-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg413-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg413-server oleg413-server: oleg413-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg413-server' oleg413-server: oleg413-server.virtnet: executing load_modules_local oleg413-server: Loading modules from /home/green/git/lustre-release/lustre oleg413-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg413-server: detected 4 online CPUs by sysfs oleg413-server: MODOPTS_LIBCFS= oleg413-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg413-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg413-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 630 (time 1787828020.31 total 10.01 last 62.95) - open/close 1268 (time 1787828030.32 total 20.02 last 63.75) - open/close 1987 (time 1787828040.32 total 30.02 last 71.88) - open/close 2603 (time 1787828050.33 total 40.03 last 61.54) - open/close 3169 (time 1787828060.34 total 50.04 last 56.54) - open/close 3816 (time 1787828070.35 total 60.05 last 64.66) - open/close 4420 (time 1787828080.36 total 70.05 last 60.35) total: 4500 open/close in 71.33 seconds: 63.08 ops/second - /unlink 1224 (time 1787828096.15 total 10.02 last 122.21) - /unlink 2373 (time 1787828106.16 total 20.02 last 114.85) - /unlink 3574 (time 1787828116.16 total 30.03 last 120.03) total: 4500 /unlink in 37.53 seconds: 119.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 514 (time 1787828139.14 total 10.03 last 51.23) - open/close 1118 (time 1787828149.14 total 20.04 last 60.39) - open/close 1653 (time 1787828159.14 total 30.04 last 53.48) - open/close 2283 (time 1787828169.15 total 40.05 last 62.95) - open/close 2877 (time 1787828179.17 total 50.06 last 59.30) - open/close 3455 (time 1787828189.18 total 60.07 last 57.73) - open/close 4071 (time 1787828199.20 total 70.09 last 61.49) total: 4500 open/close in 77.35 seconds: 58.17 ops/second - /unlink 1281 (time 1787828221.52 total 10.00 last 128.05) - /unlink 2433 (time 1787828231.53 total 20.00 last 115.19) - /unlink 3691 (time 1787828241.53 total 30.01 last 125.77) total: 4500 /unlink in 36.27 seconds: 124.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 671 (time 1787828262.18 total 10.00 last 67.07) - open/close 1337 (time 1787828272.19 total 20.02 last 66.53) - open/close 1982 (time 1787828282.20 total 30.02 last 64.48) - open/close 2612 (time 1787828292.20 total 40.03 last 62.96) - open/close 3266 (time 1787828302.20 total 50.03 last 65.39) - open/close 3915 (time 1787828312.22 total 60.04 last 64.82) total: 4500 open/close in 69.42 seconds: 64.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1107 (time 1787828336.13 total 10.00 last 110.67) - /unlink 2290 (time 1787828346.14 total 20.01 last 118.22) - /unlink 3492 (time 1787828356.14 total 30.01 last 120.15) total: 4500 /unlink in 38.31 seconds: 117.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 655 (time 1787828379.96 total 10.02 last 65.34) - open/close 1299 (time 1787828389.98 total 20.05 last 64.26) - open/close 2027 (time 1787828399.99 total 30.06 last 72.73) - open/close 2727 (time 1787828409.99 total 40.06 last 69.97) - open/close 3380 (time 1787828420.00 total 50.06 last 65.27) - open/close 4063 (time 1787828430.02 total 60.08 last 68.16) total: 4500 open/close in 66.58 seconds: 67.59 ops/second - /unlink 1302 (time 1787828450.67 total 10.00 last 130.19) - /unlink 2595 (time 1787828460.67 total 20.00 last 129.29) - /unlink 3833 (time 1787828470.67 total 30.00 last 123.79) total: 4500 /unlink in 35.97 seconds: 125.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 693 (time 1787828490.89 total 10.00 last 69.27) - open/close 1397 (time 1787828500.91 total 20.02 last 70.31) - open/close 2026 (time 1787828510.93 total 30.04 last 62.78) - open/close 2710 (time 1787828520.97 total 40.08 last 68.08) - open/close 3351 (time 1787828530.98 total 50.09 last 64.06) - open/close 4031 (time 1787828540.99 total 60.10 last 67.90) total: 4500 open/close in 67.63 seconds: 66.54 ops/second - /unlink 1243 (time 1787828563.66 total 10.00 last 124.26) - /unlink 2527 (time 1787828573.66 total 20.01 last 128.35) - /unlink 3806 (time 1787828583.66 total 30.01 last 127.89) total: 4500 /unlink in 35.36 seconds: 127.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 649 (time 1787828603.31 total 10.01 last 64.85) - open/close 1337 (time 1787828613.31 total 20.01 last 68.80) - open/close 1974 (time 1787828623.32 total 30.01 last 63.66) - open/close 2695 (time 1787828633.33 total 40.02 last 72.06) - open/close 3353 (time 1787828643.34 total 50.03 last 65.73) - open/close 4062 (time 1787828653.34 total 60.04 last 70.85) total: 4500 open/close in 66.74 seconds: 67.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1154 (time 1787828674.35 total 10.00 last 115.35) - /unlink 2348 (time 1787828684.35 total 20.01 last 119.38) - /unlink 3545 (time 1787828694.36 total 30.02 last 119.55) total: 4500 /unlink in 37.94 seconds: 118.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 696 (time 1787828716.80 total 10.02 last 69.46) - open/close 1343 (time 1787828726.81 total 20.03 last 64.64) - open/close 2000 (time 1787828736.82 total 30.04 last 65.65) - open/close 2677 (time 1787828746.83 total 40.04 last 67.67) - open/close 3319 (time 1787828756.84 total 50.05 last 64.14) - open/close 4046 (time 1787828766.84 total 60.06 last 72.65) total: 4500 open/close in 67.08 seconds: 67.09 ops/second - /unlink 1032 (time 1787828789.36 total 10.00 last 103.18) - /unlink 2023 (time 1787828799.37 total 20.01 last 99.03) - /unlink 3025 (time 1787828809.37 total 30.01 last 100.15) - /unlink 4181 (time 1787828819.37 total 40.02 last 115.56) total: 4500 /unlink in 42.69 seconds: 105.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 607 (time 1787828836.87 total 10.01 last 60.64) - open/close 1253 (time 1787828846.88 total 20.02 last 64.52) - open/close 1921 (time 1787828856.88 total 30.02 last 66.80) - open/close 2564 (time 1787828866.89 total 40.03 last 64.24) - open/close 3254 (time 1787828876.89 total 50.03 last 68.98) - open/close 3932 (time 1787828886.90 total 60.04 last 67.77) total: 4500 open/close in 68.33 seconds: 65.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1294 (time 1787828909.60 total 10.00 last 129.34) - /unlink 2490 (time 1787828919.60 total 20.01 last 119.58) - /unlink 3650 (time 1787828929.61 total 30.01 last 115.93) total: 4500 /unlink in 36.93 seconds: 121.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 666 (time 1787828951.22 total 10.01 last 66.52) - open/close 1275 (time 1787828961.25 total 20.04 last 60.73) - open/close 1928 (time 1787828971.25 total 30.04 last 65.28) - open/close 2562 (time 1787828981.26 total 40.06 last 63.32) - open/close 3210 (time 1787828991.27 total 50.06 last 64.75) - open/close 3794 (time 1787829001.28 total 60.07 last 58.36) - open/close 4455 (time 1787829011.29 total 70.08 last 66.01) total: 4500 open/close in 70.88 seconds: 63.49 ops/second - /unlink 1178 (time 1787829027.13 total 10.00 last 117.77) - /unlink 2380 (time 1787829037.13 total 20.01 last 120.15) - /unlink 3609 (time 1787829047.14 total 30.01 last 122.84) total: 4500 /unlink in 37.95 seconds: 118.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 697 (time 1787829069.65 total 10.00 last 69.67) - open/close 1386 (time 1787829079.65 total 20.01 last 68.84) - open/close 2079 (time 1787829089.65 total 30.01 last 69.29) - open/close 2720 (time 1787829099.66 total 40.01 last 64.09) - open/close 3376 (time 1787829109.69 total 50.05 last 65.38) - open/close 4015 (time 1787829119.69 total 60.05 last 63.88) total: 4500 open/close in 67.50 seconds: 66.66 ops/second - /unlink 1280 (time 1787829141.21 total 10.00 last 127.99) - /unlink 2481 (time 1787829151.22 total 20.01 last 120.00) - /unlink 3701 (time 1787829161.22 total 30.01 last 121.99) total: 4500 /unlink in 37.76 seconds: 119.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 549 (time 1787829184.76 total 10.03 last 54.74) - open/close 1110 (time 1787829194.76 total 20.04 last 56.06) - open/close 1732 (time 1787829204.78 total 30.05 last 62.12) - open/close 2416 (time 1787829214.79 total 40.06 last 68.30) - open/close 3064 (time 1787829224.80 total 50.07 last 64.76) - open/close 3680 (time 1787829234.82 total 60.10 last 61.44) - open/close 4180 (time 1787829244.84 total 70.11 last 49.94) total: 4500 open/close in 75.85 seconds: 59.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1040 (time 1787829266.66 total 10.00 last 103.99) - /unlink 2207 (time 1787829276.66 total 20.00 last 116.67) - /unlink 3228 (time 1787829286.66 total 30.01 last 102.07) - /unlink 4223 (time 1787829296.67 total 40.01 last 99.45) total: 4500 /unlink in 43.51 seconds: 103.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 539 (time 1787829316.66 total 10.04 last 53.67) - open/close 1003 (time 1787829326.66 total 20.05 last 46.38) - open/close 1583 (time 1787829336.68 total 30.06 last 57.92) - open/close 2163 (time 1787829346.69 total 40.07 last 57.93) - open/close 2802 (time 1787829356.70 total 50.08 last 63.84) - open/close 3446 (time 1787829366.71 total 60.09 last 64.34) - open/close 4124 (time 1787829376.72 total 70.11 last 67.70) total: 4500 open/close in 75.96 seconds: 59.24 ops/second - /unlink 1172 (time 1787829397.07 total 10.00 last 117.19) - /unlink 2454 (time 1787829407.07 total 20.00 last 128.20) - /unlink 3631 (time 1787829417.08 total 30.01 last 117.61) total: 4500 /unlink in 38.41 seconds: 117.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 488 (time 1787829441.86 total 10.01 last 48.77) - open/close 1027 (time 1787829451.87 total 20.02 last 53.82) - open/close 1649 (time 1787829461.88 total 30.03 last 62.14) - open/close 2213 (time 1787829471.89 total 40.04 last 56.37) - open/close 2830 (time 1787829481.89 total 50.04 last 61.69) - open/close 3495 (time 1787829491.90 total 60.05 last 66.41) - open/close 4140 (time 1787829501.90 total 70.05 last 64.50) total: 4500 open/close in 75.22 seconds: 59.82 ops/second - /unlink 1270 (time 1787829521.31 total 10.01 last 126.91) - /unlink 2582 (time 1787829531.32 total 20.01 last 131.13) - /unlink 3848 (time 1787829541.32 total 30.01 last 126.60) total: 4500 /unlink in 36.20 seconds: 124.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 638 (time 1787829562.26 total 10.02 last 63.70) - open/close 1305 (time 1787829572.27 total 20.03 last 66.60) - open/close 1969 (time 1787829582.29 total 30.04 last 66.31) - open/close 2638 (time 1787829592.30 total 40.06 last 66.80) - open/close 3310 (time 1787829602.30 total 50.06 last 67.19) - open/close 3904 (time 1787829612.30 total 60.06 last 59.40) total: 4500 open/close in 69.91 seconds: 64.37 ops/second - /unlink 1299 (time 1787829637.86 total 10.01 last 129.73) - /unlink 2459 (time 1787829647.87 total 20.02 last 115.92) - /unlink 3633 (time 1787829657.87 total 30.02 last 117.38) total: 4500 /unlink in 36.79 seconds: 122.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 611 (time 1787829679.87 total 10.01 last 61.05) - open/close 1200 (time 1787829689.88 total 20.02 last 58.84) - open/close 1835 (time 1787829699.89 total 30.03 last 63.41) - open/close 2394 (time 1787829709.92 total 40.05 last 55.78) - open/close 2941 (time 1787829719.92 total 50.06 last 54.69) - open/close 3535 (time 1787829729.94 total 60.08 last 59.28) - open/close 4138 (time 1787829739.96 total 70.10 last 60.17) total: 4500 open/close in 75.73 seconds: 59.42 ops/second - /unlink 1289 (time 1787829760.72 total 10.00 last 128.84) - /unlink 2405 (time 1787829770.73 total 20.01 last 111.56) - /unlink 3622 (time 1787829780.74 total 30.02 last 121.60) total: 4500 /unlink in 38.34 seconds: 117.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 590 (time 1787829805.29 total 10.01 last 58.92) - open/close 1190 (time 1787829815.30 total 20.03 last 59.91) - open/close 1868 (time 1787829825.31 total 30.03 last 67.78) - open/close 2514 (time 1787829835.31 total 40.04 last 64.57) - open/close 3159 (time 1787829845.32 total 50.05 last 64.44) - open/close 3879 (time 1787829855.33 total 60.05 last 71.96) total: 4500 open/close in 68.93 seconds: 65.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1217 (time 1787829878.77 total 10.00 last 121.65) - /unlink 2365 (time 1787829888.77 total 20.00 last 114.79) - /unlink 3637 (time 1787829898.77 total 30.01 last 127.14) total: 4500 /unlink in 37.38 seconds: 120.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 698 (time 1787829920.80 total 10.01 last 69.73) - open/close 1338 (time 1787829930.81 total 20.01 last 63.97) - open/close 1975 (time 1787829940.81 total 30.02 last 63.65) - open/close 2654 (time 1787829950.82 total 40.03 last 67.85) - open/close 3300 (time 1787829960.84 total 50.05 last 64.48) - open/close 3930 (time 1787829970.84 total 60.05 last 62.97) total: 4500 open/close in 68.38 seconds: 65.81 ops/second - /unlink 1339 (time 1787829994.21 total 10.00 last 133.89) - /unlink 2591 (time 1787830004.22 total 20.01 last 125.07) - /unlink 3810 (time 1787830014.23 total 30.02 last 121.77) total: 4500 /unlink in 36.58 seconds: 123.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 570 (time 1787830036.22 total 10.02 last 56.88) - open/close 1199 (time 1787830046.23 total 20.04 last 62.80) - open/close 1872 (time 1787830056.23 total 30.04 last 67.30) - open/close 2449 (time 1787830066.23 total 40.04 last 57.68) - open/close 3064 (time 1787830076.26 total 50.06 last 61.35) - open/close 3644 (time 1787830086.27 total 60.08 last 57.93) - open/close 4290 (time 1787830096.27 total 70.08 last 64.59) total: 4500 open/close in 73.21 seconds: 61.47 ops/second - /unlink 1076 (time 1787830114.67 total 10.01 last 107.53) - /unlink 2089 (time 1787830124.67 total 20.01 last 101.27) - /unlink 3123 (time 1787830134.67 total 30.01 last 103.39) - /unlink 4103 (time 1787830144.68 total 40.02 last 97.94) total: 4500 /unlink in 43.55 seconds: 103.32 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11330: 75484 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg413-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg413-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg413-server stop mds service on oleg413-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg413-server unloading modules via unload_modules_local on: 'oleg413-server' oleg413-server: oleg413-server.virtnet: executing unload_modules_local oleg413-server: modules unloaded.