== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:28:01 (1787844481) Stopping clients: oleg624-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg624-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg624-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg624-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg624-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg624-server oleg624-server: oleg624-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing load_modules_local oleg624-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg624-server: Loading modules from /home/green/git/lustre-release/lustre oleg624-server: detected 4 online CPUs by sysfs oleg624-server: MODOPTS_LIBCFS= oleg624-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg624-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg624-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg624-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 770 (time 1787844679.97 total 10.00 last 76.97) - open/close 1534 (time 1787844689.97 total 20.00 last 76.40) - open/close 2271 (time 1787844699.99 total 30.03 last 73.51) - open/close 3003 (time 1787844710.00 total 40.04 last 73.14) - open/close 3759 (time 1787844720.02 total 50.06 last 75.44) - open/close 4482 (time 1787844730.04 total 60.07 last 72.20) total: 4500 open/close in 60.43 seconds: 74.47 ops/second - /unlink 1371 (time 1787844744.79 total 10.00 last 137.06) - /unlink 2770 (time 1787844754.79 total 20.00 last 139.89) - /unlink 4222 (time 1787844764.79 total 30.00 last 145.19) total: 4500 /unlink in 32.09 seconds: 140.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 741 (time 1787844780.97 total 10.01 last 74.03) - open/close 1499 (time 1787844790.98 total 20.02 last 75.74) - open/close 2253 (time 1787844800.99 total 30.03 last 75.33) - open/close 2982 (time 1787844810.99 total 40.03 last 72.88) - open/close 3766 (time 1787844821.00 total 50.04 last 78.32) total: 4500 open/close in 58.96 seconds: 76.33 ops/second - /unlink 1509 (time 1787844843.56 total 10.00 last 150.88) - /unlink 2599 (time 1787844853.57 total 20.01 last 108.89) - /unlink 3474 (time 1787844863.57 total 30.02 last 87.45) total: 4500 /unlink in 39.47 seconds: 114.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 704 (time 1787844888.21 total 10.05 last 70.05) - open/close 1402 (time 1787844898.22 total 20.06 last 69.72) - open/close 2173 (time 1787844908.23 total 30.06 last 77.08) - open/close 2921 (time 1787844918.23 total 40.07 last 74.76) - open/close 3623 (time 1787844928.25 total 50.08 last 70.10) - open/close 4328 (time 1787844938.26 total 60.10 last 70.38) total: 4500 open/close in 62.48 seconds: 72.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1438 (time 1787844954.71 total 10.01 last 143.72) - /unlink 2734 (time 1787844964.71 total 20.01 last 129.58) - /unlink 4116 (time 1787844974.72 total 30.01 last 138.10) total: 4500 /unlink in 32.90 seconds: 136.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 771 (time 1787844992.31 total 10.00 last 77.08) - open/close 1508 (time 1787845002.31 total 20.01 last 73.68) - open/close 2247 (time 1787845012.32 total 30.01 last 73.86) - open/close 3022 (time 1787845022.32 total 40.01 last 77.49) - open/close 3722 (time 1787845032.34 total 50.03 last 69.90) - open/close 4450 (time 1787845042.34 total 60.03 last 72.79) total: 4500 open/close in 60.48 seconds: 74.41 ops/second - /unlink 1383 (time 1787845057.02 total 10.00 last 138.26) - /unlink 2633 (time 1787845067.02 total 20.01 last 124.94) - /unlink 4068 (time 1787845077.02 total 30.01 last 143.47) total: 4500 /unlink in 33.11 seconds: 135.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 723 (time 1787845095.14 total 10.01 last 72.23) - open/close 1436 (time 1787845105.14 total 20.01 last 71.28) - open/close 2275 (time 1787845115.15 total 30.03 last 83.77) - open/close 2991 (time 1787845125.16 total 40.03 last 71.57) - open/close 3746 (time 1787845135.17 total 50.04 last 75.44) total: 4500 open/close in 59.73 seconds: 75.33 ops/second - /unlink 1394 (time 1787845159.14 total 10.00 last 139.34) - /unlink 2723 (time 1787845169.14 total 20.01 last 132.85) - /unlink 4108 (time 1787845179.16 total 30.03 last 138.21) total: 4500 /unlink in 33.05 seconds: 136.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 751 (time 1787845196.20 total 10.00 last 75.09) - open/close 1500 (time 1787845206.20 total 20.00 last 74.89) - open/close 2208 (time 1787845216.22 total 30.02 last 70.68) - open/close 2979 (time 1787845226.23 total 40.03 last 77.04) - open/close 3795 (time 1787845236.24 total 50.04 last 81.49) - open/close 4488 (time 1787845246.25 total 60.05 last 69.25) total: 4500 open/close in 60.17 seconds: 74.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1379 (time 1787845260.90 total 10.00 last 137.85) - /unlink 2829 (time 1787845270.90 total 20.01 last 144.93) - /unlink 4204 (time 1787845280.91 total 30.02 last 137.40) total: 4500 /unlink in 32.59 seconds: 138.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 731 (time 1787845297.79 total 10.01 last 73.01) - open/close 1441 (time 1787845307.80 total 20.02 last 70.94) - open/close 2115 (time 1787845317.80 total 30.02 last 67.37) - open/close 2789 (time 1787845327.80 total 40.03 last 67.39) - open/close 3538 (time 1787845337.81 total 50.03 last 74.86) - open/close 4255 (time 1787845347.81 total 60.03 last 71.67) total: 4500 open/close in 63.39 seconds: 70.99 ops/second - /unlink 1475 (time 1787845365.65 total 10.01 last 147.39) - /unlink 2556 (time 1787845375.65 total 20.01 last 108.04) - /unlink 3429 (time 1787845385.66 total 30.02 last 87.27) - /unlink 4327 (time 1787845395.67 total 40.02 last 89.73) total: 4500 /unlink in 41.99 seconds: 107.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 607 (time 1787845413.54 total 10.02 last 60.57) - open/close 1429 (time 1787845423.55 total 20.03 last 82.14) - open/close 2100 (time 1787845433.56 total 30.04 last 67.04) - open/close 2837 (time 1787845443.56 total 40.04 last 73.69) - open/close 3564 (time 1787845453.57 total 50.05 last 72.59) - open/close 4306 (time 1787845463.58 total 60.06 last 74.16) total: 4500 open/close in 62.48 seconds: 72.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1484 (time 1787845479.84 total 10.01 last 148.29) - /unlink 2910 (time 1787845489.85 total 20.01 last 142.55) - /unlink 4272 (time 1787845499.86 total 30.02 last 136.02) total: 4500 /unlink in 31.80 seconds: 141.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 763 (time 1787845515.66 total 10.01 last 76.26) - open/close 1502 (time 1787845525.67 total 20.01 last 73.86) - open/close 2318 (time 1787845535.68 total 30.02 last 81.55) - open/close 3052 (time 1787845545.68 total 40.03 last 73.34) - open/close 3776 (time 1787845555.69 total 50.03 last 72.34) total: 4500 open/close in 59.69 seconds: 75.39 ops/second - /unlink 1422 (time 1787845579.55 total 10.00 last 142.14) - /unlink 2789 (time 1787845589.56 total 20.01 last 136.65) - /unlink 4246 (time 1787845599.56 total 30.02 last 145.58) total: 4500 /unlink in 32.00 seconds: 140.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 792 (time 1787845615.68 total 10.00 last 79.17) - open/close 1527 (time 1787845625.69 total 20.01 last 73.47) - open/close 2264 (time 1787845635.69 total 30.02 last 73.64) - open/close 2946 (time 1787845645.70 total 40.02 last 68.15) - open/close 3677 (time 1787845655.72 total 50.04 last 72.96) - open/close 4355 (time 1787845665.73 total 60.06 last 67.71) total: 4500 open/close in 62.37 seconds: 72.15 ops/second - /unlink 1366 (time 1787845682.30 total 10.00 last 136.58) - /unlink 2684 (time 1787845692.30 total 20.00 last 131.79) - /unlink 4154 (time 1787845702.30 total 30.01 last 146.95) total: 4500 /unlink in 32.96 seconds: 136.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 666 (time 1787845719.56 total 10.01 last 66.54) - open/close 1271 (time 1787845729.57 total 20.02 last 60.42) - open/close 1801 (time 1787845739.60 total 30.05 last 52.85) - open/close 2432 (time 1787845749.60 total 40.05 last 63.08) - open/close 3036 (time 1787845759.60 total 50.05 last 60.39) - open/close 3632 (time 1787845769.61 total 60.06 last 59.54) - open/close 4288 (time 1787845779.62 total 70.08 last 65.53) total: 4500 open/close in 72.80 seconds: 61.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1335 (time 1787845797.52 total 10.01 last 133.42) - /unlink 2519 (time 1787845807.53 total 20.01 last 118.31) - /unlink 3857 (time 1787845817.55 total 30.03 last 133.56) total: 4500 /unlink in 35.00 seconds: 128.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 679 (time 1787845837.26 total 10.01 last 67.84) - open/close 1360 (time 1787845847.27 total 20.01 last 68.06) - open/close 2039 (time 1787845857.27 total 30.02 last 67.88) - open/close 2752 (time 1787845867.29 total 40.03 last 71.20) - open/close 3515 (time 1787845877.30 total 50.04 last 76.24) - open/close 4222 (time 1787845887.30 total 60.05 last 70.64) total: 4500 open/close in 63.77 seconds: 70.57 ops/second - /unlink 1291 (time 1787845905.38 total 10.00 last 129.09) - /unlink 2529 (time 1787845915.40 total 20.01 last 123.64) - /unlink 3736 (time 1787845925.40 total 30.01 last 120.69) total: 4500 /unlink in 36.63 seconds: 122.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 608 (time 1787845947.21 total 10.00 last 60.79) - open/close 1214 (time 1787845957.22 total 20.02 last 60.51) - open/close 1827 (time 1787845967.23 total 30.03 last 61.25) - open/close 2455 (time 1787845977.23 total 40.03 last 62.80) - open/close 3128 (time 1787845987.23 total 50.03 last 67.28) - open/close 3839 (time 1787845997.23 total 60.03 last 71.09) total: 4500 open/close in 69.61 seconds: 64.65 ops/second - /unlink 1134 (time 1787846021.56 total 10.00 last 113.36) - /unlink 2385 (time 1787846031.57 total 20.01 last 124.98) - /unlink 3633 (time 1787846041.57 total 30.01 last 124.80) total: 4500 /unlink in 37.23 seconds: 120.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 715 (time 1787846063.56 total 10.01 last 71.42) - open/close 1415 (time 1787846073.57 total 20.02 last 69.94) - open/close 2075 (time 1787846083.58 total 30.02 last 65.97) - open/close 2755 (time 1787846093.58 total 40.03 last 67.98) - open/close 3507 (time 1787846103.59 total 50.04 last 75.11) - open/close 4221 (time 1787846113.60 total 60.05 last 71.35) total: 4500 open/close in 64.43 seconds: 69.85 ops/second - /unlink 1299 (time 1787846132.69 total 10.00 last 129.88) - /unlink 2591 (time 1787846142.69 total 20.01 last 129.15) - /unlink 3904 (time 1787846152.70 total 30.01 last 131.26) total: 4500 /unlink in 34.77 seconds: 129.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 606 (time 1787846172.57 total 10.00 last 60.58) - open/close 1271 (time 1787846182.57 total 20.00 last 66.49) - open/close 1809 (time 1787846192.58 total 30.01 last 53.79) - open/close 2346 (time 1787846202.58 total 40.01 last 53.66) - open/close 3087 (time 1787846212.61 total 50.04 last 73.91) - open/close 3783 (time 1787846222.62 total 60.05 last 69.54) - open/close 4467 (time 1787846232.62 total 70.05 last 68.38) total: 4500 open/close in 70.37 seconds: 63.95 ops/second - /unlink 1478 (time 1787846247.91 total 10.00 last 147.77) - /unlink 2872 (time 1787846257.93 total 20.02 last 139.18) - /unlink 4139 (time 1787846267.93 total 30.02 last 126.68) total: 4500 /unlink in 32.74 seconds: 137.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 587 (time 1787846286.86 total 10.00 last 58.70) - open/close 1240 (time 1787846296.86 total 20.01 last 65.27) - open/close 1864 (time 1787846306.87 total 30.01 last 62.36) - open/close 2520 (time 1787846316.87 total 40.02 last 65.58) - open/close 3164 (time 1787846326.89 total 50.03 last 64.28) - open/close 3816 (time 1787846336.91 total 60.05 last 65.10) total: 4500 open/close in 68.27 seconds: 65.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1331 (time 1787846359.69 total 10.01 last 132.94) - /unlink 2534 (time 1787846369.70 total 20.02 last 120.23) - /unlink 3864 (time 1787846379.70 total 30.02 last 132.99) total: 4500 /unlink in 34.25 seconds: 131.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 707 (time 1787846399.67 total 10.00 last 70.67) - open/close 1409 (time 1787846409.68 total 20.01 last 70.15) - open/close 2066 (time 1787846419.69 total 30.02 last 65.66) - open/close 2693 (time 1787846429.69 total 40.02 last 62.66) - open/close 3455 (time 1787846439.70 total 50.03 last 76.14) - open/close 4139 (time 1787846449.70 total 60.03 last 68.38) total: 4500 open/close in 64.89 seconds: 69.35 ops/second - /unlink 1536 (time 1787846469.16 total 10.00 last 153.55) - /unlink 2826 (time 1787846479.16 total 20.01 last 128.91) - /unlink 4164 (time 1787846489.17 total 30.02 last 133.72) total: 4500 /unlink in 32.59 seconds: 138.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 661 (time 1787846507.59 total 10.02 last 65.95) - open/close 1337 (time 1787846517.59 total 20.02 last 67.59) - open/close 1999 (time 1787846527.60 total 30.04 last 66.12) - open/close 2781 (time 1787846537.62 total 40.05 last 78.08) - open/close 3506 (time 1787846547.62 total 50.06 last 72.46) - open/close 4222 (time 1787846557.63 total 60.07 last 71.50) total: 4500 open/close in 63.55 seconds: 70.81 ops/second - /unlink 1336 (time 1787846575.26 total 10.00 last 133.58) - /unlink 2627 (time 1787846585.28 total 20.02 last 128.82) - /unlink 3791 (time 1787846595.28 total 30.02 last 116.39) total: 4500 /unlink in 35.49 seconds: 126.80 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95227 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg624-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg624-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server stop mds service on oleg624-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg624-server stop mds service on oleg624-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg624-server unloading modules via unload_modules_local on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing unload_modules_local oleg624-server: modules unloaded.