== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:22:26 (1787397746) Stopping clients: oleg114-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg114-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg114-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg114-server oleg114-server: oleg114-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg114-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg114-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 849 (time 1787397957.24 total 10.00 last 84.87) - open/close 1670 (time 1787397967.25 total 20.01 last 82.00) - open/close 2544 (time 1787397977.26 total 30.02 last 87.32) - open/close 3498 (time 1787397987.26 total 40.03 last 95.37) - open/close 4407 (time 1787397997.27 total 50.04 last 90.80) total: 4500 open/close in 51.17 seconds: 87.94 ops/second - /unlink 1803 (time 1787398012.43 total 10.00 last 180.25) - /unlink 3391 (time 1787398022.43 total 20.01 last 158.76) total: 4500 /unlink in 27.14 seconds: 165.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 971 (time 1787398044.82 total 10.00 last 97.06) - open/close 1782 (time 1787398054.82 total 20.01 last 81.09) - open/close 2690 (time 1787398064.82 total 30.01 last 90.78) - open/close 3618 (time 1787398074.82 total 40.01 last 92.77) - open/close 4431 (time 1787398084.84 total 50.02 last 81.20) total: 4500 open/close in 50.81 seconds: 88.56 ops/second - /unlink 1768 (time 1787398099.15 total 10.00 last 176.75) - /unlink 3455 (time 1787398109.15 total 20.00 last 168.69) total: 4500 /unlink in 26.42 seconds: 170.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 969 (time 1787398129.54 total 10.01 last 96.76) - open/close 1966 (time 1787398139.55 total 20.02 last 99.69) - open/close 2966 (time 1787398149.55 total 30.02 last 99.93) - open/close 3906 (time 1787398159.55 total 40.02 last 93.98) total: 4500 open/close in 47.34 seconds: 95.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1655 (time 1787398180.87 total 10.00 last 165.49) - /unlink 3160 (time 1787398190.88 total 20.01 last 150.36) total: 4500 /unlink in 28.71 seconds: 156.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 925 (time 1787398214.90 total 10.01 last 92.44) - open/close 1715 (time 1787398224.91 total 20.02 last 78.86) - open/close 2512 (time 1787398234.92 total 30.03 last 79.64) - open/close 3403 (time 1787398244.93 total 40.04 last 89.06) - open/close 4210 (time 1787398254.93 total 50.04 last 80.70) total: 4500 open/close in 53.82 seconds: 83.62 ops/second - /unlink 1522 (time 1787398273.10 total 10.00 last 152.17) - /unlink 2925 (time 1787398283.10 total 20.00 last 140.26) - /unlink 4085 (time 1787398293.12 total 30.02 last 115.81) total: 4500 /unlink in 33.40 seconds: 134.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 819 (time 1787398311.83 total 10.00 last 81.88) - open/close 1746 (time 1787398321.84 total 20.01 last 92.62) - open/close 2637 (time 1787398331.85 total 30.02 last 89.04) - open/close 3546 (time 1787398341.85 total 40.02 last 90.89) - open/close 4391 (time 1787398351.85 total 50.02 last 84.50) total: 4500 open/close in 51.32 seconds: 87.69 ops/second - /unlink 1470 (time 1787398367.37 total 10.00 last 146.98) - /unlink 2936 (time 1787398377.37 total 20.01 last 146.54) - /unlink 4289 (time 1787398387.37 total 30.01 last 135.28) total: 4500 /unlink in 31.48 seconds: 142.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 835 (time 1787398403.70 total 10.02 last 83.37) - open/close 1610 (time 1787398413.71 total 20.03 last 77.43) - open/close 2453 (time 1787398423.71 total 30.03 last 84.26) - open/close 3329 (time 1787398433.75 total 40.06 last 87.30) - open/close 4208 (time 1787398443.75 total 50.07 last 87.85) total: 4500 open/close in 53.71 seconds: 83.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1582 (time 1787398461.65 total 10.00 last 158.19) - /unlink 3024 (time 1787398471.65 total 20.01 last 144.11) - /unlink 4476 (time 1787398481.66 total 30.02 last 145.05) total: 4500 /unlink in 30.40 seconds: 148.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 772 (time 1787398497.20 total 10.01 last 77.16) - open/close 1588 (time 1787398507.20 total 20.01 last 81.58) - open/close 2425 (time 1787398517.21 total 30.02 last 83.64) - open/close 3137 (time 1787398527.21 total 40.02 last 71.18) - open/close 3843 (time 1787398537.21 total 50.02 last 70.60) total: 4500 open/close in 57.07 seconds: 78.85 ops/second - /unlink 1503 (time 1787398559.29 total 10.01 last 150.22) - /unlink 3195 (time 1787398569.30 total 20.01 last 169.04) total: 4500 /unlink in 29.35 seconds: 153.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 763 (time 1787398593.53 total 10.01 last 76.23) - open/close 1476 (time 1787398603.54 total 20.02 last 71.22) - open/close 2201 (time 1787398613.55 total 30.03 last 72.46) - open/close 3035 (time 1787398623.55 total 40.03 last 83.39) - open/close 3800 (time 1787398633.55 total 50.03 last 76.48) total: 4500 open/close in 59.89 seconds: 75.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1303 (time 1787398657.67 total 10.01 last 130.22) - /unlink 2657 (time 1787398667.68 total 20.02 last 135.24) - /unlink 4099 (time 1787398677.69 total 30.02 last 144.10) total: 4500 /unlink in 33.27 seconds: 135.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 745 (time 1787398697.05 total 10.01 last 74.45) - open/close 1650 (time 1787398707.06 total 20.02 last 90.39) - open/close 2610 (time 1787398717.07 total 30.03 last 95.94) - open/close 3486 (time 1787398727.08 total 40.04 last 87.50) - open/close 4314 (time 1787398737.08 total 50.04 last 82.77) total: 4500 open/close in 52.15 seconds: 86.28 ops/second - /unlink 1247 (time 1787398753.14 total 10.02 last 124.49) - /unlink 2630 (time 1787398763.14 total 20.02 last 138.30) - /unlink 4310 (time 1787398773.15 total 30.03 last 167.77) total: 4500 /unlink in 31.89 seconds: 141.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 782 (time 1787398790.00 total 10.01 last 78.14) - open/close 1652 (time 1787398800.02 total 20.02 last 86.86) - open/close 2451 (time 1787398810.04 total 30.04 last 79.75) - open/close 3304 (time 1787398820.04 total 40.04 last 85.28) - open/close 4104 (time 1787398830.04 total 50.05 last 79.96) total: 4500 open/close in 55.13 seconds: 81.62 ops/second - /unlink 1589 (time 1787398849.50 total 10.01 last 158.80) - /unlink 3248 (time 1787398859.51 total 20.01 last 165.78) - /unlink 4392 (time 1787398869.52 total 30.03 last 114.23) total: 4500 /unlink in 30.95 seconds: 145.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 776 (time 1787398885.24 total 10.01 last 77.51) - open/close 1519 (time 1787398895.25 total 20.02 last 74.22) - open/close 2256 (time 1787398905.26 total 30.03 last 73.64) - open/close 3086 (time 1787398915.26 total 40.04 last 82.95) - open/close 4049 (time 1787398925.27 total 50.04 last 96.24) total: 4500 open/close in 56.20 seconds: 80.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1369 (time 1787398946.71 total 10.00 last 136.87) - /unlink 2759 (time 1787398956.71 total 20.00 last 138.98) - /unlink 4300 (time 1787398966.71 total 30.00 last 154.09) total: 4500 /unlink in 31.71 seconds: 141.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 875 (time 1787398983.50 total 10.00 last 87.49) - open/close 1737 (time 1787398993.54 total 20.04 last 85.87) - open/close 2614 (time 1787399003.54 total 30.04 last 87.67) - open/close 3421 (time 1787399013.56 total 40.06 last 80.54) - open/close 4154 (time 1787399023.57 total 50.07 last 73.23) total: 4500 open/close in 55.99 seconds: 80.38 ops/second - /unlink 1377 (time 1787399045.02 total 10.00 last 137.69) - /unlink 2707 (time 1787399055.02 total 20.01 last 132.92) - /unlink 4268 (time 1787399065.03 total 30.01 last 156.03) total: 4500 /unlink in 31.67 seconds: 142.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 600 (time 1787399082.32 total 10.01 last 59.95) - open/close 1313 (time 1787399092.33 total 20.02 last 71.25) - open/close 2041 (time 1787399102.34 total 30.02 last 72.74) - open/close 2854 (time 1787399112.36 total 40.05 last 81.12) - open/close 3700 (time 1787399122.37 total 50.06 last 84.50) total: 4500 open/close in 58.52 seconds: 76.89 ops/second - /unlink 1559 (time 1787399145.57 total 10.01 last 155.75) - /unlink 2634 (time 1787399155.57 total 20.01 last 107.46) - /unlink 3864 (time 1787399165.57 total 30.02 last 122.96) total: 4500 /unlink in 36.43 seconds: 123.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 769 (time 1787399188.60 total 10.00 last 76.89) - open/close 1653 (time 1787399198.61 total 20.01 last 88.30) - open/close 2245 (time 1787399208.62 total 30.01 last 59.19) - open/close 3051 (time 1787399218.62 total 40.02 last 80.56) - open/close 3808 (time 1787399228.62 total 50.02 last 75.68) total: 4500 open/close in 58.67 seconds: 76.70 ops/second - /unlink 1583 (time 1787399252.24 total 10.00 last 158.29) - /unlink 3062 (time 1787399262.25 total 20.01 last 147.73) - /unlink 4499 (time 1787399272.25 total 30.01 last 143.70) total: 4500 /unlink in 30.01 seconds: 149.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 826 (time 1787399287.25 total 10.00 last 82.57) - open/close 1567 (time 1787399297.26 total 20.01 last 74.04) - open/close 2341 (time 1787399307.27 total 30.02 last 77.36) - open/close 3058 (time 1787399317.27 total 40.02 last 71.69) - open/close 3992 (time 1787399327.28 total 50.03 last 93.29) total: 4500 open/close in 55.58 seconds: 80.97 ops/second - /unlink 1595 (time 1787399347.27 total 10.00 last 159.44) - /unlink 3078 (time 1787399357.27 total 20.00 last 148.29) total: 4500 /unlink in 29.11 seconds: 154.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 808 (time 1787399380.62 total 10.00 last 80.77) - open/close 1718 (time 1787399390.63 total 20.01 last 90.93) - open/close 2665 (time 1787399400.63 total 30.01 last 94.68) - open/close 3538 (time 1787399410.64 total 40.02 last 87.25) - open/close 4439 (time 1787399420.64 total 50.02 last 90.04) total: 4500 open/close in 50.70 seconds: 88.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1709 (time 1787399435.62 total 10.00 last 170.86) - /unlink 3266 (time 1787399445.62 total 20.01 last 155.64) total: 4500 /unlink in 27.23 seconds: 165.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 879 (time 1787399466.17 total 10.00 last 87.87) - open/close 1721 (time 1787399476.20 total 20.04 last 83.93) - open/close 2499 (time 1787399486.20 total 30.04 last 77.80) - open/close 3266 (time 1787399496.20 total 40.04 last 76.69) - open/close 4177 (time 1787399506.21 total 50.04 last 91.05) total: 4500 open/close in 53.91 seconds: 83.47 ops/second - /unlink 1671 (time 1787399524.06 total 10.00 last 167.06) - /unlink 3222 (time 1787399534.06 total 20.01 last 155.05) total: 4500 /unlink in 26.99 seconds: 166.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 834 (time 1787399554.91 total 10.01 last 83.35) - open/close 1758 (time 1787399564.91 total 20.01 last 92.38) - open/close 2733 (time 1787399574.92 total 30.02 last 97.38) - open/close 3583 (time 1787399584.92 total 40.02 last 84.99) total: 4500 open/close in 49.04 seconds: 91.76 ops/second - /unlink 1367 (time 1787399609.15 total 10.00 last 136.66) - /unlink 2806 (time 1787399619.15 total 20.00 last 143.90) - /unlink 4350 (time 1787399629.15 total 30.01 last 154.36) total: 4500 /unlink in 30.91 seconds: 145.56 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94778 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server unloading modules via unload_modules_local on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: modules unloaded.