== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:56:52 (1786039012) Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg120-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg120-server oleg120-server: oleg120-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg120-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg120-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 1003 (time 1786039238.45 total 10.01 last 100.20) - open/close 1806 (time 1786039248.45 total 20.01 last 80.30) - open/close 2608 (time 1786039258.46 total 30.02 last 80.12) - open/close 3376 (time 1786039268.46 total 40.02 last 76.78) - open/close 4274 (time 1786039278.46 total 50.02 last 89.80) total: 4500 open/close in 52.42 seconds: 85.84 ops/second - /unlink 1764 (time 1786039294.59 total 10.00 last 176.37) - /unlink 3103 (time 1786039304.59 total 20.01 last 133.84) total: 4500 /unlink in 28.43 seconds: 158.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 691 (time 1786039329.08 total 10.00 last 69.08) - open/close 1477 (time 1786039339.08 total 20.01 last 78.53) - open/close 2435 (time 1786039349.09 total 30.02 last 95.74) - open/close 3464 (time 1786039359.10 total 40.02 last 102.82) total: 4500 open/close in 49.94 seconds: 90.11 ops/second - /unlink 1710 (time 1786039383.10 total 10.00 last 170.98) - /unlink 3358 (time 1786039393.10 total 20.00 last 164.75) total: 4500 /unlink in 26.61 seconds: 169.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 823 (time 1786039415.15 total 10.02 last 82.13) - open/close 1772 (time 1786039425.16 total 20.02 last 94.87) - open/close 2716 (time 1786039435.16 total 30.03 last 94.37) - open/close 3607 (time 1786039445.16 total 40.03 last 89.06) total: 4500 open/close in 48.74 seconds: 92.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1450 (time 1786039469.08 total 10.00 last 144.93) - /unlink 3025 (time 1786039479.09 total 20.01 last 157.47) total: 4500 /unlink in 28.16 seconds: 159.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 948 (time 1786039503.04 total 10.00 last 94.77) - open/close 1988 (time 1786039513.04 total 20.01 last 103.95) - open/close 3072 (time 1786039523.05 total 30.01 last 108.38) - open/close 4010 (time 1786039533.05 total 40.01 last 93.77) total: 4500 open/close in 44.77 seconds: 100.51 ops/second - /unlink 1727 (time 1786039551.58 total 10.01 last 172.58) - /unlink 3497 (time 1786039561.58 total 20.01 last 176.95) total: 4500 /unlink in 25.21 seconds: 178.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 897 (time 1786039580.82 total 10.00 last 89.66) - open/close 1780 (time 1786039590.82 total 20.01 last 88.27) - open/close 2787 (time 1786039600.82 total 30.01 last 100.69) - open/close 3835 (time 1786039610.83 total 40.01 last 104.75) total: 4500 open/close in 46.36 seconds: 97.07 ops/second - /unlink 1727 (time 1786039632.43 total 10.01 last 172.47) - /unlink 3399 (time 1786039642.44 total 20.02 last 167.13) total: 4500 /unlink in 26.44 seconds: 170.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 874 (time 1786039663.07 total 10.03 last 87.18) - open/close 1799 (time 1786039673.07 total 20.03 last 92.46) - open/close 2779 (time 1786039683.07 total 30.03 last 97.99) - open/close 3779 (time 1786039693.09 total 40.04 last 99.88) total: 4500 open/close in 47.73 seconds: 94.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1710 (time 1786039714.80 total 10.00 last 170.94) - /unlink 3227 (time 1786039724.80 total 20.00 last 151.69) total: 4500 /unlink in 27.62 seconds: 162.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 954 (time 1786039747.14 total 10.00 last 95.39) - open/close 1886 (time 1786039757.14 total 20.00 last 93.18) - open/close 2788 (time 1786039767.15 total 30.01 last 90.13) - open/close 3705 (time 1786039777.16 total 40.02 last 91.67) total: 4500 open/close in 47.67 seconds: 94.41 ops/second - /unlink 1688 (time 1786039798.89 total 10.00 last 168.73) - /unlink 3493 (time 1786039808.90 total 20.02 last 180.29) total: 4500 /unlink in 26.22 seconds: 171.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 739 (time 1786039830.50 total 10.00 last 73.89) - open/close 1550 (time 1786039840.50 total 20.00 last 81.08) - open/close 2500 (time 1786039850.52 total 30.02 last 94.89) - open/close 3395 (time 1786039860.52 total 40.02 last 89.47) - open/close 4356 (time 1786039870.53 total 50.03 last 96.04) total: 4500 open/close in 51.31 seconds: 87.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1617 (time 1786039885.62 total 10.02 last 161.43) - /unlink 3282 (time 1786039895.62 total 20.02 last 166.45) total: 4500 /unlink in 27.60 seconds: 163.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 749 (time 1786039919.68 total 10.02 last 74.78) - open/close 1733 (time 1786039929.69 total 20.02 last 98.33) - open/close 2582 (time 1786039939.69 total 30.03 last 84.84) - open/close 3513 (time 1786039949.72 total 40.05 last 92.90) - open/close 4377 (time 1786039959.72 total 50.05 last 86.37) total: 4500 open/close in 51.75 seconds: 86.96 ops/second - /unlink 1740 (time 1786039976.32 total 10.00 last 173.94) - /unlink 3445 (time 1786039986.32 total 20.01 last 170.47) total: 4500 /unlink in 27.43 seconds: 164.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 807 (time 1786040008.84 total 10.01 last 80.64) - open/close 1764 (time 1786040018.85 total 20.02 last 95.61) - open/close 2692 (time 1786040028.86 total 30.03 last 92.71) - open/close 3758 (time 1786040038.86 total 40.03 last 106.59) total: 4500 open/close in 47.14 seconds: 95.46 ops/second - /unlink 1455 (time 1786040060.58 total 10.00 last 145.44) - /unlink 3289 (time 1786040070.59 total 20.01 last 183.30) total: 4500 /unlink in 27.35 seconds: 164.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 870 (time 1786040092.46 total 10.02 last 86.84) - open/close 1640 (time 1786040102.48 total 20.04 last 76.86) - open/close 2402 (time 1786040112.48 total 30.04 last 76.18) - open/close 3359 (time 1786040122.49 total 40.04 last 95.68) - open/close 4359 (time 1786040132.49 total 50.05 last 99.91) total: 4500 open/close in 51.52 seconds: 87.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1919 (time 1786040147.60 total 10.00 last 191.90) - /unlink 3456 (time 1786040157.61 total 20.01 last 153.61) total: 4500 /unlink in 28.03 seconds: 160.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 920 (time 1786040179.29 total 10.01 last 91.91) - open/close 1635 (time 1786040189.30 total 20.01 last 71.46) - open/close 2538 (time 1786040199.30 total 30.02 last 90.25) - open/close 3405 (time 1786040209.31 total 40.03 last 86.63) - open/close 4114 (time 1786040219.32 total 50.03 last 70.87) total: 4500 open/close in 55.04 seconds: 81.75 ops/second - /unlink 1546 (time 1786040239.69 total 10.01 last 154.48) - /unlink 3084 (time 1786040249.69 total 20.01 last 153.75) total: 4500 /unlink in 29.18 seconds: 154.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 804 (time 1786040273.36 total 10.00 last 80.38) - open/close 1621 (time 1786040283.36 total 20.00 last 81.70) - open/close 2496 (time 1786040293.36 total 30.01 last 87.47) - open/close 3373 (time 1786040303.36 total 40.01 last 87.67) - open/close 4170 (time 1786040313.36 total 50.01 last 79.69) total: 4500 open/close in 53.82 seconds: 83.61 ops/second - /unlink 1413 (time 1786040331.72 total 10.02 last 141.06) - /unlink 2454 (time 1786040341.73 total 20.02 last 104.06) - /unlink 3993 (time 1786040351.73 total 30.02 last 153.85) total: 4500 /unlink in 32.75 seconds: 137.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 887 (time 1786040369.28 total 10.00 last 88.67) - open/close 1745 (time 1786040379.31 total 20.03 last 85.59) - open/close 2552 (time 1786040389.31 total 30.03 last 80.64) - open/close 3376 (time 1786040399.32 total 40.04 last 82.37) - open/close 4303 (time 1786040409.32 total 50.04 last 92.68) total: 4500 open/close in 52.67 seconds: 85.44 ops/second - /unlink 1478 (time 1786040426.30 total 10.00 last 147.74) - /unlink 3115 (time 1786040436.31 total 20.01 last 163.61) - /unlink 4357 (time 1786040446.34 total 30.04 last 123.85) total: 4500 /unlink in 31.67 seconds: 142.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 750 (time 1786040462.64 total 10.00 last 75.00) - open/close 1550 (time 1786040472.64 total 20.00 last 79.97) - open/close 2407 (time 1786040482.66 total 30.02 last 85.53) - open/close 3166 (time 1786040492.67 total 40.03 last 75.87) - open/close 4080 (time 1786040502.67 total 50.03 last 91.34) total: 4500 open/close in 54.78 seconds: 82.15 ops/second - /unlink 1567 (time 1786040522.40 total 10.00 last 156.69) - /unlink 3015 (time 1786040532.41 total 20.01 last 144.73) total: 4500 /unlink in 29.92 seconds: 150.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 764 (time 1786040557.42 total 10.01 last 76.35) - open/close 1534 (time 1786040567.42 total 20.01 last 76.98) - open/close 2326 (time 1786040577.43 total 30.02 last 79.12) - open/close 3270 (time 1786040587.45 total 40.03 last 94.27) - open/close 4073 (time 1786040597.47 total 50.05 last 80.14) total: 4500 open/close in 55.19 seconds: 81.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1542 (time 1786040616.58 total 10.01 last 154.12) - /unlink 3002 (time 1786040626.59 total 20.01 last 145.95) total: 4500 /unlink in 29.76 seconds: 151.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 914 (time 1786040650.82 total 10.00 last 91.36) - open/close 1822 (time 1786040660.82 total 20.01 last 90.76) - open/close 2737 (time 1786040670.83 total 30.02 last 91.38) - open/close 3485 (time 1786040680.84 total 40.03 last 74.76) - open/close 4335 (time 1786040690.84 total 50.03 last 84.97) total: 4500 open/close in 51.95 seconds: 86.62 ops/second - /unlink 1336 (time 1786040707.01 total 10.01 last 133.50) - /unlink 2851 (time 1786040717.01 total 20.01 last 151.50) - /unlink 4242 (time 1786040727.01 total 30.01 last 139.07) total: 4500 /unlink in 31.67 seconds: 142.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 869 (time 1786040742.76 total 10.00 last 86.89) - open/close 1744 (time 1786040752.77 total 20.01 last 87.44) - open/close 2629 (time 1786040762.78 total 30.02 last 88.45) - open/close 3472 (time 1786040772.78 total 40.02 last 84.23) - open/close 4334 (time 1786040782.79 total 50.03 last 86.14) total: 4500 open/close in 51.96 seconds: 86.60 ops/second - /unlink 1759 (time 1786040798.72 total 10.00 last 175.82) - /unlink 3237 (time 1786040808.72 total 20.01 last 147.77) total: 4500 /unlink in 29.72 seconds: 151.43 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95236 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: modules unloaded.