== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:14:32 (1787796872) Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg438-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg438-server oleg438-server: oleg438-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg438-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg438-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg438-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 841 (time 1787797083.61 total 10.03 last 83.87) - open/close 1419 (time 1787797093.61 total 20.03 last 57.77) - open/close 2043 (time 1787797103.62 total 30.04 last 62.33) - open/close 2764 (time 1787797113.63 total 40.05 last 72.07) - open/close 3522 (time 1787797123.64 total 50.05 last 75.76) - open/close 4309 (time 1787797133.64 total 60.06 last 78.64) total: 4500 open/close in 62.50 seconds: 72.00 ops/second - /unlink 1534 (time 1787797150.34 total 10.00 last 153.34) - /unlink 2785 (time 1787797160.34 total 20.01 last 125.08) - /unlink 4306 (time 1787797170.35 total 30.01 last 152.04) total: 4500 /unlink in 31.41 seconds: 143.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 625 (time 1787797187.78 total 10.00 last 62.48) - open/close 1294 (time 1787797197.78 total 20.01 last 66.88) - open/close 1932 (time 1787797207.79 total 30.02 last 63.74) - open/close 2770 (time 1787797217.80 total 40.02 last 83.79) - open/close 3535 (time 1787797227.80 total 50.02 last 76.46) - open/close 4301 (time 1787797237.80 total 60.02 last 76.59) total: 4500 open/close in 62.30 seconds: 72.23 ops/second - /unlink 1196 (time 1787797255.31 total 10.00 last 119.60) - /unlink 2734 (time 1787797265.32 total 20.00 last 153.80) - /unlink 4217 (time 1787797275.32 total 30.00 last 148.23) total: 4500 /unlink in 33.09 seconds: 135.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 723 (time 1787797293.97 total 10.03 last 72.09) - open/close 1465 (time 1787797303.97 total 20.03 last 74.15) - open/close 2095 (time 1787797313.98 total 30.04 last 62.96) - open/close 2796 (time 1787797324.01 total 40.07 last 69.89) - open/close 3597 (time 1787797334.01 total 50.07 last 80.07) - open/close 4272 (time 1787797344.02 total 60.08 last 67.47) total: 4500 open/close in 62.65 seconds: 71.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1587 (time 1787797360.66 total 10.01 last 158.60) - /unlink 2862 (time 1787797370.66 total 20.01 last 127.47) - /unlink 4165 (time 1787797380.66 total 30.01 last 130.26) total: 4500 /unlink in 32.03 seconds: 140.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 803 (time 1787797397.79 total 10.01 last 80.22) - open/close 1560 (time 1787797407.79 total 20.02 last 75.66) - open/close 2370 (time 1787797417.81 total 30.03 last 80.91) - open/close 3204 (time 1787797427.81 total 40.03 last 83.39) - open/close 4073 (time 1787797437.81 total 50.03 last 86.86) total: 4500 open/close in 55.54 seconds: 81.02 ops/second - /unlink 1475 (time 1787797457.94 total 10.00 last 147.46) - /unlink 2990 (time 1787797467.95 total 20.01 last 151.39) - /unlink 4425 (time 1787797477.96 total 30.02 last 143.39) total: 4500 /unlink in 30.76 seconds: 146.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 818 (time 1787797493.46 total 10.02 last 81.63) - open/close 1412 (time 1787797503.46 total 20.02 last 59.40) - open/close 2127 (time 1787797513.46 total 30.03 last 71.47) - open/close 2758 (time 1787797523.47 total 40.03 last 63.05) - open/close 3457 (time 1787797533.48 total 50.04 last 69.82) - open/close 4034 (time 1787797543.50 total 60.06 last 57.61) total: 4500 open/close in 66.15 seconds: 68.03 ops/second - /unlink 1385 (time 1787797564.50 total 10.00 last 138.48) - /unlink 2732 (time 1787797574.50 total 20.00 last 134.69) - /unlink 4228 (time 1787797584.50 total 30.00 last 149.60) total: 4500 /unlink in 31.95 seconds: 140.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 782 (time 1787797601.51 total 10.02 last 78.07) - open/close 1560 (time 1787797611.53 total 20.04 last 77.65) - open/close 2385 (time 1787797621.53 total 30.04 last 82.48) - open/close 3200 (time 1787797631.54 total 40.05 last 81.40) - open/close 3995 (time 1787797641.57 total 50.08 last 79.27) total: 4500 open/close in 55.91 seconds: 80.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1446 (time 1787797661.69 total 10.00 last 144.55) - /unlink 2970 (time 1787797671.69 total 20.00 last 152.39) - /unlink 4465 (time 1787797681.70 total 30.02 last 149.29) total: 4500 /unlink in 30.70 seconds: 146.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 820 (time 1787797697.29 total 10.00 last 81.97) - open/close 1519 (time 1787797707.29 total 20.01 last 69.87) - open/close 2336 (time 1787797717.30 total 30.01 last 81.65) - open/close 3131 (time 1787797727.31 total 40.02 last 79.44) - open/close 3972 (time 1787797737.31 total 50.02 last 84.10) total: 4500 open/close in 56.05 seconds: 80.29 ops/second - /unlink 1557 (time 1787797757.64 total 10.00 last 155.69) - /unlink 3034 (time 1787797767.64 total 20.01 last 147.60) total: 4500 /unlink in 29.59 seconds: 152.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 795 (time 1787797792.51 total 10.00 last 79.47) - open/close 1574 (time 1787797802.51 total 20.00 last 77.89) - open/close 2389 (time 1787797812.51 total 30.01 last 81.49) - open/close 3204 (time 1787797822.52 total 40.01 last 81.49) - open/close 4022 (time 1787797832.52 total 50.01 last 81.80) total: 4500 open/close in 55.61 seconds: 80.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1422 (time 1787797852.76 total 10.01 last 142.12) - /unlink 2878 (time 1787797862.77 total 20.02 last 145.41) - /unlink 4360 (time 1787797872.77 total 30.02 last 148.18) total: 4500 /unlink in 31.13 seconds: 144.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 715 (time 1787797888.27 total 10.02 last 71.39) - open/close 1513 (time 1787797898.27 total 20.02 last 79.80) - open/close 2325 (time 1787797908.28 total 30.03 last 81.09) - open/close 3067 (time 1787797918.29 total 40.03 last 74.17) - open/close 3790 (time 1787797928.29 total 50.04 last 72.27) total: 4500 open/close in 58.59 seconds: 76.81 ops/second - /unlink 1549 (time 1787797951.01 total 10.00 last 154.85) - /unlink 3064 (time 1787797961.02 total 20.01 last 151.43) total: 4500 /unlink in 29.01 seconds: 155.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 743 (time 1787797985.27 total 10.00 last 74.27) - open/close 1516 (time 1787797995.27 total 20.01 last 77.23) - open/close 2355 (time 1787798005.29 total 30.03 last 83.81) - open/close 3173 (time 1787798015.29 total 40.03 last 81.80) - open/close 4067 (time 1787798025.29 total 50.03 last 89.37) total: 4500 open/close in 55.20 seconds: 81.53 ops/second - /unlink 1502 (time 1787798046.34 total 10.00 last 150.16) - /unlink 2942 (time 1787798056.34 total 20.01 last 143.94) - /unlink 4460 (time 1787798066.34 total 30.01 last 151.77) total: 4500 /unlink in 30.54 seconds: 147.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 810 (time 1787798081.73 total 10.01 last 80.94) - open/close 1585 (time 1787798091.74 total 20.02 last 77.37) - open/close 2418 (time 1787798101.75 total 30.03 last 83.22) - open/close 3243 (time 1787798111.76 total 40.04 last 82.46) - open/close 4038 (time 1787798121.77 total 50.05 last 79.42) total: 4500 open/close in 57.07 seconds: 78.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1664 (time 1787798143.72 total 10.00 last 166.40) - /unlink 3174 (time 1787798153.72 total 20.00 last 150.96) total: 4500 /unlink in 28.64 seconds: 157.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 863 (time 1787798177.62 total 10.01 last 86.18) - open/close 1664 (time 1787798187.63 total 20.02 last 80.05) - open/close 2389 (time 1787798197.64 total 30.03 last 72.46) - open/close 3083 (time 1787798207.64 total 40.03 last 69.37) - open/close 3980 (time 1787798217.64 total 50.03 last 89.66) total: 4500 open/close in 56.48 seconds: 79.68 ops/second - /unlink 1411 (time 1787798238.72 total 10.00 last 141.09) - /unlink 2959 (time 1787798248.72 total 20.00 last 154.77) - /unlink 4314 (time 1787798258.73 total 30.01 last 135.43) total: 4500 /unlink in 31.52 seconds: 142.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 808 (time 1787798276.03 total 10.00 last 80.77) - open/close 1579 (time 1787798286.04 total 20.01 last 77.03) - open/close 2327 (time 1787798296.04 total 30.02 last 74.78) - open/close 3123 (time 1787798306.06 total 40.04 last 79.43) - open/close 3897 (time 1787798316.06 total 50.04 last 77.38) total: 4500 open/close in 56.99 seconds: 78.96 ops/second - /unlink 1556 (time 1787798337.26 total 10.00 last 155.55) - /unlink 2925 (time 1787798347.26 total 20.01 last 136.86) - /unlink 4441 (time 1787798357.26 total 30.01 last 151.60) total: 4500 /unlink in 30.54 seconds: 147.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 845 (time 1787798372.62 total 10.02 last 84.34) - open/close 1708 (time 1787798382.62 total 20.02 last 86.26) - open/close 2592 (time 1787798392.65 total 30.05 last 88.18) - open/close 3431 (time 1787798402.65 total 40.05 last 83.88) - open/close 4335 (time 1787798412.65 total 50.05 last 90.38) total: 4500 open/close in 52.06 seconds: 86.44 ops/second - /unlink 1655 (time 1787798429.25 total 10.02 last 165.19) - /unlink 3048 (time 1787798439.25 total 20.02 last 139.28) total: 4500 /unlink in 29.50 seconds: 152.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 822 (time 1787798463.05 total 10.01 last 82.14) - open/close 1705 (time 1787798473.05 total 20.01 last 88.27) - open/close 2543 (time 1787798483.06 total 30.02 last 83.75) - open/close 3380 (time 1787798493.06 total 40.02 last 83.69) - open/close 4215 (time 1787798503.08 total 50.03 last 83.36) total: 4500 open/close in 54.25 seconds: 82.94 ops/second - /unlink 1510 (time 1787798521.53 total 10.00 last 150.98) - /unlink 2785 (time 1787798531.54 total 20.01 last 127.35) - /unlink 4115 (time 1787798541.54 total 30.02 last 132.96) total: 4500 /unlink in 32.90 seconds: 136.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 767 (time 1787798559.41 total 10.01 last 76.66) - open/close 1392 (time 1787798569.42 total 20.02 last 62.44) - open/close 2067 (time 1787798579.42 total 30.02 last 67.49) - open/close 2835 (time 1787798589.43 total 40.02 last 76.77) - open/close 3614 (time 1787798599.44 total 50.03 last 77.82) - open/close 4409 (time 1787798609.45 total 60.04 last 79.43) total: 4500 open/close in 61.08 seconds: 73.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1252 (time 1787798625.13 total 10.01 last 125.04) - /unlink 2719 (time 1787798635.13 total 20.02 last 146.66) - /unlink 4282 (time 1787798645.13 total 30.02 last 156.28) total: 4500 /unlink in 31.81 seconds: 141.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 835 (time 1787798661.30 total 10.00 last 83.49) - open/close 1692 (time 1787798671.31 total 20.01 last 85.61) - open/close 2579 (time 1787798681.32 total 30.02 last 88.61) - open/close 3436 (time 1787798691.33 total 40.03 last 85.59) - open/close 4275 (time 1787798701.33 total 50.04 last 83.89) total: 4500 open/close in 53.14 seconds: 84.68 ops/second - /unlink 1529 (time 1787798718.95 total 10.00 last 152.88) - /unlink 2911 (time 1787798728.95 total 20.00 last 138.15) - /unlink 4369 (time 1787798738.96 total 30.02 last 145.65) total: 4500 /unlink in 31.02 seconds: 145.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 634 (time 1787798754.14 total 10.02 last 63.27) - open/close 1273 (time 1787798764.16 total 20.04 last 63.75) - open/close 1939 (time 1787798774.17 total 30.05 last 66.58) - open/close 2541 (time 1787798784.17 total 40.05 last 60.19) - open/close 3287 (time 1787798794.18 total 50.06 last 74.52) - open/close 4080 (time 1787798804.19 total 60.07 last 79.26) total: 4500 open/close in 65.37 seconds: 68.84 ops/second - /unlink 1440 (time 1787798824.39 total 10.01 last 143.86) - /unlink 2895 (time 1787798834.39 total 20.01 last 145.47) - /unlink 4122 (time 1787798844.40 total 30.02 last 122.59) total: 4500 /unlink in 32.73 seconds: 137.48 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95243 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg438-server unloading modules via unload_modules_local on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local oleg438-server: modules unloaded.