== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:14:05 (1787390045) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg339-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg339-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 864 (time 1787390227.78 total 10.02 last 86.25) - open/close 1711 (time 1787390237.78 total 20.02 last 84.68) - open/close 2599 (time 1787390247.79 total 30.03 last 88.71) - open/close 3490 (time 1787390257.80 total 40.04 last 89.05) - open/close 4352 (time 1787390267.80 total 50.04 last 86.16) total: 4500 open/close in 51.83 seconds: 86.82 ops/second - /unlink 1341 (time 1787390283.37 total 10.00 last 134.09) - /unlink 2833 (time 1787390293.38 total 20.01 last 149.10) - /unlink 4332 (time 1787390303.38 total 30.01 last 149.89) total: 4500 /unlink in 31.03 seconds: 145.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 888 (time 1787390317.51 total 10.00 last 88.79) - open/close 1685 (time 1787390327.52 total 20.01 last 79.61) - open/close 2574 (time 1787390337.52 total 30.02 last 88.87) - open/close 3395 (time 1787390347.53 total 40.03 last 82.00) - open/close 4299 (time 1787390357.54 total 50.03 last 90.38) total: 4500 open/close in 52.36 seconds: 85.94 ops/second - /unlink 1560 (time 1787390373.45 total 10.00 last 155.96) - /unlink 2952 (time 1787390383.46 total 20.01 last 139.11) - /unlink 4361 (time 1787390393.46 total 30.02 last 140.81) total: 4500 /unlink in 31.00 seconds: 145.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 799 (time 1787390408.83 total 10.01 last 79.86) - open/close 1626 (time 1787390418.84 total 20.01 last 82.65) - open/close 2495 (time 1787390428.84 total 30.01 last 86.88) - open/close 3372 (time 1787390438.86 total 40.03 last 87.57) - open/close 4291 (time 1787390448.86 total 50.03 last 91.89) total: 4500 open/close in 52.25 seconds: 86.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1677 (time 1787390464.17 total 10.00 last 167.64) - /unlink 3189 (time 1787390474.18 total 20.01 last 151.15) total: 4500 /unlink in 28.50 seconds: 157.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 819 (time 1787390496.97 total 10.01 last 81.83) - open/close 1665 (time 1787390506.98 total 20.02 last 84.51) - open/close 2452 (time 1787390517.00 total 30.03 last 78.57) - open/close 3260 (time 1787390527.00 total 40.04 last 80.79) - open/close 3989 (time 1787390537.01 total 50.05 last 72.80) total: 4500 open/close in 57.16 seconds: 78.72 ops/second - /unlink 1685 (time 1787390559.55 total 10.00 last 168.46) - /unlink 3254 (time 1787390569.55 total 20.01 last 156.82) total: 4500 /unlink in 27.87 seconds: 161.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 846 (time 1787390591.73 total 10.03 last 84.34) - open/close 1653 (time 1787390601.74 total 20.04 last 80.66) - open/close 2498 (time 1787390611.74 total 30.04 last 84.46) - open/close 3296 (time 1787390621.75 total 40.05 last 79.75) - open/close 4081 (time 1787390631.76 total 50.06 last 78.39) total: 4500 open/close in 55.83 seconds: 80.60 ops/second - /unlink 1324 (time 1787390651.24 total 10.00 last 132.38) - /unlink 2862 (time 1787390661.25 total 20.01 last 153.66) - /unlink 4135 (time 1787390671.25 total 30.01 last 127.30) total: 4500 /unlink in 32.72 seconds: 137.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 705 (time 1787390688.94 total 10.01 last 70.39) - open/close 1458 (time 1787390698.95 total 20.02 last 75.27) - open/close 2143 (time 1787390708.95 total 30.02 last 68.50) - open/close 2826 (time 1787390718.96 total 40.03 last 68.21) - open/close 3636 (time 1787390728.97 total 50.04 last 80.95) - open/close 4451 (time 1787390738.98 total 60.05 last 81.41) total: 4500 open/close in 60.58 seconds: 74.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1534 (time 1787390752.76 total 10.01 last 153.25) - /unlink 2998 (time 1787390762.77 total 20.02 last 146.21) total: 4500 /unlink in 29.86 seconds: 150.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 681 (time 1787390787.57 total 10.02 last 67.98) - open/close 1367 (time 1787390797.58 total 20.03 last 68.53) - open/close 2082 (time 1787390807.60 total 30.04 last 71.40) - open/close 2931 (time 1787390817.60 total 40.04 last 84.87) - open/close 3678 (time 1787390827.61 total 50.05 last 74.65) - open/close 4488 (time 1787390837.62 total 60.06 last 80.94) total: 4500 open/close in 60.28 seconds: 74.66 ops/second - /unlink 1239 (time 1787390852.79 total 10.00 last 123.85) - /unlink 2686 (time 1787390862.79 total 20.01 last 144.66) - /unlink 4179 (time 1787390872.79 total 30.01 last 149.25) total: 4500 /unlink in 32.26 seconds: 139.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 853 (time 1787390889.24 total 10.01 last 85.24) - open/close 1618 (time 1787390899.25 total 20.02 last 76.39) - open/close 2506 (time 1787390909.26 total 30.03 last 88.74) - open/close 3341 (time 1787390919.26 total 40.03 last 83.49) - open/close 4066 (time 1787390929.30 total 50.07 last 72.23) total: 4500 open/close in 55.95 seconds: 80.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1430 (time 1787390950.10 total 10.00 last 142.93) - /unlink 2888 (time 1787390960.11 total 20.01 last 145.75) - /unlink 4394 (time 1787390970.11 total 30.01 last 150.56) total: 4500 /unlink in 30.84 seconds: 145.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 816 (time 1787390985.06 total 10.01 last 81.49) - open/close 1584 (time 1787390995.07 total 20.02 last 76.75) - open/close 2356 (time 1787391005.09 total 30.04 last 77.04) - open/close 3244 (time 1787391015.11 total 40.06 last 88.67) - open/close 4032 (time 1787391025.12 total 50.07 last 78.69) total: 4500 open/close in 55.99 seconds: 80.37 ops/second - /unlink 1423 (time 1787391045.10 total 10.00 last 142.27) - /unlink 2862 (time 1787391055.10 total 20.01 last 143.82) - /unlink 4372 (time 1787391065.12 total 30.02 last 150.83) total: 4500 /unlink in 31.22 seconds: 144.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 722 (time 1787391080.79 total 10.00 last 72.18) - open/close 1230 (time 1787391090.80 total 20.01 last 50.75) - open/close 1725 (time 1787391100.81 total 30.02 last 49.47) - open/close 2339 (time 1787391110.82 total 40.03 last 61.34) - open/close 3104 (time 1787391120.82 total 50.04 last 76.43) - open/close 3971 (time 1787391130.83 total 60.04 last 86.68) total: 4500 open/close in 66.69 seconds: 67.47 ops/second - /unlink 1530 (time 1787391151.69 total 10.01 last 152.77) - /unlink 2894 (time 1787391161.69 total 20.02 last 136.38) - /unlink 4245 (time 1787391171.69 total 30.02 last 135.10) total: 4500 /unlink in 31.74 seconds: 141.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 825 (time 1787391187.05 total 10.01 last 82.42) - open/close 1658 (time 1787391197.05 total 20.02 last 83.26) - open/close 2541 (time 1787391207.06 total 30.02 last 88.24) - open/close 3430 (time 1787391217.06 total 40.02 last 88.90) - open/close 4318 (time 1787391227.06 total 50.03 last 88.75) total: 4500 open/close in 52.54 seconds: 85.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1117 (time 1787391244.77 total 10.02 last 111.46) - /unlink 2521 (time 1787391254.77 total 20.02 last 140.37) - /unlink 4046 (time 1787391264.78 total 30.03 last 152.46) total: 4500 /unlink in 33.13 seconds: 135.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 897 (time 1787391281.79 total 10.00 last 89.69) - open/close 1777 (time 1787391291.79 total 20.01 last 87.94) - open/close 2584 (time 1787391301.81 total 30.02 last 80.61) - open/close 3454 (time 1787391311.81 total 40.03 last 86.95) - open/close 4289 (time 1787391321.82 total 50.03 last 83.45) total: 4500 open/close in 52.22 seconds: 86.18 ops/second - /unlink 1493 (time 1787391337.70 total 10.00 last 149.28) - /unlink 3066 (time 1787391347.71 total 20.01 last 157.20) - /unlink 4497 (time 1787391357.71 total 30.01 last 143.08) total: 4500 /unlink in 30.03 seconds: 149.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 806 (time 1787391372.07 total 10.00 last 80.57) - open/close 1657 (time 1787391382.08 total 20.02 last 84.99) - open/close 2454 (time 1787391392.09 total 30.02 last 79.65) - open/close 3300 (time 1787391402.09 total 40.03 last 84.54) - open/close 4128 (time 1787391412.10 total 50.03 last 82.77) total: 4500 open/close in 54.38 seconds: 82.75 ops/second - /unlink 1565 (time 1787391430.01 total 10.02 last 156.18) - /unlink 3054 (time 1787391440.01 total 20.02 last 148.87) - /unlink 4342 (time 1787391450.01 total 30.02 last 128.77) total: 4500 /unlink in 31.44 seconds: 143.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 880 (time 1787391465.92 total 10.00 last 87.99) - open/close 1708 (time 1787391475.92 total 20.00 last 82.79) - open/close 2628 (time 1787391485.92 total 30.01 last 91.95) - open/close 3494 (time 1787391495.94 total 40.02 last 86.48) - open/close 4348 (time 1787391505.94 total 50.02 last 85.39) total: 4500 open/close in 51.72 seconds: 87.01 ops/second - /unlink 1414 (time 1787391522.34 total 10.00 last 141.36) - /unlink 3020 (time 1787391532.34 total 20.01 last 160.52) - /unlink 4438 (time 1787391542.35 total 30.01 last 141.72) total: 4500 /unlink in 30.44 seconds: 147.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 867 (time 1787391556.95 total 10.00 last 86.67) - open/close 1735 (time 1787391566.96 total 20.02 last 86.69) - open/close 2602 (time 1787391576.98 total 30.03 last 86.58) - open/close 3424 (time 1787391586.98 total 40.03 last 82.17) - open/close 4286 (time 1787391596.99 total 50.04 last 86.09) total: 4500 open/close in 52.05 seconds: 86.45 ops/second - /unlink 1262 (time 1787391612.91 total 10.01 last 126.13) - /unlink 2803 (time 1787391622.91 total 20.01 last 154.10) - /unlink 4062 (time 1787391632.91 total 30.01 last 125.83) total: 4500 /unlink in 33.57 seconds: 134.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 654 (time 1787391652.31 total 10.02 last 65.30) - open/close 1312 (time 1787391662.31 total 20.02 last 65.79) - open/close 2065 (time 1787391672.32 total 30.03 last 75.23) - open/close 2805 (time 1787391682.33 total 40.04 last 73.92) - open/close 3534 (time 1787391692.34 total 50.04 last 72.86) - open/close 4280 (time 1787391702.36 total 60.06 last 74.45) total: 4500 open/close in 62.52 seconds: 71.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1473 (time 1787391719.01 total 10.00 last 147.29) - /unlink 2683 (time 1787391729.02 total 20.01 last 120.91) - /unlink 4146 (time 1787391739.04 total 30.03 last 145.99) total: 4500 /unlink in 32.96 seconds: 136.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 616 (time 1787391756.79 total 10.02 last 61.45) - open/close 1268 (time 1787391766.79 total 20.03 last 65.17) - open/close 2043 (time 1787391776.79 total 30.03 last 77.47) - open/close 2816 (time 1787391786.80 total 40.03 last 77.27) - open/close 3592 (time 1787391796.81 total 50.05 last 77.51) - open/close 4389 (time 1787391806.81 total 60.05 last 79.67) total: 4500 open/close in 61.37 seconds: 73.33 ops/second - /unlink 1442 (time 1787391822.76 total 10.01 last 144.09) - /unlink 2737 (time 1787391832.76 total 20.01 last 129.43) - /unlink 3798 (time 1787391842.79 total 30.04 last 105.78) total: 4500 /unlink in 36.70 seconds: 122.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 736 (time 1787391864.61 total 10.01 last 73.55) - open/close 1565 (time 1787391874.61 total 20.01 last 82.87) - open/close 2292 (time 1787391884.64 total 30.03 last 72.54) - open/close 3054 (time 1787391894.64 total 40.03 last 76.19) - open/close 3895 (time 1787391904.64 total 50.04 last 84.04) total: 4500 open/close in 58.59 seconds: 76.81 ops/second - /unlink 1387 (time 1787391927.47 total 10.00 last 138.63) - /unlink 2747 (time 1787391937.48 total 20.01 last 135.87) - /unlink 4171 (time 1787391947.48 total 30.02 last 142.29) total: 4500 /unlink in 32.42 seconds: 138.82 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94905 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded.