== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:24:15 (1787822655) Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg241-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server oleg241-server: oleg241-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg241-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg241-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg241-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 729 (time 1787822865.00 total 10.00 last 72.88) - open/close 1426 (time 1787822875.00 total 20.01 last 69.66) - open/close 2233 (time 1787822885.02 total 30.02 last 80.60) - open/close 3021 (time 1787822895.03 total 40.03 last 78.73) - open/close 3761 (time 1787822905.04 total 50.04 last 73.89) total: 4500 open/close in 59.33 seconds: 75.84 ops/second - /unlink 1543 (time 1787822929.16 total 10.00 last 154.30) - /unlink 2977 (time 1787822939.16 total 20.00 last 143.38) - /unlink 4232 (time 1787822949.17 total 30.01 last 125.44) total: 4500 /unlink in 32.98 seconds: 136.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 647 (time 1787822969.46 total 10.00 last 64.69) - open/close 1326 (time 1787822979.46 total 20.00 last 67.88) - open/close 2108 (time 1787822989.49 total 30.03 last 78.02) - open/close 2941 (time 1787822999.49 total 40.03 last 83.25) - open/close 3628 (time 1787823009.50 total 50.04 last 68.66) - open/close 4443 (time 1787823019.50 total 60.04 last 81.47) total: 4500 open/close in 60.75 seconds: 74.08 ops/second - /unlink 1412 (time 1787823034.75 total 10.00 last 141.14) - /unlink 2782 (time 1787823044.76 total 20.01 last 136.88) - /unlink 4190 (time 1787823054.76 total 30.01 last 140.77) total: 4500 /unlink in 32.59 seconds: 138.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 645 (time 1787823072.79 total 10.02 last 64.36) - open/close 1439 (time 1787823082.80 total 20.03 last 79.29) - open/close 2126 (time 1787823092.80 total 30.04 last 68.70) - open/close 2878 (time 1787823102.81 total 40.05 last 75.12) - open/close 3596 (time 1787823112.81 total 50.05 last 71.80) - open/close 4424 (time 1787823122.83 total 60.06 last 82.71) total: 4500 open/close in 60.91 seconds: 73.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1346 (time 1787823138.43 total 10.00 last 134.59) - /unlink 2400 (time 1787823148.44 total 20.00 last 105.36) - /unlink 3530 (time 1787823158.45 total 30.02 last 112.86) total: 4500 /unlink in 38.77 seconds: 116.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 614 (time 1787823184.23 total 10.00 last 61.38) - open/close 1316 (time 1787823194.24 total 20.01 last 70.14) - open/close 1976 (time 1787823204.24 total 30.01 last 65.99) - open/close 2605 (time 1787823214.25 total 40.03 last 62.81) - open/close 3412 (time 1787823224.26 total 50.03 last 80.67) - open/close 4179 (time 1787823234.27 total 60.04 last 76.60) total: 4500 open/close in 64.60 seconds: 69.66 ops/second - /unlink 1319 (time 1787823253.29 total 10.00 last 131.85) - /unlink 2478 (time 1787823263.29 total 20.01 last 115.87) - /unlink 3319 (time 1787823273.30 total 30.02 last 84.03) - /unlink 4388 (time 1787823283.31 total 40.02 last 106.85) total: 4500 /unlink in 41.47 seconds: 108.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 689 (time 1787823299.76 total 10.00 last 68.88) - open/close 1395 (time 1787823309.76 total 20.01 last 70.58) - open/close 2088 (time 1787823319.77 total 30.01 last 69.26) - open/close 2735 (time 1787823329.77 total 40.02 last 64.64) - open/close 3427 (time 1787823339.78 total 50.02 last 69.17) - open/close 4058 (time 1787823349.81 total 60.05 last 62.93) total: 4500 open/close in 67.62 seconds: 66.55 ops/second - /unlink 1100 (time 1787823372.42 total 10.01 last 109.87) - /unlink 2415 (time 1787823382.42 total 20.01 last 131.50) - /unlink 3696 (time 1787823392.42 total 30.01 last 128.07) total: 4500 /unlink in 35.28 seconds: 127.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 842 (time 1787823412.61 total 10.00 last 84.19) - open/close 1611 (time 1787823422.61 total 20.00 last 76.88) - open/close 2382 (time 1787823432.62 total 30.01 last 77.04) - open/close 3210 (time 1787823442.62 total 40.01 last 82.78) - open/close 3988 (time 1787823452.64 total 50.03 last 77.69) total: 4500 open/close in 56.62 seconds: 79.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1449 (time 1787823474.37 total 10.00 last 144.86) - /unlink 2610 (time 1787823484.37 total 20.01 last 116.06) - /unlink 3920 (time 1787823494.38 total 30.01 last 130.93) total: 4500 /unlink in 34.44 seconds: 130.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 786 (time 1787823514.46 total 10.01 last 78.52) - open/close 1483 (time 1787823524.46 total 20.02 last 69.66) - open/close 2146 (time 1787823534.47 total 30.02 last 66.27) - open/close 2767 (time 1787823544.48 total 40.03 last 62.03) - open/close 3436 (time 1787823554.48 total 50.03 last 66.90) - open/close 4087 (time 1787823564.49 total 60.04 last 65.06) total: 4500 open/close in 65.50 seconds: 68.70 ops/second - /unlink 1109 (time 1787823585.88 total 10.01 last 110.77) - /unlink 2335 (time 1787823595.88 total 20.02 last 122.54) - /unlink 3740 (time 1787823605.88 total 30.02 last 140.49) total: 4500 /unlink in 35.47 seconds: 126.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 649 (time 1787823626.15 total 10.03 last 64.72) - open/close 1262 (time 1787823636.16 total 20.04 last 61.22) - open/close 1963 (time 1787823646.16 total 30.05 last 70.06) - open/close 2499 (time 1787823656.18 total 40.06 last 53.54) - open/close 3202 (time 1787823666.20 total 50.08 last 70.17) - open/close 3878 (time 1787823676.20 total 60.08 last 67.56) total: 4500 open/close in 69.26 seconds: 64.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1320 (time 1787823700.45 total 10.00 last 131.97) - /unlink 2531 (time 1787823710.45 total 20.01 last 121.04) - /unlink 3563 (time 1787823720.45 total 30.01 last 103.17) total: 4500 /unlink in 36.96 seconds: 121.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 777 (time 1787823742.56 total 10.02 last 77.54) - open/close 1607 (time 1787823752.57 total 20.03 last 82.96) - open/close 2382 (time 1787823762.57 total 30.03 last 77.49) - open/close 3020 (time 1787823772.58 total 40.04 last 63.75) - open/close 3710 (time 1787823782.58 total 50.04 last 68.95) - open/close 4450 (time 1787823792.59 total 60.05 last 73.97) total: 4500 open/close in 60.65 seconds: 74.19 ops/second - /unlink 1006 (time 1787823810.22 total 10.01 last 100.54) - /unlink 2097 (time 1787823820.24 total 20.02 last 108.93) - /unlink 3246 (time 1787823830.24 total 30.02 last 114.89) total: 4500 /unlink in 39.50 seconds: 113.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 620 (time 1787823855.48 total 10.00 last 61.99) - open/close 1273 (time 1787823865.49 total 20.01 last 65.23) - open/close 1959 (time 1787823875.50 total 30.02 last 68.54) - open/close 2670 (time 1787823885.50 total 40.02 last 71.10) - open/close 3471 (time 1787823895.50 total 50.02 last 80.09) - open/close 4367 (time 1787823905.50 total 60.03 last 89.57) total: 4500 open/close in 61.59 seconds: 73.06 ops/second - /unlink 1541 (time 1787823920.97 total 10.00 last 154.03) - /unlink 2719 (time 1787823930.97 total 20.01 last 117.71) - /unlink 4091 (time 1787823940.98 total 30.02 last 137.10) total: 4500 /unlink in 33.47 seconds: 134.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 719 (time 1787823959.46 total 10.01 last 71.85) - open/close 1361 (time 1787823969.47 total 20.02 last 64.13) - open/close 1927 (time 1787823979.47 total 30.02 last 56.56) - open/close 2656 (time 1787823989.48 total 40.03 last 72.87) - open/close 3395 (time 1787823999.49 total 50.04 last 73.81) - open/close 4023 (time 1787824009.50 total 60.05 last 62.76) total: 4500 open/close in 67.22 seconds: 66.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1536 (time 1787824032.30 total 10.01 last 153.40) - /unlink 2974 (time 1787824042.31 total 20.02 last 143.72) - /unlink 4393 (time 1787824052.31 total 30.02 last 141.87) total: 4500 /unlink in 30.76 seconds: 146.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 730 (time 1787824067.58 total 10.00 last 72.97) - open/close 1456 (time 1787824077.60 total 20.02 last 72.50) - open/close 2132 (time 1787824087.61 total 30.03 last 67.54) - open/close 2810 (time 1787824097.62 total 40.05 last 67.67) - open/close 3386 (time 1787824107.64 total 50.06 last 57.50) - open/close 4123 (time 1787824117.64 total 60.06 last 73.69) total: 4500 open/close in 66.78 seconds: 67.38 ops/second - /unlink 1472 (time 1787824139.43 total 10.00 last 147.18) - /unlink 2777 (time 1787824149.44 total 20.01 last 130.38) - /unlink 4025 (time 1787824159.45 total 30.02 last 124.71) total: 4500 /unlink in 33.32 seconds: 135.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 840 (time 1787824178.22 total 10.01 last 83.93) - open/close 1693 (time 1787824188.23 total 20.03 last 85.15) - open/close 2436 (time 1787824198.25 total 30.04 last 74.19) - open/close 3083 (time 1787824208.25 total 40.04 last 64.69) - open/close 3743 (time 1787824218.25 total 50.05 last 65.96) total: 4500 open/close in 59.58 seconds: 75.52 ops/second - /unlink 1504 (time 1787824243.08 total 10.00 last 150.37) - /unlink 2886 (time 1787824253.09 total 20.01 last 138.05) - /unlink 4301 (time 1787824263.10 total 30.02 last 141.40) total: 4500 /unlink in 31.46 seconds: 143.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 697 (time 1787824279.18 total 10.01 last 69.64) - open/close 1468 (time 1787824289.18 total 20.01 last 77.07) - open/close 2117 (time 1787824299.19 total 30.02 last 64.85) - open/close 2759 (time 1787824309.19 total 40.02 last 64.18) - open/close 3426 (time 1787824319.20 total 50.03 last 66.66) - open/close 4171 (time 1787824329.20 total 60.03 last 74.49) total: 4500 open/close in 64.88 seconds: 69.36 ops/second - /unlink 1145 (time 1787824349.04 total 10.00 last 114.48) - /unlink 2344 (time 1787824359.05 total 20.01 last 119.79) - /unlink 3945 (time 1787824369.06 total 30.02 last 160.02) total: 4500 /unlink in 34.63 seconds: 129.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 685 (time 1787824388.69 total 10.01 last 68.44) - open/close 1408 (time 1787824398.69 total 20.01 last 72.27) - open/close 2223 (time 1787824408.71 total 30.02 last 81.40) - open/close 3024 (time 1787824418.71 total 40.03 last 80.06) - open/close 3717 (time 1787824428.71 total 50.03 last 69.28) - open/close 4456 (time 1787824438.72 total 60.04 last 73.84) total: 4500 open/close in 60.64 seconds: 74.21 ops/second - /unlink 1787 (time 1787824453.04 total 10.00 last 178.67) - /unlink 3182 (time 1787824463.04 total 20.00 last 139.47) total: 4500 /unlink in 29.14 seconds: 154.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 831 (time 1787824487.22 total 10.00 last 83.06) - open/close 1572 (time 1787824497.23 total 20.02 last 74.02) - open/close 2350 (time 1787824507.24 total 30.02 last 77.74) - open/close 3110 (time 1787824517.25 total 40.04 last 75.86) - open/close 3876 (time 1787824527.26 total 50.05 last 76.56) total: 4500 open/close in 58.24 seconds: 77.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1633 (time 1787824550.22 total 10.00 last 163.27) - /unlink 3117 (time 1787824560.23 total 20.01 last 148.33) total: 4500 /unlink in 29.34 seconds: 153.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 727 (time 1787824583.76 total 10.01 last 72.62) - open/close 1455 (time 1787824593.77 total 20.02 last 72.75) - open/close 2256 (time 1787824603.78 total 30.03 last 79.99) - open/close 2917 (time 1787824613.79 total 40.04 last 66.08) - open/close 3653 (time 1787824623.80 total 50.05 last 73.51) - open/close 4311 (time 1787824633.80 total 60.05 last 65.76) total: 4500 open/close in 62.94 seconds: 71.50 ops/second - /unlink 1374 (time 1787824651.22 total 10.00 last 137.35) - /unlink 2701 (time 1787824661.22 total 20.01 last 132.62) - /unlink 4160 (time 1787824671.23 total 30.01 last 145.84) total: 4500 /unlink in 33.26 seconds: 135.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 841 (time 1787824689.77 total 10.00 last 84.09) - open/close 1683 (time 1787824699.80 total 20.03 last 83.97) - open/close 2543 (time 1787824709.80 total 30.03 last 85.98) - open/close 3317 (time 1787824719.81 total 40.04 last 77.34) - open/close 4091 (time 1787824729.82 total 50.04 last 77.37) total: 4500 open/close in 55.35 seconds: 81.31 ops/second - /unlink 1434 (time 1787824749.33 total 10.00 last 143.35) - /unlink 2958 (time 1787824759.33 total 20.01 last 152.37) - /unlink 4307 (time 1787824769.34 total 30.01 last 134.83) total: 4500 /unlink in 31.57 seconds: 142.53 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95243 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg241-server unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local oleg241-server: modules unloaded.