== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:43:51 (1787701431) Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg344-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg344-server oleg344-server: oleg344-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg344-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg344-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg344-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 723 (time 1787701645.33 total 10.01 last 72.21) - open/close 1361 (time 1787701655.34 total 20.02 last 63.74) - open/close 1883 (time 1787701665.36 total 30.04 last 52.10) - open/close 2469 (time 1787701675.38 total 40.06 last 58.50) - open/close 3073 (time 1787701685.38 total 50.06 last 60.38) - open/close 3824 (time 1787701695.39 total 60.07 last 75.05) total: 4500 open/close in 68.37 seconds: 65.82 ops/second - /unlink 1418 (time 1787701718.57 total 10.00 last 141.77) - /unlink 2681 (time 1787701728.58 total 20.00 last 126.29) - /unlink 4007 (time 1787701738.59 total 30.01 last 132.47) total: 4500 /unlink in 33.52 seconds: 134.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 702 (time 1787701757.85 total 10.00 last 70.17) - open/close 1426 (time 1787701767.85 total 20.01 last 72.39) - open/close 2382 (time 1787701777.86 total 30.01 last 95.54) - open/close 3280 (time 1787701787.86 total 40.02 last 89.77) - open/close 4078 (time 1787701797.87 total 50.02 last 79.73) total: 4500 open/close in 56.50 seconds: 79.64 ops/second - /unlink 1555 (time 1787701819.73 total 10.00 last 155.44) - /unlink 2759 (time 1787701829.74 total 20.01 last 120.35) - /unlink 4197 (time 1787701839.74 total 30.02 last 143.69) total: 4500 /unlink in 31.98 seconds: 140.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 837 (time 1787701855.85 total 10.01 last 83.65) - open/close 1567 (time 1787701865.89 total 20.05 last 72.69) - open/close 2447 (time 1787701875.91 total 30.07 last 87.84) - open/close 3145 (time 1787701885.92 total 40.07 last 69.74) - open/close 3916 (time 1787701895.92 total 50.08 last 77.05) total: 4500 open/close in 57.14 seconds: 78.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1641 (time 1787701917.46 total 10.00 last 164.06) - /unlink 3202 (time 1787701927.46 total 20.00 last 156.07) total: 4500 /unlink in 27.93 seconds: 161.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 793 (time 1787701950.08 total 10.01 last 79.22) - open/close 1671 (time 1787701960.08 total 20.01 last 87.78) - open/close 2533 (time 1787701970.08 total 30.01 last 86.20) - open/close 3317 (time 1787701980.09 total 40.02 last 78.31) - open/close 3982 (time 1787701990.10 total 50.03 last 66.48) total: 4500 open/close in 58.66 seconds: 76.71 ops/second - /unlink 1501 (time 1787702015.80 total 10.00 last 150.08) - /unlink 2950 (time 1787702025.80 total 20.00 last 144.87) - /unlink 4351 (time 1787702035.81 total 30.01 last 140.06) total: 4500 /unlink in 31.17 seconds: 144.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 835 (time 1787702051.22 total 10.00 last 83.49) - open/close 1715 (time 1787702061.22 total 20.01 last 87.94) - open/close 2520 (time 1787702071.24 total 30.02 last 80.40) - open/close 3321 (time 1787702081.24 total 40.03 last 80.05) - open/close 4205 (time 1787702091.25 total 50.03 last 88.34) total: 4500 open/close in 53.66 seconds: 83.86 ops/second - /unlink 1635 (time 1787702109.10 total 10.00 last 163.49) - /unlink 3208 (time 1787702119.10 total 20.00 last 157.25) total: 4500 /unlink in 27.90 seconds: 161.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 781 (time 1787702142.59 total 10.01 last 78.04) - open/close 1416 (time 1787702152.60 total 20.02 last 63.40) - open/close 2243 (time 1787702162.61 total 30.03 last 82.69) - open/close 3096 (time 1787702172.61 total 40.03 last 85.26) - open/close 3887 (time 1787702182.61 total 50.03 last 79.08) total: 4500 open/close in 57.91 seconds: 77.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1556 (time 1787702204.80 total 10.00 last 155.55) - /unlink 3285 (time 1787702214.80 total 20.01 last 172.87) total: 4500 /unlink in 26.88 seconds: 167.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 716 (time 1787702236.59 total 10.01 last 71.53) - open/close 1380 (time 1787702246.62 total 20.03 last 66.24) - open/close 2013 (time 1787702256.63 total 30.04 last 63.24) - open/close 2717 (time 1787702266.63 total 40.05 last 70.38) - open/close 3468 (time 1787702276.65 total 50.07 last 74.95) - open/close 4184 (time 1787702286.65 total 60.07 last 71.57) total: 4500 open/close in 64.12 seconds: 70.18 ops/second - /unlink 1583 (time 1787702305.48 total 10.00 last 158.26) - /unlink 3180 (time 1787702315.49 total 20.00 last 159.68) total: 4500 /unlink in 28.84 seconds: 156.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 752 (time 1787702339.52 total 10.01 last 75.13) - open/close 1375 (time 1787702349.53 total 20.01 last 62.27) - open/close 2192 (time 1787702359.53 total 30.02 last 81.66) - open/close 2986 (time 1787702369.54 total 40.02 last 79.38) - open/close 3729 (time 1787702379.54 total 50.02 last 74.29) total: 4500 open/close in 59.57 seconds: 75.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1535 (time 1787702403.61 total 10.00 last 153.46) - /unlink 3066 (time 1787702413.61 total 20.01 last 153.03) total: 4500 /unlink in 29.02 seconds: 155.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 752 (time 1787702438.30 total 10.01 last 75.10) - open/close 1325 (time 1787702448.30 total 20.02 last 57.28) - open/close 2210 (time 1787702458.31 total 30.02 last 88.49) - open/close 3045 (time 1787702468.31 total 40.02 last 83.47) - open/close 3924 (time 1787702478.32 total 50.03 last 87.83) total: 4500 open/close in 57.31 seconds: 78.52 ops/second - /unlink 1100 (time 1787702499.89 total 10.01 last 109.91) - /unlink 2441 (time 1787702509.89 total 20.01 last 134.07) - /unlink 3958 (time 1787702519.90 total 30.02 last 151.57) total: 4500 /unlink in 33.98 seconds: 132.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 596 (time 1787702538.74 total 10.00 last 59.58) - open/close 1340 (time 1787702548.75 total 20.01 last 74.32) - open/close 2160 (time 1787702558.76 total 30.02 last 81.97) - open/close 2932 (time 1787702568.76 total 40.02 last 77.19) - open/close 3693 (time 1787702578.76 total 50.02 last 76.07) total: 4500 open/close in 59.39 seconds: 75.77 ops/second - /unlink 1429 (time 1787702602.97 total 10.01 last 142.80) - /unlink 3074 (time 1787702612.97 total 20.01 last 164.46) total: 4500 /unlink in 29.33 seconds: 153.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 829 (time 1787702636.81 total 10.02 last 82.77) - open/close 1643 (time 1787702646.81 total 20.02 last 81.38) - open/close 2426 (time 1787702656.82 total 30.03 last 78.24) - open/close 3292 (time 1787702666.83 total 40.04 last 86.51) - open/close 4111 (time 1787702676.84 total 50.05 last 81.80) total: 4500 open/close in 54.84 seconds: 82.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1383 (time 1787702696.48 total 10.00 last 138.26) - /unlink 2850 (time 1787702706.48 total 20.00 last 146.69) - /unlink 4264 (time 1787702716.48 total 30.01 last 141.37) total: 4500 /unlink in 31.72 seconds: 141.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 737 (time 1787702733.43 total 10.01 last 73.66) - open/close 1407 (time 1787702743.43 total 20.01 last 66.98) - open/close 2057 (time 1787702753.44 total 30.02 last 64.94) - open/close 2740 (time 1787702763.45 total 40.03 last 68.26) - open/close 3450 (time 1787702773.46 total 50.03 last 70.95) - open/close 4177 (time 1787702783.47 total 60.04 last 72.62) total: 4500 open/close in 64.87 seconds: 69.37 ops/second - /unlink 1414 (time 1787702803.41 total 10.00 last 141.37) - /unlink 2901 (time 1787702813.42 total 20.01 last 148.63) - /unlink 4391 (time 1787702823.43 total 30.02 last 148.82) total: 4500 /unlink in 30.92 seconds: 145.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 727 (time 1787702838.17 total 10.01 last 72.63) - open/close 1588 (time 1787702848.17 total 20.01 last 86.10) - open/close 2377 (time 1787702858.18 total 30.02 last 78.79) - open/close 3257 (time 1787702868.19 total 40.03 last 87.95) - open/close 4033 (time 1787702878.19 total 50.03 last 77.59) total: 4500 open/close in 56.16 seconds: 80.13 ops/second - /unlink 1583 (time 1787702898.71 total 10.00 last 158.28) - /unlink 3195 (time 1787702908.71 total 20.00 last 161.19) total: 4500 /unlink in 28.58 seconds: 157.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 821 (time 1787702931.79 total 10.02 last 81.95) - open/close 1653 (time 1787702941.80 total 20.03 last 83.14) - open/close 2400 (time 1787702951.81 total 30.03 last 74.63) - open/close 3207 (time 1787702961.83 total 40.05 last 80.56) - open/close 3911 (time 1787702971.85 total 50.07 last 70.27) total: 4500 open/close in 58.08 seconds: 77.48 ops/second - /unlink 1650 (time 1787702994.29 total 10.00 last 164.96) - /unlink 3226 (time 1787703004.29 total 20.01 last 157.55) total: 4500 /unlink in 28.70 seconds: 156.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 842 (time 1787703027.26 total 10.00 last 84.16) - open/close 1680 (time 1787703037.27 total 20.01 last 83.77) - open/close 2539 (time 1787703047.29 total 30.03 last 85.69) - open/close 3317 (time 1787703057.30 total 40.04 last 77.75) - open/close 4049 (time 1787703067.30 total 50.04 last 73.18) total: 4500 open/close in 56.87 seconds: 79.13 ops/second - /unlink 1055 (time 1787703090.24 total 10.01 last 105.44) - /unlink 2241 (time 1787703100.24 total 20.01 last 118.56) - /unlink 3658 (time 1787703110.25 total 30.01 last 141.68) total: 4500 /unlink in 36.24 seconds: 124.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 678 (time 1787703132.36 total 10.02 last 67.64) - open/close 1424 (time 1787703142.36 total 20.03 last 74.56) - open/close 2137 (time 1787703152.37 total 30.04 last 71.24) - open/close 2986 (time 1787703162.39 total 40.06 last 84.72) - open/close 3839 (time 1787703172.40 total 50.06 last 85.27) total: 4500 open/close in 57.41 seconds: 78.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1616 (time 1787703194.48 total 10.01 last 161.51) - /unlink 2963 (time 1787703204.48 total 20.01 last 134.63) total: 4500 /unlink in 29.84 seconds: 150.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 819 (time 1787703228.93 total 10.01 last 81.83) - open/close 1512 (time 1787703238.95 total 20.03 last 69.18) - open/close 2303 (time 1787703248.95 total 30.03 last 79.06) - open/close 3120 (time 1787703258.96 total 40.04 last 81.64) - open/close 3929 (time 1787703268.97 total 50.05 last 80.84) total: 4500 open/close in 57.15 seconds: 78.74 ops/second - /unlink 1556 (time 1787703290.44 total 10.01 last 155.44) - /unlink 2812 (time 1787703300.45 total 20.01 last 125.55) - /unlink 4256 (time 1787703310.45 total 30.02 last 144.36) total: 4500 /unlink in 31.78 seconds: 141.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 739 (time 1787703327.14 total 10.01 last 73.81) - open/close 1383 (time 1787703337.15 total 20.02 last 64.37) - open/close 2211 (time 1787703347.15 total 30.02 last 82.77) - open/close 3020 (time 1787703357.16 total 40.02 last 80.86) - open/close 3754 (time 1787703367.16 total 50.03 last 73.39) - open/close 4402 (time 1787703377.17 total 60.03 last 64.75) total: 4500 open/close in 61.62 seconds: 73.03 ops/second - /unlink 1089 (time 1787703395.12 total 10.01 last 108.82) - /unlink 2515 (time 1787703405.13 total 20.01 last 142.51) - /unlink 4019 (time 1787703415.13 total 30.02 last 150.30) total: 4500 /unlink in 32.89 seconds: 136.82 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95315 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg344-server unloading modules via unload_modules_local on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local oleg344-server: modules unloaded.