== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:49:54 (1786189794) Stopping clients: oleg203-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg203-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg203-server oleg203-server: oleg203-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg203-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg203-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg203-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 778 (time 1786190041.29 total 10.00 last 77.78) - open/close 1431 (time 1786190051.30 total 20.01 last 65.25) - open/close 2247 (time 1786190061.31 total 30.02 last 81.51) - open/close 2948 (time 1786190071.31 total 40.02 last 70.10) - open/close 3781 (time 1786190081.31 total 50.03 last 83.26) total: 4500 open/close in 59.90 seconds: 75.12 ops/second - /unlink 1273 (time 1786190105.72 total 10.01 last 127.19) - /unlink 2723 (time 1786190115.73 total 20.02 last 144.86) - /unlink 3948 (time 1786190125.73 total 30.02 last 122.45) total: 4500 /unlink in 33.70 seconds: 133.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 754 (time 1786190146.66 total 10.01 last 75.35) - open/close 1410 (time 1786190156.66 total 20.01 last 65.59) - open/close 2194 (time 1786190166.66 total 30.01 last 78.37) - open/close 2860 (time 1786190176.66 total 40.01 last 66.59) - open/close 3600 (time 1786190186.67 total 50.02 last 73.95) - open/close 4124 (time 1786190196.69 total 60.03 last 52.32) total: 4500 open/close in 67.74 seconds: 66.43 ops/second - /unlink 998 (time 1786190221.90 total 10.02 last 99.61) - /unlink 2029 (time 1786190231.92 total 20.04 last 102.86) - /unlink 2956 (time 1786190241.93 total 30.04 last 92.68) - /unlink 4287 (time 1786190251.93 total 40.05 last 133.04) total: 4500 /unlink in 41.83 seconds: 107.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 674 (time 1786190269.86 total 10.01 last 67.33) - open/close 1273 (time 1786190279.87 total 20.02 last 59.85) - open/close 1887 (time 1786190289.88 total 30.03 last 61.34) - open/close 2435 (time 1786190299.90 total 40.05 last 54.67) - open/close 3057 (time 1786190309.91 total 50.06 last 62.17) - open/close 3673 (time 1786190319.91 total 60.06 last 61.56) - open/close 4321 (time 1786190329.92 total 70.07 last 64.78) total: 4500 open/close in 72.62 seconds: 61.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1465 (time 1786190348.55 total 10.00 last 146.48) - /unlink 2740 (time 1786190358.55 total 20.00 last 127.49) - /unlink 4126 (time 1786190368.56 total 30.01 last 138.47) total: 4500 /unlink in 32.94 seconds: 136.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 750 (time 1786190388.00 total 10.00 last 74.97) - open/close 1359 (time 1786190398.01 total 20.02 last 60.82) - open/close 2055 (time 1786190408.01 total 30.02 last 69.57) - open/close 2700 (time 1786190418.04 total 40.04 last 64.36) - open/close 3355 (time 1786190428.05 total 50.05 last 65.45) - open/close 3978 (time 1786190438.06 total 60.07 last 62.22) total: 4500 open/close in 68.29 seconds: 65.89 ops/second - /unlink 920 (time 1786190463.39 total 10.00 last 91.98) - /unlink 2077 (time 1786190473.39 total 20.00 last 115.69) - /unlink 3540 (time 1786190483.39 total 30.01 last 146.26) total: 4500 /unlink in 38.59 seconds: 116.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 654 (time 1786190508.34 total 10.01 last 65.30) - open/close 1260 (time 1786190518.34 total 20.02 last 60.60) - open/close 1796 (time 1786190528.35 total 30.03 last 53.54) - open/close 2385 (time 1786190538.36 total 40.04 last 58.84) - open/close 2951 (time 1786190548.36 total 50.04 last 56.57) - open/close 3519 (time 1786190558.38 total 60.06 last 56.72) - open/close 4163 (time 1786190568.38 total 70.06 last 64.39) total: 4500 open/close in 75.11 seconds: 59.91 ops/second - /unlink 1058 (time 1786190589.46 total 10.00 last 105.76) - /unlink 2274 (time 1786190599.47 total 20.01 last 121.56) - /unlink 3679 (time 1786190609.47 total 30.01 last 140.46) total: 4500 /unlink in 36.25 seconds: 124.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 596 (time 1786190631.40 total 10.00 last 59.59) - open/close 1214 (time 1786190641.43 total 20.03 last 61.60) - open/close 1748 (time 1786190651.44 total 30.04 last 53.38) - open/close 2244 (time 1786190661.46 total 40.06 last 49.47) - open/close 2911 (time 1786190671.46 total 50.06 last 66.70) - open/close 3477 (time 1786190681.52 total 60.12 last 56.31) - open/close 3974 (time 1786190691.53 total 70.13 last 49.64) total: 4500 open/close in 77.90 seconds: 57.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1317 (time 1786190714.92 total 10.00 last 131.65) - /unlink 2664 (time 1786190724.92 total 20.01 last 134.64) - /unlink 3965 (time 1786190734.93 total 30.01 last 130.04) total: 4500 /unlink in 35.17 seconds: 127.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 651 (time 1786190755.28 total 10.02 last 64.97) - open/close 1266 (time 1786190765.29 total 20.03 last 61.43) - open/close 1902 (time 1786190775.31 total 30.05 last 63.50) - open/close 2689 (time 1786190785.31 total 40.05 last 78.67) - open/close 3220 (time 1786190795.32 total 50.06 last 53.08) - open/close 3864 (time 1786190805.34 total 60.08 last 64.25) total: 4500 open/close in 69.65 seconds: 64.61 ops/second - /unlink 1162 (time 1786190831.13 total 10.00 last 116.17) - /unlink 2386 (time 1786190841.14 total 20.01 last 122.32) - /unlink 3468 (time 1786190851.14 total 30.01 last 108.14) total: 4500 /unlink in 39.68 seconds: 113.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 653 (time 1786190876.47 total 10.00 last 65.28) - open/close 1326 (time 1786190886.47 total 20.00 last 67.29) - open/close 2155 (time 1786190896.47 total 30.01 last 82.88) - open/close 2947 (time 1786190906.49 total 40.02 last 79.07) - open/close 3422 (time 1786190916.51 total 50.05 last 47.39) - open/close 4094 (time 1786190926.52 total 60.05 last 67.20) total: 4500 open/close in 65.58 seconds: 68.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1106 (time 1786190947.97 total 10.00 last 110.56) - /unlink 2475 (time 1786190957.97 total 20.01 last 136.85) - /unlink 3718 (time 1786190967.97 total 30.01 last 124.27) total: 4500 /unlink in 36.07 seconds: 124.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 696 (time 1786190989.05 total 10.01 last 69.51) - open/close 1372 (time 1786190999.06 total 20.01 last 67.59) - open/close 2095 (time 1786191009.06 total 30.02 last 72.29) - open/close 2773 (time 1786191019.07 total 40.02 last 67.74) - open/close 3400 (time 1786191029.07 total 50.03 last 62.67) - open/close 4271 (time 1786191039.07 total 60.03 last 87.07) total: 4500 open/close in 62.63 seconds: 71.85 ops/second - /unlink 1376 (time 1786191056.41 total 10.01 last 137.50) - /unlink 2853 (time 1786191066.41 total 20.01 last 147.70) - /unlink 4381 (time 1786191076.41 total 30.01 last 152.75) total: 4500 /unlink in 31.29 seconds: 143.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 790 (time 1786191092.42 total 10.01 last 78.88) - open/close 1578 (time 1786191102.43 total 20.02 last 78.77) - open/close 2415 (time 1786191112.43 total 30.02 last 83.67) - open/close 3234 (time 1786191122.44 total 40.03 last 81.83) - open/close 4010 (time 1786191132.45 total 50.04 last 77.56) total: 4500 open/close in 56.21 seconds: 80.06 ops/second - /unlink 1479 (time 1786191152.69 total 10.00 last 147.87) - /unlink 2991 (time 1786191162.69 total 20.00 last 151.17) - /unlink 4393 (time 1786191172.69 total 30.01 last 140.18) total: 4500 /unlink in 30.82 seconds: 146.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 809 (time 1786191188.33 total 10.00 last 80.87) - open/close 1642 (time 1786191198.34 total 20.01 last 83.28) - open/close 2440 (time 1786191208.34 total 30.01 last 79.78) - open/close 3251 (time 1786191218.35 total 40.02 last 81.04) - open/close 4034 (time 1786191228.35 total 50.02 last 78.30) total: 4500 open/close in 55.17 seconds: 81.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1588 (time 1786191248.55 total 10.00 last 158.79) - /unlink 3328 (time 1786191258.56 total 20.00 last 173.93) total: 4500 /unlink in 27.60 seconds: 163.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 880 (time 1786191280.83 total 10.00 last 87.99) - open/close 1797 (time 1786191290.83 total 20.00 last 91.69) - open/close 2686 (time 1786191300.84 total 30.01 last 88.82) - open/close 3569 (time 1786191310.84 total 40.01 last 88.29) - open/close 4367 (time 1786191320.84 total 50.01 last 79.78) total: 4500 open/close in 51.55 seconds: 87.29 ops/second - /unlink 1489 (time 1786191335.96 total 10.00 last 148.85) - /unlink 3099 (time 1786191345.96 total 20.00 last 160.98) total: 4500 /unlink in 28.82 seconds: 156.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 1026 (time 1786191368.87 total 10.00 last 102.56) - open/close 1944 (time 1786191378.89 total 20.01 last 91.70) - open/close 2929 (time 1786191388.90 total 30.02 last 98.40) - open/close 3859 (time 1786191398.90 total 40.03 last 92.93) total: 4500 open/close in 46.32 seconds: 97.15 ops/second - /unlink 1828 (time 1786191418.28 total 10.00 last 182.74) - /unlink 3612 (time 1786191428.28 total 20.01 last 178.36) total: 4500 /unlink in 25.46 seconds: 176.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 963 (time 1786191447.46 total 10.01 last 96.25) - open/close 1964 (time 1786191457.46 total 20.01 last 100.07) - open/close 3081 (time 1786191467.47 total 30.02 last 111.61) - open/close 4036 (time 1786191477.47 total 40.02 last 95.46) total: 4500 open/close in 45.45 seconds: 99.02 ops/second - /unlink 1861 (time 1786191496.23 total 10.00 last 186.03) - /unlink 3778 (time 1786191506.24 total 20.01 last 191.66) total: 4500 /unlink in 23.66 seconds: 190.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 1064 (time 1786191523.16 total 10.01 last 106.32) - open/close 2206 (time 1786191533.17 total 20.01 last 114.14) - open/close 3342 (time 1786191543.17 total 30.02 last 113.54) - open/close 4417 (time 1786191553.17 total 40.02 last 107.46) total: 4500 open/close in 40.69 seconds: 110.58 ops/second - /unlink 2231 (time 1786191566.46 total 10.00 last 223.08) - /unlink 4310 (time 1786191576.46 total 20.00 last 207.88) total: 4500 /unlink in 21.02 seconds: 214.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1016 (time 1786191590.14 total 10.00 last 101.59) - open/close 1939 (time 1786191600.15 total 20.01 last 92.26) - open/close 2821 (time 1786191610.15 total 30.01 last 88.20) - open/close 3779 (time 1786191620.15 total 40.01 last 95.79) total: 4500 open/close in 47.02 seconds: 95.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 2213 (time 1786191639.78 total 10.00 last 221.28) - /unlink 4213 (time 1786191649.78 total 20.00 last 199.93) total: 4500 /unlink in 21.33 seconds: 211.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1070 (time 1786191663.60 total 10.01 last 106.93) - open/close 2159 (time 1786191673.61 total 20.01 last 108.82) - open/close 3461 (time 1786191683.61 total 30.02 last 130.18) total: 4500 open/close in 38.45 seconds: 117.03 ops/second - /unlink 2170 (time 1786191704.61 total 10.00 last 216.90) - /unlink 3893 (time 1786191714.61 total 20.01 last 172.23) total: 4500 /unlink in 23.70 seconds: 189.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 1039 (time 1786191731.65 total 10.01 last 103.84) - open/close 2141 (time 1786191741.65 total 20.01 last 110.15) - open/close 3220 (time 1786191751.66 total 30.01 last 107.86) - open/close 4414 (time 1786191761.66 total 40.02 last 119.38) total: 4500 open/close in 40.91 seconds: 109.98 ops/second - /unlink 2131 (time 1786191774.88 total 10.00 last 213.07) - /unlink 4239 (time 1786191784.88 total 20.00 last 210.80) total: 4500 /unlink in 20.97 seconds: 214.55 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95375 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg203-server unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local oleg203-server: modules unloaded.