== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:00:32 (1779206432) Stopping clients: oleg658-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg658-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg658-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg658-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg658-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg658-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg658-server oleg658-server: oleg658-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing load_modules_local oleg658-server: Loading modules from /home/green/git/lustre-release/lustre oleg658-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg658-server: detected 4 online CPUs by sysfs oleg658-server: MODOPTS_LIBCFS= oleg658-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg658-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg658-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg658-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg658-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg658-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg658-client.virtnet: -o user_xattr,flock 192.168.206.158@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 578 (time 1779206634.63 total 10.00 last 57.79) - open/close 1200 (time 1779206644.65 total 20.02 last 62.06) - open/close 1811 (time 1779206654.65 total 30.03 last 61.09) - open/close 2384 (time 1779206664.65 total 40.03 last 57.28) - open/close 2889 (time 1779206674.65 total 50.03 last 50.50) - open/close 3398 (time 1779206684.66 total 60.04 last 50.86) - open/close 3849 (time 1779206694.68 total 70.05 last 45.04) - open/close 4386 (time 1779206704.68 total 80.06 last 53.67) total: 4500 open/close in 82.05 seconds: 54.85 ops/second - /unlink 1090 (time 1779206722.13 total 10.00 last 108.97) - /unlink 2163 (time 1779206732.15 total 20.03 last 107.04) - /unlink 3133 (time 1779206742.15 total 30.03 last 96.99) - /unlink 4215 (time 1779206752.15 total 40.03 last 108.17) total: 4500 /unlink in 42.30 seconds: 106.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 640 (time 1779206769.77 total 10.02 last 63.86) - open/close 1256 (time 1779206779.78 total 20.03 last 61.56) - open/close 1764 (time 1779206789.80 total 30.05 last 50.71) - open/close 2289 (time 1779206799.81 total 40.06 last 52.42) - open/close 2788 (time 1779206809.84 total 50.08 last 49.79) - open/close 3280 (time 1779206819.84 total 60.09 last 49.17) - open/close 3804 (time 1779206829.85 total 70.09 last 52.38) - open/close 4400 (time 1779206839.85 total 80.09 last 59.59) total: 4500 open/close in 81.92 seconds: 54.93 ops/second - /unlink 1119 (time 1779206856.65 total 10.01 last 111.75) - /unlink 2358 (time 1779206866.65 total 20.02 last 123.87) - /unlink 3406 (time 1779206876.66 total 30.03 last 104.70) - /unlink 4474 (time 1779206886.67 total 40.03 last 106.75) total: 4500 /unlink in 40.39 seconds: 111.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 606 (time 1779206901.85 total 10.01 last 60.55) - open/close 1203 (time 1779206911.85 total 20.01 last 59.69) - open/close 1844 (time 1779206921.86 total 30.01 last 64.07) - open/close 2456 (time 1779206931.86 total 40.02 last 61.18) - open/close 3055 (time 1779206941.87 total 50.02 last 59.87) - open/close 3597 (time 1779206951.87 total 60.02 last 54.20) - open/close 4209 (time 1779206961.88 total 70.04 last 61.11) total: 4500 open/close in 75.71 seconds: 59.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1085 (time 1779206982.39 total 10.00 last 108.46) - /unlink 2250 (time 1779206992.39 total 20.01 last 116.45) - /unlink 3407 (time 1779207002.40 total 30.01 last 115.67) total: 4500 /unlink in 39.28 seconds: 114.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 622 (time 1779207026.38 total 10.01 last 62.11) - open/close 1252 (time 1779207036.38 total 20.01 last 63.00) - open/close 1812 (time 1779207046.38 total 30.02 last 55.99) - open/close 2407 (time 1779207056.38 total 40.02 last 59.48) - open/close 3029 (time 1779207066.39 total 50.03 last 62.16) - open/close 3682 (time 1779207076.39 total 60.03 last 65.29) - open/close 4299 (time 1779207086.40 total 70.04 last 61.64) total: 4500 open/close in 73.09 seconds: 61.57 ops/second - /unlink 1146 (time 1779207104.16 total 10.01 last 114.49) - /unlink 2280 (time 1779207114.16 total 20.02 last 113.33) - /unlink 3412 (time 1779207124.17 total 30.02 last 113.11) - /unlink 4482 (time 1779207134.18 total 40.03 last 106.92) total: 4500 /unlink in 40.38 seconds: 111.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 565 (time 1779207150.37 total 10.01 last 56.45) - open/close 1156 (time 1779207160.37 total 20.01 last 59.08) - open/close 1669 (time 1779207170.39 total 30.03 last 51.21) - open/close 2229 (time 1779207180.39 total 40.03 last 56.00) - open/close 2820 (time 1779207190.40 total 50.04 last 59.07) - open/close 3359 (time 1779207200.41 total 60.05 last 53.81) - open/close 3883 (time 1779207210.41 total 70.05 last 52.39) total: 4500 open/close in 80.00 seconds: 56.25 ops/second - /unlink 1026 (time 1779207234.98 total 10.00 last 102.58) - /unlink 2100 (time 1779207244.98 total 20.00 last 107.37) - /unlink 3136 (time 1779207254.98 total 30.01 last 103.59) - /unlink 4263 (time 1779207264.98 total 40.01 last 112.66) total: 4500 /unlink in 42.56 seconds: 105.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 653 (time 1779207283.33 total 10.01 last 65.27) - open/close 1222 (time 1779207293.34 total 20.02 last 56.82) - open/close 1726 (time 1779207303.35 total 30.03 last 50.35) - open/close 2274 (time 1779207313.36 total 40.03 last 54.77) - open/close 2800 (time 1779207323.39 total 50.07 last 52.43) - open/close 3307 (time 1779207333.42 total 60.10 last 50.55) - open/close 3822 (time 1779207343.42 total 70.10 last 51.50) - open/close 4406 (time 1779207353.43 total 80.11 last 58.32) total: 4500 open/close in 81.56 seconds: 55.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1176 (time 1779207370.34 total 10.00 last 117.57) - /unlink 2324 (time 1779207380.35 total 20.01 last 114.72) - /unlink 3430 (time 1779207390.37 total 30.02 last 110.45) - /unlink 4494 (time 1779207400.37 total 40.03 last 106.32) total: 4500 /unlink in 40.05 seconds: 112.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 491 (time 1779207415.57 total 10.01 last 49.06) - open/close 1004 (time 1779207425.59 total 20.03 last 51.21) - open/close 1526 (time 1779207435.60 total 30.03 last 52.15) - open/close 2150 (time 1779207445.61 total 40.04 last 62.37) - open/close 2708 (time 1779207455.61 total 50.05 last 55.77) - open/close 3289 (time 1779207465.63 total 60.06 last 57.99) - open/close 3877 (time 1779207475.66 total 70.09 last 58.62) - open/close 4419 (time 1779207485.67 total 80.10 last 54.17) total: 4500 open/close in 81.52 seconds: 55.20 ops/second - /unlink 1029 (time 1779207502.98 total 10.01 last 102.84) - /unlink 2034 (time 1779207512.99 total 20.01 last 100.45) - /unlink 3058 (time 1779207522.99 total 30.02 last 102.32) - /unlink 4078 (time 1779207533.00 total 40.02 last 101.95) total: 4500 /unlink in 44.03 seconds: 102.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 526 (time 1779207552.44 total 10.00 last 52.58) - open/close 1093 (time 1779207562.45 total 20.01 last 56.68) - open/close 1647 (time 1779207572.45 total 30.01 last 55.37) - open/close 2076 (time 1779207582.46 total 40.02 last 42.86) - open/close 2481 (time 1779207592.47 total 50.03 last 40.46) - open/close 3029 (time 1779207602.48 total 60.04 last 54.78) - open/close 3556 (time 1779207612.48 total 70.04 last 52.69) - open/close 4144 (time 1779207622.50 total 80.06 last 58.68) total: 4500 open/close in 86.26 seconds: 52.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1097 (time 1779207643.49 total 10.00 last 109.68) - /unlink 2182 (time 1779207653.49 total 20.00 last 108.48) - /unlink 3196 (time 1779207663.50 total 30.01 last 101.38) - /unlink 4235 (time 1779207673.50 total 40.01 last 103.81) total: 4500 /unlink in 42.44 seconds: 106.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 572 (time 1779207691.21 total 10.01 last 57.16) - open/close 1148 (time 1779207701.21 total 20.01 last 57.59) - open/close 1725 (time 1779207711.21 total 30.01 last 57.70) - open/close 2298 (time 1779207721.22 total 40.02 last 57.24) - open/close 2872 (time 1779207731.24 total 50.03 last 57.33) - open/close 3436 (time 1779207741.25 total 60.05 last 56.31) - open/close 3943 (time 1779207751.25 total 70.05 last 50.69) total: 4500 open/close in 79.61 seconds: 56.52 ops/second - /unlink 1106 (time 1779207776.41 total 10.00 last 110.56) - /unlink 2182 (time 1779207786.42 total 20.01 last 107.55) - /unlink 3226 (time 1779207796.42 total 30.01 last 104.37) - /unlink 4255 (time 1779207806.43 total 40.02 last 102.81) total: 4500 /unlink in 42.23 seconds: 106.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 538 (time 1779207823.44 total 10.00 last 53.79) - open/close 1210 (time 1779207833.45 total 20.01 last 67.18) - open/close 1813 (time 1779207843.45 total 30.01 last 60.26) - open/close 2387 (time 1779207853.46 total 40.02 last 57.36) - open/close 2978 (time 1779207863.47 total 50.03 last 59.04) - open/close 3543 (time 1779207873.48 total 60.03 last 56.47) - open/close 4104 (time 1779207883.51 total 70.07 last 55.89) total: 4500 open/close in 77.17 seconds: 58.31 ops/second - /unlink 998 (time 1779207908.11 total 10.00 last 99.77) - /unlink 2160 (time 1779207918.11 total 20.00 last 116.18) - /unlink 3315 (time 1779207928.12 total 30.01 last 115.45) - /unlink 4381 (time 1779207938.12 total 40.01 last 106.60) total: 4500 /unlink in 41.14 seconds: 109.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 585 (time 1779207954.25 total 10.00 last 58.49) - open/close 1178 (time 1779207964.26 total 20.01 last 59.24) - open/close 1752 (time 1779207974.27 total 30.03 last 57.32) - open/close 2321 (time 1779207984.27 total 40.03 last 56.89) - open/close 2894 (time 1779207994.29 total 50.04 last 57.21) - open/close 3523 (time 1779208004.29 total 60.05 last 62.87) - open/close 4128 (time 1779208014.32 total 70.07 last 60.36) total: 4500 open/close in 77.84 seconds: 57.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1113 (time 1779208037.51 total 10.00 last 111.27) - /unlink 2193 (time 1779208047.51 total 20.01 last 107.94) - /unlink 3400 (time 1779208057.51 total 30.01 last 120.69) total: 4500 /unlink in 39.37 seconds: 114.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 631 (time 1779208081.57 total 10.01 last 63.04) - open/close 1252 (time 1779208091.59 total 20.03 last 61.98) - open/close 1863 (time 1779208101.60 total 30.04 last 61.02) - open/close 2385 (time 1779208111.61 total 40.04 last 52.19) - open/close 2965 (time 1779208121.61 total 50.05 last 57.98) - open/close 3547 (time 1779208131.62 total 60.06 last 58.13) - open/close 4105 (time 1779208141.63 total 70.07 last 55.72) total: 4500 open/close in 76.91 seconds: 58.51 ops/second - /unlink 1109 (time 1779208163.54 total 10.01 last 110.81) - /unlink 2184 (time 1779208173.55 total 20.01 last 107.44) - /unlink 3220 (time 1779208183.55 total 30.02 last 103.57) - /unlink 4386 (time 1779208193.55 total 40.02 last 116.56) total: 4500 /unlink in 41.02 seconds: 109.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 638 (time 1779208208.59 total 10.01 last 63.76) - open/close 1297 (time 1779208218.59 total 20.01 last 65.88) - open/close 1920 (time 1779208228.60 total 30.02 last 62.25) - open/close 2537 (time 1779208238.60 total 40.03 last 61.66) - open/close 3186 (time 1779208248.62 total 50.04 last 64.80) - open/close 3764 (time 1779208258.63 total 60.05 last 57.77) - open/close 4331 (time 1779208268.65 total 70.07 last 56.58) total: 4500 open/close in 73.51 seconds: 61.22 ops/second - /unlink 976 (time 1779208288.29 total 10.01 last 97.51) - /unlink 1975 (time 1779208298.29 total 20.01 last 99.90) - /unlink 2943 (time 1779208308.30 total 30.01 last 96.74) - /unlink 3878 (time 1779208318.31 total 40.02 last 93.42) total: 4500 /unlink in 45.58 seconds: 98.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 570 (time 1779208338.47 total 10.00 last 56.98) - open/close 1139 (time 1779208348.48 total 20.01 last 56.87) - open/close 1785 (time 1779208358.52 total 30.05 last 64.32) - open/close 2328 (time 1779208368.54 total 40.07 last 54.19) - open/close 2922 (time 1779208378.55 total 50.08 last 59.36) - open/close 3530 (time 1779208388.55 total 60.08 last 60.79) - open/close 4121 (time 1779208398.56 total 70.09 last 59.06) total: 4500 open/close in 76.16 seconds: 59.08 ops/second - /unlink 1081 (time 1779208419.90 total 10.00 last 108.08) - /unlink 2055 (time 1779208429.91 total 20.01 last 97.30) - /unlink 3081 (time 1779208439.91 total 30.01 last 102.60) - /unlink 4102 (time 1779208449.92 total 40.02 last 102.07) total: 4500 /unlink in 43.99 seconds: 102.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 581 (time 1779208469.43 total 10.00 last 58.09) - open/close 1166 (time 1779208479.44 total 20.01 last 58.44) - open/close 1714 (time 1779208489.45 total 30.03 last 54.73) - open/close 2256 (time 1779208499.46 total 40.03 last 54.15) - open/close 2805 (time 1779208509.47 total 50.04 last 54.85) - open/close 3392 (time 1779208519.49 total 60.06 last 58.59) - open/close 3992 (time 1779208529.51 total 70.08 last 59.88) total: 4500 open/close in 78.95 seconds: 57.00 ops/second - /unlink 1006 (time 1779208553.44 total 10.00 last 100.59) - /unlink 2104 (time 1779208563.44 total 20.00 last 109.77) - /unlink 3034 (time 1779208573.45 total 30.01 last 92.94) - /unlink 4016 (time 1779208583.45 total 40.02 last 98.13) total: 4500 /unlink in 44.57 seconds: 100.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 490 (time 1779208604.00 total 10.02 last 48.90) - open/close 954 (time 1779208614.00 total 20.03 last 46.37) - open/close 1442 (time 1779208624.01 total 30.03 last 48.79) - open/close 2017 (time 1779208634.03 total 40.05 last 57.37) - open/close 2580 (time 1779208644.03 total 50.05 last 56.29) - open/close 3179 (time 1779208654.04 total 60.06 last 59.86) - open/close 3752 (time 1779208664.04 total 70.06 last 57.28) - open/close 4355 (time 1779208674.04 total 80.07 last 60.29) total: 4500 open/close in 82.64 seconds: 54.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1090 (time 1779208691.35 total 10.00 last 108.96) - /unlink 2189 (time 1779208701.36 total 20.01 last 109.78) - /unlink 3295 (time 1779208711.36 total 30.02 last 110.59) - /unlink 4407 (time 1779208721.37 total 40.02 last 111.10) total: 4500 /unlink in 41.11 seconds: 109.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 550 (time 1779208738.31 total 10.00 last 54.99) - open/close 1161 (time 1779208748.33 total 20.03 last 60.94) - open/close 1757 (time 1779208758.35 total 30.05 last 59.49) - open/close 2418 (time 1779208768.37 total 40.07 last 65.98) - open/close 2918 (time 1779208778.38 total 50.08 last 49.94) - open/close 3456 (time 1779208788.43 total 60.13 last 53.54) - open/close 4020 (time 1779208798.46 total 70.15 last 56.25) total: 4500 open/close in 77.69 seconds: 57.92 ops/second - /unlink 1095 (time 1779208821.37 total 10.01 last 109.43) - /unlink 2163 (time 1779208831.37 total 20.01 last 106.79) - /unlink 3224 (time 1779208841.38 total 30.02 last 106.02) - /unlink 4264 (time 1779208851.38 total 40.02 last 103.99) total: 4500 /unlink in 42.56 seconds: 105.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 564 (time 1779208869.80 total 10.02 last 56.27) - open/close 1155 (time 1779208879.80 total 20.03 last 59.05) - open/close 1751 (time 1779208889.81 total 30.03 last 59.59) - open/close 2365 (time 1779208899.81 total 40.04 last 61.36) - open/close 2983 (time 1779208909.82 total 50.04 last 61.78) - open/close 3563 (time 1779208919.82 total 60.04 last 58.00) - open/close 4159 (time 1779208929.82 total 70.05 last 59.57) total: 4500 open/close in 75.85 seconds: 59.33 ops/second - /unlink 1091 (time 1779208950.58 total 10.02 last 108.93) - /unlink 2171 (time 1779208960.59 total 20.03 last 107.88) - /unlink 3171 (time 1779208970.60 total 30.04 last 99.89) - /unlink 4199 (time 1779208980.62 total 40.05 last 102.63) total: 4500 /unlink in 43.34 seconds: 103.83 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11217: 94579 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg658-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg658-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg658-server stop mds service on oleg658-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg658-server stop mds service on oleg658-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg658-server unloading modules via unload_modules_local on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing unload_modules_local oleg658-server: modules unloaded.