== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:23:29 (1763335409) Stopping clients: oleg205-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg205-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg205-server oleg205-server: oleg205-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: detected 4 online CPUs by sysfs oleg205-server: MODOPTS_LIBCFS= oleg205-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg205-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg205-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg205-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 646 (time 1763335569.52 total 10.01 last 64.53) - open/close 1391 (time 1763335579.53 total 20.02 last 74.41) - open/close 2128 (time 1763335589.54 total 30.03 last 73.68) - open/close 2932 (time 1763335599.55 total 40.04 last 80.28) - open/close 3682 (time 1763335609.55 total 50.04 last 75.00) - open/close 4465 (time 1763335619.56 total 60.05 last 78.21) total: 4500 open/close in 60.54 seconds: 74.33 ops/second - /unlink 1391 (time 1763335633.29 total 10.01 last 138.93) - /unlink 2707 (time 1763335643.29 total 20.01 last 131.59) - /unlink 3982 (time 1763335653.29 total 30.01 last 127.50) total: 4500 /unlink in 33.91 seconds: 132.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 719 (time 1763335670.78 total 10.01 last 71.81) - open/close 1449 (time 1763335680.79 total 20.02 last 72.97) - open/close 2202 (time 1763335690.79 total 30.02 last 75.27) - open/close 2911 (time 1763335700.80 total 40.03 last 70.85) - open/close 3580 (time 1763335710.81 total 50.04 last 66.80) - open/close 4252 (time 1763335720.83 total 60.05 last 67.12) total: 4500 open/close in 63.48 seconds: 70.89 ops/second - /unlink 1356 (time 1763335737.22 total 10.01 last 135.49) - /unlink 2594 (time 1763335747.23 total 20.01 last 123.77) - /unlink 3852 (time 1763335757.23 total 30.02 last 125.73) total: 4500 /unlink in 34.69 seconds: 129.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 682 (time 1763335776.44 total 10.01 last 68.14) - open/close 1307 (time 1763335786.45 total 20.02 last 62.42) - open/close 1973 (time 1763335796.46 total 30.03 last 66.54) - open/close 2642 (time 1763335806.47 total 40.04 last 66.86) - open/close 3365 (time 1763335816.48 total 50.05 last 72.21) - open/close 4107 (time 1763335826.48 total 60.05 last 74.17) total: 4500 open/close in 65.33 seconds: 68.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1418 (time 1763335845.40 total 10.00 last 141.77) - /unlink 2744 (time 1763335855.40 total 20.01 last 132.53) - /unlink 3999 (time 1763335865.41 total 30.01 last 125.49) total: 4500 /unlink in 33.90 seconds: 132.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 715 (time 1763335883.24 total 10.01 last 71.42) - open/close 1536 (time 1763335893.24 total 20.02 last 82.05) - open/close 2314 (time 1763335903.25 total 30.03 last 77.73) - open/close 3128 (time 1763335913.26 total 40.03 last 81.36) - open/close 3918 (time 1763335923.26 total 50.04 last 78.96) total: 4500 open/close in 57.15 seconds: 78.75 ops/second - /unlink 1423 (time 1763335943.52 total 10.00 last 142.25) - /unlink 2766 (time 1763335953.53 total 20.01 last 134.23) - /unlink 4067 (time 1763335963.53 total 30.01 last 130.07) total: 4500 /unlink in 33.15 seconds: 135.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 754 (time 1763335980.69 total 10.00 last 75.39) - open/close 1533 (time 1763335990.70 total 20.00 last 77.87) - open/close 2324 (time 1763336000.70 total 30.01 last 79.06) - open/close 3121 (time 1763336010.71 total 40.02 last 79.66) - open/close 3936 (time 1763336020.72 total 50.03 last 81.41) total: 4500 open/close in 56.88 seconds: 79.12 ops/second - /unlink 1380 (time 1763336040.76 total 10.00 last 137.94) - /unlink 2766 (time 1763336050.76 total 20.01 last 138.55) - /unlink 4022 (time 1763336060.77 total 30.01 last 125.58) total: 4500 /unlink in 33.57 seconds: 134.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 676 (time 1763336078.58 total 10.01 last 67.51) - open/close 1386 (time 1763336088.59 total 20.03 last 70.92) - open/close 2103 (time 1763336098.59 total 30.03 last 71.70) - open/close 2838 (time 1763336108.59 total 40.03 last 73.47) - open/close 3556 (time 1763336118.62 total 50.05 last 71.64) - open/close 4332 (time 1763336128.63 total 60.06 last 77.50) total: 4500 open/close in 62.45 seconds: 72.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1309 (time 1763336144.55 total 10.00 last 130.84) - /unlink 2641 (time 1763336154.56 total 20.01 last 133.08) - /unlink 3956 (time 1763336164.56 total 30.02 last 131.46) total: 4500 /unlink in 34.37 seconds: 130.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 682 (time 1763336182.84 total 10.02 last 68.08) - open/close 1338 (time 1763336192.86 total 20.04 last 65.47) - open/close 1976 (time 1763336202.87 total 30.05 last 63.72) - open/close 2614 (time 1763336212.89 total 40.07 last 63.67) - open/close 3251 (time 1763336222.91 total 50.08 last 63.61) - open/close 3921 (time 1763336232.91 total 60.09 last 66.99) total: 4500 open/close in 68.61 seconds: 65.59 ops/second - /unlink 1227 (time 1763336256.37 total 10.00 last 122.69) - /unlink 2528 (time 1763336266.37 total 20.01 last 130.04) - /unlink 3781 (time 1763336276.38 total 30.01 last 125.25) total: 4500 /unlink in 35.84 seconds: 125.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 742 (time 1763336296.07 total 10.01 last 74.14) - open/close 1484 (time 1763336306.08 total 20.01 last 74.15) - open/close 2196 (time 1763336316.09 total 30.03 last 71.13) - open/close 2948 (time 1763336326.11 total 40.05 last 75.05) - open/close 3652 (time 1763336336.12 total 50.05 last 70.34) - open/close 4359 (time 1763336346.13 total 60.06 last 70.63) total: 4500 open/close in 62.14 seconds: 72.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1343 (time 1763336361.67 total 10.01 last 134.12) - /unlink 2661 (time 1763336371.68 total 20.02 last 131.65) - /unlink 3920 (time 1763336381.69 total 30.03 last 125.84) total: 4500 /unlink in 34.46 seconds: 130.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 776 (time 1763336399.92 total 10.01 last 77.56) - open/close 1492 (time 1763336409.92 total 20.01 last 71.55) - open/close 2257 (time 1763336419.93 total 30.02 last 76.45) - open/close 3012 (time 1763336429.93 total 40.02 last 75.48) - open/close 3713 (time 1763336439.93 total 50.02 last 70.08) - open/close 4382 (time 1763336449.94 total 60.03 last 66.83) total: 4500 open/close in 61.57 seconds: 73.09 ops/second - /unlink 1216 (time 1763336466.10 total 10.00 last 121.56) - /unlink 2491 (time 1763336476.10 total 20.01 last 127.45) - /unlink 3789 (time 1763336486.11 total 30.01 last 129.76) total: 4500 /unlink in 35.86 seconds: 125.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 716 (time 1763336505.12 total 10.01 last 71.55) - open/close 1393 (time 1763336515.13 total 20.02 last 67.62) - open/close 2173 (time 1763336525.14 total 30.02 last 77.96) - open/close 2925 (time 1763336535.15 total 40.03 last 75.15) - open/close 3640 (time 1763336545.16 total 50.04 last 71.43) - open/close 4358 (time 1763336555.16 total 60.04 last 71.79) total: 4500 open/close in 61.83 seconds: 72.78 ops/second - /unlink 1300 (time 1763336570.63 total 10.02 last 129.78) - /unlink 2543 (time 1763336580.63 total 20.02 last 124.23) - /unlink 3836 (time 1763336590.64 total 30.03 last 129.18) total: 4500 /unlink in 35.01 seconds: 128.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 666 (time 1763336609.49 total 10.01 last 66.56) - open/close 1313 (time 1763336619.51 total 20.03 last 64.57) - open/close 2037 (time 1763336629.51 total 30.03 last 72.39) - open/close 2670 (time 1763336639.55 total 40.06 last 63.09) - open/close 3254 (time 1763336649.56 total 50.07 last 58.33) - open/close 3937 (time 1763336659.56 total 60.07 last 68.30) total: 4500 open/close in 67.86 seconds: 66.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1415 (time 1763336680.81 total 10.00 last 141.44) - /unlink 2807 (time 1763336690.81 total 20.01 last 139.18) - /unlink 4092 (time 1763336700.81 total 30.01 last 128.50) total: 4500 /unlink in 33.29 seconds: 135.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 778 (time 1763336718.89 total 10.00 last 77.77) - open/close 1438 (time 1763336728.90 total 20.01 last 65.96) - open/close 2160 (time 1763336738.90 total 30.01 last 72.20) - open/close 2929 (time 1763336748.90 total 40.01 last 76.90) - open/close 3579 (time 1763336758.92 total 50.03 last 64.87) - open/close 4165 (time 1763336768.93 total 60.04 last 58.53) total: 4500 open/close in 65.02 seconds: 69.21 ops/second - /unlink 1315 (time 1763336787.99 total 10.00 last 131.47) - /unlink 2705 (time 1763336798.00 total 20.01 last 138.91) - /unlink 4173 (time 1763336808.00 total 30.01 last 146.79) total: 4500 /unlink in 32.15 seconds: 139.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 740 (time 1763336822.98 total 10.00 last 73.98) - open/close 1443 (time 1763336832.98 total 20.00 last 70.29) - open/close 2159 (time 1763336842.99 total 30.01 last 71.53) - open/close 2960 (time 1763336853.00 total 40.03 last 80.01) - open/close 3717 (time 1763336863.00 total 50.03 last 75.69) - open/close 4496 (time 1763336873.01 total 60.03 last 77.87) total: 4500 open/close in 60.06 seconds: 74.93 ops/second - /unlink 1405 (time 1763336886.14 total 10.00 last 140.49) - /unlink 2760 (time 1763336896.15 total 20.01 last 135.40) - /unlink 3945 (time 1763336906.15 total 30.01 last 118.43) total: 4500 /unlink in 35.55 seconds: 126.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 723 (time 1763336925.96 total 10.01 last 72.26) - open/close 1436 (time 1763336935.96 total 20.01 last 71.27) - open/close 2226 (time 1763336945.97 total 30.02 last 78.94) - open/close 2975 (time 1763336955.97 total 40.02 last 74.87) - open/close 3700 (time 1763336966.00 total 50.05 last 72.33) - open/close 4363 (time 1763336976.00 total 60.05 last 66.26) total: 4500 open/close in 61.88 seconds: 72.72 ops/second - /unlink 1287 (time 1763336991.46 total 10.00 last 128.68) - /unlink 2712 (time 1763337001.46 total 20.00 last 142.49) - /unlink 3979 (time 1763337011.47 total 30.01 last 126.63) total: 4500 /unlink in 33.98 seconds: 132.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 796 (time 1763337028.96 total 10.01 last 79.52) - open/close 1621 (time 1763337038.96 total 20.02 last 82.46) - open/close 2385 (time 1763337048.97 total 30.02 last 76.35) - open/close 3177 (time 1763337058.97 total 40.02 last 79.17) - open/close 3936 (time 1763337068.98 total 50.03 last 75.88) total: 4500 open/close in 57.33 seconds: 78.50 ops/second - /unlink 1381 (time 1763337089.67 total 10.00 last 138.09) - /unlink 2748 (time 1763337099.67 total 20.01 last 136.61) - /unlink 4061 (time 1763337109.68 total 30.01 last 131.27) total: 4500 /unlink in 33.36 seconds: 134.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 759 (time 1763337126.93 total 10.00 last 75.89) - open/close 1519 (time 1763337136.94 total 20.02 last 75.89) - open/close 2274 (time 1763337146.95 total 30.02 last 75.47) - open/close 3051 (time 1763337156.97 total 40.04 last 77.54) - open/close 3877 (time 1763337166.97 total 50.04 last 82.56) total: 4500 open/close in 57.05 seconds: 78.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1478 (time 1763337186.83 total 10.00 last 147.73) - /unlink 2878 (time 1763337196.83 total 20.01 last 139.99) - /unlink 4217 (time 1763337206.83 total 30.01 last 133.87) total: 4500 /unlink in 32.39 seconds: 138.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 701 (time 1763337223.53 total 10.01 last 70.04) - open/close 1432 (time 1763337233.54 total 20.02 last 73.02) - open/close 2186 (time 1763337243.54 total 30.02 last 75.39) - open/close 2917 (time 1763337253.55 total 40.03 last 73.04) - open/close 3675 (time 1763337263.56 total 50.04 last 75.74) - open/close 4414 (time 1763337273.56 total 60.04 last 73.88) total: 4500 open/close in 61.24 seconds: 73.48 ops/second - /unlink 1300 (time 1763337288.59 total 10.00 last 129.97) - /unlink 2691 (time 1763337298.59 total 20.00 last 139.09) - /unlink 4136 (time 1763337308.59 total 30.01 last 144.46) total: 4500 /unlink in 32.40 seconds: 138.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 858 (time 1763337324.05 total 10.01 last 85.75) - open/close 1687 (time 1763337334.05 total 20.01 last 82.90) - open/close 2414 (time 1763337344.05 total 30.01 last 72.69) - open/close 3169 (time 1763337354.06 total 40.01 last 75.45) - open/close 3844 (time 1763337364.07 total 50.02 last 67.44) total: 4500 open/close in 59.98 seconds: 75.03 ops/second - /unlink 1361 (time 1763337387.00 total 10.00 last 136.08) - /unlink 2617 (time 1763337397.00 total 20.01 last 125.55) - /unlink 3910 (time 1763337407.00 total 30.01 last 129.28) total: 4500 /unlink in 34.06 seconds: 132.12 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 93847 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing unload_modules_local oleg205-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg205-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.