== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:33:55 (1789158835) Stopping clients: oleg209-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg209-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg209-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg209-server oleg209-server: oleg209-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg209-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg209-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 626 (time 1789159048.62 total 10.01 last 62.56) - open/close 1490 (time 1789159058.62 total 20.01 last 86.40) - open/close 2230 (time 1789159068.62 total 30.01 last 74.00) - open/close 3009 (time 1789159078.64 total 40.03 last 77.76) - open/close 3736 (time 1789159088.65 total 50.04 last 72.63) - open/close 4480 (time 1789159098.66 total 60.04 last 74.33) total: 4500 open/close in 60.25 seconds: 74.69 ops/second - /unlink 1610 (time 1789159112.96 total 10.00 last 160.92) - /unlink 3125 (time 1789159122.97 total 20.01 last 151.40) total: 4500 /unlink in 29.41 seconds: 153.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 801 (time 1789159147.03 total 10.02 last 79.91) - open/close 1538 (time 1789159157.04 total 20.04 last 73.61) - open/close 2196 (time 1789159167.05 total 30.05 last 65.69) - open/close 2999 (time 1789159177.07 total 40.07 last 80.19) - open/close 3728 (time 1789159187.07 total 50.07 last 72.86) - open/close 4333 (time 1789159197.09 total 60.08 last 60.42) total: 4500 open/close in 62.30 seconds: 72.24 ops/second - /unlink 1447 (time 1789159215.17 total 10.00 last 144.65) - /unlink 2809 (time 1789159225.17 total 20.01 last 136.16) total: 4500 /unlink in 29.90 seconds: 150.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 860 (time 1789159248.78 total 10.01 last 85.91) - open/close 1688 (time 1789159258.78 total 20.01 last 82.77) - open/close 2584 (time 1789159268.78 total 30.01 last 89.59) - open/close 3545 (time 1789159278.79 total 40.02 last 96.09) - open/close 4438 (time 1789159288.79 total 50.02 last 89.24) total: 4500 open/close in 50.55 seconds: 89.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1702 (time 1789159303.25 total 10.00 last 170.18) - /unlink 3222 (time 1789159313.25 total 20.00 last 151.95) total: 4500 /unlink in 28.82 seconds: 156.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 696 (time 1789159338.13 total 10.02 last 69.47) - open/close 1433 (time 1789159348.13 total 20.02 last 73.70) - open/close 2290 (time 1789159358.14 total 30.03 last 85.57) - open/close 3087 (time 1789159368.15 total 40.04 last 79.64) - open/close 3874 (time 1789159378.16 total 50.05 last 78.66) total: 4500 open/close in 57.48 seconds: 78.29 ops/second - /unlink 1721 (time 1789159399.59 total 10.00 last 172.04) - /unlink 3504 (time 1789159409.59 total 20.01 last 178.24) total: 4500 /unlink in 25.71 seconds: 175.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 890 (time 1789159429.54 total 10.00 last 88.96) - open/close 1661 (time 1789159439.54 total 20.01 last 77.09) - open/close 2489 (time 1789159449.54 total 30.01 last 82.74) - open/close 3288 (time 1789159459.55 total 40.01 last 79.90) - open/close 4103 (time 1789159469.55 total 50.02 last 81.46) total: 4500 open/close in 54.83 seconds: 82.06 ops/second - /unlink 1638 (time 1789159488.50 total 10.02 last 163.53) - /unlink 3033 (time 1789159498.51 total 20.02 last 139.44) total: 4500 /unlink in 28.90 seconds: 155.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 802 (time 1789159521.52 total 10.00 last 80.19) - open/close 1735 (time 1789159531.52 total 20.00 last 93.28) - open/close 2593 (time 1789159541.54 total 30.02 last 85.68) - open/close 3435 (time 1789159551.54 total 40.02 last 84.16) - open/close 4299 (time 1789159561.55 total 50.03 last 86.35) total: 4500 open/close in 53.03 seconds: 84.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1346 (time 1789159579.92 total 10.02 last 134.29) - /unlink 2672 (time 1789159589.92 total 20.02 last 132.59) - /unlink 4151 (time 1789159599.92 total 30.02 last 147.90) total: 4500 /unlink in 32.48 seconds: 138.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 666 (time 1789159617.19 total 10.01 last 66.55) - open/close 1538 (time 1789159627.21 total 20.02 last 87.05) - open/close 2435 (time 1789159637.22 total 30.04 last 89.60) - open/close 3294 (time 1789159647.23 total 40.04 last 85.83) - open/close 4107 (time 1789159657.24 total 50.05 last 81.23) total: 4500 open/close in 55.16 seconds: 81.58 ops/second - /unlink 1450 (time 1789159677.61 total 10.00 last 144.93) - /unlink 3097 (time 1789159687.62 total 20.01 last 164.66) total: 4500 /unlink in 28.70 seconds: 156.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 848 (time 1789159711.79 total 10.00 last 84.77) - open/close 1697 (time 1789159721.80 total 20.02 last 84.76) - open/close 2576 (time 1789159731.81 total 30.03 last 87.85) - open/close 3419 (time 1789159741.84 total 40.05 last 84.07) - open/close 4258 (time 1789159751.84 total 50.06 last 83.82) total: 4500 open/close in 52.64 seconds: 85.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1444 (time 1789159768.90 total 10.00 last 144.39) - /unlink 2930 (time 1789159778.91 total 20.01 last 148.48) - /unlink 4315 (time 1789159788.91 total 30.01 last 138.45) total: 4500 /unlink in 31.31 seconds: 143.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 632 (time 1789159805.25 total 10.02 last 63.06) - open/close 1374 (time 1789159815.26 total 20.03 last 74.11) - open/close 2031 (time 1789159825.27 total 30.04 last 65.65) - open/close 2903 (time 1789159835.28 total 40.05 last 87.11) - open/close 3876 (time 1789159845.28 total 50.05 last 97.30) total: 4500 open/close in 57.86 seconds: 77.77 ops/second - /unlink 1539 (time 1789159867.00 total 10.00 last 153.86) - /unlink 2939 (time 1789159877.01 total 20.01 last 139.90) total: 4500 /unlink in 29.21 seconds: 154.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 739 (time 1789159901.31 total 10.00 last 73.90) - open/close 1649 (time 1789159911.32 total 20.01 last 90.93) - open/close 2604 (time 1789159921.32 total 30.01 last 95.45) - open/close 3491 (time 1789159931.34 total 40.03 last 88.53) - open/close 4381 (time 1789159941.35 total 50.04 last 88.92) total: 4500 open/close in 51.34 seconds: 87.65 ops/second - /unlink 2008 (time 1789159956.69 total 10.01 last 200.65) - /unlink 3527 (time 1789159966.69 total 20.01 last 151.83) total: 4500 /unlink in 25.97 seconds: 173.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 804 (time 1789159987.01 total 10.00 last 80.37) - open/close 1641 (time 1789159997.02 total 20.01 last 83.62) - open/close 2453 (time 1789160007.03 total 30.02 last 81.15) - open/close 3247 (time 1789160017.03 total 40.02 last 79.40) - open/close 4038 (time 1789160027.05 total 50.04 last 78.96) total: 4500 open/close in 57.92 seconds: 77.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 2068 (time 1789160049.43 total 10.00 last 206.75) - /unlink 3773 (time 1789160059.43 total 20.00 last 170.47) total: 4500 /unlink in 24.48 seconds: 183.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 994 (time 1789160078.27 total 10.01 last 99.27) - open/close 1998 (time 1789160088.28 total 20.02 last 100.33) - open/close 2850 (time 1789160098.28 total 30.03 last 85.14) - open/close 3572 (time 1789160108.29 total 40.03 last 72.18) - open/close 4355 (time 1789160118.29 total 50.03 last 78.27) total: 4500 open/close in 51.80 seconds: 86.88 ops/second - /unlink 1486 (time 1789160134.97 total 10.00 last 148.55) - /unlink 3097 (time 1789160144.97 total 20.00 last 161.08) total: 4500 /unlink in 29.70 seconds: 151.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 820 (time 1789160169.30 total 10.00 last 81.98) - open/close 1654 (time 1789160179.32 total 20.02 last 83.24) - open/close 2547 (time 1789160189.32 total 30.03 last 89.26) - open/close 3420 (time 1789160199.34 total 40.04 last 87.14) - open/close 4330 (time 1789160209.35 total 50.05 last 90.91) total: 4500 open/close in 52.58 seconds: 85.58 ops/second - /unlink 1572 (time 1789160226.97 total 10.00 last 157.18) - /unlink 2844 (time 1789160236.97 total 20.00 last 127.17) - /unlink 4141 (time 1789160246.98 total 30.01 last 129.66) total: 4500 /unlink in 32.29 seconds: 139.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 945 (time 1789160264.04 total 10.00 last 94.48) - open/close 1877 (time 1789160274.05 total 20.01 last 93.17) - open/close 2730 (time 1789160284.07 total 30.03 last 85.09) - open/close 3571 (time 1789160294.07 total 40.03 last 84.08) - open/close 4429 (time 1789160304.08 total 50.04 last 85.77) total: 4500 open/close in 50.94 seconds: 88.34 ops/second - /unlink 1631 (time 1789160319.33 total 10.00 last 163.07) - /unlink 3379 (time 1789160329.33 total 20.00 last 174.75) total: 4500 /unlink in 26.46 seconds: 170.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 779 (time 1789160349.45 total 10.00 last 77.88) - open/close 1717 (time 1789160359.45 total 20.00 last 93.80) - open/close 2584 (time 1789160369.45 total 30.01 last 86.66) - open/close 3396 (time 1789160379.47 total 40.03 last 81.02) - open/close 4200 (time 1789160389.48 total 50.04 last 80.36) total: 4500 open/close in 53.66 seconds: 83.86 ops/second - /unlink 1462 (time 1789160408.49 total 10.00 last 146.20) - /unlink 2700 (time 1789160418.50 total 20.01 last 123.66) - /unlink 3940 (time 1789160428.50 total 30.01 last 123.97) total: 4500 /unlink in 33.95 seconds: 132.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 881 (time 1789160447.44 total 10.01 last 88.05) - open/close 1678 (time 1789160457.47 total 20.03 last 79.51) - open/close 2550 (time 1789160467.47 total 30.04 last 87.13) - open/close 3507 (time 1789160477.47 total 40.04 last 95.69) - open/close 4240 (time 1789160487.48 total 50.05 last 73.24) total: 4500 open/close in 53.44 seconds: 84.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1448 (time 1789160506.21 total 10.00 last 144.78) - /unlink 2767 (time 1789160516.22 total 20.01 last 131.77) - /unlink 4415 (time 1789160526.22 total 30.01 last 164.79) total: 4500 /unlink in 30.61 seconds: 147.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 947 (time 1789160541.96 total 10.00 last 94.66) - open/close 1758 (time 1789160551.97 total 20.01 last 81.03) - open/close 2578 (time 1789160561.98 total 30.02 last 81.96) - open/close 3346 (time 1789160572.01 total 40.05 last 76.57) - open/close 4140 (time 1789160582.02 total 50.06 last 79.27) total: 4500 open/close in 53.99 seconds: 83.34 ops/second - /unlink 1511 (time 1789160600.13 total 10.00 last 151.05) - /unlink 2984 (time 1789160610.14 total 20.01 last 147.21) total: 4500 /unlink in 28.72 seconds: 156.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 879 (time 1789160633.53 total 10.01 last 87.78) - open/close 1910 (time 1789160643.55 total 20.03 last 102.89) - open/close 2607 (time 1789160653.55 total 30.04 last 69.69) - open/close 3282 (time 1789160663.57 total 40.05 last 67.41) - open/close 3974 (time 1789160673.58 total 50.06 last 69.13) total: 4500 open/close in 56.31 seconds: 79.91 ops/second - /unlink 1519 (time 1789160693.86 total 10.01 last 151.79) - /unlink 3094 (time 1789160703.86 total 20.01 last 157.47) total: 4500 /unlink in 28.32 seconds: 158.90 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95250 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: modules unloaded.