== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:57:52 (1764107872) Stopping clients: oleg443-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg443-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg443-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg443-server oleg443-server: oleg443-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg443-server: detected 4 online CPUs by sysfs oleg443-server: MODOPTS_LIBCFS= oleg443-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg443-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg443-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg443-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_loc=0x1312 fail_val=5 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 771 (time 1764108022.41 total 10.00 last 77.07) - open/close 1563 (time 1764108032.42 total 20.01 last 79.11) - open/close 2392 (time 1764108042.43 total 30.02 last 82.82) - open/close 3200 (time 1764108052.44 total 40.03 last 80.77) - open/close 3819 (time 1764108062.44 total 50.03 last 61.90) - open/close 4332 (time 1764108072.45 total 60.04 last 51.24) total: 4500 open/close in 64.04 seconds: 70.27 ops/second - /unlink 1097 (time 1764108092.71 total 10.00 last 109.68) - /unlink 2467 (time 1764108102.72 total 20.01 last 136.93) - /unlink 3994 (time 1764108112.72 total 30.01 last 152.64) total: 4500 /unlink in 33.22 seconds: 135.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 916 (time 1764108129.18 total 10.01 last 91.50) - open/close 1768 (time 1764108139.18 total 20.01 last 85.18) - open/close 2676 (time 1764108149.19 total 30.02 last 90.72) - open/close 3562 (time 1764108159.19 total 40.03 last 88.58) - open/close 4465 (time 1764108169.19 total 50.03 last 90.29) total: 4500 open/close in 50.46 seconds: 89.18 ops/second - /unlink 1593 (time 1764108183.05 total 10.01 last 159.21) - /unlink 2978 (time 1764108193.05 total 20.01 last 138.44) - /unlink 4263 (time 1764108203.06 total 30.02 last 128.35) total: 4500 /unlink in 32.47 seconds: 138.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 803 (time 1764108220.12 total 10.01 last 80.23) - open/close 1624 (time 1764108230.12 total 20.02 last 82.04) - open/close 2533 (time 1764108240.13 total 30.02 last 90.87) - open/close 3358 (time 1764108250.13 total 40.02 last 82.46) - open/close 4220 (time 1764108260.13 total 50.03 last 86.19) total: 4500 open/close in 53.35 seconds: 84.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1606 (time 1764108277.08 total 10.00 last 160.60) - /unlink 3068 (time 1764108287.08 total 20.00 last 146.18) total: 4500 /unlink in 28.63 seconds: 157.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 873 (time 1764108309.36 total 10.01 last 87.24) - open/close 1754 (time 1764108319.37 total 20.02 last 88.02) - open/close 2578 (time 1764108329.37 total 30.02 last 82.39) - open/close 3507 (time 1764108339.37 total 40.02 last 92.89) - open/close 4397 (time 1764108349.38 total 50.02 last 88.96) total: 4500 open/close in 51.12 seconds: 88.02 ops/second - /unlink 1697 (time 1764108363.39 total 10.00 last 169.65) - /unlink 3247 (time 1764108373.39 total 20.00 last 154.98) total: 4500 /unlink in 27.65 seconds: 162.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 903 (time 1764108394.13 total 10.01 last 90.25) - open/close 1752 (time 1764108404.13 total 20.01 last 84.86) - open/close 2653 (time 1764108414.14 total 30.02 last 90.03) - open/close 3580 (time 1764108424.14 total 40.02 last 92.66) total: 4500 open/close in 49.81 seconds: 90.34 ops/second - /unlink 1599 (time 1764108447.18 total 10.00 last 159.86) - /unlink 3272 (time 1764108457.18 total 20.00 last 167.28) total: 4500 /unlink in 27.52 seconds: 163.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 905 (time 1764108477.70 total 10.00 last 90.46) - open/close 1811 (time 1764108487.70 total 20.01 last 90.57) - open/close 2734 (time 1764108497.70 total 30.01 last 92.27) - open/close 3684 (time 1764108507.71 total 40.01 last 94.97) total: 4500 open/close in 48.95 seconds: 91.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1633 (time 1764108529.79 total 10.00 last 163.28) - /unlink 3359 (time 1764108539.79 total 20.00 last 172.54) total: 4500 /unlink in 27.15 seconds: 165.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 539 (time 1764108559.89 total 10.02 last 53.78) - open/close 956 (time 1764108569.90 total 20.03 last 41.67) - open/close 1415 (time 1764108579.92 total 30.06 last 45.79) - open/close 2010 (time 1764108589.93 total 40.06 last 59.47) - open/close 2853 (time 1764108599.93 total 50.06 last 84.27) - open/close 3670 (time 1764108609.93 total 60.07 last 81.68) - open/close 4444 (time 1764108619.94 total 70.07 last 77.34) total: 4500 open/close in 70.68 seconds: 63.66 ops/second - /unlink 1520 (time 1764108634.10 total 10.01 last 151.91) - /unlink 3039 (time 1764108644.10 total 20.01 last 151.88) - /unlink 4447 (time 1764108654.11 total 30.02 last 140.67) total: 4500 /unlink in 30.32 seconds: 148.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 819 (time 1764108667.88 total 10.00 last 81.89) - open/close 1610 (time 1764108677.89 total 20.01 last 79.03) - open/close 2433 (time 1764108687.89 total 30.02 last 82.24) - open/close 3265 (time 1764108697.90 total 40.02 last 83.18) - open/close 4135 (time 1764108707.90 total 50.02 last 86.97) total: 4500 open/close in 54.13 seconds: 83.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1663 (time 1764108724.91 total 10.00 last 166.22) - /unlink 3214 (time 1764108734.91 total 20.01 last 155.08) total: 4500 /unlink in 27.96 seconds: 160.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 949 (time 1764108755.80 total 10.01 last 94.82) - open/close 1938 (time 1764108765.80 total 20.01 last 98.86) - open/close 3104 (time 1764108775.81 total 30.01 last 116.57) - open/close 4060 (time 1764108785.82 total 40.03 last 95.43) total: 4500 open/close in 44.92 seconds: 100.19 ops/second - /unlink 1814 (time 1764108803.43 total 10.00 last 181.34) - /unlink 3518 (time 1764108813.43 total 20.01 last 170.28) total: 4500 /unlink in 25.60 seconds: 175.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 907 (time 1764108831.22 total 10.00 last 90.69) - open/close 1793 (time 1764108841.23 total 20.01 last 88.55) - open/close 2704 (time 1764108851.23 total 30.01 last 91.10) - open/close 3606 (time 1764108861.23 total 40.01 last 90.17) - open/close 4450 (time 1764108871.24 total 50.02 last 84.36) total: 4500 open/close in 50.63 seconds: 88.88 ops/second - /unlink 1775 (time 1764108885.34 total 10.00 last 177.50) - /unlink 3354 (time 1764108895.35 total 20.00 last 157.87) total: 4500 /unlink in 26.82 seconds: 167.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1044 (time 1764108915.78 total 10.00 last 104.38) - open/close 2004 (time 1764108925.79 total 20.01 last 95.96) - open/close 2956 (time 1764108935.81 total 30.02 last 95.02) - open/close 3935 (time 1764108945.81 total 40.03 last 97.82) total: 4500 open/close in 46.38 seconds: 97.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1625 (time 1764108965.30 total 10.00 last 162.49) - /unlink 3192 (time 1764108975.30 total 20.00 last 156.67) total: 4500 /unlink in 27.61 seconds: 162.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 896 (time 1764108995.54 total 10.00 last 89.59) - open/close 1831 (time 1764109005.54 total 20.00 last 93.50) - open/close 2887 (time 1764109015.54 total 30.00 last 105.58) - open/close 3913 (time 1764109025.54 total 40.00 last 102.60) total: 4500 open/close in 45.41 seconds: 99.09 ops/second - /unlink 1961 (time 1764109043.37 total 10.00 last 196.01) - /unlink 3665 (time 1764109053.38 total 20.01 last 170.32) total: 4500 /unlink in 25.23 seconds: 178.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 822 (time 1764109071.26 total 10.02 last 82.04) - open/close 1615 (time 1764109081.27 total 20.03 last 79.24) - open/close 2390 (time 1764109091.27 total 30.03 last 77.46) - open/close 2943 (time 1764109101.30 total 40.06 last 55.16) - open/close 3486 (time 1764109111.31 total 50.07 last 54.22) - open/close 4249 (time 1764109121.33 total 60.09 last 76.14) total: 4500 open/close in 63.08 seconds: 71.33 ops/second - /unlink 1579 (time 1764109137.53 total 10.01 last 157.79) - /unlink 3161 (time 1764109147.53 total 20.01 last 158.19) total: 4500 /unlink in 28.93 seconds: 155.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 952 (time 1764109170.24 total 10.01 last 95.13) - open/close 1845 (time 1764109180.26 total 20.02 last 89.18) - open/close 2746 (time 1764109190.26 total 30.02 last 90.09) - open/close 3577 (time 1764109200.26 total 40.02 last 83.08) - open/close 4341 (time 1764109210.27 total 50.03 last 76.36) total: 4500 open/close in 51.92 seconds: 86.68 ops/second - /unlink 1537 (time 1764109225.69 total 10.01 last 153.52) - /unlink 3397 (time 1764109235.70 total 20.01 last 185.94) total: 4500 /unlink in 25.44 seconds: 176.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 990 (time 1764109253.71 total 10.00 last 98.97) - open/close 1890 (time 1764109263.71 total 20.00 last 89.98) - open/close 2784 (time 1764109273.71 total 30.01 last 89.38) - open/close 3729 (time 1764109283.71 total 40.01 last 94.46) total: 4500 open/close in 48.18 seconds: 93.39 ops/second - /unlink 1642 (time 1764109304.61 total 10.00 last 164.18) - /unlink 3348 (time 1764109314.61 total 20.00 last 170.58) total: 4500 /unlink in 26.37 seconds: 170.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1048 (time 1764109333.61 total 10.01 last 104.72) - open/close 2054 (time 1764109343.62 total 20.02 last 100.50) - open/close 3075 (time 1764109353.63 total 30.03 last 102.02) - open/close 4045 (time 1764109363.64 total 40.03 last 96.94) total: 4500 open/close in 45.15 seconds: 99.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1575 (time 1764109383.17 total 10.00 last 157.46) - /unlink 2845 (time 1764109393.17 total 20.00 last 126.98) - /unlink 4296 (time 1764109403.17 total 30.00 last 145.10) total: 4500 /unlink in 31.49 seconds: 142.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 774 (time 1764109418.73 total 10.01 last 77.33) - open/close 1670 (time 1764109428.73 total 20.01 last 89.60) - open/close 2673 (time 1764109438.74 total 30.01 last 100.25) - open/close 3721 (time 1764109448.74 total 40.02 last 104.75) total: 4500 open/close in 48.11 seconds: 93.54 ops/second - /unlink 1498 (time 1764109470.37 total 10.00 last 149.78) - /unlink 3504 (time 1764109480.37 total 20.00 last 200.56) total: 4500 /unlink in 25.16 seconds: 178.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 1019 (time 1764109498.04 total 10.01 last 101.81) - open/close 2100 (time 1764109508.05 total 20.01 last 108.05) - open/close 3138 (time 1764109518.06 total 30.03 last 103.65) - open/close 4124 (time 1764109528.07 total 40.03 last 98.54) total: 4500 open/close in 43.99 seconds: 102.31 ops/second - /unlink 2227 (time 1764109544.37 total 10.00 last 222.67) - /unlink 4107 (time 1764109554.37 total 20.01 last 187.91) total: 4500 /unlink in 21.88 seconds: 205.69 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 93167 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg443-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local oleg443-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg443-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.