== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:46:51 (1768564011) Stopping clients: oleg349-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg349-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg349-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg349-server oleg349-server: oleg349-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg349-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg349-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg349-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg349-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 715 (time 1768564192.58 total 10.01 last 71.43) - open/close 1591 (time 1768564202.61 total 20.04 last 87.31) - open/close 2399 (time 1768564212.61 total 30.04 last 80.80) - open/close 3184 (time 1768564222.61 total 40.04 last 78.50) - open/close 3945 (time 1768564232.62 total 50.05 last 76.04) total: 4500 open/close in 57.16 seconds: 78.72 ops/second - /unlink 1399 (time 1768564253.74 total 10.00 last 139.86) - /unlink 2774 (time 1768564263.74 total 20.01 last 137.45) - /unlink 4182 (time 1768564273.75 total 30.01 last 140.70) total: 4500 /unlink in 32.36 seconds: 139.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 871 (time 1768564290.25 total 10.00 last 87.08) - open/close 1717 (time 1768564300.26 total 20.01 last 84.56) - open/close 2535 (time 1768564310.26 total 30.01 last 81.79) - open/close 3373 (time 1768564320.26 total 40.01 last 83.76) - open/close 4204 (time 1768564330.27 total 50.02 last 83.03) total: 4500 open/close in 53.35 seconds: 84.34 ops/second - /unlink 1637 (time 1768564347.61 total 10.00 last 163.64) - /unlink 3419 (time 1768564357.62 total 20.01 last 178.02) total: 4500 /unlink in 27.43 seconds: 164.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 902 (time 1768564378.96 total 10.00 last 90.17) - open/close 1725 (time 1768564388.97 total 20.01 last 82.25) - open/close 2598 (time 1768564398.97 total 30.02 last 87.23) - open/close 3493 (time 1768564408.98 total 40.03 last 89.43) - open/close 4425 (time 1768564418.99 total 50.03 last 93.15) total: 4500 open/close in 50.88 seconds: 88.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1592 (time 1768564433.77 total 10.01 last 159.06) - /unlink 3155 (time 1768564443.77 total 20.01 last 156.29) total: 4500 /unlink in 27.76 seconds: 162.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 869 (time 1768564465.53 total 10.00 last 86.88) - open/close 1735 (time 1768564475.53 total 20.00 last 86.60) - open/close 2694 (time 1768564485.54 total 30.01 last 95.86) - open/close 3589 (time 1768564495.54 total 40.01 last 89.50) - open/close 4453 (time 1768564505.55 total 50.02 last 86.32) total: 4500 open/close in 50.55 seconds: 89.02 ops/second - /unlink 1584 (time 1768564519.62 total 10.01 last 158.30) - /unlink 3263 (time 1768564529.62 total 20.01 last 167.89) total: 4500 /unlink in 27.88 seconds: 161.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 812 (time 1768564551.65 total 10.01 last 81.11) - open/close 1681 (time 1768564561.66 total 20.02 last 86.81) - open/close 2520 (time 1768564571.67 total 30.04 last 83.78) - open/close 3329 (time 1768564581.68 total 40.04 last 80.87) - open/close 4270 (time 1768564591.68 total 50.05 last 94.02) total: 4500 open/close in 52.22 seconds: 86.18 ops/second - /unlink 1864 (time 1768564607.30 total 10.00 last 186.36) - /unlink 3420 (time 1768564617.31 total 20.01 last 155.44) total: 4500 /unlink in 26.50 seconds: 169.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 973 (time 1768564637.29 total 10.00 last 97.26) - open/close 1922 (time 1768564647.29 total 20.01 last 94.87) - open/close 2978 (time 1768564657.29 total 30.01 last 105.59) - open/close 3885 (time 1768564667.30 total 40.01 last 90.64) total: 4500 open/close in 46.49 seconds: 96.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1716 (time 1768564686.80 total 10.00 last 171.54) - /unlink 3439 (time 1768564696.81 total 20.01 last 172.27) total: 4500 /unlink in 26.79 seconds: 168.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 852 (time 1768564717.39 total 10.00 last 85.17) - open/close 1696 (time 1768564727.40 total 20.01 last 84.32) - open/close 2565 (time 1768564737.41 total 30.02 last 86.86) - open/close 3485 (time 1768564747.41 total 40.02 last 91.98) - open/close 4497 (time 1768564757.42 total 50.03 last 101.14) total: 4500 open/close in 50.04 seconds: 89.93 ops/second - /unlink 1558 (time 1768564770.71 total 10.00 last 155.80) - /unlink 3073 (time 1768564780.72 total 20.01 last 151.39) total: 4500 /unlink in 29.53 seconds: 152.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 860 (time 1768564803.57 total 10.01 last 85.90) - open/close 1754 (time 1768564813.58 total 20.02 last 89.34) - open/close 2599 (time 1768564823.58 total 30.02 last 84.48) - open/close 3444 (time 1768564833.59 total 40.03 last 84.45) - open/close 4254 (time 1768564843.59 total 50.04 last 80.93) total: 4500 open/close in 52.90 seconds: 85.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1536 (time 1768564859.61 total 10.01 last 153.48) - /unlink 3063 (time 1768564869.61 total 20.01 last 152.62) total: 4500 /unlink in 28.88 seconds: 155.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 921 (time 1768564891.51 total 10.01 last 92.01) - open/close 1891 (time 1768564901.52 total 20.02 last 96.91) - open/close 2905 (time 1768564911.53 total 30.03 last 101.32) - open/close 3817 (time 1768564921.55 total 40.04 last 91.04) total: 4500 open/close in 47.49 seconds: 94.76 ops/second - /unlink 1578 (time 1768564941.94 total 10.00 last 157.77) - /unlink 3225 (time 1768564951.94 total 20.00 last 164.66) total: 4500 /unlink in 26.98 seconds: 166.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 888 (time 1768564972.28 total 10.01 last 88.69) - open/close 1692 (time 1768564982.29 total 20.01 last 80.38) - open/close 2512 (time 1768564992.29 total 30.02 last 82.00) - open/close 3423 (time 1768565002.29 total 40.02 last 91.09) - open/close 4299 (time 1768565012.29 total 50.02 last 87.56) total: 4500 open/close in 51.84 seconds: 86.80 ops/second - /unlink 1620 (time 1768565027.58 total 10.00 last 161.99) - /unlink 3281 (time 1768565037.59 total 20.01 last 165.94) total: 4500 /unlink in 27.09 seconds: 166.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1074 (time 1768565057.71 total 10.00 last 107.35) - open/close 1975 (time 1768565067.72 total 20.01 last 90.06) - open/close 2904 (time 1768565077.72 total 30.01 last 92.89) - open/close 3870 (time 1768565087.72 total 40.01 last 96.58) total: 4500 open/close in 46.53 seconds: 96.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1866 (time 1768565107.34 total 10.00 last 186.59) - /unlink 3610 (time 1768565117.34 total 20.01 last 174.31) total: 4500 /unlink in 24.91 seconds: 180.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 880 (time 1768565136.04 total 10.00 last 87.96) - open/close 1877 (time 1768565146.05 total 20.01 last 99.67) - open/close 2759 (time 1768565156.06 total 30.02 last 88.07) - open/close 3736 (time 1768565166.07 total 40.03 last 97.62) total: 4500 open/close in 47.69 seconds: 94.36 ops/second - /unlink 1808 (time 1768565186.57 total 10.00 last 180.79) - /unlink 3789 (time 1768565196.58 total 20.01 last 197.92) total: 4500 /unlink in 23.59 seconds: 190.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 1010 (time 1768565213.18 total 10.00 last 100.96) - open/close 2047 (time 1768565223.19 total 20.01 last 103.60) - open/close 3095 (time 1768565233.20 total 30.02 last 104.72) - open/close 4106 (time 1768565243.20 total 40.02 last 101.08) total: 4500 open/close in 43.73 seconds: 102.91 ops/second - /unlink 2110 (time 1768565259.68 total 10.00 last 210.94) - /unlink 3939 (time 1768565269.68 total 20.00 last 182.87) total: 4500 /unlink in 23.01 seconds: 195.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 1078 (time 1768565285.63 total 10.02 last 107.57) - open/close 2113 (time 1768565295.65 total 20.04 last 103.34) - open/close 3128 (time 1768565305.66 total 30.05 last 101.35) - open/close 4236 (time 1768565315.68 total 40.07 last 110.59) total: 4500 open/close in 42.59 seconds: 105.67 ops/second - /unlink 1969 (time 1768565330.88 total 10.00 last 196.89) - /unlink 4044 (time 1768565340.88 total 20.00 last 207.43) total: 4500 /unlink in 22.14 seconds: 203.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 1290 (time 1768565355.69 total 10.00 last 128.97) - open/close 2685 (time 1768565365.69 total 20.01 last 139.42) - open/close 4150 (time 1768565375.70 total 30.02 last 146.36) total: 4500 open/close in 32.80 seconds: 137.21 ops/second - /unlink 2138 (time 1768565390.83 total 10.00 last 213.72) - /unlink 4208 (time 1768565400.83 total 20.01 last 206.94) total: 4500 /unlink in 21.59 seconds: 208.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1153 (time 1768565415.36 total 10.01 last 115.19) - open/close 2203 (time 1768565425.37 total 20.01 last 104.94) - open/close 3442 (time 1768565435.37 total 30.02 last 123.87) total: 4500 open/close in 38.44 seconds: 117.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 2469 (time 1768565456.07 total 10.01 last 246.77) total: 4500 /unlink in 19.83 seconds: 226.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1200 (time 1768565478.38 total 10.01 last 119.92) - open/close 2263 (time 1768565488.39 total 20.01 last 106.22) - open/close 3471 (time 1768565498.39 total 30.01 last 120.80) total: 4500 open/close in 37.48 seconds: 120.07 ops/second - /unlink 2580 (time 1768565518.21 total 10.01 last 257.86) total: 4500 /unlink in 17.36 seconds: 259.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 1251 (time 1768565537.89 total 10.00 last 125.10) - open/close 2415 (time 1768565547.90 total 20.01 last 116.33) - open/close 3552 (time 1768565557.90 total 30.01 last 113.65) total: 4500 open/close in 37.39 seconds: 120.34 ops/second - /unlink 2331 (time 1768565577.38 total 10.00 last 233.06) total: 4500 /unlink in 19.59 seconds: 229.69 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11101: 94057 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg349-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local oleg349-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg349-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.