== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:35:18 (1781238918) Stopping clients: oleg119-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg119-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg119-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg119-server oleg119-server: oleg119-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg119-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg119-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 847 (time 1781239111.79 total 10.01 last 84.63) - open/close 1663 (time 1781239121.80 total 20.02 last 81.54) - open/close 2539 (time 1781239131.80 total 30.02 last 87.57) - open/close 3561 (time 1781239141.81 total 40.03 last 102.13) - open/close 4404 (time 1781239151.81 total 50.03 last 84.29) total: 4500 open/close in 51.27 seconds: 87.77 ops/second - /unlink 1698 (time 1781239167.04 total 10.00 last 169.79) - /unlink 3361 (time 1781239177.04 total 20.00 last 166.29) total: 4500 /unlink in 26.86 seconds: 167.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 838 (time 1781239198.15 total 10.00 last 83.78) - open/close 1729 (time 1781239208.16 total 20.01 last 89.03) - open/close 2532 (time 1781239218.17 total 30.02 last 80.25) - open/close 3324 (time 1781239228.17 total 40.02 last 79.20) - open/close 4296 (time 1781239238.17 total 50.02 last 97.19) total: 4500 open/close in 52.28 seconds: 86.08 ops/second - /unlink 1536 (time 1781239254.23 total 10.00 last 153.56) - /unlink 3220 (time 1781239264.23 total 20.01 last 168.35) total: 4500 /unlink in 28.46 seconds: 158.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 774 (time 1781239286.94 total 10.02 last 77.22) - open/close 1574 (time 1781239296.95 total 20.03 last 79.92) - open/close 2452 (time 1781239306.97 total 30.05 last 87.64) - open/close 3240 (time 1781239316.98 total 40.06 last 78.70) - open/close 4138 (time 1781239326.99 total 50.07 last 89.76) total: 4500 open/close in 53.76 seconds: 83.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1523 (time 1781239344.78 total 10.00 last 152.27) - /unlink 3083 (time 1781239354.78 total 20.00 last 155.96) total: 4500 /unlink in 28.69 seconds: 156.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 972 (time 1781239378.06 total 10.01 last 97.15) - open/close 1857 (time 1781239388.07 total 20.01 last 88.45) - open/close 2674 (time 1781239398.07 total 30.02 last 81.65) - open/close 3353 (time 1781239408.08 total 40.03 last 67.84) - open/close 4213 (time 1781239418.09 total 50.04 last 85.90) total: 4500 open/close in 53.25 seconds: 84.51 ops/second - /unlink 1693 (time 1781239435.99 total 10.00 last 169.23) - /unlink 3237 (time 1781239445.99 total 20.01 last 154.36) total: 4500 /unlink in 29.02 seconds: 155.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 973 (time 1781239469.17 total 10.01 last 97.20) - open/close 1912 (time 1781239479.18 total 20.01 last 93.86) - open/close 2851 (time 1781239489.18 total 30.02 last 93.85) - open/close 3673 (time 1781239499.19 total 40.03 last 82.12) - open/close 4304 (time 1781239509.20 total 50.03 last 63.08) total: 4500 open/close in 53.03 seconds: 84.86 ops/second - /unlink 1283 (time 1781239527.08 total 10.01 last 128.19) - /unlink 2683 (time 1781239537.08 total 20.01 last 139.99) - /unlink 3878 (time 1781239547.09 total 30.02 last 119.41) total: 4500 /unlink in 34.58 seconds: 130.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 653 (time 1781239568.05 total 10.01 last 65.26) - open/close 1430 (time 1781239578.06 total 20.02 last 77.63) - open/close 2267 (time 1781239588.10 total 30.05 last 83.43) - open/close 2980 (time 1781239598.11 total 40.06 last 71.19) - open/close 3717 (time 1781239608.13 total 50.08 last 73.58) - open/close 4451 (time 1781239618.15 total 60.10 last 73.23) total: 4500 open/close in 60.72 seconds: 74.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1525 (time 1781239633.96 total 10.00 last 152.47) - /unlink 3039 (time 1781239643.97 total 20.01 last 151.32) - /unlink 4158 (time 1781239653.98 total 30.02 last 111.72) total: 4500 /unlink in 32.77 seconds: 137.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 499 (time 1781239672.93 total 10.01 last 49.87) - open/close 928 (time 1781239682.94 total 20.02 last 42.86) - open/close 1476 (time 1781239692.94 total 30.02 last 54.76) - open/close 1940 (time 1781239702.94 total 40.02 last 46.40) - open/close 2651 (time 1781239712.94 total 50.02 last 71.10) - open/close 3434 (time 1781239722.95 total 60.03 last 78.25) - open/close 4159 (time 1781239732.95 total 70.03 last 72.48) total: 4500 open/close in 74.56 seconds: 60.35 ops/second - /unlink 1298 (time 1781239751.97 total 10.00 last 129.79) - /unlink 2752 (time 1781239761.97 total 20.00 last 145.39) - /unlink 3951 (time 1781239771.97 total 30.00 last 119.88) total: 4500 /unlink in 34.44 seconds: 130.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 700 (time 1781239791.35 total 10.00 last 69.98) - open/close 1375 (time 1781239801.36 total 20.01 last 67.48) - open/close 2151 (time 1781239811.36 total 30.01 last 77.60) - open/close 2933 (time 1781239821.38 total 40.03 last 78.01) - open/close 3696 (time 1781239831.39 total 50.04 last 76.20) - open/close 4439 (time 1781239841.40 total 60.05 last 74.24) total: 4500 open/close in 60.65 seconds: 74.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1521 (time 1781239855.62 total 10.00 last 152.10) - /unlink 3001 (time 1781239865.62 total 20.00 last 147.94) - /unlink 4268 (time 1781239875.63 total 30.01 last 126.57) total: 4500 /unlink in 31.92 seconds: 140.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 584 (time 1781239892.95 total 10.00 last 58.39) - open/close 1229 (time 1781239902.95 total 20.00 last 64.48) - open/close 1919 (time 1781239912.97 total 30.02 last 68.86) - open/close 2590 (time 1781239922.98 total 40.03 last 67.05) - open/close 3287 (time 1781239932.99 total 50.04 last 69.65) - open/close 3948 (time 1781239942.99 total 60.05 last 66.06) total: 4500 open/close in 68.37 seconds: 65.82 ops/second - /unlink 1703 (time 1781239966.23 total 10.00 last 170.30) - /unlink 3067 (time 1781239976.24 total 20.00 last 136.35) total: 4500 /unlink in 29.90 seconds: 150.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 796 (time 1781240000.64 total 10.01 last 79.51) - open/close 1566 (time 1781240010.64 total 20.01 last 76.98) - open/close 2312 (time 1781240020.65 total 30.02 last 74.54) - open/close 3036 (time 1781240030.65 total 40.03 last 72.36) - open/close 3995 (time 1781240040.66 total 50.04 last 95.81) total: 4500 open/close in 57.28 seconds: 78.56 ops/second - /unlink 1162 (time 1781240062.77 total 10.00 last 116.19) - /unlink 2587 (time 1781240072.78 total 20.01 last 142.34) - /unlink 3937 (time 1781240082.79 total 30.02 last 134.89) total: 4500 /unlink in 33.80 seconds: 133.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 791 (time 1781240100.93 total 10.01 last 79.05) - open/close 1767 (time 1781240110.93 total 20.01 last 97.60) - open/close 2591 (time 1781240120.96 total 30.03 last 82.18) - open/close 3391 (time 1781240130.98 total 40.06 last 79.82) - open/close 4132 (time 1781240140.99 total 50.07 last 74.02) total: 4500 open/close in 55.29 seconds: 81.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1419 (time 1781240160.22 total 10.00 last 141.84) - /unlink 3020 (time 1781240170.22 total 20.01 last 160.01) total: 4500 /unlink in 29.38 seconds: 153.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 691 (time 1781240194.88 total 10.02 last 68.96) - open/close 1504 (time 1781240204.88 total 20.03 last 81.22) - open/close 2306 (time 1781240214.88 total 30.03 last 80.20) - open/close 3034 (time 1781240224.89 total 40.04 last 72.75) - open/close 3812 (time 1781240234.90 total 50.04 last 77.75) total: 4500 open/close in 58.80 seconds: 76.53 ops/second - /unlink 1569 (time 1781240257.90 total 10.01 last 156.76) - /unlink 3127 (time 1781240267.91 total 20.01 last 155.77) total: 4500 /unlink in 28.99 seconds: 155.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 749 (time 1781240291.82 total 10.02 last 74.79) - open/close 1477 (time 1781240301.85 total 20.04 last 72.60) - open/close 2220 (time 1781240311.85 total 30.05 last 74.28) - open/close 3074 (time 1781240321.85 total 40.05 last 85.39) - open/close 4006 (time 1781240331.86 total 50.06 last 93.09) total: 4500 open/close in 55.05 seconds: 81.74 ops/second - /unlink 1704 (time 1781240350.13 total 10.00 last 170.36) - /unlink 3303 (time 1781240360.14 total 20.01 last 159.73) total: 4500 /unlink in 27.19 seconds: 165.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 910 (time 1781240381.38 total 10.01 last 90.93) - open/close 1773 (time 1781240391.39 total 20.01 last 86.28) - open/close 2785 (time 1781240401.39 total 30.02 last 101.10) - open/close 3648 (time 1781240411.40 total 40.03 last 86.24) total: 4500 open/close in 49.65 seconds: 90.63 ops/second - /unlink 1669 (time 1781240434.63 total 10.00 last 166.88) - /unlink 3281 (time 1781240444.63 total 20.00 last 161.16) total: 4500 /unlink in 27.52 seconds: 163.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 579 (time 1781240466.14 total 10.02 last 57.79) - open/close 1210 (time 1781240476.15 total 20.03 last 63.06) - open/close 2068 (time 1781240486.16 total 30.04 last 85.72) - open/close 3106 (time 1781240496.17 total 40.05 last 103.69) - open/close 3869 (time 1781240506.18 total 50.05 last 76.24) total: 4500 open/close in 57.77 seconds: 77.89 ops/second - /unlink 1661 (time 1781240527.93 total 10.00 last 166.10) - /unlink 3249 (time 1781240537.94 total 20.00 last 158.75) total: 4500 /unlink in 26.85 seconds: 167.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 819 (time 1781240559.59 total 10.01 last 81.80) - open/close 1635 (time 1781240569.59 total 20.01 last 81.58) - open/close 2510 (time 1781240579.60 total 30.03 last 87.39) - open/close 3425 (time 1781240589.61 total 40.04 last 91.43) - open/close 4280 (time 1781240599.61 total 50.04 last 85.49) total: 4500 open/close in 52.28 seconds: 86.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1299 (time 1781240616.75 total 10.01 last 129.75) - /unlink 3145 (time 1781240626.76 total 20.02 last 184.48) total: 4500 /unlink in 28.61 seconds: 157.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 946 (time 1781240649.83 total 10.00 last 94.59) - open/close 1940 (time 1781240659.84 total 20.01 last 99.35) - open/close 2871 (time 1781240669.85 total 30.02 last 93.00) - open/close 3748 (time 1781240679.86 total 40.03 last 87.60) total: 4500 open/close in 49.18 seconds: 91.49 ops/second - /unlink 1361 (time 1781240703.51 total 10.02 last 135.88) - /unlink 2999 (time 1781240713.51 total 20.02 last 163.75) total: 4500 /unlink in 29.16 seconds: 154.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 964 (time 1781240736.97 total 10.00 last 96.39) - open/close 1986 (time 1781240747.00 total 20.03 last 101.95) - open/close 2932 (time 1781240757.01 total 30.04 last 94.47) - open/close 3776 (time 1781240767.03 total 40.05 last 84.27) total: 4500 open/close in 48.15 seconds: 93.45 ops/second - /unlink 1430 (time 1781240789.38 total 10.00 last 143.00) - /unlink 2957 (time 1781240799.39 total 20.00 last 152.66) - /unlink 4488 (time 1781240809.39 total 30.01 last 153.06) total: 4500 /unlink in 30.10 seconds: 149.49 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11209: 94043 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local oleg119-server: modules unloaded.