== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:12:59 (1788275579) Stopping clients: oleg235-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg235-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg235-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg235-server oleg235-server: oleg235-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg235-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg235-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg235-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 593 (time 1788275817.37 total 10.01 last 59.22) - open/close 929 (time 1788275827.38 total 20.02 last 33.58) - open/close 1377 (time 1788275837.38 total 30.02 last 44.79) - open/close 2002 (time 1788275847.39 total 40.02 last 62.48) - open/close 2901 (time 1788275857.39 total 50.03 last 89.88) - open/close 3817 (time 1788275867.40 total 60.04 last 91.48) total: 4500 open/close in 67.98 seconds: 66.19 ops/second - /unlink 1613 (time 1788275889.46 total 10.01 last 161.20) - /unlink 3202 (time 1788275899.46 total 20.01 last 158.89) total: 4500 /unlink in 28.69 seconds: 156.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 654 (time 1788275924.03 total 10.00 last 65.37) - open/close 1284 (time 1788275934.03 total 20.01 last 62.97) - open/close 1978 (time 1788275944.06 total 30.04 last 69.21) - open/close 2566 (time 1788275954.06 total 40.04 last 58.79) - open/close 3329 (time 1788275964.07 total 50.04 last 76.26) - open/close 4000 (time 1788275974.08 total 60.05 last 67.04) total: 4500 open/close in 67.39 seconds: 66.77 ops/second - /unlink 1476 (time 1788275996.97 total 10.00 last 147.57) - /unlink 2945 (time 1788276006.97 total 20.00 last 146.88) - /unlink 4399 (time 1788276016.98 total 30.01 last 145.32) total: 4500 /unlink in 31.05 seconds: 144.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 762 (time 1788276033.36 total 10.00 last 76.17) - open/close 1475 (time 1788276043.36 total 20.01 last 71.26) - open/close 2181 (time 1788276053.37 total 30.01 last 70.58) - open/close 2782 (time 1788276063.38 total 40.02 last 60.03) - open/close 3367 (time 1788276073.38 total 50.03 last 58.48) - open/close 3799 (time 1788276083.40 total 60.05 last 43.11) - open/close 4292 (time 1788276093.41 total 70.05 last 49.27) total: 4500 open/close in 72.99 seconds: 61.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1657 (time 1788276111.86 total 10.01 last 165.60) - /unlink 3103 (time 1788276121.86 total 20.01 last 144.57) total: 4500 /unlink in 29.32 seconds: 153.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 771 (time 1788276146.87 total 10.00 last 77.07) - open/close 1506 (time 1788276156.87 total 20.01 last 73.49) - open/close 2203 (time 1788276166.88 total 30.01 last 69.64) - open/close 3006 (time 1788276176.88 total 40.02 last 80.29) - open/close 3761 (time 1788276186.90 total 50.03 last 75.36) - open/close 4364 (time 1788276196.91 total 60.04 last 60.26) total: 4500 open/close in 62.69 seconds: 71.78 ops/second - /unlink 1460 (time 1788276215.13 total 10.00 last 145.98) - /unlink 3094 (time 1788276225.13 total 20.01 last 163.28) total: 4500 /unlink in 28.77 seconds: 156.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 841 (time 1788276249.05 total 10.01 last 84.05) - open/close 1568 (time 1788276259.07 total 20.02 last 72.57) - open/close 2267 (time 1788276269.07 total 30.02 last 69.90) - open/close 2962 (time 1788276279.08 total 40.04 last 69.41) - open/close 3596 (time 1788276289.09 total 50.04 last 63.37) - open/close 4096 (time 1788276299.09 total 60.04 last 49.99) total: 4500 open/close in 65.51 seconds: 68.69 ops/second - /unlink 1518 (time 1788276320.15 total 10.00 last 151.78) - /unlink 2882 (time 1788276330.15 total 20.00 last 136.38) total: 4500 /unlink in 29.82 seconds: 150.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 658 (time 1788276355.69 total 10.08 last 65.29) - open/close 1312 (time 1788276365.70 total 20.08 last 65.38) - open/close 2058 (time 1788276375.71 total 30.10 last 74.50) - open/close 2814 (time 1788276385.71 total 40.10 last 75.57) - open/close 3723 (time 1788276395.72 total 50.10 last 90.85) total: 4500 open/close in 58.67 seconds: 76.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1710 (time 1788276418.94 total 10.00 last 170.97) - /unlink 3429 (time 1788276428.94 total 20.00 last 171.84) total: 4500 /unlink in 27.00 seconds: 166.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 804 (time 1788276450.82 total 10.02 last 80.27) - open/close 1737 (time 1788276460.85 total 20.04 last 93.06) - open/close 2517 (time 1788276470.87 total 30.06 last 77.86) - open/close 3285 (time 1788276480.87 total 40.06 last 76.79) - open/close 4141 (time 1788276490.87 total 50.06 last 85.60) total: 4500 open/close in 53.94 seconds: 83.42 ops/second - /unlink 1508 (time 1788276509.25 total 10.00 last 150.79) - /unlink 3298 (time 1788276519.25 total 20.00 last 178.93) total: 4500 /unlink in 26.77 seconds: 168.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 675 (time 1788276541.09 total 10.01 last 67.45) - open/close 1464 (time 1788276551.10 total 20.01 last 78.85) - open/close 2275 (time 1788276561.11 total 30.02 last 81.06) - open/close 3072 (time 1788276571.11 total 40.02 last 79.69) - open/close 3653 (time 1788276581.12 total 50.03 last 58.04) - open/close 4487 (time 1788276591.12 total 60.03 last 83.36) total: 4500 open/close in 60.28 seconds: 74.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1423 (time 1788276605.92 total 10.00 last 142.26) - /unlink 3132 (time 1788276615.92 total 20.01 last 170.84) total: 4500 /unlink in 28.68 seconds: 156.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 923 (time 1788276639.99 total 10.01 last 92.25) - open/close 1621 (time 1788276650.00 total 20.01 last 69.76) - open/close 2238 (time 1788276660.02 total 30.03 last 61.57) - open/close 2864 (time 1788276670.03 total 40.04 last 62.54) - open/close 3486 (time 1788276680.05 total 50.06 last 62.08) - open/close 4138 (time 1788276690.06 total 60.07 last 65.13) total: 4500 open/close in 65.10 seconds: 69.13 ops/second - /unlink 1549 (time 1788276709.91 total 10.00 last 154.89) - /unlink 3044 (time 1788276719.92 total 20.00 last 149.47) total: 4500 /unlink in 28.04 seconds: 160.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 669 (time 1788276742.78 total 10.01 last 66.82) - open/close 1340 (time 1788276752.80 total 20.03 last 66.99) - open/close 2059 (time 1788276762.80 total 30.03 last 71.86) - open/close 2844 (time 1788276772.81 total 40.04 last 78.46) - open/close 3580 (time 1788276782.82 total 50.05 last 73.51) - open/close 4344 (time 1788276792.84 total 60.06 last 76.29) total: 4500 open/close in 62.32 seconds: 72.20 ops/second - /unlink 1790 (time 1788276809.44 total 10.00 last 178.94) - /unlink 3470 (time 1788276819.45 total 20.01 last 167.94) total: 4500 /unlink in 26.28 seconds: 171.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 786 (time 1788276840.96 total 10.01 last 78.55) - open/close 1651 (time 1788276850.96 total 20.01 last 86.46) - open/close 2561 (time 1788276860.97 total 30.02 last 90.94) - open/close 3389 (time 1788276870.97 total 40.02 last 82.75) - open/close 4247 (time 1788276880.98 total 50.03 last 85.72) total: 4500 open/close in 52.96 seconds: 84.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1961 (time 1788276898.10 total 10.00 last 196.01) - /unlink 3673 (time 1788276908.11 total 20.01 last 171.10) total: 4500 /unlink in 24.64 seconds: 182.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 871 (time 1788276927.37 total 10.00 last 87.08) - open/close 1573 (time 1788276937.38 total 20.01 last 70.15) - open/close 2481 (time 1788276947.38 total 30.02 last 90.74) - open/close 3447 (time 1788276957.40 total 40.04 last 96.39) - open/close 4417 (time 1788276967.41 total 50.04 last 96.98) total: 4500 open/close in 51.11 seconds: 88.05 ops/second - /unlink 1504 (time 1788276983.06 total 10.00 last 150.38) - /unlink 2851 (time 1788276993.06 total 20.00 last 134.67) total: 4500 /unlink in 29.03 seconds: 154.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 677 (time 1788277017.99 total 10.01 last 67.66) - open/close 1507 (time 1788277027.99 total 20.01 last 82.95) - open/close 2315 (time 1788277038.00 total 30.02 last 80.73) - open/close 3174 (time 1788277048.02 total 40.04 last 85.75) - open/close 4119 (time 1788277058.02 total 50.04 last 94.49) total: 4500 open/close in 54.57 seconds: 82.46 ops/second - /unlink 1164 (time 1788277077.82 total 10.00 last 116.37) - /unlink 2317 (time 1788277087.82 total 20.00 last 115.29) - /unlink 3556 (time 1788277097.82 total 30.01 last 123.84) total: 4500 /unlink in 38.39 seconds: 117.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 753 (time 1788277122.54 total 10.00 last 75.28) - open/close 1558 (time 1788277132.55 total 20.01 last 80.45) - open/close 2504 (time 1788277142.55 total 30.01 last 94.56) - open/close 3347 (time 1788277152.56 total 40.02 last 84.24) - open/close 4068 (time 1788277162.60 total 50.07 last 71.77) total: 4500 open/close in 57.63 seconds: 78.09 ops/second - /unlink 1607 (time 1788277185.99 total 10.00 last 160.64) - /unlink 3102 (time 1788277195.99 total 20.01 last 149.45) total: 4500 /unlink in 28.41 seconds: 158.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 832 (time 1788277219.13 total 10.00 last 83.17) - open/close 1644 (time 1788277229.14 total 20.01 last 81.11) - open/close 2454 (time 1788277239.14 total 30.02 last 81.00) - open/close 3145 (time 1788277249.16 total 40.03 last 68.98) - open/close 3924 (time 1788277259.16 total 50.04 last 77.86) total: 4500 open/close in 58.23 seconds: 77.27 ops/second - /unlink 1289 (time 1788277282.56 total 10.00 last 128.90) - /unlink 2649 (time 1788277292.56 total 20.00 last 135.99) - /unlink 4211 (time 1788277302.56 total 30.00 last 156.19) total: 4500 /unlink in 31.81 seconds: 141.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 806 (time 1788277319.33 total 10.01 last 80.53) - open/close 1556 (time 1788277329.34 total 20.02 last 74.94) - open/close 2395 (time 1788277339.36 total 30.03 last 83.75) - open/close 3194 (time 1788277349.37 total 40.04 last 79.85) - open/close 3979 (time 1788277359.38 total 50.05 last 78.42) total: 4500 open/close in 56.05 seconds: 80.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1350 (time 1788277381.37 total 10.00 last 134.99) - /unlink 2390 (time 1788277391.38 total 20.01 last 103.94) - /unlink 3332 (time 1788277401.38 total 30.01 last 94.15) total: 4500 /unlink in 36.84 seconds: 122.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 810 (time 1788277422.63 total 10.03 last 80.77) - open/close 1641 (time 1788277432.63 total 20.03 last 83.05) - open/close 2361 (time 1788277442.65 total 30.05 last 71.89) - open/close 3190 (time 1788277452.66 total 40.06 last 82.84) - open/close 4045 (time 1788277462.66 total 50.06 last 85.44) total: 4500 open/close in 54.63 seconds: 82.38 ops/second - /unlink 1730 (time 1788277482.71 total 10.00 last 172.97) - /unlink 3456 (time 1788277492.72 total 20.01 last 172.52) total: 4500 /unlink in 26.36 seconds: 170.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 874 (time 1788277513.04 total 10.00 last 87.38) - open/close 1727 (time 1788277523.04 total 20.01 last 85.27) - open/close 2346 (time 1788277533.04 total 30.01 last 61.87) - open/close 3190 (time 1788277543.05 total 40.01 last 84.37) - open/close 4077 (time 1788277553.05 total 50.02 last 88.67) total: 4500 open/close in 54.30 seconds: 82.87 ops/second - /unlink 1671 (time 1788277571.39 total 10.01 last 166.94) - /unlink 3323 (time 1788277581.39 total 20.01 last 165.15) total: 4500 /unlink in 27.41 seconds: 164.19 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95270 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg235-server unloading modules via unload_modules_local on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing unload_modules_local oleg235-server: modules unloaded.