== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:14:17 (1789481657) Stopping clients: oleg233-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg233-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg233-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg233-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg233-server oleg233-server: oleg233-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing load_modules_local oleg233-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg233-server: Loading modules from /home/green/git/lustre-release/lustre oleg233-server: detected 4 online CPUs by sysfs oleg233-server: MODOPTS_LIBCFS= oleg233-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg233-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg233-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg233-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 665 (time 1789481858.12 total 10.01 last 66.41) - open/close 1318 (time 1789481868.16 total 20.06 last 65.02) - open/close 2111 (time 1789481878.16 total 30.06 last 79.29) - open/close 2810 (time 1789481888.16 total 40.06 last 69.88) - open/close 3567 (time 1789481898.17 total 50.06 last 75.67) - open/close 4321 (time 1789481908.17 total 60.07 last 75.38) total: 4500 open/close in 62.76 seconds: 71.70 ops/second - /unlink 1411 (time 1789481925.32 total 10.01 last 140.96) - /unlink 2701 (time 1789481935.32 total 20.02 last 128.89) - /unlink 3895 (time 1789481945.33 total 30.02 last 119.37) total: 4500 /unlink in 34.61 seconds: 130.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 746 (time 1789481964.96 total 10.01 last 74.54) - open/close 1374 (time 1789481974.97 total 20.02 last 62.75) - open/close 2078 (time 1789481984.99 total 30.03 last 70.26) - open/close 2788 (time 1789481995.01 total 40.06 last 70.84) - open/close 3468 (time 1789482005.02 total 50.07 last 67.92) - open/close 4115 (time 1789482015.04 total 60.09 last 64.59) total: 4500 open/close in 65.55 seconds: 68.65 ops/second - /unlink 1376 (time 1789482035.65 total 10.01 last 137.42) - /unlink 2681 (time 1789482045.65 total 20.01 last 130.48) - /unlink 4051 (time 1789482055.65 total 30.02 last 136.98) total: 4500 /unlink in 33.52 seconds: 134.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 643 (time 1789482074.69 total 10.01 last 64.23) - open/close 1222 (time 1789482084.70 total 20.02 last 57.86) - open/close 1864 (time 1789482094.71 total 30.03 last 64.12) - open/close 2584 (time 1789482104.73 total 40.05 last 71.89) - open/close 3166 (time 1789482114.73 total 50.05 last 58.17) - open/close 3930 (time 1789482124.74 total 60.06 last 76.36) total: 4500 open/close in 68.95 seconds: 65.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1112 (time 1789482150.11 total 10.00 last 111.15) - /unlink 2384 (time 1789482160.11 total 20.00 last 127.20) - /unlink 3583 (time 1789482170.12 total 30.01 last 119.83) total: 4500 /unlink in 38.45 seconds: 117.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 689 (time 1789482195.06 total 10.00 last 68.89) - open/close 1421 (time 1789482205.06 total 20.00 last 73.18) - open/close 2036 (time 1789482215.07 total 30.01 last 61.46) - open/close 2639 (time 1789482225.12 total 40.06 last 60.03) - open/close 3156 (time 1789482235.13 total 50.07 last 51.65) - open/close 3832 (time 1789482245.13 total 60.07 last 67.58) total: 4500 open/close in 69.48 seconds: 64.77 ops/second - /unlink 1408 (time 1789482269.26 total 10.00 last 140.74) - /unlink 2671 (time 1789482279.26 total 20.01 last 126.23) - /unlink 3976 (time 1789482289.28 total 30.02 last 130.31) total: 4500 /unlink in 33.92 seconds: 132.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 703 (time 1789482308.03 total 10.01 last 70.25) - open/close 1418 (time 1789482318.04 total 20.01 last 71.45) - open/close 2197 (time 1789482328.05 total 30.02 last 77.85) - open/close 2886 (time 1789482338.05 total 40.03 last 68.86) - open/close 3515 (time 1789482348.06 total 50.04 last 62.84) - open/close 4072 (time 1789482358.06 total 60.04 last 55.68) total: 4500 open/close in 68.34 seconds: 65.85 ops/second - /unlink 1179 (time 1789482382.76 total 10.00 last 117.89) - /unlink 2189 (time 1789482392.77 total 20.01 last 100.93) - /unlink 3353 (time 1789482402.77 total 30.01 last 116.36) total: 4500 /unlink in 39.68 seconds: 113.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 717 (time 1789482428.11 total 10.01 last 71.66) - open/close 1314 (time 1789482438.12 total 20.01 last 59.65) - open/close 2001 (time 1789482448.14 total 30.04 last 68.51) - open/close 2674 (time 1789482458.15 total 40.04 last 67.27) - open/close 3391 (time 1789482468.15 total 50.05 last 71.66) - open/close 4247 (time 1789482478.15 total 60.05 last 85.59) total: 4500 open/close in 63.13 seconds: 71.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1209 (time 1789482495.58 total 10.01 last 120.82) - /unlink 2443 (time 1789482505.59 total 20.02 last 123.25) - /unlink 3686 (time 1789482515.60 total 30.02 last 124.25) total: 4500 /unlink in 35.69 seconds: 126.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 709 (time 1789482535.99 total 10.01 last 70.84) - open/close 1480 (time 1789482546.00 total 20.02 last 77.04) - open/close 2167 (time 1789482556.01 total 30.02 last 68.67) - open/close 2958 (time 1789482566.01 total 40.02 last 79.10) - open/close 3525 (time 1789482576.04 total 50.05 last 56.53) - open/close 4152 (time 1789482586.06 total 60.08 last 62.54) total: 4500 open/close in 65.31 seconds: 68.90 ops/second - /unlink 1108 (time 1789482607.22 total 10.00 last 110.77) - /unlink 2235 (time 1789482617.22 total 20.00 last 112.68) - /unlink 3504 (time 1789482627.22 total 30.01 last 126.85) total: 4500 /unlink in 37.57 seconds: 119.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 688 (time 1789482649.15 total 10.01 last 68.76) - open/close 1360 (time 1789482659.15 total 20.01 last 67.19) - open/close 2074 (time 1789482669.16 total 30.01 last 71.35) - open/close 2715 (time 1789482679.16 total 40.01 last 64.10) - open/close 3527 (time 1789482689.18 total 50.04 last 81.02) - open/close 4283 (time 1789482699.19 total 60.04 last 75.56) total: 4500 open/close in 62.52 seconds: 71.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1225 (time 1789482717.41 total 10.00 last 122.46) - /unlink 2256 (time 1789482727.43 total 20.02 last 102.92) - /unlink 3426 (time 1789482737.43 total 30.02 last 116.99) total: 4500 /unlink in 38.73 seconds: 116.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 633 (time 1789482761.54 total 10.01 last 63.26) - open/close 1220 (time 1789482771.54 total 20.01 last 58.67) - open/close 1733 (time 1789482781.56 total 30.02 last 51.24) - open/close 2349 (time 1789482791.58 total 40.04 last 61.48) - open/close 3008 (time 1789482801.58 total 50.05 last 65.88) - open/close 3659 (time 1789482811.59 total 60.05 last 65.04) - open/close 4266 (time 1789482821.60 total 70.07 last 60.63) total: 4500 open/close in 73.54 seconds: 61.19 ops/second - /unlink 1273 (time 1789482839.87 total 10.01 last 127.13) - /unlink 2609 (time 1789482849.88 total 20.02 last 133.53) - /unlink 3966 (time 1789482859.88 total 30.02 last 135.69) total: 4500 /unlink in 34.53 seconds: 130.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 716 (time 1789482880.90 total 10.00 last 71.59) - open/close 1382 (time 1789482890.90 total 20.01 last 66.56) - open/close 2246 (time 1789482900.91 total 30.01 last 86.34) - open/close 2997 (time 1789482910.91 total 40.02 last 75.06) - open/close 3843 (time 1789482920.92 total 50.03 last 84.55) total: 4500 open/close in 59.48 seconds: 75.65 ops/second - /unlink 1369 (time 1789482944.49 total 10.01 last 136.71) - /unlink 2740 (time 1789482954.49 total 20.02 last 137.05) - /unlink 4219 (time 1789482964.49 total 30.02 last 147.87) total: 4500 /unlink in 31.71 seconds: 141.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 776 (time 1789482981.21 total 10.01 last 77.51) - open/close 1572 (time 1789482991.22 total 20.02 last 79.54) - open/close 2320 (time 1789483001.23 total 30.03 last 74.74) - open/close 3129 (time 1789483011.23 total 40.03 last 80.89) - open/close 3889 (time 1789483021.24 total 50.04 last 75.94) total: 4500 open/close in 57.86 seconds: 77.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1498 (time 1789483043.35 total 10.00 last 149.75) - /unlink 2847 (time 1789483053.35 total 20.00 last 134.88) - /unlink 4155 (time 1789483063.36 total 30.01 last 130.70) total: 4500 /unlink in 32.51 seconds: 138.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 700 (time 1789483080.67 total 10.01 last 69.96) - open/close 1430 (time 1789483090.69 total 20.03 last 72.84) - open/close 2250 (time 1789483100.70 total 30.03 last 81.97) - open/close 2976 (time 1789483110.70 total 40.04 last 72.54) - open/close 3712 (time 1789483120.72 total 50.06 last 73.46) - open/close 4387 (time 1789483130.72 total 60.06 last 67.49) total: 4500 open/close in 61.33 seconds: 73.38 ops/second - /unlink 1310 (time 1789483147.13 total 10.00 last 130.99) - /unlink 2399 (time 1789483157.13 total 20.00 last 108.88) - /unlink 3632 (time 1789483167.14 total 30.01 last 123.25) total: 4500 /unlink in 36.14 seconds: 124.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 825 (time 1789483187.65 total 10.00 last 82.48) - open/close 1576 (time 1789483197.66 total 20.01 last 75.03) - open/close 2321 (time 1789483207.67 total 30.02 last 74.41) - open/close 3122 (time 1789483217.67 total 40.02 last 80.10) - open/close 3898 (time 1789483227.68 total 50.03 last 77.56) total: 4500 open/close in 58.33 seconds: 77.15 ops/second - /unlink 1330 (time 1789483251.31 total 10.01 last 132.90) - /unlink 2463 (time 1789483261.32 total 20.02 last 113.20) - /unlink 3797 (time 1789483271.34 total 30.03 last 133.20) total: 4500 /unlink in 34.94 seconds: 128.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 682 (time 1789483292.25 total 10.04 last 67.96) - open/close 1370 (time 1789483302.25 total 20.04 last 68.79) - open/close 2157 (time 1789483312.25 total 30.04 last 78.64) - open/close 2896 (time 1789483322.27 total 40.05 last 73.82) - open/close 3600 (time 1789483332.27 total 50.06 last 70.39) - open/close 4445 (time 1789483342.27 total 60.06 last 84.49) total: 4500 open/close in 60.72 seconds: 74.10 ops/second - /unlink 1322 (time 1789483357.29 total 10.00 last 132.17) - /unlink 2736 (time 1789483367.29 total 20.01 last 141.35) - /unlink 3888 (time 1789483377.30 total 30.01 last 115.15) total: 4500 /unlink in 36.05 seconds: 124.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 669 (time 1789483399.15 total 10.01 last 66.86) - open/close 1288 (time 1789483409.16 total 20.01 last 61.88) - open/close 2042 (time 1789483419.16 total 30.01 last 75.39) - open/close 2754 (time 1789483429.18 total 40.03 last 71.03) - open/close 3482 (time 1789483439.19 total 50.04 last 72.72) - open/close 4249 (time 1789483449.19 total 60.05 last 76.68) total: 4500 open/close in 63.46 seconds: 70.91 ops/second - /unlink 1430 (time 1789483467.04 total 10.00 last 142.98) - /unlink 2769 (time 1789483477.04 total 20.01 last 133.82) - /unlink 4002 (time 1789483487.04 total 30.01 last 123.29) total: 4500 /unlink in 33.96 seconds: 132.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 627 (time 1789483506.05 total 10.01 last 62.61) - open/close 1369 (time 1789483516.07 total 20.03 last 74.04) - open/close 2120 (time 1789483526.07 total 30.04 last 75.08) - open/close 2888 (time 1789483536.08 total 40.04 last 76.75) - open/close 3633 (time 1789483546.09 total 50.06 last 74.41) - open/close 4483 (time 1789483556.10 total 60.07 last 84.90) total: 4500 open/close in 60.23 seconds: 74.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1394 (time 1789483571.06 total 10.00 last 139.36) - /unlink 2790 (time 1789483581.06 total 20.01 last 139.54) - /unlink 4093 (time 1789483591.06 total 30.01 last 130.25) total: 4500 /unlink in 32.83 seconds: 137.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 681 (time 1789483609.30 total 10.00 last 68.08) - open/close 1324 (time 1789483619.30 total 20.01 last 64.26) - open/close 1962 (time 1789483629.31 total 30.02 last 63.77) - open/close 2585 (time 1789483639.32 total 40.03 last 62.22) - open/close 3249 (time 1789483649.32 total 50.03 last 66.38) - open/close 3905 (time 1789483659.32 total 60.03 last 65.60) total: 4500 open/close in 68.12 seconds: 66.06 ops/second - /unlink 1474 (time 1789483682.74 total 10.00 last 147.34) - /unlink 2744 (time 1789483692.74 total 20.01 last 126.98) - /unlink 3901 (time 1789483702.74 total 30.01 last 115.66) total: 4500 /unlink in 35.48 seconds: 126.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 571 (time 1789483723.33 total 10.00 last 57.10) - open/close 1235 (time 1789483733.34 total 20.01 last 66.32) - open/close 2066 (time 1789483743.34 total 30.01 last 83.10) - open/close 2787 (time 1789483753.35 total 40.02 last 72.05) - open/close 3494 (time 1789483763.35 total 50.02 last 70.67) - open/close 4310 (time 1789483773.35 total 60.03 last 81.58) total: 4500 open/close in 62.73 seconds: 71.73 ops/second - /unlink 1269 (time 1789483791.39 total 10.00 last 126.86) - /unlink 2424 (time 1789483801.40 total 20.01 last 115.40) - /unlink 3495 (time 1789483811.40 total 30.02 last 107.04) - /unlink 4472 (time 1789483821.41 total 40.02 last 97.66) total: 4500 /unlink in 40.51 seconds: 111.08 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95336 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg233-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg233-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg233-server unloading modules via unload_modules_local on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing unload_modules_local oleg233-server: modules unloaded.