== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:28:23 (1788168503) Stopping clients: oleg345-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg345-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg345-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg345-server oleg345-server: oleg345-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg345-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg345-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg345-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 661 (time 1788168704.07 total 10.00 last 66.09) - open/close 1350 (time 1788168714.08 total 20.01 last 68.84) - open/close 2104 (time 1788168724.09 total 30.02 last 75.35) - open/close 2764 (time 1788168734.10 total 40.03 last 65.94) - open/close 3412 (time 1788168744.11 total 50.03 last 64.76) - open/close 4072 (time 1788168754.11 total 60.04 last 65.96) total: 4500 open/close in 65.56 seconds: 68.64 ops/second - /unlink 1139 (time 1788168773.67 total 10.01 last 113.84) - /unlink 2339 (time 1788168783.68 total 20.01 last 119.90) - /unlink 3500 (time 1788168793.68 total 30.02 last 116.07) total: 4500 /unlink in 38.03 seconds: 118.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 718 (time 1788168815.68 total 10.00 last 71.79) - open/close 1370 (time 1788168825.68 total 20.00 last 65.19) - open/close 1984 (time 1788168835.69 total 30.01 last 61.39) - open/close 2625 (time 1788168845.70 total 40.02 last 64.02) - open/close 3217 (time 1788168855.71 total 50.02 last 59.16) - open/close 3879 (time 1788168865.71 total 60.02 last 66.19) total: 4500 open/close in 69.34 seconds: 64.89 ops/second - /unlink 1240 (time 1788168889.72 total 10.01 last 123.84) - /unlink 2591 (time 1788168899.73 total 20.02 last 135.02) - /unlink 3899 (time 1788168909.73 total 30.02 last 130.77) total: 4500 /unlink in 34.86 seconds: 129.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 679 (time 1788168928.95 total 10.01 last 67.86) - open/close 1424 (time 1788168938.96 total 20.01 last 74.47) - open/close 2141 (time 1788168948.96 total 30.01 last 71.68) - open/close 2815 (time 1788168958.97 total 40.03 last 67.32) - open/close 3524 (time 1788168968.98 total 50.03 last 70.87) - open/close 4221 (time 1788168978.99 total 60.04 last 69.62) total: 4500 open/close in 63.97 seconds: 70.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1212 (time 1788168996.83 total 10.00 last 121.20) - /unlink 2432 (time 1788169006.83 total 20.00 last 121.97) - /unlink 3609 (time 1788169016.83 total 30.01 last 117.67) total: 4500 /unlink in 37.90 seconds: 118.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 649 (time 1788169039.70 total 10.01 last 64.84) - open/close 1327 (time 1788169049.72 total 20.03 last 67.67) - open/close 1967 (time 1788169059.74 total 30.05 last 63.87) - open/close 2584 (time 1788169069.75 total 40.06 last 61.65) - open/close 3244 (time 1788169079.76 total 50.07 last 65.91) - open/close 3832 (time 1788169089.78 total 60.08 last 58.72) - open/close 4328 (time 1788169099.79 total 70.10 last 49.54) total: 4500 open/close in 73.12 seconds: 61.54 ops/second - /unlink 1111 (time 1788169118.90 total 10.01 last 111.01) - /unlink 2195 (time 1788169128.90 total 20.01 last 108.36) - /unlink 3263 (time 1788169138.90 total 30.01 last 106.78) total: 4500 /unlink in 39.97 seconds: 112.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 683 (time 1788169163.36 total 10.01 last 68.23) - open/close 1383 (time 1788169173.38 total 20.03 last 69.89) - open/close 2125 (time 1788169183.38 total 30.04 last 74.13) - open/close 2776 (time 1788169193.39 total 40.04 last 65.05) - open/close 3351 (time 1788169203.40 total 50.05 last 57.48) - open/close 3907 (time 1788169213.41 total 60.06 last 55.50) - open/close 4416 (time 1788169223.42 total 70.07 last 50.88) total: 4500 open/close in 71.80 seconds: 62.68 ops/second - /unlink 1120 (time 1788169239.68 total 10.00 last 111.94) - /unlink 2212 (time 1788169249.69 total 20.01 last 109.16) - /unlink 3342 (time 1788169259.70 total 30.02 last 112.91) total: 4500 /unlink in 39.15 seconds: 114.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 647 (time 1788169283.57 total 10.01 last 64.66) - open/close 1307 (time 1788169293.58 total 20.01 last 65.96) - open/close 1940 (time 1788169303.59 total 30.02 last 63.24) - open/close 2508 (time 1788169313.61 total 40.05 last 56.67) - open/close 3149 (time 1788169323.62 total 50.06 last 64.02) - open/close 3795 (time 1788169333.63 total 60.07 last 64.51) - open/close 4391 (time 1788169343.64 total 70.07 last 59.59) total: 4500 open/close in 71.73 seconds: 62.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1062 (time 1788169360.17 total 10.00 last 106.15) - /unlink 2139 (time 1788169370.18 total 20.01 last 107.61) - /unlink 3276 (time 1788169380.18 total 30.01 last 113.69) - /unlink 4419 (time 1788169390.18 total 40.01 last 114.29) total: 4500 /unlink in 40.83 seconds: 110.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 692 (time 1788169405.25 total 10.01 last 69.14) - open/close 1322 (time 1788169415.26 total 20.02 last 62.93) - open/close 1967 (time 1788169425.27 total 30.03 last 64.45) - open/close 2634 (time 1788169435.27 total 40.03 last 66.66) - open/close 3206 (time 1788169445.29 total 50.06 last 57.06) - open/close 3785 (time 1788169455.31 total 60.07 last 57.80) - open/close 4423 (time 1788169465.31 total 70.07 last 63.78) total: 4500 open/close in 71.06 seconds: 63.33 ops/second - /unlink 1229 (time 1788169481.32 total 10.00 last 122.87) - /unlink 2479 (time 1788169491.32 total 20.01 last 124.96) - /unlink 3565 (time 1788169501.32 total 30.01 last 108.59) total: 4500 /unlink in 38.47 seconds: 116.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 662 (time 1788169525.52 total 10.01 last 66.15) - open/close 1321 (time 1788169535.52 total 20.02 last 65.85) - open/close 2009 (time 1788169545.53 total 30.03 last 68.73) - open/close 2742 (time 1788169555.54 total 40.03 last 73.25) - open/close 3367 (time 1788169565.55 total 50.04 last 62.44) - open/close 4002 (time 1788169575.56 total 60.05 last 63.47) total: 4500 open/close in 69.15 seconds: 65.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1260 (time 1788169600.67 total 10.02 last 125.69) - /unlink 2461 (time 1788169610.68 total 20.03 last 119.99) - /unlink 3636 (time 1788169620.69 total 30.04 last 117.39) total: 4500 /unlink in 37.76 seconds: 119.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 571 (time 1788169644.05 total 10.01 last 57.04) - open/close 1188 (time 1788169654.06 total 20.02 last 61.63) - open/close 1796 (time 1788169664.06 total 30.02 last 60.79) - open/close 2345 (time 1788169674.07 total 40.03 last 54.84) - open/close 2880 (time 1788169684.08 total 50.04 last 53.46) - open/close 3435 (time 1788169694.08 total 60.04 last 55.49) - open/close 4073 (time 1788169704.14 total 70.10 last 63.45) total: 4500 open/close in 77.17 seconds: 58.31 ops/second - /unlink 1157 (time 1788169726.29 total 10.00 last 115.67) - /unlink 2473 (time 1788169736.30 total 20.01 last 131.55) - /unlink 3659 (time 1788169746.30 total 30.01 last 118.51) total: 4500 /unlink in 37.15 seconds: 121.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 615 (time 1788169768.27 total 10.00 last 61.47) - open/close 1307 (time 1788169778.28 total 20.02 last 69.11) - open/close 1942 (time 1788169788.30 total 30.03 last 63.41) - open/close 2567 (time 1788169798.32 total 40.06 last 62.34) - open/close 3215 (time 1788169808.33 total 50.06 last 64.76) - open/close 3924 (time 1788169818.37 total 60.11 last 70.57) total: 4500 open/close in 68.67 seconds: 65.53 ops/second - /unlink 1246 (time 1788169841.69 total 10.03 last 124.25) - /unlink 2412 (time 1788169851.69 total 20.03 last 116.59) - /unlink 3580 (time 1788169861.69 total 30.04 last 116.73) total: 4500 /unlink in 37.79 seconds: 119.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 614 (time 1788169884.30 total 10.01 last 61.33) - open/close 1169 (time 1788169894.31 total 20.02 last 55.46) - open/close 1909 (time 1788169904.33 total 30.04 last 73.86) - open/close 2577 (time 1788169914.34 total 40.05 last 66.69) - open/close 3242 (time 1788169924.35 total 50.06 last 66.48) - open/close 3877 (time 1788169934.35 total 60.06 last 63.49) total: 4500 open/close in 69.29 seconds: 64.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1302 (time 1788169958.12 total 10.00 last 130.14) - /unlink 2501 (time 1788169968.12 total 20.01 last 119.88) - /unlink 3688 (time 1788169978.12 total 30.01 last 118.68) total: 4500 /unlink in 37.47 seconds: 120.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 612 (time 1788170001.22 total 10.01 last 61.15) - open/close 1216 (time 1788170011.23 total 20.02 last 60.33) - open/close 1882 (time 1788170021.24 total 30.03 last 66.51) - open/close 2543 (time 1788170031.26 total 40.05 last 65.98) - open/close 3250 (time 1788170041.27 total 50.06 last 70.64) - open/close 3901 (time 1788170051.28 total 60.07 last 65.00) total: 4500 open/close in 68.59 seconds: 65.60 ops/second - /unlink 942 (time 1788170074.39 total 10.00 last 94.19) - /unlink 2009 (time 1788170084.40 total 20.01 last 106.57) - /unlink 3336 (time 1788170094.41 total 30.02 last 132.64) total: 4500 /unlink in 39.71 seconds: 113.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 672 (time 1788170118.94 total 10.00 last 67.19) - open/close 1340 (time 1788170128.94 total 20.00 last 66.79) - open/close 2067 (time 1788170138.95 total 30.01 last 72.67) - open/close 2726 (time 1788170148.97 total 40.03 last 65.73) - open/close 3398 (time 1788170158.97 total 50.03 last 67.20) - open/close 3972 (time 1788170168.98 total 60.04 last 57.36) total: 4500 open/close in 68.94 seconds: 65.28 ops/second - /unlink 1255 (time 1788170193.41 total 10.00 last 125.47) - /unlink 2519 (time 1788170203.42 total 20.01 last 126.36) - /unlink 3668 (time 1788170213.42 total 30.01 last 114.89) total: 4500 /unlink in 37.26 seconds: 120.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 723 (time 1788170235.65 total 10.01 last 72.23) - open/close 1396 (time 1788170245.66 total 20.02 last 67.24) - open/close 2106 (time 1788170255.67 total 30.03 last 70.92) - open/close 2751 (time 1788170265.69 total 40.04 last 64.42) - open/close 3431 (time 1788170275.69 total 50.04 last 67.99) - open/close 4124 (time 1788170285.71 total 60.06 last 69.19) total: 4500 open/close in 65.56 seconds: 68.63 ops/second - /unlink 1184 (time 1788170305.75 total 10.00 last 118.37) - /unlink 2481 (time 1788170315.75 total 20.01 last 129.66) - /unlink 3699 (time 1788170325.76 total 30.01 last 121.70) total: 4500 /unlink in 36.99 seconds: 121.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 616 (time 1788170347.21 total 10.00 last 61.58) - open/close 1289 (time 1788170357.22 total 20.01 last 67.24) - open/close 1917 (time 1788170367.23 total 30.02 last 62.72) - open/close 2519 (time 1788170377.24 total 40.03 last 60.14) - open/close 3111 (time 1788170387.24 total 50.04 last 59.20) - open/close 3763 (time 1788170397.25 total 60.05 last 65.13) - open/close 4420 (time 1788170407.27 total 70.06 last 65.60) total: 4500 open/close in 71.25 seconds: 63.16 ops/second - /unlink 1083 (time 1788170423.26 total 10.01 last 108.22) - /unlink 2315 (time 1788170433.26 total 20.01 last 123.16) - /unlink 3420 (time 1788170443.27 total 30.02 last 110.44) total: 4500 /unlink in 39.84 seconds: 112.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 642 (time 1788170469.46 total 10.00 last 64.18) - open/close 1353 (time 1788170479.46 total 20.01 last 71.05) - open/close 2019 (time 1788170489.47 total 30.01 last 66.58) - open/close 2659 (time 1788170499.47 total 40.01 last 64.00) - open/close 3261 (time 1788170509.48 total 50.03 last 60.10) - open/close 3888 (time 1788170519.49 total 60.03 last 62.68) total: 4500 open/close in 69.23 seconds: 65.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1328 (time 1788170543.99 total 10.01 last 132.71) - /unlink 2557 (time 1788170554.00 total 20.01 last 122.89) - /unlink 3528 (time 1788170564.00 total 30.02 last 97.02) total: 4500 /unlink in 38.44 seconds: 117.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 590 (time 1788170589.11 total 10.00 last 58.99) - open/close 1106 (time 1788170599.12 total 20.01 last 51.56) - open/close 1671 (time 1788170609.15 total 30.04 last 56.32) - open/close 2376 (time 1788170619.16 total 40.05 last 70.43) - open/close 3129 (time 1788170629.17 total 50.06 last 75.23) - open/close 3783 (time 1788170639.19 total 60.07 last 65.31) - open/close 4357 (time 1788170649.21 total 70.09 last 57.29) total: 4500 open/close in 72.33 seconds: 62.21 ops/second - /unlink 1262 (time 1788170667.04 total 10.02 last 126.01) - /unlink 2543 (time 1788170677.05 total 20.02 last 128.05) - /unlink 3814 (time 1788170687.05 total 30.03 last 127.00) total: 4500 /unlink in 35.50 seconds: 126.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 745 (time 1788170707.19 total 10.00 last 74.49) - open/close 1363 (time 1788170717.21 total 20.02 last 61.68) - open/close 2062 (time 1788170727.22 total 30.03 last 69.84) - open/close 2655 (time 1788170737.22 total 40.03 last 59.29) - open/close 3231 (time 1788170747.23 total 50.04 last 57.55) - open/close 3840 (time 1788170757.24 total 60.05 last 60.83) - open/close 4410 (time 1788170767.24 total 70.05 last 56.99) total: 4500 open/close in 71.30 seconds: 63.11 ops/second - /unlink 1188 (time 1788170783.71 total 10.01 last 118.70) - /unlink 2363 (time 1788170793.72 total 20.02 last 117.38) - /unlink 3572 (time 1788170803.73 total 30.03 last 120.83) total: 4500 /unlink in 38.09 seconds: 118.13 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95233 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg345-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server unloading modules via unload_modules_local on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing unload_modules_local oleg345-server: modules unloaded.