== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:47:21 (1788310041) Stopping clients: oleg110-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg110-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg110-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg110-server oleg110-server: oleg110-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing load_modules_local oleg110-server: Loading modules from /home/green/git/lustre-release/lustre oleg110-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg110-server: detected 4 online CPUs by sysfs oleg110-server: MODOPTS_LIBCFS= oleg110-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg110-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg110-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg110-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 707 (time 1788310253.07 total 10.01 last 70.65) - open/close 1470 (time 1788310263.07 total 20.01 last 76.30) - open/close 2128 (time 1788310273.08 total 30.02 last 65.73) - open/close 2882 (time 1788310283.09 total 40.02 last 75.39) - open/close 3674 (time 1788310293.09 total 50.02 last 79.19) - open/close 4331 (time 1788310303.10 total 60.03 last 65.64) total: 4500 open/close in 62.38 seconds: 72.14 ops/second - /unlink 1645 (time 1788310319.53 total 10.01 last 164.41) - /unlink 3187 (time 1788310329.54 total 20.01 last 154.14) total: 4500 /unlink in 28.80 seconds: 156.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 793 (time 1788310352.57 total 10.00 last 79.29) - open/close 1563 (time 1788310362.57 total 20.00 last 76.99) - open/close 2350 (time 1788310372.58 total 30.01 last 78.62) - open/close 3190 (time 1788310382.59 total 40.02 last 83.91) - open/close 3979 (time 1788310392.60 total 50.03 last 78.83) total: 4500 open/close in 57.11 seconds: 78.80 ops/second - /unlink 1455 (time 1788310414.06 total 10.00 last 145.49) - /unlink 2818 (time 1788310424.06 total 20.00 last 136.26) - /unlink 4066 (time 1788310434.06 total 30.01 last 124.77) total: 4500 /unlink in 33.57 seconds: 134.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 729 (time 1788310452.59 total 10.01 last 72.83) - open/close 1430 (time 1788310462.60 total 20.02 last 70.05) - open/close 2135 (time 1788310472.61 total 30.02 last 70.46) - open/close 2891 (time 1788310482.61 total 40.03 last 75.55) - open/close 3625 (time 1788310492.62 total 50.03 last 73.37) - open/close 4316 (time 1788310502.62 total 60.04 last 69.09) total: 4500 open/close in 63.19 seconds: 71.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1332 (time 1788310521.00 total 10.01 last 133.11) - /unlink 2595 (time 1788310531.00 total 20.01 last 126.25) - /unlink 4052 (time 1788310541.00 total 30.01 last 145.67) total: 4500 /unlink in 33.02 seconds: 136.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 715 (time 1788310559.54 total 10.01 last 71.43) - open/close 1460 (time 1788310569.54 total 20.01 last 74.49) - open/close 2280 (time 1788310579.54 total 30.01 last 81.99) - open/close 2964 (time 1788310589.55 total 40.02 last 68.33) - open/close 3724 (time 1788310599.56 total 50.03 last 75.96) - open/close 4416 (time 1788310609.57 total 60.04 last 69.12) total: 4500 open/close in 61.20 seconds: 73.53 ops/second - /unlink 1343 (time 1788310625.53 total 10.00 last 134.25) - /unlink 2476 (time 1788310635.53 total 20.01 last 113.26) - /unlink 3792 (time 1788310645.54 total 30.01 last 131.51) total: 4500 /unlink in 36.71 seconds: 122.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 523 (time 1788310668.45 total 10.00 last 52.28) - open/close 1198 (time 1788310678.48 total 20.04 last 67.26) - open/close 1957 (time 1788310688.48 total 30.04 last 75.89) - open/close 2671 (time 1788310698.50 total 40.06 last 71.25) - open/close 3434 (time 1788310708.52 total 50.07 last 76.20) - open/close 4150 (time 1788310718.54 total 60.10 last 71.41) total: 4500 open/close in 65.10 seconds: 69.12 ops/second - /unlink 1322 (time 1788310738.17 total 10.01 last 132.08) - /unlink 2724 (time 1788310748.18 total 20.01 last 140.13) - /unlink 4209 (time 1788310758.19 total 30.02 last 148.34) total: 4500 /unlink in 32.13 seconds: 140.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 820 (time 1788310774.63 total 10.02 last 81.86) - open/close 1592 (time 1788310784.64 total 20.03 last 77.11) - open/close 2374 (time 1788310794.66 total 30.04 last 78.08) - open/close 3110 (time 1788310804.66 total 40.05 last 73.56) - open/close 3838 (time 1788310814.67 total 50.06 last 72.75) total: 4500 open/close in 58.60 seconds: 76.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1311 (time 1788310838.80 total 10.00 last 131.10) - /unlink 2454 (time 1788310848.80 total 20.00 last 114.30) - /unlink 3821 (time 1788310858.80 total 30.00 last 136.66) total: 4500 /unlink in 35.45 seconds: 126.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 777 (time 1788310878.58 total 10.00 last 77.69) - open/close 1431 (time 1788310888.59 total 20.01 last 65.35) - open/close 2136 (time 1788310898.60 total 30.02 last 70.42) - open/close 2775 (time 1788310908.60 total 40.02 last 63.89) - open/close 3512 (time 1788310918.64 total 50.06 last 73.45) - open/close 4162 (time 1788310928.65 total 60.07 last 64.94) total: 4500 open/close in 65.72 seconds: 68.47 ops/second - /unlink 1340 (time 1788310949.74 total 10.00 last 134.00) - /unlink 2571 (time 1788310959.74 total 20.00 last 123.07) - /unlink 3737 (time 1788310969.75 total 30.02 last 116.45) total: 4500 /unlink in 35.80 seconds: 125.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 735 (time 1788310989.99 total 10.01 last 73.45) - open/close 1453 (time 1788311000.00 total 20.02 last 71.74) - open/close 2160 (time 1788311010.01 total 30.03 last 70.61) - open/close 2983 (time 1788311020.02 total 40.03 last 82.27) - open/close 3741 (time 1788311030.02 total 50.04 last 75.74) - open/close 4396 (time 1788311040.03 total 60.05 last 65.43) total: 4500 open/close in 61.37 seconds: 73.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1364 (time 1788311056.59 total 10.01 last 136.30) - /unlink 2812 (time 1788311066.59 total 20.01 last 144.80) - /unlink 4184 (time 1788311076.59 total 30.01 last 137.14) total: 4500 /unlink in 32.32 seconds: 139.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 784 (time 1788311094.28 total 10.01 last 78.32) - open/close 1593 (time 1788311104.28 total 20.01 last 80.87) - open/close 2397 (time 1788311114.28 total 30.02 last 80.37) - open/close 3190 (time 1788311124.30 total 40.03 last 79.16) - open/close 3929 (time 1788311134.31 total 50.04 last 73.84) total: 4500 open/close in 57.59 seconds: 78.13 ops/second - /unlink 1316 (time 1788311156.76 total 10.00 last 131.56) - /unlink 2764 (time 1788311166.77 total 20.01 last 144.69) - /unlink 4276 (time 1788311176.77 total 30.01 last 151.20) total: 4500 /unlink in 31.87 seconds: 141.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 753 (time 1788311193.51 total 10.01 last 75.26) - open/close 1519 (time 1788311203.51 total 20.01 last 76.57) - open/close 2329 (time 1788311213.53 total 30.02 last 80.90) - open/close 3023 (time 1788311223.53 total 40.03 last 69.35) - open/close 3734 (time 1788311233.54 total 50.04 last 71.05) total: 4500 open/close in 59.90 seconds: 75.13 ops/second - /unlink 1477 (time 1788311257.57 total 10.00 last 147.69) - /unlink 2878 (time 1788311267.58 total 20.00 last 140.06) - /unlink 4256 (time 1788311277.58 total 30.01 last 137.77) total: 4500 /unlink in 31.73 seconds: 141.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 681 (time 1788311293.05 total 10.00 last 68.07) - open/close 1429 (time 1788311303.05 total 20.01 last 74.78) - open/close 2067 (time 1788311313.05 total 30.01 last 63.79) - open/close 2771 (time 1788311323.06 total 40.02 last 70.35) - open/close 3591 (time 1788311333.06 total 50.02 last 81.98) - open/close 4310 (time 1788311343.07 total 60.03 last 71.85) total: 4500 open/close in 62.33 seconds: 72.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1815 (time 1788311359.13 total 10.02 last 181.20) - /unlink 3449 (time 1788311369.13 total 20.02 last 163.38) total: 4500 /unlink in 27.13 seconds: 165.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 815 (time 1788311389.88 total 10.02 last 81.30) - open/close 1534 (time 1788311399.89 total 20.03 last 71.85) - open/close 2338 (time 1788311409.90 total 30.04 last 80.32) - open/close 3298 (time 1788311419.90 total 40.04 last 96.00) - open/close 4140 (time 1788311429.91 total 50.05 last 84.16) total: 4500 open/close in 54.58 seconds: 82.45 ops/second - /unlink 1262 (time 1788311449.17 total 10.00 last 126.16) - /unlink 2639 (time 1788311459.17 total 20.01 last 137.66) - /unlink 4168 (time 1788311469.17 total 30.01 last 152.87) total: 4500 /unlink in 32.54 seconds: 138.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 895 (time 1788311486.87 total 10.00 last 89.47) - open/close 1636 (time 1788311496.87 total 20.01 last 74.06) - open/close 2424 (time 1788311506.88 total 30.02 last 78.72) - open/close 3215 (time 1788311516.89 total 40.03 last 79.03) - open/close 4100 (time 1788311526.90 total 50.04 last 88.41) total: 4500 open/close in 55.09 seconds: 81.68 ops/second - /unlink 1665 (time 1788311545.90 total 10.00 last 166.46) - /unlink 3128 (time 1788311555.91 total 20.01 last 146.27) - /unlink 4436 (time 1788311565.91 total 30.01 last 130.78) total: 4500 /unlink in 30.56 seconds: 147.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 750 (time 1788311579.81 total 10.01 last 74.95) - open/close 1648 (time 1788311589.82 total 20.01 last 89.73) - open/close 2367 (time 1788311599.82 total 30.02 last 71.87) - open/close 3193 (time 1788311609.82 total 40.02 last 82.60) - open/close 3879 (time 1788311619.83 total 50.03 last 68.54) total: 4500 open/close in 58.82 seconds: 76.51 ops/second - /unlink 1425 (time 1788311643.95 total 10.01 last 142.39) - /unlink 2919 (time 1788311653.95 total 20.01 last 149.38) - /unlink 4233 (time 1788311663.95 total 30.01 last 131.40) total: 4500 /unlink in 31.95 seconds: 140.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 585 (time 1788311681.21 total 10.01 last 58.41) - open/close 1081 (time 1788311691.22 total 20.03 last 49.54) - open/close 1731 (time 1788311701.22 total 30.03 last 65.00) - open/close 2548 (time 1788311711.23 total 40.04 last 81.63) - open/close 3327 (time 1788311721.25 total 50.06 last 77.73) - open/close 4117 (time 1788311731.26 total 60.06 last 78.95) total: 4500 open/close in 64.83 seconds: 69.41 ops/second - /unlink 1355 (time 1788311750.01 total 10.01 last 135.40) - /unlink 2696 (time 1788311760.02 total 20.01 last 134.05) - /unlink 4264 (time 1788311770.02 total 30.01 last 156.78) total: 4500 /unlink in 31.87 seconds: 141.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 805 (time 1788311785.84 total 10.01 last 80.44) - open/close 1549 (time 1788311795.85 total 20.02 last 74.33) - open/close 2197 (time 1788311805.85 total 30.02 last 64.78) - open/close 2789 (time 1788311815.86 total 40.03 last 59.12) - open/close 3510 (time 1788311825.87 total 50.04 last 72.05) - open/close 4149 (time 1788311835.89 total 60.05 last 63.81) total: 4500 open/close in 65.51 seconds: 68.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1361 (time 1788311855.76 total 10.00 last 136.10) - /unlink 2827 (time 1788311865.77 total 20.00 last 146.57) - /unlink 4362 (time 1788311875.77 total 30.00 last 153.48) total: 4500 /unlink in 31.04 seconds: 144.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 660 (time 1788311891.42 total 10.00 last 65.97) - open/close 1532 (time 1788311901.43 total 20.01 last 87.15) - open/close 2317 (time 1788311911.43 total 30.01 last 78.48) - open/close 3149 (time 1788311921.44 total 40.03 last 83.09) - open/close 4025 (time 1788311931.45 total 50.03 last 87.55) total: 4500 open/close in 55.96 seconds: 80.41 ops/second - /unlink 1304 (time 1788311951.15 total 10.00 last 130.35) - /unlink 2750 (time 1788311961.16 total 20.01 last 144.47) - /unlink 4155 (time 1788311971.16 total 30.01 last 140.49) total: 4500 /unlink in 32.60 seconds: 138.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 756 (time 1788311988.25 total 10.01 last 75.50) - open/close 1567 (time 1788311998.25 total 20.02 last 81.09) - open/close 2406 (time 1788312008.26 total 30.02 last 83.84) - open/close 3169 (time 1788312018.28 total 40.05 last 76.12) - open/close 3925 (time 1788312028.30 total 50.06 last 75.48) total: 4500 open/close in 57.66 seconds: 78.04 ops/second - /unlink 1378 (time 1788312050.20 total 10.00 last 137.78) - /unlink 2509 (time 1788312060.20 total 20.00 last 113.07) - /unlink 3960 (time 1788312070.21 total 30.02 last 144.92) total: 4500 /unlink in 34.44 seconds: 130.66 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95235 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg110-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg110-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server unloading modules via unload_modules_local on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing unload_modules_local oleg110-server: modules unloaded.