== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:07:07 (1788156427) Stopping clients: oleg646-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg646-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg646-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg646-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg646-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg646-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg646-server oleg646-server: oleg646-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing load_modules_local oleg646-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg646-server: Loading modules from /home/green/git/lustre-release/lustre oleg646-server: detected 4 online CPUs by sysfs oleg646-server: MODOPTS_LIBCFS= oleg646-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg646-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg646-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg646-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg646-client.virtnet: -o user_xattr,flock 192.168.206.146@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 682 (time 1788156628.50 total 10.02 last 68.07) - open/close 1359 (time 1788156638.50 total 20.02 last 67.70) - open/close 2090 (time 1788156648.50 total 30.02 last 73.10) - open/close 2769 (time 1788156658.50 total 40.02 last 67.88) - open/close 3444 (time 1788156668.52 total 50.04 last 67.37) - open/close 4178 (time 1788156678.52 total 60.04 last 73.39) total: 4500 open/close in 64.32 seconds: 69.96 ops/second - /unlink 1264 (time 1788156697.82 total 10.00 last 126.40) - /unlink 2537 (time 1788156707.83 total 20.01 last 127.18) - /unlink 3826 (time 1788156717.84 total 30.02 last 128.73) total: 4500 /unlink in 34.85 seconds: 129.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 662 (time 1788156737.57 total 10.01 last 66.15) - open/close 1326 (time 1788156747.58 total 20.02 last 66.33) - open/close 1973 (time 1788156757.58 total 30.02 last 64.69) - open/close 2627 (time 1788156767.59 total 40.02 last 65.37) - open/close 3396 (time 1788156777.61 total 50.05 last 76.68) - open/close 4025 (time 1788156787.62 total 60.06 last 62.85) total: 4500 open/close in 66.97 seconds: 67.19 ops/second - /unlink 1136 (time 1788156810.28 total 10.00 last 113.57) - /unlink 2347 (time 1788156820.29 total 20.01 last 121.00) - /unlink 3632 (time 1788156830.30 total 30.02 last 128.39) total: 4500 /unlink in 36.55 seconds: 123.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 621 (time 1788156850.61 total 10.00 last 62.08) - open/close 1224 (time 1788156860.61 total 20.01 last 60.27) - open/close 1878 (time 1788156870.62 total 30.02 last 65.33) - open/close 2531 (time 1788156880.64 total 40.03 last 65.21) - open/close 3075 (time 1788156890.65 total 50.04 last 54.34) - open/close 3752 (time 1788156900.66 total 60.06 last 67.62) - open/close 4435 (time 1788156910.66 total 70.06 last 68.29) total: 4500 open/close in 71.12 seconds: 63.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1245 (time 1788156925.33 total 10.01 last 124.36) - /unlink 2510 (time 1788156935.34 total 20.02 last 126.38) - /unlink 3604 (time 1788156945.36 total 30.04 last 109.19) total: 4500 /unlink in 37.66 seconds: 119.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 688 (time 1788156967.14 total 10.01 last 68.75) - open/close 1335 (time 1788156977.18 total 20.05 last 64.44) - open/close 1888 (time 1788156987.19 total 30.06 last 55.24) - open/close 2489 (time 1788156997.24 total 40.10 last 59.84) - open/close 3082 (time 1788157007.24 total 50.11 last 59.27) - open/close 3793 (time 1788157017.25 total 60.11 last 71.06) - open/close 4431 (time 1788157027.25 total 70.12 last 63.78) total: 4500 open/close in 71.06 seconds: 63.32 ops/second - /unlink 1278 (time 1788157042.12 total 10.00 last 127.74) - /unlink 2545 (time 1788157052.12 total 20.01 last 126.69) - /unlink 3999 (time 1788157062.13 total 30.01 last 145.30) total: 4500 /unlink in 33.43 seconds: 134.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 729 (time 1788157081.14 total 10.01 last 72.85) - open/close 1437 (time 1788157091.15 total 20.02 last 70.72) - open/close 2087 (time 1788157101.18 total 30.04 last 64.83) - open/close 2567 (time 1788157111.19 total 40.06 last 47.92) - open/close 3127 (time 1788157121.20 total 50.07 last 55.94) - open/close 3711 (time 1788157131.22 total 60.09 last 58.29) - open/close 4405 (time 1788157141.22 total 70.09 last 69.38) total: 4500 open/close in 71.50 seconds: 62.93 ops/second - /unlink 1192 (time 1788157157.27 total 10.01 last 119.08) - /unlink 2452 (time 1788157167.27 total 20.01 last 125.97) - /unlink 3740 (time 1788157177.27 total 30.01 last 128.77) total: 4500 /unlink in 36.13 seconds: 124.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 710 (time 1788157197.80 total 10.01 last 70.96) - open/close 1428 (time 1788157207.81 total 20.02 last 71.73) - open/close 2134 (time 1788157217.82 total 30.02 last 70.56) - open/close 2814 (time 1788157227.82 total 40.03 last 67.96) - open/close 3336 (time 1788157237.83 total 50.04 last 52.15) - open/close 3891 (time 1788157247.85 total 60.05 last 55.42) total: 4500 open/close in 69.75 seconds: 64.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1281 (time 1788157271.78 total 10.00 last 128.07) - /unlink 2635 (time 1788157281.78 total 20.00 last 135.38) - /unlink 3967 (time 1788157291.79 total 30.01 last 133.16) total: 4500 /unlink in 33.83 seconds: 133.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 718 (time 1788157310.56 total 10.01 last 71.72) - open/close 1365 (time 1788157320.58 total 20.02 last 64.62) - open/close 2125 (time 1788157330.58 total 30.03 last 75.99) - open/close 2777 (time 1788157340.58 total 40.03 last 65.15) - open/close 3408 (time 1788157350.59 total 50.04 last 63.07) - open/close 4000 (time 1788157360.60 total 60.05 last 59.11) total: 4500 open/close in 67.49 seconds: 66.68 ops/second - /unlink 1252 (time 1788157382.47 total 10.00 last 125.19) - /unlink 2497 (time 1788157392.47 total 20.00 last 124.47) - /unlink 3705 (time 1788157402.47 total 30.01 last 120.75) total: 4500 /unlink in 36.79 seconds: 122.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 702 (time 1788157424.34 total 10.00 last 70.17) - open/close 1429 (time 1788157434.34 total 20.01 last 72.66) - open/close 2138 (time 1788157444.35 total 30.02 last 70.84) - open/close 2871 (time 1788157454.35 total 40.02 last 73.29) - open/close 3492 (time 1788157464.35 total 50.02 last 62.09) - open/close 4035 (time 1788157474.37 total 60.04 last 54.21) total: 4500 open/close in 68.94 seconds: 65.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1176 (time 1788157499.54 total 10.01 last 117.47) - /unlink 2188 (time 1788157509.54 total 20.01 last 101.17) - /unlink 3381 (time 1788157519.54 total 30.02 last 119.27) - /unlink 4464 (time 1788157529.55 total 40.02 last 108.27) total: 4500 /unlink in 40.52 seconds: 111.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 620 (time 1788157545.42 total 10.01 last 61.93) - open/close 1250 (time 1788157555.42 total 20.02 last 62.96) - open/close 1909 (time 1788157565.43 total 30.02 last 65.88) - open/close 2513 (time 1788157575.43 total 40.03 last 60.35) - open/close 3121 (time 1788157585.48 total 50.07 last 60.54) - open/close 3725 (time 1788157595.48 total 60.07 last 60.38) - open/close 4380 (time 1788157605.50 total 70.10 last 65.35) total: 4500 open/close in 72.11 seconds: 62.41 ops/second - /unlink 1262 (time 1788157622.49 total 10.00 last 126.20) - /unlink 2490 (time 1788157632.50 total 20.01 last 122.70) - /unlink 3796 (time 1788157642.50 total 30.01 last 130.52) total: 4500 /unlink in 35.70 seconds: 126.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 676 (time 1788157664.28 total 10.00 last 67.59) - open/close 1363 (time 1788157674.28 total 20.00 last 68.70) - open/close 2030 (time 1788157684.29 total 30.01 last 66.64) - open/close 2669 (time 1788157694.30 total 40.03 last 63.82) - open/close 3328 (time 1788157704.31 total 50.03 last 65.85) - open/close 4073 (time 1788157714.31 total 60.04 last 74.47) total: 4500 open/close in 66.69 seconds: 67.48 ops/second - /unlink 1287 (time 1788157735.22 total 10.01 last 128.51) - /unlink 2548 (time 1788157745.23 total 20.02 last 126.06) - /unlink 3709 (time 1788157755.23 total 30.02 last 116.07) total: 4500 /unlink in 36.59 seconds: 122.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 665 (time 1788157777.55 total 10.01 last 66.45) - open/close 1367 (time 1788157787.56 total 20.02 last 70.13) - open/close 2002 (time 1788157797.57 total 30.02 last 63.48) - open/close 2668 (time 1788157807.58 total 40.03 last 66.53) - open/close 3348 (time 1788157817.59 total 50.05 last 67.89) - open/close 4020 (time 1788157827.61 total 60.06 last 67.09) total: 4500 open/close in 66.98 seconds: 67.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1219 (time 1788157849.60 total 10.00 last 121.88) - /unlink 2510 (time 1788157859.60 total 20.00 last 129.08) - /unlink 3679 (time 1788157869.61 total 30.01 last 116.76) total: 4500 /unlink in 36.85 seconds: 122.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 661 (time 1788157891.49 total 10.00 last 66.09) - open/close 1199 (time 1788157901.49 total 20.00 last 53.79) - open/close 1800 (time 1788157911.51 total 30.02 last 60.01) - open/close 2416 (time 1788157921.51 total 40.02 last 61.59) - open/close 2983 (time 1788157931.52 total 50.04 last 56.60) - open/close 3504 (time 1788157941.52 total 60.04 last 52.09) - open/close 4054 (time 1788157951.53 total 70.04 last 54.99) total: 4500 open/close in 76.93 seconds: 58.49 ops/second - /unlink 1192 (time 1788157973.87 total 10.01 last 119.12) - /unlink 2355 (time 1788157983.87 total 20.01 last 116.28) - /unlink 3606 (time 1788157993.89 total 30.02 last 124.93) total: 4500 /unlink in 37.06 seconds: 121.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 707 (time 1788158015.81 total 10.01 last 70.66) - open/close 1449 (time 1788158025.82 total 20.01 last 74.14) - open/close 2144 (time 1788158035.83 total 30.02 last 69.45) - open/close 2826 (time 1788158045.83 total 40.03 last 68.16) - open/close 3547 (time 1788158055.84 total 50.04 last 72.03) - open/close 4188 (time 1788158065.85 total 60.04 last 64.07) total: 4500 open/close in 64.56 seconds: 69.70 ops/second - /unlink 1446 (time 1788158085.34 total 10.00 last 144.55) - /unlink 2690 (time 1788158095.35 total 20.01 last 124.30) - /unlink 3995 (time 1788158105.35 total 30.02 last 130.44) total: 4500 /unlink in 34.19 seconds: 131.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 564 (time 1788158125.73 total 10.01 last 56.37) - open/close 1120 (time 1788158135.73 total 20.01 last 55.59) - open/close 1661 (time 1788158145.73 total 30.01 last 54.07) - open/close 2185 (time 1788158155.75 total 40.03 last 52.30) - open/close 2795 (time 1788158165.76 total 50.04 last 60.95) - open/close 3435 (time 1788158175.77 total 60.05 last 63.96) - open/close 4093 (time 1788158185.77 total 70.05 last 65.80) total: 4500 open/close in 76.10 seconds: 59.14 ops/second - /unlink 1189 (time 1788158206.91 total 10.00 last 118.84) - /unlink 2336 (time 1788158216.92 total 20.01 last 114.65) - /unlink 3483 (time 1788158226.93 total 30.02 last 114.52) total: 4500 /unlink in 39.04 seconds: 115.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 751 (time 1788158250.65 total 10.01 last 74.99) - open/close 1438 (time 1788158260.66 total 20.03 last 68.61) - open/close 2161 (time 1788158270.67 total 30.04 last 72.24) - open/close 2865 (time 1788158280.68 total 40.04 last 70.35) - open/close 3575 (time 1788158290.68 total 50.05 last 70.98) - open/close 4371 (time 1788158300.68 total 60.05 last 79.56) total: 4500 open/close in 61.87 seconds: 72.73 ops/second - /unlink 1277 (time 1788158317.73 total 10.00 last 127.68) - /unlink 2456 (time 1788158327.73 total 20.01 last 117.84) - /unlink 3659 (time 1788158337.74 total 30.02 last 120.19) total: 4500 /unlink in 36.37 seconds: 123.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 725 (time 1788158359.79 total 10.01 last 72.42) - open/close 1369 (time 1788158369.80 total 20.02 last 64.37) - open/close 2113 (time 1788158379.81 total 30.03 last 74.30) - open/close 2852 (time 1788158389.82 total 40.03 last 73.86) - open/close 3532 (time 1788158399.82 total 50.04 last 67.96) - open/close 4265 (time 1788158409.83 total 60.05 last 73.26) total: 4500 open/close in 62.90 seconds: 71.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1273 (time 1788158426.81 total 10.01 last 127.21) - /unlink 2620 (time 1788158436.81 total 20.01 last 134.66) - /unlink 3946 (time 1788158446.82 total 30.01 last 132.58) total: 4500 /unlink in 34.45 seconds: 130.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 732 (time 1788158465.84 total 10.00 last 73.19) - open/close 1384 (time 1788158475.84 total 20.00 last 65.19) - open/close 2047 (time 1788158485.89 total 30.05 last 65.98) - open/close 2693 (time 1788158495.89 total 40.05 last 64.59) - open/close 3351 (time 1788158505.91 total 50.07 last 65.68) - open/close 3999 (time 1788158515.92 total 60.08 last 64.72) total: 4500 open/close in 67.81 seconds: 66.36 ops/second - /unlink 1362 (time 1788158538.31 total 10.01 last 136.12) - /unlink 2614 (time 1788158548.31 total 20.01 last 125.17) - /unlink 3880 (time 1788158558.31 total 30.01 last 126.57) total: 4500 /unlink in 34.26 seconds: 131.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 761 (time 1788158577.67 total 10.01 last 76.06) - open/close 1469 (time 1788158587.68 total 20.01 last 70.74) - open/close 2144 (time 1788158597.69 total 30.02 last 67.44) - open/close 2831 (time 1788158607.69 total 40.03 last 68.69) - open/close 3520 (time 1788158617.70 total 50.04 last 68.82) - open/close 4143 (time 1788158627.71 total 60.05 last 62.24) total: 4500 open/close in 65.16 seconds: 69.07 ops/second - /unlink 1179 (time 1788158647.50 total 10.00 last 117.88) - /unlink 2363 (time 1788158657.51 total 20.00 last 118.37) - /unlink 3670 (time 1788158667.51 total 30.00 last 130.70) total: 4500 /unlink in 36.81 seconds: 122.24 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95355 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg646-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg646-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg646-server stop mds service on oleg646-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg646-server stop mds service on oleg646-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg646-server unloading modules via unload_modules_local on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing unload_modules_local oleg646-server: modules unloaded.