== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:37:22 (1789123042) Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg254-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg254-server oleg254-server: oleg254-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg254-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg254-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg254-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 639 (time 1789123279.26 total 10.01 last 63.82) - open/close 1278 (time 1789123289.26 total 20.02 last 63.87) - open/close 1895 (time 1789123299.30 total 30.06 last 61.46) - open/close 2462 (time 1789123309.32 total 40.08 last 56.60) - open/close 3096 (time 1789123319.33 total 50.08 last 63.34) - open/close 3806 (time 1789123329.35 total 60.11 last 70.84) - open/close 4496 (time 1789123339.35 total 70.11 last 68.96) total: 4500 open/close in 70.14 seconds: 64.16 ops/second - /unlink 1365 (time 1789123354.95 total 10.02 last 136.18) - /unlink 2567 (time 1789123364.96 total 20.03 last 120.08) - /unlink 3477 (time 1789123374.96 total 30.04 last 90.96) total: 4500 /unlink in 39.62 seconds: 113.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 597 (time 1789123400.56 total 10.01 last 59.65) - open/close 1129 (time 1789123410.59 total 20.03 last 53.06) - open/close 1595 (time 1789123420.61 total 30.05 last 46.51) - open/close 2035 (time 1789123430.62 total 40.07 last 43.92) - open/close 2446 (time 1789123440.63 total 50.08 last 41.08) - open/close 2926 (time 1789123450.63 total 60.08 last 47.99) - open/close 3657 (time 1789123460.64 total 70.09 last 73.01) - open/close 4306 (time 1789123470.65 total 80.10 last 64.84) total: 4500 open/close in 82.94 seconds: 54.25 ops/second - /unlink 1309 (time 1789123489.50 total 10.01 last 130.82) - /unlink 2535 (time 1789123499.51 total 20.02 last 122.45) - /unlink 3780 (time 1789123509.52 total 30.02 last 124.48) total: 4500 /unlink in 35.40 seconds: 127.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 545 (time 1789123530.72 total 10.02 last 54.39) - open/close 1177 (time 1789123540.73 total 20.03 last 63.12) - open/close 1747 (time 1789123550.74 total 30.04 last 56.96) - open/close 2448 (time 1789123560.75 total 40.05 last 70.02) - open/close 3128 (time 1789123570.77 total 50.07 last 67.84) - open/close 3796 (time 1789123580.77 total 60.08 last 66.78) - open/close 4481 (time 1789123590.78 total 70.08 last 68.45) total: 4500 open/close in 70.48 seconds: 63.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1066 (time 1789123606.83 total 10.01 last 106.53) - /unlink 2299 (time 1789123616.85 total 20.02 last 123.09) - /unlink 3596 (time 1789123626.85 total 30.03 last 129.68) total: 4500 /unlink in 37.14 seconds: 121.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 609 (time 1789123649.94 total 10.01 last 60.87) - open/close 1147 (time 1789123659.94 total 20.01 last 53.77) - open/close 1879 (time 1789123669.95 total 30.02 last 73.17) - open/close 2482 (time 1789123679.95 total 40.02 last 60.27) - open/close 3034 (time 1789123689.96 total 50.03 last 55.16) - open/close 3601 (time 1789123699.96 total 60.03 last 56.67) - open/close 4236 (time 1789123709.97 total 70.04 last 63.46) total: 4500 open/close in 74.15 seconds: 60.69 ops/second - /unlink 1153 (time 1789123729.92 total 10.00 last 115.24) - /unlink 2335 (time 1789123739.92 total 20.01 last 118.16) - /unlink 3247 (time 1789123749.93 total 30.02 last 91.13) - /unlink 4276 (time 1789123759.94 total 40.03 last 102.80) total: 4500 /unlink in 42.63 seconds: 105.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 676 (time 1789123778.49 total 10.01 last 67.51) - open/close 1256 (time 1789123788.50 total 20.03 last 57.91) - open/close 1820 (time 1789123798.53 total 30.06 last 56.24) - open/close 2434 (time 1789123808.55 total 40.08 last 61.29) - open/close 3153 (time 1789123818.56 total 50.09 last 71.85) - open/close 3802 (time 1789123828.57 total 60.09 last 64.85) - open/close 4476 (time 1789123838.58 total 70.11 last 67.27) total: 4500 open/close in 70.68 seconds: 63.67 ops/second - /unlink 1105 (time 1789123855.52 total 10.01 last 110.37) - /unlink 2202 (time 1789123865.52 total 20.02 last 109.67) - /unlink 3285 (time 1789123875.54 total 30.03 last 108.15) - /unlink 4132 (time 1789123885.54 total 40.03 last 84.66) total: 4500 /unlink in 42.24 seconds: 106.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 770 (time 1789123903.40 total 10.01 last 76.95) - open/close 1419 (time 1789123913.41 total 20.01 last 64.85) - open/close 2029 (time 1789123923.42 total 30.03 last 60.92) - open/close 2750 (time 1789123933.45 total 40.06 last 71.89) - open/close 3336 (time 1789123943.49 total 50.09 last 58.41) - open/close 3981 (time 1789123953.49 total 60.10 last 64.45) total: 4500 open/close in 68.77 seconds: 65.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1449 (time 1789123977.11 total 10.00 last 144.84) - /unlink 2594 (time 1789123987.11 total 20.01 last 114.43) - /unlink 3754 (time 1789123997.11 total 30.01 last 115.98) total: 4500 /unlink in 38.56 seconds: 116.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 469 (time 1789124022.62 total 10.00 last 46.89) - open/close 950 (time 1789124032.62 total 20.00 last 48.10) - open/close 1505 (time 1789124042.63 total 30.01 last 55.45) - open/close 2172 (time 1789124052.64 total 40.02 last 66.66) - open/close 2706 (time 1789124062.65 total 50.03 last 53.34) - open/close 3317 (time 1789124072.65 total 60.03 last 61.07) - open/close 4055 (time 1789124082.65 total 70.03 last 73.80) total: 4500 open/close in 75.96 seconds: 59.24 ops/second - /unlink 1325 (time 1789124103.78 total 10.00 last 132.47) - /unlink 2471 (time 1789124113.79 total 20.01 last 114.48) - /unlink 3728 (time 1789124123.79 total 30.02 last 125.65) total: 4500 /unlink in 37.15 seconds: 121.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 598 (time 1789124146.62 total 10.01 last 59.75) - open/close 1223 (time 1789124156.63 total 20.01 last 62.48) - open/close 1896 (time 1789124166.64 total 30.02 last 67.23) - open/close 2536 (time 1789124176.64 total 40.03 last 63.97) - open/close 3203 (time 1789124186.64 total 50.03 last 66.69) - open/close 3869 (time 1789124196.65 total 60.03 last 66.56) - open/close 4480 (time 1789124206.65 total 70.04 last 61.07) total: 4500 open/close in 70.50 seconds: 63.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1293 (time 1789124222.13 total 10.00 last 129.29) - /unlink 2543 (time 1789124232.13 total 20.01 last 124.94) - /unlink 3797 (time 1789124242.15 total 30.02 last 125.21) total: 4500 /unlink in 36.06 seconds: 124.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 642 (time 1789124264.75 total 10.00 last 64.18) - open/close 1312 (time 1789124274.78 total 20.04 last 66.78) - open/close 2047 (time 1789124284.79 total 30.04 last 73.49) - open/close 2786 (time 1789124294.79 total 40.04 last 73.86) - open/close 3486 (time 1789124304.79 total 50.04 last 69.99) - open/close 4242 (time 1789124314.81 total 60.06 last 75.49) total: 4500 open/close in 63.89 seconds: 70.44 ops/second - /unlink 1412 (time 1789124333.78 total 10.01 last 141.09) - /unlink 2464 (time 1789124343.80 total 20.03 last 105.01) - /unlink 3606 (time 1789124353.80 total 30.03 last 114.16) total: 4500 /unlink in 36.98 seconds: 121.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 719 (time 1789124376.24 total 10.03 last 71.70) - open/close 1430 (time 1789124386.24 total 20.03 last 71.07) - open/close 2207 (time 1789124396.25 total 30.03 last 77.68) - open/close 2936 (time 1789124406.26 total 40.04 last 72.83) - open/close 3486 (time 1789124416.26 total 50.05 last 54.97) - open/close 4062 (time 1789124426.27 total 60.05 last 57.57) total: 4500 open/close in 68.05 seconds: 66.13 ops/second - /unlink 1265 (time 1789124450.06 total 10.00 last 126.49) - /unlink 2206 (time 1789124460.06 total 20.00 last 94.06) - /unlink 3234 (time 1789124470.07 total 30.02 last 102.68) total: 4500 /unlink in 39.94 seconds: 112.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 592 (time 1789124495.79 total 10.00 last 59.17) - open/close 1292 (time 1789124505.80 total 20.01 last 69.93) - open/close 2014 (time 1789124515.80 total 30.02 last 72.19) - open/close 2683 (time 1789124525.81 total 40.03 last 66.84) - open/close 3362 (time 1789124535.82 total 50.04 last 67.84) - open/close 4048 (time 1789124545.83 total 60.05 last 68.53) total: 4500 open/close in 67.36 seconds: 66.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1565 (time 1789124568.77 total 10.01 last 156.34) - /unlink 2981 (time 1789124578.77 total 20.01 last 141.53) - /unlink 4147 (time 1789124588.78 total 30.02 last 116.53) total: 4500 /unlink in 33.46 seconds: 134.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 647 (time 1789124608.28 total 10.02 last 64.59) - open/close 1287 (time 1789124618.29 total 20.03 last 63.95) - open/close 2039 (time 1789124628.31 total 30.04 last 75.08) - open/close 2745 (time 1789124638.31 total 40.05 last 70.57) - open/close 3458 (time 1789124648.32 total 50.05 last 71.25) - open/close 4162 (time 1789124658.32 total 60.06 last 70.34) total: 4500 open/close in 64.70 seconds: 69.55 ops/second - /unlink 1069 (time 1789124677.93 total 10.00 last 106.86) - /unlink 2248 (time 1789124687.93 total 20.01 last 117.84) - /unlink 3434 (time 1789124697.94 total 30.01 last 118.56) total: 4500 /unlink in 38.19 seconds: 117.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 544 (time 1789124722.16 total 10.00 last 54.38) - open/close 1156 (time 1789124732.16 total 20.01 last 61.19) - open/close 1908 (time 1789124742.17 total 30.01 last 75.16) - open/close 2512 (time 1789124752.17 total 40.01 last 60.39) - open/close 3227 (time 1789124762.17 total 50.02 last 71.47) - open/close 3997 (time 1789124772.19 total 60.03 last 76.90) total: 4500 open/close in 67.84 seconds: 66.33 ops/second - /unlink 1104 (time 1789124794.88 total 10.00 last 110.35) - /unlink 2136 (time 1789124804.89 total 20.02 last 103.06) - /unlink 3247 (time 1789124814.90 total 30.02 last 111.06) total: 4500 /unlink in 38.31 seconds: 117.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 718 (time 1789124838.08 total 10.00 last 71.77) - open/close 1412 (time 1789124848.10 total 20.02 last 69.29) - open/close 2081 (time 1789124858.12 total 30.04 last 66.79) - open/close 2687 (time 1789124868.12 total 40.04 last 60.59) - open/close 3334 (time 1789124878.14 total 50.06 last 64.58) - open/close 3984 (time 1789124888.14 total 60.06 last 64.99) total: 4500 open/close in 66.72 seconds: 67.45 ops/second - /unlink 1390 (time 1789124909.40 total 10.00 last 138.93) - /unlink 2562 (time 1789124919.40 total 20.01 last 117.18) - /unlink 3783 (time 1789124929.40 total 30.01 last 122.09) total: 4500 /unlink in 34.68 seconds: 129.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 605 (time 1789124949.46 total 10.00 last 60.48) - open/close 1352 (time 1789124959.47 total 20.01 last 74.66) - open/close 2004 (time 1789124969.48 total 30.02 last 65.15) - open/close 2609 (time 1789124979.48 total 40.02 last 60.48) - open/close 3393 (time 1789124989.49 total 50.03 last 78.29) - open/close 4039 (time 1789124999.50 total 60.04 last 64.55) total: 4500 open/close in 66.02 seconds: 68.17 ops/second - /unlink 1298 (time 1789125020.50 total 10.00 last 129.79) - /unlink 2776 (time 1789125030.50 total 20.00 last 147.79) - /unlink 4329 (time 1789125040.51 total 30.00 last 155.26) total: 4500 /unlink in 31.27 seconds: 143.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 642 (time 1789125058.04 total 10.01 last 64.13) - open/close 1186 (time 1789125068.05 total 20.02 last 54.32) - open/close 1893 (time 1789125078.06 total 30.03 last 70.64) - open/close 2508 (time 1789125088.08 total 40.06 last 61.36) - open/close 3069 (time 1789125098.09 total 50.06 last 56.09) - open/close 3661 (time 1789125108.10 total 60.07 last 59.12) - open/close 4322 (time 1789125118.10 total 70.08 last 66.07) total: 4500 open/close in 73.01 seconds: 61.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1178 (time 1789125136.92 total 10.02 last 117.60) - /unlink 2304 (time 1789125146.92 total 20.02 last 112.56) - /unlink 3264 (time 1789125156.92 total 30.02 last 95.97) - /unlink 4217 (time 1789125166.93 total 40.03 last 95.27) total: 4500 /unlink in 42.61 seconds: 105.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 606 (time 1789125185.16 total 10.01 last 60.56) - open/close 1376 (time 1789125195.16 total 20.01 last 77.00) - open/close 2087 (time 1789125205.18 total 30.02 last 70.98) - open/close 2825 (time 1789125215.19 total 40.03 last 73.75) - open/close 3441 (time 1789125225.19 total 50.04 last 61.57) - open/close 4114 (time 1789125235.20 total 60.05 last 67.21) total: 4500 open/close in 65.78 seconds: 68.41 ops/second - /unlink 1396 (time 1789125256.25 total 10.02 last 139.38) - /unlink 2398 (time 1789125266.26 total 20.02 last 100.14) - /unlink 3472 (time 1789125276.26 total 30.02 last 107.38) total: 4500 /unlink in 38.60 seconds: 116.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 696 (time 1789125301.08 total 10.00 last 69.58) - open/close 1334 (time 1789125311.08 total 20.01 last 63.79) - open/close 1983 (time 1789125321.10 total 30.02 last 64.82) - open/close 2661 (time 1789125331.11 total 40.03 last 67.71) - open/close 3335 (time 1789125341.12 total 50.04 last 67.35) - open/close 4045 (time 1789125351.12 total 60.04 last 70.98) total: 4500 open/close in 67.73 seconds: 66.44 ops/second - /unlink 1212 (time 1789125373.55 total 10.00 last 121.18) - /unlink 2446 (time 1789125383.55 total 20.00 last 123.38) - /unlink 3562 (time 1789125393.56 total 30.01 last 111.52) total: 4500 /unlink in 37.70 seconds: 119.36 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95425 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server unloading modules via unload_modules_local on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local oleg254-server: modules unloaded.