== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:12:10 (1789380730) Stopping clients: oleg123-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg123-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg123-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg123-server oleg123-server: oleg123-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: kmemleak disabled oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: detected 4 online CPUs by sysfs oleg123-server: MODOPTS_LIBCFS= oleg123-server: Force libcfs to create 2 CPU partitions oleg123-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg123-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 start mds service on oleg123-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg123-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 493 (time 1789380965.57 total 10.00 last 49.30) - open/close 930 (time 1789380975.59 total 20.02 last 43.61) - open/close 1298 (time 1789380985.60 total 30.02 last 36.80) - open/close 1831 (time 1789380995.60 total 40.02 last 53.29) - open/close 2295 (time 1789381005.61 total 50.03 last 46.36) - open/close 2652 (time 1789381015.61 total 60.03 last 35.70) - open/close 3125 (time 1789381025.61 total 70.04 last 47.28) - open/close 3594 (time 1789381035.62 total 80.05 last 46.86) - open/close 3994 (time 1789381045.65 total 90.08 last 39.87) total: 4500 open/close in 98.14 seconds: 45.85 ops/second - /unlink 782 (time 1789381070.07 total 10.01 last 78.15) - /unlink 1408 (time 1789381080.07 total 20.01 last 62.59) - /unlink 2319 (time 1789381090.07 total 30.01 last 91.06) - /unlink 3617 (time 1789381100.07 total 40.02 last 129.75) total: 4500 /unlink in 46.85 seconds: 96.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 595 (time 1789381122.35 total 10.00 last 59.49) - open/close 1128 (time 1789381132.36 total 20.01 last 53.23) - open/close 1663 (time 1789381142.38 total 30.03 last 53.41) - open/close 2228 (time 1789381152.38 total 40.04 last 56.47) - open/close 2766 (time 1789381162.39 total 50.05 last 53.75) - open/close 3299 (time 1789381172.39 total 60.05 last 53.30) - open/close 3908 (time 1789381182.40 total 70.05 last 60.88) - open/close 4499 (time 1789381192.42 total 80.07 last 59.00) total: 4500 open/close in 80.07 seconds: 56.20 ops/second - /unlink 1325 (time 1789381207.75 total 10.00 last 132.45) - /unlink 2485 (time 1789381217.75 total 20.00 last 116.00) - /unlink 3822 (time 1789381227.75 total 30.01 last 133.63) total: 4500 /unlink in 35.40 seconds: 127.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 775 (time 1789381248.15 total 10.01 last 77.42) - open/close 1533 (time 1789381258.17 total 20.03 last 75.64) - open/close 2165 (time 1789381268.19 total 30.05 last 63.08) - open/close 2938 (time 1789381278.19 total 40.05 last 77.29) - open/close 3535 (time 1789381288.20 total 50.06 last 59.65) - open/close 4238 (time 1789381298.21 total 60.07 last 70.22) total: 4500 open/close in 64.32 seconds: 69.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1327 (time 1789381318.21 total 10.01 last 132.59) - /unlink 2488 (time 1789381328.21 total 20.01 last 116.07) - /unlink 3615 (time 1789381338.22 total 30.02 last 112.65) total: 4500 /unlink in 36.67 seconds: 122.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 769 (time 1789381360.48 total 10.01 last 76.82) - open/close 1488 (time 1789381370.49 total 20.02 last 71.84) - open/close 2040 (time 1789381380.49 total 30.02 last 55.19) - open/close 2678 (time 1789381390.50 total 40.03 last 63.73) - open/close 3342 (time 1789381400.50 total 50.04 last 66.37) - open/close 4089 (time 1789381410.51 total 60.05 last 74.62) total: 4500 open/close in 65.59 seconds: 68.61 ops/second - /unlink 1500 (time 1789381431.43 total 10.02 last 149.77) - /unlink 2651 (time 1789381441.44 total 20.02 last 115.03) - /unlink 3830 (time 1789381451.44 total 30.02 last 117.87) total: 4500 /unlink in 35.01 seconds: 128.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 641 (time 1789381472.84 total 10.04 last 63.82) - open/close 1387 (time 1789381482.85 total 20.05 last 74.52) - open/close 2054 (time 1789381492.86 total 30.07 last 66.58) - open/close 2725 (time 1789381502.88 total 40.09 last 66.97) - open/close 3251 (time 1789381512.90 total 50.10 last 52.54) - open/close 3642 (time 1789381522.90 total 60.11 last 39.09) - open/close 4211 (time 1789381532.90 total 70.11 last 56.90) total: 4500 open/close in 73.89 seconds: 60.90 ops/second - /unlink 1085 (time 1789381553.30 total 10.01 last 108.39) - /unlink 2179 (time 1789381563.30 total 20.01 last 109.37) - /unlink 3351 (time 1789381573.31 total 30.02 last 117.11) total: 4500 /unlink in 38.93 seconds: 115.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 677 (time 1789381597.72 total 10.01 last 67.66) - open/close 1172 (time 1789381607.73 total 20.02 last 49.46) - open/close 1855 (time 1789381617.73 total 30.02 last 68.27) - open/close 2545 (time 1789381627.75 total 40.04 last 68.89) - open/close 3133 (time 1789381637.75 total 50.04 last 58.78) - open/close 3744 (time 1789381647.75 total 60.04 last 61.10) - open/close 4338 (time 1789381657.76 total 70.05 last 59.34) total: 4500 open/close in 72.54 seconds: 62.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1612 (time 1789381674.79 total 10.01 last 161.11) - /unlink 2861 (time 1789381684.79 total 20.01 last 124.80) - /unlink 4158 (time 1789381694.79 total 30.01 last 129.68) total: 4500 /unlink in 33.27 seconds: 135.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 526 (time 1789381715.11 total 10.01 last 52.57) - open/close 1005 (time 1789381725.12 total 20.02 last 47.82) - open/close 1511 (time 1789381735.14 total 30.04 last 50.54) - open/close 1998 (time 1789381745.16 total 40.06 last 48.59) - open/close 2684 (time 1789381755.16 total 50.06 last 68.59) - open/close 3356 (time 1789381765.16 total 60.06 last 67.18) - open/close 4004 (time 1789381775.17 total 70.07 last 64.76) total: 4500 open/close in 77.20 seconds: 58.29 ops/second - /unlink 979 (time 1789381798.59 total 10.02 last 97.70) - /unlink 1792 (time 1789381808.59 total 20.02 last 81.29) - /unlink 2670 (time 1789381818.60 total 30.03 last 87.75) - /unlink 3757 (time 1789381828.60 total 40.03 last 108.64) total: 4500 /unlink in 46.40 seconds: 96.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 558 (time 1789381850.42 total 10.00 last 55.78) - open/close 1170 (time 1789381860.43 total 20.01 last 61.19) - open/close 1736 (time 1789381870.44 total 30.02 last 56.55) - open/close 2451 (time 1789381880.46 total 40.04 last 71.32) - open/close 3128 (time 1789381890.47 total 50.05 last 67.63) - open/close 3964 (time 1789381900.48 total 60.06 last 83.55) total: 4500 open/close in 66.96 seconds: 67.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1451 (time 1789381921.92 total 10.00 last 145.05) - /unlink 2725 (time 1789381931.93 total 20.02 last 127.21) - /unlink 4008 (time 1789381941.94 total 30.02 last 128.25) total: 4500 /unlink in 33.79 seconds: 133.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 723 (time 1789381961.12 total 10.01 last 72.24) - open/close 1472 (time 1789381971.13 total 20.01 last 74.89) - open/close 2194 (time 1789381981.13 total 30.01 last 72.18) - open/close 2816 (time 1789381991.13 total 40.01 last 62.20) - open/close 3473 (time 1789382001.13 total 50.01 last 65.69) - open/close 4096 (time 1789382011.14 total 60.03 last 62.22) total: 4500 open/close in 66.51 seconds: 67.66 ops/second - /unlink 731 (time 1789382035.13 total 10.01 last 73.06) - /unlink 2059 (time 1789382045.14 total 20.01 last 132.69) - /unlink 3065 (time 1789382055.15 total 30.02 last 100.56) - /unlink 4134 (time 1789382065.17 total 40.04 last 106.67) total: 4500 /unlink in 43.57 seconds: 103.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 717 (time 1789382085.15 total 10.02 last 71.53) - open/close 1476 (time 1789382095.16 total 20.03 last 75.84) - open/close 2153 (time 1789382105.17 total 30.05 last 67.61) - open/close 2996 (time 1789382115.18 total 40.05 last 84.28) - open/close 3733 (time 1789382125.19 total 50.06 last 73.63) total: 4500 open/close in 59.96 seconds: 75.05 ops/second - /unlink 1166 (time 1789382150.77 total 10.00 last 116.56) - /unlink 2134 (time 1789382160.79 total 20.02 last 96.65) - /unlink 3145 (time 1789382170.79 total 30.03 last 101.01) - /unlink 4265 (time 1789382180.80 total 40.03 last 112.00) total: 4500 /unlink in 42.10 seconds: 106.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 668 (time 1789382198.70 total 10.00 last 66.79) - open/close 1165 (time 1789382208.72 total 20.02 last 49.62) - open/close 1737 (time 1789382218.74 total 30.04 last 57.07) - open/close 2390 (time 1789382228.74 total 40.04 last 65.28) - open/close 3018 (time 1789382238.74 total 50.05 last 62.78) - open/close 3647 (time 1789382248.75 total 60.06 last 62.84) - open/close 4329 (time 1789382258.76 total 70.06 last 68.17) total: 4500 open/close in 72.95 seconds: 61.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1150 (time 1789382278.27 total 10.00 last 115.00) - /unlink 2115 (time 1789382288.29 total 20.01 last 96.38) - /unlink 3258 (time 1789382298.29 total 30.02 last 114.26) - /unlink 4482 (time 1789382308.29 total 40.02 last 122.37) total: 4500 /unlink in 40.20 seconds: 111.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 667 (time 1789382324.72 total 10.00 last 66.67) - open/close 1252 (time 1789382334.72 total 20.01 last 58.48) - open/close 1992 (time 1789382344.72 total 30.01 last 73.97) - open/close 2663 (time 1789382354.73 total 40.01 last 67.08) - open/close 3408 (time 1789382364.73 total 50.02 last 74.47) - open/close 4145 (time 1789382374.73 total 60.02 last 73.66) total: 4500 open/close in 65.89 seconds: 68.29 ops/second - /unlink 1136 (time 1789382395.51 total 10.00 last 113.55) - /unlink 2446 (time 1789382405.51 total 20.00 last 131.00) - /unlink 3539 (time 1789382415.51 total 30.01 last 109.29) - /unlink 4419 (time 1789382425.52 total 40.02 last 87.91) total: 4500 /unlink in 41.10 seconds: 109.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 677 (time 1789382442.46 total 10.00 last 67.70) - open/close 1325 (time 1789382452.46 total 20.01 last 64.76) - open/close 2056 (time 1789382462.48 total 30.02 last 72.98) - open/close 2724 (time 1789382472.49 total 40.03 last 66.77) - open/close 3416 (time 1789382482.49 total 50.04 last 69.14) - open/close 4111 (time 1789382492.50 total 60.05 last 69.43) total: 4500 open/close in 65.45 seconds: 68.76 ops/second - /unlink 1300 (time 1789382513.10 total 10.00 last 129.94) - /unlink 2753 (time 1789382523.10 total 20.01 last 145.23) - /unlink 4052 (time 1789382533.11 total 30.01 last 129.84) total: 4500 /unlink in 33.83 seconds: 133.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 733 (time 1789382552.10 total 10.00 last 73.27) - open/close 1467 (time 1789382562.10 total 20.00 last 73.40) - open/close 2210 (time 1789382572.11 total 30.02 last 74.21) - open/close 3005 (time 1789382582.12 total 40.02 last 79.46) - open/close 3649 (time 1789382592.12 total 50.03 last 64.36) - open/close 4374 (time 1789382602.13 total 60.04 last 72.44) total: 4500 open/close in 61.54 seconds: 73.12 ops/second - /unlink 1507 (time 1789382617.88 total 10.00 last 150.66) - /unlink 2719 (time 1789382627.90 total 20.02 last 121.04) - /unlink 3880 (time 1789382637.90 total 30.02 last 116.07) total: 4500 /unlink in 34.53 seconds: 130.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 880 (time 1789382657.49 total 10.00 last 87.96) - open/close 1651 (time 1789382667.49 total 20.01 last 77.09) - open/close 2330 (time 1789382677.52 total 30.04 last 67.69) - open/close 2977 (time 1789382687.53 total 40.05 last 64.64) - open/close 3649 (time 1789382697.53 total 50.05 last 67.20) - open/close 4427 (time 1789382707.56 total 60.07 last 77.63) total: 4500 open/close in 61.71 seconds: 72.92 ops/second - /unlink 1112 (time 1789382725.15 total 10.00 last 111.18) - /unlink 2560 (time 1789382735.17 total 20.01 last 144.63) - /unlink 3888 (time 1789382745.18 total 30.03 last 132.63) total: 4500 /unlink in 35.14 seconds: 128.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 669 (time 1789382765.18 total 10.02 last 66.79) - open/close 1207 (time 1789382775.19 total 20.02 last 53.78) - open/close 1745 (time 1789382785.19 total 30.02 last 53.78) - open/close 2336 (time 1789382795.19 total 40.03 last 59.09) - open/close 2888 (time 1789382805.21 total 50.04 last 55.12) - open/close 3512 (time 1789382815.22 total 60.05 last 62.33) - open/close 4291 (time 1789382825.22 total 70.06 last 77.85) total: 4500 open/close in 73.35 seconds: 61.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 842 (time 1789382846.26 total 10.00 last 84.16) - /unlink 1757 (time 1789382856.26 total 20.01 last 91.48) - /unlink 3234 (time 1789382866.26 total 30.01 last 147.70) total: 4500 /unlink in 39.54 seconds: 113.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 609 (time 1789382890.55 total 10.01 last 60.84) - open/close 1225 (time 1789382900.56 total 20.02 last 61.54) - open/close 2021 (time 1789382910.57 total 30.03 last 79.54) - open/close 2719 (time 1789382920.57 total 40.03 last 69.80) - open/close 3370 (time 1789382930.58 total 50.04 last 65.02) - open/close 3935 (time 1789382940.59 total 60.05 last 56.45) - open/close 4462 (time 1789382950.61 total 70.07 last 52.60) total: 4500 open/close in 70.60 seconds: 63.74 ops/second - /unlink 1477 (time 1789382966.26 total 10.00 last 147.66) - /unlink 2768 (time 1789382976.27 total 20.01 last 129.07) - /unlink 4473 (time 1789382986.27 total 30.01 last 170.41) total: 4500 /unlink in 30.35 seconds: 148.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 481 (time 1789383001.35 total 10.01 last 48.05) - open/close 1108 (time 1789383011.38 total 20.04 last 62.51) - open/close 1861 (time 1789383021.40 total 30.06 last 75.17) - open/close 2528 (time 1789383031.40 total 40.06 last 66.68) - open/close 3324 (time 1789383041.41 total 50.08 last 79.49) - open/close 3922 (time 1789383051.42 total 60.08 last 59.76) total: 4500 open/close in 69.61 seconds: 64.64 ops/second - /unlink 1362 (time 1789383076.69 total 10.00 last 136.13) - /unlink 2643 (time 1789383086.69 total 20.01 last 128.05) - /unlink 3881 (time 1789383096.70 total 30.01 last 123.72) total: 4500 /unlink in 34.81 seconds: 129.26 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11415: 96350 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) oleg123-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 123 allow_register=unlimited stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg123-server unloading modules via unload_modules_local on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local oleg123-server: kmemleak disabled modules unloaded.