== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:03:44 (1777579424) Stopping clients: oleg355-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg355-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg355-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg355-server oleg355-server: oleg355-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing load_modules_local oleg355-server: Loading modules from /home/green/git/lustre-release/lustre oleg355-server: detected 4 online CPUs by sysfs oleg355-server: MODOPTS_LIBCFS= oleg355-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg355-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg355-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg355-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg355-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 686 (time 1777579651.86 total 10.01 last 68.55) - open/close 1338 (time 1777579661.88 total 20.02 last 65.09) - open/close 1937 (time 1777579671.90 total 30.04 last 59.81) - open/close 2403 (time 1777579681.92 total 40.07 last 46.47) - open/close 2793 (time 1777579691.96 total 50.11 last 38.85) - open/close 3329 (time 1777579701.97 total 60.11 last 53.55) - open/close 3971 (time 1777579711.98 total 70.12 last 64.15) total: 4500 open/close in 77.43 seconds: 58.12 ops/second - /unlink 1174 (time 1777579734.25 total 10.00 last 117.34) - /unlink 2290 (time 1777579744.25 total 20.01 last 111.59) - /unlink 3185 (time 1777579754.26 total 30.01 last 89.49) - /unlink 4087 (time 1777579764.26 total 40.01 last 90.18) total: 4500 /unlink in 43.51 seconds: 103.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 581 (time 1777579783.32 total 10.01 last 58.01) - open/close 1173 (time 1777579793.32 total 20.02 last 59.19) - open/close 1823 (time 1777579803.32 total 30.02 last 64.99) - open/close 2458 (time 1777579813.33 total 40.02 last 63.47) - open/close 3080 (time 1777579823.35 total 50.04 last 62.09) - open/close 3723 (time 1777579833.35 total 60.04 last 64.28) - open/close 4377 (time 1777579843.35 total 70.05 last 65.36) total: 4500 open/close in 72.11 seconds: 62.40 ops/second - /unlink 1217 (time 1777579860.47 total 10.01 last 121.60) - /unlink 2392 (time 1777579870.47 total 20.01 last 117.47) - /unlink 3515 (time 1777579880.47 total 30.01 last 112.29) total: 4500 /unlink in 39.33 seconds: 114.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 616 (time 1777579905.71 total 10.02 last 61.49) - open/close 1180 (time 1777579915.72 total 20.03 last 56.32) - open/close 1681 (time 1777579925.73 total 30.04 last 50.04) - open/close 2150 (time 1777579935.75 total 40.06 last 46.83) - open/close 2765 (time 1777579945.75 total 50.06 last 61.49) - open/close 3459 (time 1777579955.75 total 60.06 last 69.39) - open/close 3969 (time 1777579965.77 total 70.08 last 50.91) - open/close 4495 (time 1777579975.77 total 80.08 last 52.58) total: 4500 open/close in 80.16 seconds: 56.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 805 (time 1777579992.29 total 10.01 last 80.42) - /unlink 1654 (time 1777580002.30 total 20.02 last 84.84) - /unlink 2693 (time 1777580012.30 total 30.02 last 103.89) - /unlink 3700 (time 1777580022.30 total 40.02 last 100.68) total: 4500 /unlink in 46.88 seconds: 95.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 546 (time 1777580044.95 total 10.00 last 54.57) - open/close 1131 (time 1777580054.96 total 20.01 last 58.45) - open/close 1828 (time 1777580064.99 total 30.04 last 69.49) - open/close 2445 (time 1777580075.00 total 40.06 last 61.62) - open/close 3056 (time 1777580085.01 total 50.07 last 61.03) - open/close 3580 (time 1777580095.02 total 60.07 last 52.38) - open/close 3998 (time 1777580105.04 total 70.09 last 41.72) - open/close 4350 (time 1777580115.09 total 80.15 last 35.00) total: 4500 open/close in 84.75 seconds: 53.10 ops/second - /unlink 1120 (time 1777580136.33 total 10.02 last 111.77) - /unlink 1976 (time 1777580146.33 total 20.02 last 85.59) - /unlink 2817 (time 1777580156.35 total 30.04 last 83.95) - /unlink 3675 (time 1777580166.36 total 40.04 last 85.76) total: 4500 /unlink in 48.44 seconds: 92.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 539 (time 1777580190.72 total 10.00 last 53.89) - open/close 1154 (time 1777580200.74 total 20.02 last 61.40) - open/close 1764 (time 1777580210.75 total 30.03 last 60.94) - open/close 2359 (time 1777580220.75 total 40.03 last 59.47) - open/close 2887 (time 1777580230.75 total 50.04 last 52.79) - open/close 3520 (time 1777580240.76 total 60.04 last 63.25) - open/close 4098 (time 1777580250.78 total 70.07 last 57.67) total: 4500 open/close in 76.54 seconds: 58.79 ops/second - /unlink 1115 (time 1777580272.45 total 10.00 last 111.49) - /unlink 2161 (time 1777580282.46 total 20.01 last 104.49) - /unlink 3156 (time 1777580292.47 total 30.02 last 99.45) - /unlink 4150 (time 1777580302.48 total 40.03 last 99.31) total: 4500 /unlink in 44.02 seconds: 102.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 544 (time 1777580322.23 total 10.02 last 54.29) - open/close 1091 (time 1777580332.25 total 20.04 last 54.59) - open/close 1709 (time 1777580342.27 total 30.05 last 61.72) - open/close 2291 (time 1777580352.28 total 40.06 last 58.15) - open/close 2881 (time 1777580362.29 total 50.07 last 58.93) - open/close 3471 (time 1777580372.29 total 60.08 last 58.99) - open/close 4041 (time 1777580382.30 total 70.08 last 56.96) total: 4500 open/close in 77.98 seconds: 57.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1207 (time 1777580404.99 total 10.00 last 120.68) - /unlink 2354 (time 1777580414.99 total 20.00 last 114.68) - /unlink 3521 (time 1777580425.00 total 30.01 last 116.59) total: 4500 /unlink in 38.68 seconds: 116.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 649 (time 1777580449.26 total 10.03 last 64.68) - open/close 1285 (time 1777580459.26 total 20.04 last 63.56) - open/close 1902 (time 1777580469.27 total 30.05 last 61.64) - open/close 2562 (time 1777580479.28 total 40.05 last 65.98) - open/close 3215 (time 1777580489.29 total 50.06 last 65.22) - open/close 3852 (time 1777580499.29 total 60.07 last 63.69) - open/close 4480 (time 1777580509.30 total 70.07 last 62.75) total: 4500 open/close in 70.46 seconds: 63.87 ops/second - /unlink 1046 (time 1777580524.18 total 10.02 last 104.43) - /unlink 2065 (time 1777580534.20 total 20.04 last 101.69) - /unlink 3178 (time 1777580544.20 total 30.04 last 111.25) - /unlink 4291 (time 1777580554.21 total 40.05 last 111.19) total: 4500 /unlink in 41.89 seconds: 107.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 655 (time 1777580571.17 total 10.03 last 65.34) - open/close 1285 (time 1777580581.18 total 20.04 last 62.92) - open/close 1891 (time 1777580591.19 total 30.05 last 60.54) - open/close 2458 (time 1777580601.20 total 40.06 last 56.66) - open/close 3092 (time 1777580611.21 total 50.07 last 63.33) - open/close 3777 (time 1777580621.22 total 60.07 last 68.45) - open/close 4410 (time 1777580631.23 total 70.08 last 63.25) total: 4500 open/close in 71.49 seconds: 62.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1150 (time 1777580647.80 total 10.00 last 114.99) - /unlink 2258 (time 1777580657.80 total 20.00 last 110.79) - /unlink 3469 (time 1777580667.81 total 30.01 last 121.05) total: 4500 /unlink in 38.27 seconds: 117.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 662 (time 1777580690.57 total 10.01 last 66.12) - open/close 1305 (time 1777580700.58 total 20.02 last 64.25) - open/close 1924 (time 1777580710.59 total 30.04 last 61.80) - open/close 2517 (time 1777580720.60 total 40.04 last 59.27) - open/close 3145 (time 1777580730.60 total 50.04 last 62.78) - open/close 3741 (time 1777580740.61 total 60.06 last 59.52) - open/close 4327 (time 1777580750.62 total 70.07 last 58.55) total: 4500 open/close in 73.21 seconds: 61.46 ops/second - /unlink 1204 (time 1777580768.86 total 10.01 last 120.30) - /unlink 2264 (time 1777580778.87 total 20.01 last 105.95) - /unlink 3382 (time 1777580788.87 total 30.02 last 111.72) total: 4500 /unlink in 39.95 seconds: 112.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 619 (time 1777580813.92 total 10.00 last 61.88) - open/close 1226 (time 1777580823.93 total 20.01 last 60.67) - open/close 1893 (time 1777580833.94 total 30.02 last 66.65) - open/close 2541 (time 1777580843.96 total 40.04 last 64.65) - open/close 3206 (time 1777580853.97 total 50.05 last 66.46) - open/close 3865 (time 1777580863.98 total 60.06 last 65.79) - open/close 4455 (time 1777580873.99 total 70.07 last 58.97) total: 4500 open/close in 70.70 seconds: 63.65 ops/second - /unlink 1187 (time 1777580889.64 total 10.01 last 118.53) - /unlink 2433 (time 1777580899.65 total 20.02 last 124.55) - /unlink 3706 (time 1777580909.65 total 30.02 last 127.30) total: 4500 /unlink in 36.55 seconds: 123.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 648 (time 1777580930.76 total 10.00 last 64.79) - open/close 1243 (time 1777580940.77 total 20.01 last 59.43) - open/close 1871 (time 1777580950.78 total 30.02 last 62.78) - open/close 2468 (time 1777580960.79 total 40.03 last 59.62) - open/close 3079 (time 1777580970.79 total 50.03 last 61.09) - open/close 3783 (time 1777580980.80 total 60.04 last 70.37) - open/close 4298 (time 1777580990.81 total 70.05 last 51.43) total: 4500 open/close in 74.16 seconds: 60.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1057 (time 1777581010.40 total 10.01 last 105.56) - /unlink 2138 (time 1777581020.41 total 20.02 last 107.98) - /unlink 3311 (time 1777581030.42 total 30.04 last 117.15) - /unlink 4479 (time 1777581040.42 total 40.04 last 116.78) total: 4500 /unlink in 40.30 seconds: 111.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 638 (time 1777581054.79 total 10.00 last 63.79) - open/close 1288 (time 1777581064.79 total 20.00 last 64.98) - open/close 1920 (time 1777581074.81 total 30.02 last 63.11) - open/close 2515 (time 1777581084.83 total 40.04 last 59.36) - open/close 3052 (time 1777581094.85 total 50.06 last 53.61) - open/close 3701 (time 1777581104.86 total 60.07 last 64.85) - open/close 4351 (time 1777581114.86 total 70.07 last 64.98) total: 4500 open/close in 72.14 seconds: 62.38 ops/second - /unlink 1209 (time 1777581131.51 total 10.00 last 120.85) - /unlink 2454 (time 1777581141.52 total 20.02 last 124.36) - /unlink 3684 (time 1777581151.52 total 30.02 last 123.00) total: 4500 /unlink in 36.92 seconds: 121.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 648 (time 1777581174.05 total 10.02 last 64.69) - open/close 1247 (time 1777581184.05 total 20.02 last 59.88) - open/close 1829 (time 1777581194.07 total 30.04 last 58.10) - open/close 2370 (time 1777581204.07 total 40.04 last 54.07) - open/close 2898 (time 1777581214.10 total 50.07 last 52.64) - open/close 3516 (time 1777581224.12 total 60.09 last 61.68) - open/close 4194 (time 1777581234.15 total 70.12 last 67.62) total: 4500 open/close in 74.36 seconds: 60.52 ops/second - /unlink 1307 (time 1777581252.89 total 10.01 last 130.63) - /unlink 2569 (time 1777581262.89 total 20.01 last 126.19) - /unlink 3743 (time 1777581272.89 total 30.01 last 117.40) total: 4500 /unlink in 37.04 seconds: 121.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 556 (time 1777581295.57 total 10.00 last 55.57) - open/close 1121 (time 1777581305.58 total 20.01 last 56.47) - open/close 1774 (time 1777581315.58 total 30.02 last 65.26) - open/close 2366 (time 1777581325.60 total 40.03 last 59.11) - open/close 2967 (time 1777581335.61 total 50.04 last 60.06) - open/close 3650 (time 1777581345.61 total 60.05 last 68.26) - open/close 4261 (time 1777581355.62 total 70.06 last 61.03) total: 4500 open/close in 74.15 seconds: 60.68 ops/second - /unlink 1161 (time 1777581374.32 total 10.01 last 116.03) - /unlink 2411 (time 1777581384.32 total 20.01 last 124.99) - /unlink 3594 (time 1777581394.32 total 30.01 last 118.29) total: 4500 /unlink in 37.53 seconds: 119.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 638 (time 1777581417.24 total 10.00 last 63.78) - open/close 1315 (time 1777581427.27 total 20.03 last 67.52) - open/close 1930 (time 1777581437.28 total 30.04 last 61.44) - open/close 2565 (time 1777581447.28 total 40.04 last 63.49) - open/close 3168 (time 1777581457.30 total 50.06 last 60.18) - open/close 3708 (time 1777581467.31 total 60.07 last 53.94) - open/close 4300 (time 1777581477.32 total 70.09 last 59.12) total: 4500 open/close in 73.59 seconds: 61.15 ops/second - /unlink 1191 (time 1777581496.35 total 10.00 last 119.08) - /unlink 2321 (time 1777581506.36 total 20.00 last 112.98) - /unlink 3510 (time 1777581516.36 total 30.01 last 118.87) total: 4500 /unlink in 38.53 seconds: 116.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 545 (time 1777581541.07 total 10.02 last 54.39) - open/close 1183 (time 1777581551.08 total 20.03 last 63.74) - open/close 1836 (time 1777581561.08 total 30.04 last 65.26) - open/close 2489 (time 1777581571.09 total 40.04 last 65.28) - open/close 3144 (time 1777581581.09 total 50.04 last 65.49) - open/close 3751 (time 1777581591.10 total 60.05 last 60.63) - open/close 4259 (time 1777581601.10 total 70.05 last 50.80) total: 4500 open/close in 74.70 seconds: 60.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1177 (time 1777581622.35 total 10.00 last 117.68) - /unlink 2405 (time 1777581632.38 total 20.04 last 122.35) - /unlink 3533 (time 1777581642.39 total 30.05 last 112.72) total: 4500 /unlink in 39.16 seconds: 114.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 645 (time 1777581666.50 total 10.01 last 64.46) - open/close 1342 (time 1777581676.51 total 20.01 last 69.66) - open/close 2007 (time 1777581686.52 total 30.03 last 66.41) - open/close 2639 (time 1777581696.52 total 40.03 last 63.18) - open/close 3297 (time 1777581706.54 total 50.04 last 65.72) - open/close 3981 (time 1777581716.54 total 60.04 last 68.40) total: 4500 open/close in 67.65 seconds: 66.52 ops/second - /unlink 1152 (time 1777581739.13 total 10.01 last 115.04) - /unlink 2224 (time 1777581749.13 total 20.02 last 107.17) - /unlink 2973 (time 1777581759.13 total 30.02 last 74.87) - /unlink 4132 (time 1777581769.15 total 40.03 last 115.76) total: 4500 /unlink in 43.78 seconds: 102.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 610 (time 1777581789.18 total 10.01 last 60.95) - open/close 1215 (time 1777581799.18 total 20.01 last 60.50) - open/close 1878 (time 1777581809.18 total 30.01 last 66.27) - open/close 2538 (time 1777581819.19 total 40.02 last 65.98) - open/close 3217 (time 1777581829.20 total 50.03 last 67.83) - open/close 3824 (time 1777581839.20 total 60.03 last 60.68) - open/close 4496 (time 1777581849.21 total 70.04 last 67.14) total: 4500 open/close in 70.10 seconds: 64.19 ops/second - /unlink 1262 (time 1777581863.95 total 10.00 last 126.18) - /unlink 2397 (time 1777581873.95 total 20.01 last 113.45) - /unlink 3595 (time 1777581883.96 total 30.01 last 119.77) total: 4500 /unlink in 37.66 seconds: 119.50 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11189: 94175 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg355-server unloading modules via unload_modules_local on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing unload_modules_local oleg355-server: modules unloaded.