== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:22:51 (1787138571) Stopping clients: oleg460-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg460-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg460-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg460-server oleg460-server: oleg460-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg460-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg460-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 681 (time 1787138779.11 total 10.01 last 68.05) - open/close 1297 (time 1787138789.13 total 20.03 last 61.48) - open/close 1934 (time 1787138799.13 total 30.03 last 63.67) - open/close 2627 (time 1787138809.13 total 40.03 last 69.29) - open/close 3272 (time 1787138819.14 total 50.04 last 64.43) - open/close 3882 (time 1787138829.14 total 60.04 last 60.99) total: 4500 open/close in 69.49 seconds: 64.76 ops/second - /unlink 1215 (time 1787138852.26 total 10.00 last 121.45) - /unlink 2395 (time 1787138862.27 total 20.02 last 117.85) - /unlink 3540 (time 1787138872.27 total 30.02 last 114.48) total: 4500 /unlink in 38.01 seconds: 118.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 650 (time 1787138895.53 total 10.01 last 64.90) - open/close 1278 (time 1787138905.53 total 20.02 last 62.76) - open/close 1965 (time 1787138915.55 total 30.03 last 68.62) - open/close 2678 (time 1787138925.56 total 40.05 last 71.21) - open/close 3419 (time 1787138935.57 total 50.06 last 74.02) - open/close 4169 (time 1787138945.58 total 60.07 last 74.93) total: 4500 open/close in 64.61 seconds: 69.65 ops/second - /unlink 1187 (time 1787138964.71 total 10.01 last 118.62) - /unlink 2340 (time 1787138974.71 total 20.01 last 115.30) - /unlink 3635 (time 1787138984.71 total 30.01 last 129.46) total: 4500 /unlink in 37.65 seconds: 119.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 686 (time 1787139007.33 total 10.01 last 68.54) - open/close 1349 (time 1787139017.34 total 20.02 last 66.23) - open/close 1970 (time 1787139027.34 total 30.02 last 62.09) - open/close 2680 (time 1787139037.37 total 40.06 last 70.77) - open/close 3344 (time 1787139047.37 total 50.06 last 66.39) - open/close 4060 (time 1787139057.38 total 60.06 last 71.55) total: 4500 open/close in 66.25 seconds: 67.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1328 (time 1787139077.10 total 10.01 last 132.73) - /unlink 2516 (time 1787139087.10 total 20.01 last 118.80) - /unlink 3526 (time 1787139097.11 total 30.02 last 100.87) total: 4500 /unlink in 38.45 seconds: 117.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 580 (time 1787139120.70 total 10.01 last 57.94) - open/close 1179 (time 1787139130.71 total 20.02 last 59.85) - open/close 1828 (time 1787139140.72 total 30.03 last 64.84) - open/close 2531 (time 1787139150.73 total 40.04 last 70.24) - open/close 3208 (time 1787139160.74 total 50.04 last 67.64) - open/close 3849 (time 1787139170.75 total 60.05 last 64.04) - open/close 4483 (time 1787139180.75 total 70.05 last 63.39) total: 4500 open/close in 70.29 seconds: 64.02 ops/second - /unlink 1061 (time 1787139195.66 total 10.00 last 106.07) - /unlink 2171 (time 1787139205.67 total 20.01 last 110.97) - /unlink 3239 (time 1787139215.68 total 30.01 last 106.70) - /unlink 4210 (time 1787139225.68 total 40.02 last 97.09) total: 4500 /unlink in 42.76 seconds: 105.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 620 (time 1787139244.12 total 10.01 last 61.95) - open/close 1248 (time 1787139254.13 total 20.02 last 62.70) - open/close 1852 (time 1787139264.13 total 30.02 last 60.40) - open/close 2516 (time 1787139274.14 total 40.03 last 66.34) - open/close 3147 (time 1787139284.15 total 50.04 last 63.07) - open/close 3866 (time 1787139294.15 total 60.04 last 71.87) - open/close 4476 (time 1787139304.16 total 70.05 last 60.93) total: 4500 open/close in 70.31 seconds: 64.01 ops/second - /unlink 1203 (time 1787139319.07 total 10.00 last 120.25) - /unlink 2375 (time 1787139329.08 total 20.01 last 117.13) - /unlink 3661 (time 1787139339.08 total 30.01 last 128.56) total: 4500 /unlink in 36.53 seconds: 123.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 675 (time 1787139359.88 total 10.00 last 67.47) - open/close 1356 (time 1787139369.88 total 20.01 last 68.06) - open/close 2021 (time 1787139379.89 total 30.02 last 66.42) - open/close 2591 (time 1787139389.91 total 40.03 last 56.93) - open/close 3126 (time 1787139399.92 total 50.04 last 53.46) - open/close 3618 (time 1787139409.93 total 60.05 last 49.14) - open/close 4192 (time 1787139419.93 total 70.06 last 57.39) total: 4500 open/close in 75.27 seconds: 59.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1211 (time 1787139440.14 total 10.01 last 120.98) - /unlink 2394 (time 1787139450.14 total 20.01 last 118.28) - /unlink 3574 (time 1787139460.15 total 30.02 last 117.88) total: 4500 /unlink in 39.11 seconds: 115.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 641 (time 1787139484.47 total 10.02 last 64.00) - open/close 1247 (time 1787139494.48 total 20.02 last 60.57) - open/close 1917 (time 1787139504.48 total 30.03 last 66.97) - open/close 2584 (time 1787139514.50 total 40.04 last 66.63) - open/close 3262 (time 1787139524.51 total 50.05 last 67.73) - open/close 3910 (time 1787139534.51 total 60.05 last 64.77) total: 4500 open/close in 67.71 seconds: 66.46 ops/second - /unlink 1164 (time 1787139556.55 total 10.00 last 116.39) - /unlink 2437 (time 1787139566.56 total 20.01 last 127.20) - /unlink 3780 (time 1787139576.56 total 30.02 last 134.21) total: 4500 /unlink in 36.93 seconds: 121.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 652 (time 1787139598.76 total 10.01 last 65.14) - open/close 1339 (time 1787139608.77 total 20.02 last 68.65) - open/close 2044 (time 1787139618.78 total 30.03 last 70.43) - open/close 2741 (time 1787139628.81 total 40.06 last 69.48) - open/close 3313 (time 1787139638.82 total 50.07 last 57.13) - open/close 3870 (time 1787139648.82 total 60.07 last 55.70) - open/close 4345 (time 1787139658.83 total 70.08 last 47.45) total: 4500 open/close in 72.44 seconds: 62.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 949 (time 1787139676.66 total 10.00 last 94.88) - /unlink 1950 (time 1787139686.67 total 20.00 last 100.09) - /unlink 3055 (time 1787139696.68 total 30.02 last 110.35) - /unlink 4389 (time 1787139706.68 total 40.02 last 133.34) total: 4500 /unlink in 40.96 seconds: 109.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 661 (time 1787139723.35 total 10.00 last 66.07) - open/close 1332 (time 1787139733.36 total 20.01 last 67.08) - open/close 2008 (time 1787139743.37 total 30.02 last 67.53) - open/close 2658 (time 1787139753.37 total 40.02 last 64.97) - open/close 3296 (time 1787139763.37 total 50.02 last 63.79) - open/close 4027 (time 1787139773.38 total 60.03 last 73.05) total: 4500 open/close in 66.25 seconds: 67.92 ops/second - /unlink 1150 (time 1787139794.32 total 10.00 last 114.96) - /unlink 2168 (time 1787139804.32 total 20.00 last 101.79) - /unlink 3103 (time 1787139814.32 total 30.01 last 93.46) - /unlink 4312 (time 1787139824.34 total 40.02 last 120.71) total: 4500 /unlink in 42.84 seconds: 105.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 542 (time 1787139842.82 total 10.01 last 54.15) - open/close 1123 (time 1787139852.83 total 20.03 last 58.01) - open/close 1580 (time 1787139862.83 total 30.03 last 45.69) - open/close 2240 (time 1787139872.84 total 40.03 last 65.96) - open/close 2878 (time 1787139882.85 total 50.04 last 63.75) - open/close 3521 (time 1787139892.85 total 60.05 last 64.27) - open/close 4067 (time 1787139902.86 total 70.06 last 54.53) total: 4500 open/close in 76.51 seconds: 58.82 ops/second - /unlink 1235 (time 1787139924.72 total 10.00 last 123.49) - /unlink 2431 (time 1787139934.72 total 20.00 last 119.60) - /unlink 3739 (time 1787139944.72 total 30.00 last 130.78) total: 4500 /unlink in 36.27 seconds: 124.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 613 (time 1787139965.94 total 10.01 last 61.26) - open/close 1316 (time 1787139975.94 total 20.02 last 70.23) - open/close 1933 (time 1787139985.96 total 30.03 last 61.63) - open/close 2594 (time 1787139995.96 total 40.03 last 66.09) - open/close 3216 (time 1787140005.97 total 50.04 last 62.13) - open/close 3796 (time 1787140015.98 total 60.05 last 57.95) - open/close 4473 (time 1787140025.98 total 70.05 last 67.67) total: 4500 open/close in 70.31 seconds: 64.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1287 (time 1787140040.92 total 10.00 last 128.65) - /unlink 2447 (time 1787140050.93 total 20.01 last 115.95) - /unlink 3620 (time 1787140060.93 total 30.01 last 117.29) total: 4500 /unlink in 37.08 seconds: 121.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 598 (time 1787140082.83 total 10.01 last 59.74) - open/close 1217 (time 1787140092.86 total 20.03 last 61.76) - open/close 1906 (time 1787140102.86 total 30.04 last 68.86) - open/close 2717 (time 1787140112.87 total 40.04 last 81.07) - open/close 3312 (time 1787140122.88 total 50.06 last 59.39) - open/close 3848 (time 1787140132.88 total 60.06 last 53.60) - open/close 4424 (time 1787140142.90 total 70.08 last 57.50) total: 4500 open/close in 71.17 seconds: 63.23 ops/second - /unlink 1039 (time 1787140159.32 total 10.01 last 103.81) - /unlink 2182 (time 1787140169.32 total 20.01 last 114.29) - /unlink 3490 (time 1787140179.32 total 30.01 last 130.80) total: 4500 /unlink in 38.68 seconds: 116.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 772 (time 1787140203.49 total 10.01 last 77.13) - open/close 1508 (time 1787140213.49 total 20.01 last 73.60) - open/close 2059 (time 1787140223.50 total 30.02 last 55.04) - open/close 2751 (time 1787140233.52 total 40.04 last 69.08) - open/close 3514 (time 1787140243.52 total 50.04 last 76.28) - open/close 4251 (time 1787140253.53 total 60.04 last 73.68) total: 4500 open/close in 64.33 seconds: 69.95 ops/second - /unlink 1247 (time 1787140272.15 total 10.00 last 124.68) - /unlink 2370 (time 1787140282.16 total 20.01 last 112.26) - /unlink 3577 (time 1787140292.16 total 30.01 last 120.67) total: 4500 /unlink in 37.82 seconds: 118.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 748 (time 1787140314.86 total 10.00 last 74.77) - open/close 1443 (time 1787140324.88 total 20.02 last 69.38) - open/close 2238 (time 1787140334.88 total 30.03 last 79.47) - open/close 2922 (time 1787140344.89 total 40.03 last 68.35) - open/close 3653 (time 1787140354.89 total 50.04 last 73.05) - open/close 4366 (time 1787140364.90 total 60.04 last 71.29) total: 4500 open/close in 61.74 seconds: 72.89 ops/second - /unlink 1264 (time 1787140380.94 total 10.01 last 126.32) - /unlink 2781 (time 1787140390.94 total 20.01 last 151.65) - /unlink 3851 (time 1787140400.94 total 30.01 last 107.00) total: 4500 /unlink in 36.97 seconds: 121.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 573 (time 1787140422.72 total 10.01 last 57.25) - open/close 1132 (time 1787140432.72 total 20.01 last 55.89) - open/close 1749 (time 1787140442.73 total 30.02 last 61.63) - open/close 2277 (time 1787140452.74 total 40.03 last 52.76) - open/close 2898 (time 1787140462.75 total 50.05 last 62.00) - open/close 3624 (time 1787140472.76 total 60.05 last 72.53) - open/close 4455 (time 1787140482.77 total 70.07 last 83.01) total: 4500 open/close in 70.54 seconds: 63.80 ops/second - /unlink 1140 (time 1787140498.13 total 10.00 last 113.96) - /unlink 2277 (time 1787140508.14 total 20.01 last 113.64) - /unlink 3563 (time 1787140518.15 total 30.02 last 128.49) total: 4500 /unlink in 38.19 seconds: 117.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 578 (time 1787140542.34 total 10.02 last 57.69) - open/close 1150 (time 1787140552.34 total 20.02 last 57.19) - open/close 1736 (time 1787140562.34 total 30.03 last 58.57) - open/close 2371 (time 1787140572.36 total 40.05 last 63.38) - open/close 2960 (time 1787140582.38 total 50.06 last 58.82) - open/close 3653 (time 1787140592.38 total 60.06 last 69.27) - open/close 4433 (time 1787140602.39 total 70.07 last 77.97) total: 4500 open/close in 70.83 seconds: 63.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1169 (time 1787140617.96 total 10.01 last 116.83) - /unlink 2478 (time 1787140627.96 total 20.01 last 130.87) - /unlink 3596 (time 1787140637.97 total 30.01 last 111.73) total: 4500 /unlink in 38.05 seconds: 118.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 704 (time 1787140661.09 total 10.01 last 70.33) - open/close 1303 (time 1787140671.09 total 20.01 last 59.89) - open/close 1987 (time 1787140681.09 total 30.02 last 68.36) - open/close 2554 (time 1787140691.10 total 40.03 last 56.65) - open/close 3141 (time 1787140701.10 total 50.03 last 58.68) - open/close 3764 (time 1787140711.13 total 60.06 last 62.12) - open/close 4469 (time 1787140721.15 total 70.07 last 70.41) total: 4500 open/close in 70.41 seconds: 63.91 ops/second - /unlink 1262 (time 1787140736.18 total 10.01 last 126.13) - /unlink 2264 (time 1787140746.19 total 20.01 last 100.14) - /unlink 3333 (time 1787140756.20 total 30.02 last 106.83) total: 4500 /unlink in 39.96 seconds: 112.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 578 (time 1787140781.46 total 10.02 last 57.71) - open/close 1200 (time 1787140791.47 total 20.03 last 62.13) - open/close 1830 (time 1787140801.47 total 30.03 last 62.97) - open/close 2524 (time 1787140811.49 total 40.05 last 69.27) - open/close 3258 (time 1787140821.49 total 50.05 last 73.39) - open/close 3915 (time 1787140831.51 total 60.07 last 65.62) total: 4500 open/close in 70.00 seconds: 64.28 ops/second - /unlink 1090 (time 1787140856.10 total 10.00 last 108.97) - /unlink 2228 (time 1787140866.10 total 20.01 last 113.77) - /unlink 3319 (time 1787140876.10 total 30.01 last 109.04) - /unlink 4315 (time 1787140886.11 total 40.02 last 99.55) total: 4500 /unlink in 41.93 seconds: 107.31 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11318: 95363 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: modules unloaded.