== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:15:23 (1788426923) Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg416-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg416-server oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: kmemleak disabled oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions oleg416-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg416-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg416-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg416-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 495 (time 1788427154.59 total 10.02 last 49.42) - open/close 1024 (time 1788427164.61 total 20.04 last 52.80) - open/close 1552 (time 1788427174.64 total 30.06 last 52.67) - open/close 1915 (time 1788427184.64 total 40.07 last 36.28) - open/close 2375 (time 1788427194.66 total 50.08 last 45.92) - open/close 2919 (time 1788427204.67 total 60.09 last 54.34) - open/close 3427 (time 1788427214.69 total 70.12 last 50.69) - open/close 4045 (time 1788427224.70 total 80.13 last 61.74) total: 4500 open/close in 88.49 seconds: 50.86 ops/second - /unlink 1097 (time 1788427249.85 total 10.00 last 109.69) - /unlink 2021 (time 1788427259.87 total 20.01 last 92.30) - /unlink 3116 (time 1788427269.89 total 30.04 last 109.20) - /unlink 4204 (time 1788427279.89 total 40.04 last 108.79) total: 4500 /unlink in 43.49 seconds: 103.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 445 (time 1788427299.30 total 10.00 last 44.50) - open/close 891 (time 1788427309.31 total 20.01 last 44.57) - open/close 1314 (time 1788427319.33 total 30.03 last 42.20) - open/close 1773 (time 1788427329.35 total 40.05 last 45.80) - open/close 2358 (time 1788427339.36 total 50.06 last 58.45) - open/close 2983 (time 1788427349.37 total 60.07 last 62.46) - open/close 3524 (time 1788427359.39 total 70.09 last 53.98) - open/close 4171 (time 1788427369.39 total 80.09 last 64.69) total: 4500 open/close in 84.80 seconds: 53.06 ops/second - /unlink 1203 (time 1788427390.31 total 10.01 last 120.23) - /unlink 2313 (time 1788427400.31 total 20.01 last 110.96) - /unlink 3355 (time 1788427410.32 total 30.02 last 104.14) - /unlink 4451 (time 1788427420.32 total 40.02 last 109.58) total: 4500 /unlink in 40.72 seconds: 110.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 575 (time 1788427437.01 total 10.02 last 57.41) - open/close 1142 (time 1788427447.02 total 20.02 last 56.67) - open/close 1779 (time 1788427457.02 total 30.02 last 63.69) - open/close 2422 (time 1788427467.04 total 40.04 last 64.18) - open/close 2873 (time 1788427477.06 total 50.07 last 44.98) - open/close 3327 (time 1788427487.07 total 60.07 last 45.38) - open/close 3880 (time 1788427497.07 total 70.08 last 55.27) - open/close 4402 (time 1788427507.09 total 80.09 last 52.13) total: 4500 open/close in 81.79 seconds: 55.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1172 (time 1788427524.17 total 10.01 last 117.14) - /unlink 2436 (time 1788427534.17 total 20.01 last 126.37) - /unlink 3652 (time 1788427544.17 total 30.01 last 121.59) total: 4500 /unlink in 37.27 seconds: 120.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 589 (time 1788427566.68 total 10.01 last 58.84) - open/close 1267 (time 1788427576.68 total 20.01 last 67.80) - open/close 1939 (time 1788427586.69 total 30.01 last 67.18) - open/close 2557 (time 1788427596.69 total 40.02 last 61.76) - open/close 3178 (time 1788427606.70 total 50.02 last 62.09) - open/close 3784 (time 1788427616.70 total 60.02 last 60.58) - open/close 4384 (time 1788427626.71 total 70.03 last 59.95) total: 4500 open/close in 71.91 seconds: 62.58 ops/second - /unlink 1194 (time 1788427643.43 total 10.00 last 119.39) - /unlink 2349 (time 1788427653.43 total 20.00 last 115.46) - /unlink 3497 (time 1788427663.44 total 30.01 last 114.71) total: 4500 /unlink in 38.86 seconds: 115.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 554 (time 1788427689.24 total 10.01 last 55.34) - open/close 1214 (time 1788427699.25 total 20.02 last 65.95) - open/close 1872 (time 1788427709.26 total 30.03 last 65.71) - open/close 2398 (time 1788427719.31 total 40.07 last 52.38) - open/close 2988 (time 1788427729.33 total 50.09 last 58.88) - open/close 3578 (time 1788427739.33 total 60.10 last 58.99) - open/close 4184 (time 1788427749.34 total 70.11 last 60.53) total: 4500 open/close in 74.70 seconds: 60.24 ops/second - /unlink 1203 (time 1788427768.63 total 10.00 last 120.25) - /unlink 2429 (time 1788427778.63 total 20.01 last 122.59) - /unlink 3628 (time 1788427788.63 total 30.01 last 119.83) total: 4500 /unlink in 36.08 seconds: 124.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 664 (time 1788427809.47 total 10.00 last 66.40) - open/close 1266 (time 1788427819.48 total 20.00 last 60.18) - open/close 1957 (time 1788427829.48 total 30.01 last 69.05) - open/close 2596 (time 1788427839.49 total 40.01 last 63.88) - open/close 3173 (time 1788427849.50 total 50.03 last 57.63) - open/close 3761 (time 1788427859.52 total 60.05 last 58.68) - open/close 4368 (time 1788427869.52 total 70.05 last 60.67) total: 4500 open/close in 71.77 seconds: 62.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1289 (time 1788427886.03 total 10.01 last 128.73) - /unlink 2485 (time 1788427896.03 total 20.01 last 119.58) - /unlink 3698 (time 1788427906.03 total 30.02 last 121.23) total: 4500 /unlink in 36.43 seconds: 123.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 643 (time 1788427926.98 total 10.01 last 64.25) - open/close 1257 (time 1788427936.99 total 20.02 last 61.36) - open/close 1973 (time 1788427947.00 total 30.03 last 71.52) - open/close 2586 (time 1788427957.01 total 40.03 last 61.26) - open/close 3228 (time 1788427967.02 total 50.05 last 64.12) - open/close 3822 (time 1788427977.02 total 60.05 last 59.40) - open/close 4431 (time 1788427987.02 total 70.05 last 60.88) total: 4500 open/close in 71.06 seconds: 63.33 ops/second - /unlink 1187 (time 1788428003.12 total 10.00 last 118.70) - /unlink 2133 (time 1788428013.12 total 20.00 last 94.59) - /unlink 3195 (time 1788428023.12 total 30.00 last 106.16) - /unlink 4111 (time 1788428033.12 total 40.01 last 91.57) total: 4500 /unlink in 44.17 seconds: 101.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 554 (time 1788428052.99 total 10.01 last 55.35) - open/close 1215 (time 1788428062.99 total 20.01 last 66.08) - open/close 1887 (time 1788428073.00 total 30.01 last 67.19) - open/close 2398 (time 1788428083.01 total 40.02 last 51.05) - open/close 2814 (time 1788428093.01 total 50.03 last 41.58) - open/close 3334 (time 1788428103.02 total 60.04 last 51.96) - open/close 3884 (time 1788428113.03 total 70.05 last 54.92) - open/close 4438 (time 1788428123.04 total 80.05 last 55.38) total: 4500 open/close in 81.03 seconds: 55.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1197 (time 1788428138.44 total 10.00 last 119.66) - /unlink 2463 (time 1788428148.44 total 20.01 last 126.53) - /unlink 3761 (time 1788428158.44 total 30.01 last 129.79) total: 4500 /unlink in 36.54 seconds: 123.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 640 (time 1788428179.18 total 10.01 last 63.94) - open/close 1186 (time 1788428189.21 total 20.04 last 54.45) - open/close 1720 (time 1788428199.21 total 30.04 last 53.38) - open/close 2260 (time 1788428209.22 total 40.05 last 53.95) - open/close 2878 (time 1788428219.23 total 50.06 last 61.72) - open/close 3528 (time 1788428229.23 total 60.06 last 64.99) - open/close 4143 (time 1788428239.25 total 70.08 last 61.39) total: 4500 open/close in 76.34 seconds: 58.95 ops/second - /unlink 1271 (time 1788428260.58 total 10.00 last 127.07) - /unlink 2602 (time 1788428270.58 total 20.01 last 133.06) - /unlink 3952 (time 1788428280.58 total 30.01 last 134.98) total: 4500 /unlink in 34.87 seconds: 129.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 516 (time 1788428302.85 total 10.00 last 51.58) - open/close 1181 (time 1788428312.86 total 20.01 last 66.44) - open/close 1753 (time 1788428322.86 total 30.02 last 57.17) - open/close 2242 (time 1788428332.87 total 40.02 last 48.88) - open/close 2715 (time 1788428342.87 total 50.03 last 47.29) - open/close 3216 (time 1788428352.87 total 60.03 last 50.09) - open/close 3785 (time 1788428362.88 total 70.03 last 56.87) - open/close 4388 (time 1788428372.89 total 80.05 last 60.22) total: 4500 open/close in 81.91 seconds: 54.94 ops/second - /unlink 1204 (time 1788428389.91 total 10.00 last 120.37) - /unlink 2382 (time 1788428399.91 total 20.01 last 117.74) - /unlink 3745 (time 1788428409.91 total 30.01 last 136.29) total: 4500 /unlink in 36.13 seconds: 124.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 656 (time 1788428431.46 total 10.01 last 65.56) - open/close 1302 (time 1788428441.47 total 20.01 last 64.55) - open/close 1999 (time 1788428451.47 total 30.01 last 69.70) - open/close 2590 (time 1788428461.50 total 40.04 last 58.95) - open/close 3168 (time 1788428471.50 total 50.04 last 57.79) - open/close 3733 (time 1788428481.50 total 60.05 last 56.47) - open/close 4413 (time 1788428491.52 total 70.06 last 67.91) total: 4500 open/close in 71.43 seconds: 63.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1305 (time 1788428507.43 total 10.00 last 130.49) - /unlink 2530 (time 1788428517.44 total 20.01 last 122.39) - /unlink 3840 (time 1788428527.44 total 30.01 last 130.94) total: 4500 /unlink in 35.67 seconds: 126.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 637 (time 1788428547.92 total 10.00 last 63.67) - open/close 1218 (time 1788428557.94 total 20.02 last 58.00) - open/close 1750 (time 1788428567.94 total 30.03 last 53.16) - open/close 2320 (time 1788428577.95 total 40.04 last 56.96) - open/close 2867 (time 1788428587.96 total 50.04 last 54.68) - open/close 3531 (time 1788428597.96 total 60.04 last 66.39) - open/close 4143 (time 1788428607.97 total 70.05 last 61.14) total: 4500 open/close in 75.35 seconds: 59.72 ops/second - /unlink 1337 (time 1788428626.74 total 10.00 last 133.64) - /unlink 2604 (time 1788428636.75 total 20.02 last 126.53) - /unlink 3893 (time 1788428646.76 total 30.02 last 128.89) total: 4500 /unlink in 34.72 seconds: 129.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 773 (time 1788428666.52 total 10.06 last 76.88) - open/close 1458 (time 1788428676.53 total 20.06 last 68.49) - open/close 2101 (time 1788428686.54 total 30.07 last 64.23) - open/close 2788 (time 1788428696.54 total 40.07 last 68.69) - open/close 3388 (time 1788428706.54 total 50.07 last 59.99) - open/close 3966 (time 1788428716.56 total 60.09 last 57.71) total: 4500 open/close in 68.52 seconds: 65.67 ops/second - /unlink 1310 (time 1788428739.35 total 10.01 last 130.93) - /unlink 2409 (time 1788428749.36 total 20.01 last 109.82) - /unlink 3567 (time 1788428759.37 total 30.02 last 115.75) total: 4500 /unlink in 38.58 seconds: 116.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 618 (time 1788428783.54 total 10.01 last 61.75) - open/close 1272 (time 1788428793.55 total 20.01 last 65.35) - open/close 1944 (time 1788428803.56 total 30.02 last 67.15) - open/close 2633 (time 1788428813.56 total 40.03 last 68.88) - open/close 3333 (time 1788428823.57 total 50.03 last 69.94) - open/close 4035 (time 1788428833.57 total 60.04 last 70.18) total: 4500 open/close in 66.79 seconds: 67.37 ops/second - /unlink 1192 (time 1788428854.78 total 10.01 last 119.09) - /unlink 2308 (time 1788428864.80 total 20.03 last 111.40) - /unlink 3576 (time 1788428874.80 total 30.03 last 126.79) total: 4500 /unlink in 37.92 seconds: 118.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 597 (time 1788428898.16 total 10.00 last 59.67) - open/close 1229 (time 1788428908.16 total 20.01 last 63.18) - open/close 1881 (time 1788428918.18 total 30.02 last 65.12) - open/close 2496 (time 1788428928.20 total 40.05 last 61.34) - open/close 3105 (time 1788428938.20 total 50.05 last 60.90) - open/close 3665 (time 1788428948.21 total 60.06 last 55.95) - open/close 4356 (time 1788428958.22 total 70.07 last 69.03) total: 4500 open/close in 72.62 seconds: 61.97 ops/second - /unlink 1139 (time 1788428975.69 total 10.01 last 113.79) - /unlink 2251 (time 1788428985.70 total 20.01 last 111.15) - /unlink 3313 (time 1788428995.70 total 30.02 last 106.17) - /unlink 4487 (time 1788429005.71 total 40.02 last 117.33) total: 4500 /unlink in 40.15 seconds: 112.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 345 (time 1788429022.09 total 10.01 last 34.45) - open/close 765 (time 1788429032.11 total 20.03 last 41.93) - open/close 1410 (time 1788429042.13 total 30.05 last 64.35) - open/close 1967 (time 1788429052.14 total 40.06 last 55.67) - open/close 2626 (time 1788429062.15 total 50.07 last 65.84) - open/close 3198 (time 1788429072.16 total 60.08 last 57.15) - open/close 3837 (time 1788429082.17 total 70.09 last 63.81) - open/close 4422 (time 1788429092.18 total 80.10 last 58.47) total: 4500 open/close in 81.76 seconds: 55.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1119 (time 1788429108.54 total 10.00 last 111.86) - /unlink 2340 (time 1788429118.56 total 20.02 last 121.93) - /unlink 3510 (time 1788429128.57 total 30.03 last 116.89) total: 4500 /unlink in 39.29 seconds: 114.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 628 (time 1788429153.07 total 10.02 last 62.68) - open/close 1267 (time 1788429163.08 total 20.03 last 63.81) - open/close 1907 (time 1788429173.10 total 30.05 last 63.89) - open/close 2560 (time 1788429183.12 total 40.06 last 65.21) - open/close 3110 (time 1788429193.13 total 50.08 last 54.90) - open/close 3719 (time 1788429203.14 total 60.09 last 60.84) - open/close 4378 (time 1788429213.15 total 70.10 last 65.87) total: 4500 open/close in 72.16 seconds: 62.36 ops/second - /unlink 1405 (time 1788429229.84 total 10.00 last 140.46) - /unlink 2723 (time 1788429239.85 total 20.01 last 131.72) - /unlink 3785 (time 1788429249.85 total 30.01 last 106.18) total: 4500 /unlink in 35.76 seconds: 125.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 604 (time 1788429270.41 total 10.00 last 60.40) - open/close 1261 (time 1788429280.41 total 20.00 last 65.70) - open/close 1920 (time 1788429290.42 total 30.00 last 65.87) - open/close 2573 (time 1788429300.42 total 40.01 last 65.27) - open/close 3212 (time 1788429310.43 total 50.01 last 63.87) - open/close 3743 (time 1788429320.44 total 60.03 last 53.04) - open/close 4364 (time 1788429330.45 total 70.03 last 62.05) total: 4500 open/close in 71.74 seconds: 62.73 ops/second - /unlink 1183 (time 1788429346.59 total 10.00 last 118.26) - /unlink 2266 (time 1788429356.59 total 20.00 last 108.29) - /unlink 3371 (time 1788429366.59 total 30.01 last 110.47) total: 4500 /unlink in 39.78 seconds: 113.12 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11349: 95391 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg416-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server unloading modules via unload_modules_local on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing unload_modules_local oleg416-server: kmemleak disabled modules unloaded.