== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:20:56 (1788801656) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg152-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg152-server oleg152-server: oleg152-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: kmemleak disabled oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg152-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg152-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 867 (time 1788801864.67 total 10.02 last 86.54) - open/close 1750 (time 1788801874.68 total 20.03 last 88.22) - open/close 2613 (time 1788801884.69 total 30.03 last 86.26) - open/close 3539 (time 1788801894.69 total 40.04 last 92.58) - open/close 4452 (time 1788801904.69 total 50.04 last 91.29) total: 4500 open/close in 50.66 seconds: 88.83 ops/second - /unlink 1790 (time 1788801919.20 total 10.00 last 178.95) - /unlink 3341 (time 1788801929.20 total 20.00 last 155.07) total: 4500 /unlink in 28.40 seconds: 158.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 753 (time 1788801952.97 total 10.02 last 75.15) - open/close 1332 (time 1788801962.97 total 20.02 last 57.88) - open/close 2102 (time 1788801972.98 total 30.04 last 76.89) - open/close 2871 (time 1788801983.00 total 40.05 last 76.80) - open/close 3507 (time 1788801993.03 total 50.09 last 63.37) - open/close 4297 (time 1788802003.04 total 60.09 last 78.98) total: 4500 open/close in 62.62 seconds: 71.86 ops/second - /unlink 1803 (time 1788802019.85 total 10.01 last 180.18) - /unlink 3528 (time 1788802029.86 total 20.01 last 172.48) total: 4500 /unlink in 25.77 seconds: 174.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 789 (time 1788802049.56 total 10.00 last 78.86) - open/close 1498 (time 1788802059.56 total 20.01 last 70.89) - open/close 2272 (time 1788802069.56 total 30.01 last 77.38) - open/close 3069 (time 1788802079.57 total 40.02 last 79.64) - open/close 3890 (time 1788802089.57 total 50.02 last 82.09) total: 4500 open/close in 57.86 seconds: 77.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1222 (time 1788802113.13 total 10.00 last 122.16) - /unlink 2928 (time 1788802123.14 total 20.01 last 170.51) total: 4500 /unlink in 28.67 seconds: 156.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 845 (time 1788802147.17 total 10.01 last 84.45) - open/close 1656 (time 1788802157.18 total 20.01 last 81.04) - open/close 2564 (time 1788802167.20 total 30.03 last 90.67) - open/close 3295 (time 1788802177.20 total 40.03 last 73.07) - open/close 4229 (time 1788802187.20 total 50.03 last 93.38) total: 4500 open/close in 52.80 seconds: 85.22 ops/second - /unlink 1536 (time 1788802204.58 total 10.01 last 153.49) - /unlink 3159 (time 1788802214.59 total 20.01 last 162.29) total: 4500 /unlink in 27.87 seconds: 161.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 860 (time 1788802237.40 total 10.02 last 85.83) - open/close 1564 (time 1788802247.40 total 20.02 last 70.37) - open/close 2357 (time 1788802257.41 total 30.03 last 79.25) - open/close 3139 (time 1788802267.42 total 40.05 last 78.08) - open/close 3994 (time 1788802277.43 total 50.05 last 85.46) total: 4500 open/close in 55.78 seconds: 80.68 ops/second - /unlink 1438 (time 1788802297.75 total 10.00 last 143.74) - /unlink 3117 (time 1788802307.76 total 20.01 last 167.80) total: 4500 /unlink in 28.40 seconds: 158.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 721 (time 1788802331.20 total 10.00 last 72.09) - open/close 1316 (time 1788802341.20 total 20.01 last 59.46) - open/close 2189 (time 1788802351.20 total 30.01 last 87.29) - open/close 3061 (time 1788802361.22 total 40.02 last 87.09) - open/close 3956 (time 1788802371.22 total 50.02 last 89.48) total: 4500 open/close in 55.97 seconds: 80.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1769 (time 1788802391.78 total 10.00 last 176.89) - /unlink 3345 (time 1788802401.78 total 20.00 last 157.55) total: 4500 /unlink in 28.14 seconds: 159.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 809 (time 1788802425.71 total 10.00 last 80.87) - open/close 1423 (time 1788802435.71 total 20.01 last 61.36) - open/close 2047 (time 1788802445.73 total 30.03 last 62.29) - open/close 2968 (time 1788802455.74 total 40.04 last 92.00) - open/close 3770 (time 1788802465.74 total 50.04 last 80.20) total: 4500 open/close in 57.36 seconds: 78.46 ops/second - /unlink 1581 (time 1788802487.86 total 10.00 last 158.04) - /unlink 3136 (time 1788802497.86 total 20.01 last 155.44) total: 4500 /unlink in 29.14 seconds: 154.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 954 (time 1788802521.75 total 10.01 last 95.33) - open/close 1545 (time 1788802531.75 total 20.01 last 59.07) - open/close 2252 (time 1788802541.75 total 30.01 last 70.70) - open/close 3002 (time 1788802551.76 total 40.02 last 74.98) - open/close 3836 (time 1788802561.76 total 50.02 last 83.39) total: 4500 open/close in 58.13 seconds: 77.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1620 (time 1788802583.94 total 10.00 last 161.95) - /unlink 2955 (time 1788802593.94 total 20.01 last 133.43) - /unlink 4218 (time 1788802603.94 total 30.01 last 126.29) total: 4500 /unlink in 33.03 seconds: 136.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 910 (time 1788802621.85 total 10.01 last 90.92) - open/close 1836 (time 1788802631.86 total 20.01 last 92.54) - open/close 2736 (time 1788802641.86 total 30.02 last 89.99) - open/close 3623 (time 1788802651.86 total 40.02 last 88.66) - open/close 4491 (time 1788802661.87 total 50.03 last 86.74) total: 4500 open/close in 50.13 seconds: 89.77 ops/second - /unlink 1584 (time 1788802676.77 total 10.00 last 158.34) - /unlink 3220 (time 1788802686.77 total 20.01 last 163.55) total: 4500 /unlink in 28.52 seconds: 157.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 886 (time 1788802711.10 total 10.00 last 88.57) - open/close 1621 (time 1788802721.10 total 20.00 last 73.49) - open/close 2477 (time 1788802731.11 total 30.01 last 85.57) - open/close 3446 (time 1788802741.12 total 40.02 last 96.82) - open/close 4432 (time 1788802751.12 total 50.02 last 98.55) total: 4500 open/close in 50.71 seconds: 88.75 ops/second - /unlink 1752 (time 1788802766.77 total 10.00 last 175.17) - /unlink 3267 (time 1788802776.77 total 20.00 last 151.46) total: 4500 /unlink in 28.59 seconds: 157.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 865 (time 1788802800.38 total 10.01 last 86.44) - open/close 1664 (time 1788802810.38 total 20.01 last 79.87) - open/close 2398 (time 1788802820.39 total 30.01 last 73.39) - open/close 2977 (time 1788802830.39 total 40.02 last 57.85) - open/close 3558 (time 1788802840.40 total 50.03 last 58.06) - open/close 4193 (time 1788802850.42 total 60.04 last 63.41) total: 4500 open/close in 63.91 seconds: 70.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1562 (time 1788802869.66 total 10.00 last 156.19) - /unlink 3191 (time 1788802879.66 total 20.00 last 162.88) total: 4500 /unlink in 28.74 seconds: 156.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 934 (time 1788802903.86 total 10.02 last 93.18) - open/close 1868 (time 1788802913.88 total 20.03 last 93.30) - open/close 2714 (time 1788802923.88 total 30.04 last 84.57) - open/close 3511 (time 1788802933.89 total 40.05 last 79.62) - open/close 4287 (time 1788802943.90 total 50.06 last 77.48) total: 4500 open/close in 52.98 seconds: 84.93 ops/second - /unlink 1126 (time 1788802962.38 total 10.00 last 112.58) - /unlink 2438 (time 1788802972.39 total 20.01 last 131.11) - /unlink 3926 (time 1788802982.39 total 30.01 last 148.72) total: 4500 /unlink in 34.36 seconds: 130.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 799 (time 1788803002.02 total 10.00 last 79.87) - open/close 1638 (time 1788803012.03 total 20.01 last 83.81) - open/close 2497 (time 1788803022.05 total 30.04 last 85.70) - open/close 3317 (time 1788803032.06 total 40.04 last 81.96) - open/close 4176 (time 1788803042.06 total 50.05 last 85.83) total: 4500 open/close in 53.58 seconds: 83.99 ops/second - /unlink 1525 (time 1788803060.80 total 10.00 last 152.49) - /unlink 2899 (time 1788803070.80 total 20.00 last 137.39) - /unlink 4467 (time 1788803080.80 total 30.00 last 156.78) total: 4500 /unlink in 30.27 seconds: 148.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 849 (time 1788803095.72 total 10.00 last 84.89) - open/close 1815 (time 1788803105.72 total 20.01 last 96.55) - open/close 2618 (time 1788803115.72 total 30.01 last 80.28) - open/close 3573 (time 1788803125.74 total 40.03 last 95.34) - open/close 4448 (time 1788803135.74 total 50.03 last 87.48) total: 4500 open/close in 50.63 seconds: 88.88 ops/second - /unlink 1817 (time 1788803151.05 total 10.00 last 181.66) - /unlink 3372 (time 1788803161.05 total 20.01 last 155.39) total: 4500 /unlink in 27.41 seconds: 164.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 869 (time 1788803183.39 total 10.00 last 86.89) - open/close 1729 (time 1788803193.39 total 20.00 last 85.98) - open/close 2593 (time 1788803203.39 total 30.01 last 86.35) - open/close 3481 (time 1788803213.41 total 40.02 last 88.69) - open/close 4390 (time 1788803223.41 total 50.02 last 90.88) total: 4500 open/close in 51.18 seconds: 87.92 ops/second - /unlink 1564 (time 1788803239.26 total 10.00 last 156.36) - /unlink 3127 (time 1788803249.27 total 20.01 last 156.17) total: 4500 /unlink in 29.77 seconds: 151.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 895 (time 1788803274.42 total 10.00 last 89.49) - open/close 1688 (time 1788803284.43 total 20.01 last 79.24) - open/close 2494 (time 1788803294.43 total 30.01 last 80.60) - open/close 3353 (time 1788803304.43 total 40.01 last 85.86) - open/close 4236 (time 1788803314.44 total 50.02 last 88.28) total: 4500 open/close in 53.19 seconds: 84.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1609 (time 1788803332.61 total 10.01 last 160.80) - /unlink 3079 (time 1788803342.61 total 20.01 last 146.92) total: 4500 /unlink in 28.38 seconds: 158.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 756 (time 1788803366.34 total 10.02 last 75.44) - open/close 1548 (time 1788803376.38 total 20.06 last 78.86) - open/close 2464 (time 1788803386.39 total 30.07 last 91.54) - open/close 3351 (time 1788803396.39 total 40.08 last 88.66) - open/close 4222 (time 1788803406.39 total 50.08 last 87.08) total: 4500 open/close in 54.18 seconds: 83.05 ops/second - /unlink 1269 (time 1788803424.37 total 10.00 last 126.84) - /unlink 2833 (time 1788803434.38 total 20.01 last 156.35) total: 4500 /unlink in 29.63 seconds: 151.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 667 (time 1788803458.88 total 10.01 last 66.66) - open/close 1621 (time 1788803468.88 total 20.01 last 95.37) - open/close 2353 (time 1788803478.89 total 30.02 last 73.14) - open/close 3261 (time 1788803488.90 total 40.03 last 90.68) - open/close 4046 (time 1788803498.91 total 50.03 last 78.47) total: 4500 open/close in 54.91 seconds: 81.95 ops/second - /unlink 1815 (time 1788803517.69 total 10.00 last 181.44) - /unlink 3539 (time 1788803527.71 total 20.02 last 172.12) total: 4500 /unlink in 24.94 seconds: 180.43 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95270 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: kmemleak disabled modules unloaded.