== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:59:54 (1788458394)
Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg254-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg254-server
oleg254-server: oleg254-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg254-server'
oleg254-server: oleg254-server.virtnet: executing load_modules_local
oleg254-server: kmemleak disabled
oleg254-server: Loading modules from /home/green/git/lustre-release/lustre
oleg254-server: detected 4 online CPUs by sysfs
oleg254-server: MODOPTS_LIBCFS=
oleg254-server: Force libcfs to create 2 CPU partitions
oleg254-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg254-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg254-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg254-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg254-client.virtnet:  -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 625 (time 1788458615.75 total 10.01 last 62.45)
 - open/close 1265 (time 1788458625.76 total 20.01 last 63.98)
 - open/close 1940 (time 1788458635.76 total 30.02 last 67.46)
 - open/close 2640 (time 1788458645.78 total 40.03 last 69.89)
 - open/close 3385 (time 1788458655.79 total 50.05 last 74.37)
 - open/close 4035 (time 1788458665.81 total 60.07 last 64.89)
total: 4500 open/close in 69.50 seconds: 64.75 ops/second
 - /unlink 1475 (time 1788458690.18 total 10.01 last 147.37)
 - /unlink 2465 (time 1788458700.18 total 20.01 last 98.96)
 - /unlink 3790 (time 1788458710.18 total 30.02 last 132.47)
total: 4500 /unlink in 34.61 seconds: 130.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 824 (time 1788458729.97 total 10.01 last 82.32)
 - open/close 1488 (time 1788458739.97 total 20.01 last 66.40)
 - open/close 2243 (time 1788458749.98 total 30.01 last 75.46)
 - open/close 2928 (time 1788458759.99 total 40.03 last 68.41)
 - open/close 3627 (time 1788458769.99 total 50.03 last 69.88)
 - open/close 4396 (time 1788458779.99 total 60.03 last 76.88)
total: 4500 open/close in 61.28 seconds: 73.44 ops/second
 - /unlink 1376 (time 1788458795.86 total 10.00 last 137.60)
 - /unlink 2573 (time 1788458805.86 total 20.00 last 119.67)
 - /unlink 4054 (time 1788458815.87 total 30.01 last 148.00)
total: 4500 /unlink in 33.66 seconds: 133.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 784 (time 1788458834.06 total 10.01 last 78.33)
 - open/close 1522 (time 1788458844.07 total 20.02 last 73.69)
 - open/close 2249 (time 1788458854.08 total 30.04 last 72.61)
 - open/close 3046 (time 1788458864.09 total 40.04 last 79.68)
 - open/close 3771 (time 1788458874.10 total 50.05 last 72.40)
total: 4500 open/close in 59.13 seconds: 76.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1439 (time 1788458897.99 total 10.02 last 143.66)
 - /unlink 2816 (time 1788458908.01 total 20.03 last 137.49)
 - /unlink 4106 (time 1788458918.02 total 30.05 last 128.83)
total: 4500 /unlink in 33.30 seconds: 135.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 713 (time 1788458937.04 total 10.01 last 71.20)
 - open/close 1317 (time 1788458947.06 total 20.03 last 60.28)
 - open/close 1991 (time 1788458957.07 total 30.04 last 67.34)
 - open/close 2609 (time 1788458967.08 total 40.05 last 61.76)
 - open/close 3345 (time 1788458977.09 total 50.06 last 73.50)
 - open/close 4116 (time 1788458987.10 total 60.07 last 77.03)
total: 4500 open/close in 64.65 seconds: 69.60 ops/second
 - /unlink 1322 (time 1788459005.82 total 10.00 last 132.17)
 - /unlink 2633 (time 1788459015.82 total 20.00 last 131.10)
 - /unlink 3939 (time 1788459025.82 total 30.01 last 130.52)
total: 4500 /unlink in 34.07 seconds: 132.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 808 (time 1788459044.93 total 10.01 last 80.73)
 - open/close 1504 (time 1788459054.95 total 20.02 last 69.51)
 - open/close 2293 (time 1788459064.95 total 30.02 last 78.89)
 - open/close 3042 (time 1788459074.97 total 40.04 last 74.76)
 - open/close 3751 (time 1788459084.97 total 50.05 last 70.88)
total: 4500 open/close in 58.61 seconds: 76.78 ops/second
 - /unlink 1408 (time 1788459108.40 total 10.00 last 140.74)
 - /unlink 2802 (time 1788459118.40 total 20.01 last 139.27)
 - /unlink 4294 (time 1788459128.41 total 30.02 last 149.18)
total: 4500 /unlink in 31.47 seconds: 143.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 703 (time 1788459145.02 total 10.00 last 70.29)
 - open/close 1420 (time 1788459155.04 total 20.03 last 71.52)
 - open/close 2159 (time 1788459165.05 total 30.03 last 73.87)
 - open/close 2872 (time 1788459175.05 total 40.03 last 71.27)
 - open/close 3568 (time 1788459185.05 total 50.03 last 69.60)
 - open/close 4402 (time 1788459195.06 total 60.04 last 83.36)
total: 4500 open/close in 61.14 seconds: 73.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1409 (time 1788459210.55 total 10.01 last 140.78)
 - /unlink 2576 (time 1788459220.55 total 20.01 last 116.68)
 - /unlink 3940 (time 1788459230.56 total 30.02 last 136.30)
total: 4500 /unlink in 34.97 seconds: 128.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 767 (time 1788459249.64 total 10.01 last 76.64)
 - open/close 1507 (time 1788459259.64 total 20.01 last 73.99)
 - open/close 2212 (time 1788459269.65 total 30.01 last 70.48)
 - open/close 2836 (time 1788459279.65 total 40.02 last 62.37)
 - open/close 3510 (time 1788459289.65 total 50.02 last 67.39)
 - open/close 4232 (time 1788459299.66 total 60.03 last 72.14)
total: 4500 open/close in 64.01 seconds: 70.30 ops/second
 - /unlink 1406 (time 1788459318.18 total 10.00 last 140.57)
 - /unlink 2783 (time 1788459328.18 total 20.01 last 137.64)
 - /unlink 4036 (time 1788459338.18 total 30.01 last 125.29)
total: 4500 /unlink in 33.10 seconds: 135.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 702 (time 1788459356.60 total 10.03 last 70.01)
 - open/close 1307 (time 1788459366.61 total 20.04 last 60.42)
 - open/close 2131 (time 1788459376.61 total 30.04 last 82.39)
 - open/close 2958 (time 1788459386.61 total 40.05 last 82.67)
 - open/close 3696 (time 1788459396.62 total 50.05 last 73.74)
 - open/close 4412 (time 1788459406.62 total 60.06 last 71.58)
total: 4500 open/close in 61.28 seconds: 73.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1034 (time 1788459423.38 total 10.01 last 103.31)
 - /unlink 2258 (time 1788459433.40 total 20.02 last 122.24)
 - /unlink 3578 (time 1788459443.41 total 30.03 last 131.85)
total: 4500 /unlink in 37.72 seconds: 119.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 580 (time 1788459466.70 total 10.02 last 57.88)
 - open/close 1220 (time 1788459476.71 total 20.02 last 63.99)
 - open/close 1798 (time 1788459486.72 total 30.04 last 57.73)
 - open/close 2441 (time 1788459496.73 total 40.04 last 64.25)
 - open/close 3158 (time 1788459506.73 total 50.05 last 71.66)
 - open/close 3891 (time 1788459516.74 total 60.06 last 73.24)
total: 4500 open/close in 68.74 seconds: 65.46 ops/second
 - /unlink 1298 (time 1788459539.68 total 10.00 last 129.80)
 - /unlink 2620 (time 1788459549.68 total 20.00 last 132.18)
 - /unlink 4015 (time 1788459559.69 total 30.00 last 139.47)
total: 4500 /unlink in 34.08 seconds: 132.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 471 (time 1788459579.92 total 10.00 last 47.08)
 - open/close 1070 (time 1788459589.94 total 20.02 last 59.83)
 - open/close 1680 (time 1788459599.95 total 30.03 last 60.92)
 - open/close 2241 (time 1788459609.98 total 40.06 last 55.93)
 - open/close 2937 (time 1788459620.00 total 50.08 last 69.48)
 - open/close 3728 (time 1788459630.01 total 60.09 last 79.01)
 - open/close 4370 (time 1788459640.02 total 70.10 last 64.10)
total: 4500 open/close in 72.40 seconds: 62.16 ops/second
 - /unlink 1444 (time 1788459657.43 total 10.01 last 144.30)
 - /unlink 2683 (time 1788459667.44 total 20.02 last 123.75)
 - /unlink 3971 (time 1788459677.46 total 30.04 last 128.53)
total: 4500 /unlink in 34.67 seconds: 129.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 546 (time 1788459697.56 total 10.02 last 54.47)
 - open/close 1272 (time 1788459707.60 total 20.06 last 72.33)
 - open/close 1885 (time 1788459717.61 total 30.07 last 61.24)
 - open/close 2614 (time 1788459727.62 total 40.08 last 72.81)
 - open/close 3299 (time 1788459737.62 total 50.08 last 68.49)
 - open/close 4070 (time 1788459747.64 total 60.09 last 77.01)
total: 4500 open/close in 65.85 seconds: 68.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1299 (time 1788459768.82 total 10.00 last 129.86)
 - /unlink 2698 (time 1788459778.83 total 20.01 last 139.83)
 - /unlink 4066 (time 1788459788.85 total 30.02 last 136.57)
total: 4500 /unlink in 33.19 seconds: 135.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 601 (time 1788459807.93 total 10.00 last 60.09)
 - open/close 1337 (time 1788459817.94 total 20.00 last 73.59)
 - open/close 1960 (time 1788459827.94 total 30.00 last 62.29)
 - open/close 2688 (time 1788459837.94 total 40.01 last 72.80)
 - open/close 3380 (time 1788459847.94 total 50.01 last 69.19)
 - open/close 4035 (time 1788459857.96 total 60.02 last 65.38)
total: 4500 open/close in 66.65 seconds: 67.51 ops/second
 - /unlink 1231 (time 1788459879.08 total 10.00 last 123.05)
 - /unlink 2324 (time 1788459889.09 total 20.01 last 109.27)
 - /unlink 3701 (time 1788459899.09 total 30.01 last 137.64)
total: 4500 /unlink in 37.43 seconds: 120.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 647 (time 1788459922.70 total 10.01 last 64.61)
 - open/close 1245 (time 1788459932.71 total 20.02 last 59.75)
 - open/close 1847 (time 1788459942.72 total 30.03 last 60.18)
 - open/close 2652 (time 1788459952.74 total 40.05 last 80.29)
 - open/close 3459 (time 1788459962.75 total 50.06 last 80.61)
 - open/close 4282 (time 1788459972.76 total 60.07 last 82.27)
total: 4500 open/close in 62.68 seconds: 71.80 ops/second
 - /unlink 1451 (time 1788459989.61 total 10.01 last 144.89)
 - /unlink 2693 (time 1788459999.61 total 20.02 last 124.20)
 - /unlink 4107 (time 1788460009.62 total 30.02 last 141.35)
total: 4500 /unlink in 33.65 seconds: 133.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 688 (time 1788460029.11 total 10.01 last 68.72)
 - open/close 1290 (time 1788460039.13 total 20.03 last 60.08)
 - open/close 1984 (time 1788460049.13 total 30.03 last 69.40)
 - open/close 2741 (time 1788460059.14 total 40.05 last 75.59)
 - open/close 3445 (time 1788460069.16 total 50.06 last 70.27)
 - open/close 4065 (time 1788460079.16 total 60.07 last 61.99)
total: 4500 open/close in 67.43 seconds: 66.73 ops/second
 - /unlink 1333 (time 1788460102.12 total 10.00 last 133.28)
 - /unlink 2753 (time 1788460112.12 total 20.00 last 141.98)
 - /unlink 4108 (time 1788460122.14 total 30.02 last 135.31)
total: 4500 /unlink in 32.99 seconds: 136.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 739 (time 1788460140.33 total 10.00 last 73.87)
 - open/close 1360 (time 1788460150.34 total 20.01 last 62.05)
 - open/close 1965 (time 1788460160.36 total 30.03 last 60.37)
 - open/close 2576 (time 1788460170.38 total 40.06 last 60.93)
 - open/close 3276 (time 1788460180.39 total 50.07 last 69.92)
 - open/close 3998 (time 1788460190.40 total 60.08 last 72.16)
total: 4500 open/close in 67.29 seconds: 66.87 ops/second
 - /unlink 1490 (time 1788460212.97 total 10.01 last 148.80)
 - /unlink 2896 (time 1788460222.97 total 20.02 last 140.54)
 - /unlink 4078 (time 1788460232.98 total 30.02 last 118.15)
total: 4500 /unlink in 33.50 seconds: 134.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 634 (time 1788460252.08 total 10.01 last 63.32)
 - open/close 1281 (time 1788460262.09 total 20.02 last 64.63)
 - open/close 1796 (time 1788460272.09 total 30.02 last 51.49)
 - open/close 2400 (time 1788460282.10 total 40.03 last 60.34)
 - open/close 2927 (time 1788460292.13 total 50.06 last 52.56)
 - open/close 3514 (time 1788460302.13 total 60.07 last 58.66)
 - open/close 4232 (time 1788460312.14 total 70.07 last 71.77)
total: 4500 open/close in 73.61 seconds: 61.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1303 (time 1788460330.81 total 10.01 last 130.14)
 - /unlink 2656 (time 1788460340.81 total 20.02 last 135.25)
 - /unlink 3966 (time 1788460350.82 total 30.02 last 130.97)
total: 4500 /unlink in 34.33 seconds: 131.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 583 (time 1788460370.41 total 10.01 last 58.25)
 - open/close 1246 (time 1788460380.41 total 20.02 last 66.25)
 - open/close 1939 (time 1788460390.43 total 30.03 last 69.23)
 - open/close 2597 (time 1788460400.43 total 40.03 last 65.78)
 - open/close 3361 (time 1788460410.43 total 50.03 last 76.37)
 - open/close 4061 (time 1788460420.46 total 60.06 last 69.81)
total: 4500 open/close in 65.75 seconds: 68.44 ops/second
 - /unlink 1455 (time 1788460441.58 total 10.00 last 145.45)
 - /unlink 2699 (time 1788460451.58 total 20.00 last 124.40)
 - /unlink 3989 (time 1788460461.59 total 30.00 last 128.99)
total: 4500 /unlink in 34.09 seconds: 132.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 673 (time 1788460482.20 total 10.00 last 67.29)
 - open/close 1301 (time 1788460492.21 total 20.01 last 62.73)
 - open/close 1955 (time 1788460502.22 total 30.02 last 65.37)
 - open/close 2572 (time 1788460512.22 total 40.02 last 61.67)
 - open/close 3266 (time 1788460522.22 total 50.02 last 69.39)
 - open/close 4055 (time 1788460532.22 total 60.02 last 78.89)
total: 4500 open/close in 65.42 seconds: 68.78 ops/second
 - /unlink 1242 (time 1788460553.30 total 10.00 last 124.15)
 - /unlink 2480 (time 1788460563.31 total 20.01 last 123.76)
 - /unlink 3565 (time 1788460573.31 total 30.01 last 108.44)
total: 4500 /unlink in 36.95 seconds: 121.79 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95347 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg254-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg254-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server
stop mds service on oleg254-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server
stop mds service on oleg254-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server
unloading modules via unload_modules_local on: 'oleg254-server'
oleg254-server: oleg254-server.virtnet: executing unload_modules_local
oleg254-server: kmemleak disabled
modules unloaded.