== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:17:22 (1790104642)
Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg245-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg245-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg245-server
oleg245-server: oleg245-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg245-server'
oleg245-server: oleg245-server.virtnet: executing load_modules_local
oleg245-server: kmemleak disabled
oleg245-server: Loading modules from /home/green/git/lustre-release/lustre
oleg245-server: detected 4 online CPUs by sysfs
oleg245-server: MODOPTS_LIBCFS=
oleg245-server: Force libcfs to create 2 CPU partitions
oleg245-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg245-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all
pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg245-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all
pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg245-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all
pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg245-client.virtnet:  -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 784 (time 1790104854.90 total 10.01 last 78.31)
 - open/close 1431 (time 1790104864.91 total 20.02 last 64.63)
 - open/close 2148 (time 1790104874.92 total 30.03 last 71.63)
 - open/close 2868 (time 1790104884.93 total 40.05 last 71.88)
 - open/close 3703 (time 1790104894.95 total 50.06 last 83.40)
 - open/close 4400 (time 1790104904.95 total 60.07 last 69.65)
total: 4500 open/close in 61.34 seconds: 73.37 ops/second
 - /unlink 814 (time 1790104922.07 total 10.19 last 79.92)
 - /unlink 1491 (time 1790104932.11 total 20.22 last 67.47)
 - /unlink 2635 (time 1790104942.12 total 30.23 last 114.25)
 - /unlink 3965 (time 1790104952.12 total 40.24 last 132.95)
total: 4500 /unlink in 43.65 seconds: 103.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 704 (time 1790104970.34 total 10.00 last 70.39)
 - open/close 1457 (time 1790104980.38 total 20.04 last 75.05)
 - open/close 2221 (time 1790104990.39 total 30.05 last 76.32)
 - open/close 2939 (time 1790105000.39 total 40.05 last 71.80)
 - open/close 3730 (time 1790105010.39 total 50.05 last 79.07)
 - open/close 4405 (time 1790105020.40 total 60.06 last 67.45)
total: 4500 open/close in 61.45 seconds: 73.23 ops/second
 - /unlink 1310 (time 1790105037.07 total 10.01 last 130.88)
 - /unlink 2684 (time 1790105047.07 total 20.01 last 137.38)
 - /unlink 3918 (time 1790105057.08 total 30.02 last 123.29)
total: 4500 /unlink in 35.05 seconds: 128.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 693 (time 1790105077.21 total 10.01 last 69.26)
 - open/close 1331 (time 1790105087.21 total 20.01 last 63.75)
 - open/close 2025 (time 1790105097.23 total 30.03 last 69.28)
 - open/close 2756 (time 1790105107.24 total 40.04 last 73.02)
 - open/close 3483 (time 1790105117.24 total 50.04 last 72.68)
 - open/close 4172 (time 1790105127.25 total 60.05 last 68.85)
total: 4500 open/close in 65.79 seconds: 68.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 939 (time 1790105150.09 total 10.00 last 93.90)
 - /unlink 2033 (time 1790105160.10 total 20.00 last 109.37)
 - /unlink 3178 (time 1790105170.10 total 30.01 last 114.46)
 - /unlink 4451 (time 1790105180.10 total 40.01 last 127.28)
total: 4500 /unlink in 40.61 seconds: 110.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 730 (time 1790105195.53 total 10.02 last 72.88)
 - open/close 1500 (time 1790105205.54 total 20.03 last 76.91)
 - open/close 2219 (time 1790105215.54 total 30.03 last 71.90)
 - open/close 3030 (time 1790105225.55 total 40.04 last 81.04)
 - open/close 3819 (time 1790105235.56 total 50.05 last 78.81)
total: 4500 open/close in 58.89 seconds: 76.41 ops/second
 - /unlink 1106 (time 1790105260.62 total 10.01 last 110.46)
 - /unlink 2295 (time 1790105270.63 total 20.02 last 118.84)
 - /unlink 3717 (time 1790105280.63 total 30.02 last 142.16)
total: 4500 /unlink in 36.50 seconds: 123.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 770 (time 1790105303.40 total 10.00 last 77.00)
 - open/close 1586 (time 1790105313.41 total 20.01 last 81.53)
 - open/close 2308 (time 1790105323.42 total 30.03 last 72.09)
 - open/close 3037 (time 1790105333.43 total 40.03 last 72.83)
 - open/close 3821 (time 1790105343.43 total 50.04 last 78.39)
total: 4500 open/close in 59.40 seconds: 75.75 ops/second
 - /unlink 1343 (time 1790105367.74 total 10.01 last 134.22)
 - /unlink 2637 (time 1790105377.74 total 20.01 last 129.37)
 - /unlink 3853 (time 1790105387.75 total 30.01 last 121.52)
total: 4500 /unlink in 34.71 seconds: 129.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 619 (time 1790105408.98 total 10.04 last 61.68)
 - open/close 1171 (time 1790105418.99 total 20.04 last 55.15)
 - open/close 1774 (time 1790105428.99 total 30.05 last 60.28)
 - open/close 2442 (time 1790105439.00 total 40.06 last 66.74)
 - open/close 3074 (time 1790105449.01 total 50.07 last 63.12)
 - open/close 3673 (time 1790105459.03 total 60.08 last 59.81)
 - open/close 4310 (time 1790105469.04 total 70.10 last 63.62)
total: 4500 open/close in 72.96 seconds: 61.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1430 (time 1790105487.92 total 10.00 last 143.00)
 - /unlink 2673 (time 1790105497.93 total 20.01 last 124.23)
 - /unlink 4029 (time 1790105507.93 total 30.01 last 135.56)
total: 4500 /unlink in 33.27 seconds: 135.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 582 (time 1790105525.90 total 10.01 last 58.14)
 - open/close 1218 (time 1790105535.90 total 20.01 last 63.60)
 - open/close 1894 (time 1790105545.91 total 30.02 last 67.56)
 - open/close 2552 (time 1790105555.92 total 40.03 last 65.71)
 - open/close 3307 (time 1790105565.93 total 50.04 last 75.44)
 - open/close 4027 (time 1790105575.97 total 60.08 last 71.71)
total: 4500 open/close in 66.32 seconds: 67.85 ops/second
 - /unlink 1350 (time 1790105596.77 total 10.01 last 134.83)
 - /unlink 2641 (time 1790105606.77 total 20.02 last 129.04)
 - /unlink 4090 (time 1790105616.77 total 30.02 last 144.84)
total: 4500 /unlink in 33.39 seconds: 134.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 692 (time 1790105635.25 total 10.02 last 69.09)
 - open/close 1177 (time 1790105645.25 total 20.02 last 48.48)
 - open/close 1928 (time 1790105655.26 total 30.03 last 75.06)
 - open/close 2587 (time 1790105665.26 total 40.03 last 65.87)
 - open/close 3275 (time 1790105675.27 total 50.03 last 68.78)
 - open/close 3976 (time 1790105685.27 total 60.03 last 70.10)
total: 4500 open/close in 67.52 seconds: 66.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1223 (time 1790105708.25 total 10.03 last 121.95)
 - /unlink 2516 (time 1790105718.25 total 20.03 last 129.30)
 - /unlink 3883 (time 1790105728.25 total 30.03 last 136.68)
total: 4500 /unlink in 34.18 seconds: 131.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 613 (time 1790105747.37 total 10.00 last 61.28)
 - open/close 1234 (time 1790105757.38 total 20.02 last 62.01)
 - open/close 1938 (time 1790105767.40 total 30.03 last 70.29)
 - open/close 2643 (time 1790105777.40 total 40.04 last 70.48)
 - open/close 3359 (time 1790105787.40 total 50.04 last 71.58)
 - open/close 4118 (time 1790105797.41 total 60.05 last 75.84)
total: 4500 open/close in 65.62 seconds: 68.58 ops/second
 - /unlink 1469 (time 1790105818.20 total 10.01 last 146.74)
 - /unlink 2788 (time 1790105828.20 total 20.01 last 131.85)
 - /unlink 4220 (time 1790105838.22 total 30.03 last 142.90)
total: 4500 /unlink in 32.25 seconds: 139.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 754 (time 1790105855.59 total 10.01 last 75.30)
 - open/close 1481 (time 1790105865.62 total 20.04 last 72.52)
 - open/close 2190 (time 1790105875.64 total 30.06 last 70.74)
 - open/close 2842 (time 1790105885.65 total 40.07 last 65.11)
 - open/close 3598 (time 1790105895.67 total 50.09 last 75.49)
 - open/close 4292 (time 1790105905.68 total 60.11 last 69.28)
total: 4500 open/close in 62.86 seconds: 71.58 ops/second
 - /unlink 1039 (time 1790105924.39 total 10.02 last 103.71)
 - /unlink 1970 (time 1790105934.39 total 20.02 last 93.08)
 - /unlink 3324 (time 1790105944.40 total 30.03 last 135.30)
total: 4500 /unlink in 39.73 seconds: 113.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 655 (time 1790105969.72 total 10.01 last 65.41)
 - open/close 1409 (time 1790105979.72 total 20.01 last 75.40)
 - open/close 2097 (time 1790105989.73 total 30.02 last 68.75)
 - open/close 2751 (time 1790105999.73 total 40.02 last 65.37)
 - open/close 3343 (time 1790106009.74 total 50.03 last 59.15)
 - open/close 3962 (time 1790106019.75 total 60.04 last 61.84)
total: 4500 open/close in 68.85 seconds: 65.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1347 (time 1790106043.46 total 10.00 last 134.69)
 - /unlink 2766 (time 1790106053.46 total 20.01 last 141.84)
 - /unlink 3994 (time 1790106063.47 total 30.01 last 122.76)
total: 4500 /unlink in 34.35 seconds: 131.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 624 (time 1790106084.30 total 10.00 last 62.39)
 - open/close 1264 (time 1790106094.32 total 20.02 last 63.88)
 - open/close 1915 (time 1790106104.33 total 30.03 last 65.05)
 - open/close 2681 (time 1790106114.34 total 40.04 last 76.55)
 - open/close 3316 (time 1790106124.34 total 50.04 last 63.48)
 - open/close 3959 (time 1790106134.35 total 60.05 last 64.25)
total: 4500 open/close in 68.15 seconds: 66.03 ops/second
 - /unlink 1391 (time 1790106158.75 total 10.00 last 139.03)
 - /unlink 2370 (time 1790106168.75 total 20.01 last 97.88)
 - /unlink 3224 (time 1790106178.76 total 30.01 last 85.34)
 - /unlink 3949 (time 1790106188.77 total 40.02 last 72.43)
total: 4500 /unlink in 44.99 seconds: 100.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 696 (time 1790106209.16 total 10.00 last 69.59)
 - open/close 1356 (time 1790106219.18 total 20.02 last 65.87)
 - open/close 1891 (time 1790106229.18 total 30.03 last 53.47)
 - open/close 2417 (time 1790106239.19 total 40.03 last 52.57)
 - open/close 3019 (time 1790106249.20 total 50.04 last 60.12)
 - open/close 3642 (time 1790106259.22 total 60.06 last 62.21)
 - open/close 4454 (time 1790106269.22 total 70.06 last 81.20)
total: 4500 open/close in 70.73 seconds: 63.62 ops/second
 - /unlink 1246 (time 1790106284.72 total 10.01 last 124.52)
 - /unlink 2541 (time 1790106294.72 total 20.01 last 129.46)
 - /unlink 3896 (time 1790106304.73 total 30.01 last 135.46)
total: 4500 /unlink in 34.99 seconds: 128.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 749 (time 1790106324.34 total 10.01 last 74.83)
 - open/close 1539 (time 1790106334.35 total 20.02 last 78.89)
 - open/close 2328 (time 1790106344.36 total 30.03 last 78.87)
 - open/close 3110 (time 1790106354.37 total 40.05 last 78.05)
 - open/close 3981 (time 1790106364.38 total 50.05 last 87.07)
total: 4500 open/close in 56.33 seconds: 79.89 ops/second
 - /unlink 1433 (time 1790106384.36 total 10.00 last 143.26)
 - /unlink 2808 (time 1790106394.36 total 20.01 last 137.45)
 - /unlink 4046 (time 1790106404.36 total 30.01 last 123.78)
total: 4500 /unlink in 33.63 seconds: 133.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 809 (time 1790106423.36 total 10.01 last 80.85)
 - open/close 1605 (time 1790106433.37 total 20.02 last 79.51)
 - open/close 2330 (time 1790106443.38 total 30.02 last 72.46)
 - open/close 3123 (time 1790106453.39 total 40.03 last 79.22)
 - open/close 3901 (time 1790106463.39 total 50.03 last 77.78)
total: 4500 open/close in 57.67 seconds: 78.03 ops/second
 - /unlink 1333 (time 1790106485.00 total 10.00 last 133.27)
 - /unlink 2873 (time 1790106495.00 total 20.00 last 153.96)
 - /unlink 4257 (time 1790106505.00 total 30.01 last 138.37)
total: 4500 /unlink in 31.54 seconds: 142.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 756 (time 1790106520.05 total 10.01 last 75.54)
 - open/close 1574 (time 1790106530.05 total 20.01 last 81.79)
 - open/close 2378 (time 1790106540.05 total 30.01 last 80.37)
 - open/close 3217 (time 1790106550.05 total 40.01 last 83.89)
 - open/close 4065 (time 1790106560.06 total 50.02 last 84.77)
total: 4500 open/close in 54.55 seconds: 82.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1958 (time 1790106577.78 total 10.00 last 195.75)
 - /unlink 3702 (time 1790106587.79 total 20.01 last 174.26)
total: 4500 /unlink in 24.57 seconds: 183.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 880 (time 1790106605.93 total 10.00 last 88.00)
 - open/close 1671 (time 1790106615.95 total 20.03 last 78.88)
 - open/close 2570 (time 1790106625.96 total 30.04 last 89.81)
 - open/close 3383 (time 1790106635.97 total 40.04 last 81.24)
 - open/close 4205 (time 1790106645.97 total 50.05 last 82.19)
total: 4500 open/close in 53.80 seconds: 83.64 ops/second
 - /unlink 1647 (time 1790106663.33 total 10.00 last 164.68)
 - /unlink 3226 (time 1790106673.34 total 20.01 last 157.79)
total: 4500 /unlink in 26.76 seconds: 168.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 781 (time 1790106694.31 total 10.02 last 77.96)
 - open/close 1553 (time 1790106704.32 total 20.03 last 77.11)
 - open/close 2387 (time 1790106714.33 total 30.03 last 83.37)
 - open/close 3173 (time 1790106724.33 total 40.03 last 78.60)
 - open/close 3937 (time 1790106734.33 total 50.04 last 76.38)
total: 4500 open/close in 57.05 seconds: 78.88 ops/second
 - /unlink 1814 (time 1790106754.79 total 10.00 last 181.38)
 - /unlink 3694 (time 1790106764.79 total 20.00 last 187.96)
total: 4500 /unlink in 24.59 seconds: 183.00 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11390: 95595 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg245-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg245-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server
stop mds service on oleg245-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server
stop mds service on oleg245-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg245-server
unloading modules via unload_modules_local on: 'oleg245-server'
oleg245-server: oleg245-server.virtnet: executing unload_modules_local
oleg245-server: kmemleak disabled
modules unloaded.