== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:19:21 (1788873561)
Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg150-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg150-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg150-server
oleg150-server: oleg150-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg150-server'
oleg150-server: oleg150-server.virtnet: executing load_modules_local
oleg150-server: kmemleak disabled
oleg150-server: Loading modules from /home/green/git/lustre-release/lustre
oleg150-server: detected 4 online CPUs by sysfs
oleg150-server: MODOPTS_LIBCFS=
oleg150-server: Force libcfs to create 2 CPU partitions
oleg150-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg150-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all
pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg150-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all
pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg150-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all
pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg150-client.virtnet:  -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 754 (time 1788873790.27 total 10.00 last 75.37)
 - open/close 1502 (time 1788873800.28 total 20.01 last 74.75)
 - open/close 2307 (time 1788873810.28 total 30.02 last 80.44)
 - open/close 3082 (time 1788873820.31 total 40.04 last 77.32)
 - open/close 3954 (time 1788873830.31 total 50.04 last 87.18)
total: 4500 open/close in 56.57 seconds: 79.55 ops/second
 - /unlink 1470 (time 1788873851.08 total 10.00 last 146.95)
 - /unlink 2803 (time 1788873861.08 total 20.01 last 133.27)
 - /unlink 4231 (time 1788873871.09 total 30.02 last 142.63)
total: 4500 /unlink in 31.87 seconds: 141.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 814 (time 1788873888.30 total 10.02 last 81.26)
 - open/close 1596 (time 1788873898.33 total 20.04 last 78.03)
 - open/close 2323 (time 1788873908.33 total 30.04 last 72.70)
 - open/close 3010 (time 1788873918.34 total 40.05 last 68.60)
 - open/close 3656 (time 1788873928.36 total 50.07 last 64.50)
 - open/close 4378 (time 1788873938.36 total 60.07 last 72.19)
total: 4500 open/close in 61.87 seconds: 72.73 ops/second
 - /unlink 1318 (time 1788873954.77 total 10.00 last 131.74)
 - /unlink 2749 (time 1788873964.77 total 20.00 last 143.09)
 - /unlink 4113 (time 1788873974.78 total 30.01 last 136.29)
total: 4500 /unlink in 32.96 seconds: 136.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 741 (time 1788873992.40 total 10.01 last 74.02)
 - open/close 1492 (time 1788874002.41 total 20.02 last 75.04)
 - open/close 2208 (time 1788874012.42 total 30.03 last 71.55)
 - open/close 2964 (time 1788874022.42 total 40.03 last 75.60)
 - open/close 3707 (time 1788874032.43 total 50.04 last 74.19)
 - open/close 4406 (time 1788874042.44 total 60.05 last 69.82)
total: 4500 open/close in 61.32 seconds: 73.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1328 (time 1788874059.45 total 10.02 last 132.52)
 - /unlink 2550 (time 1788874069.45 total 20.02 last 122.20)
 - /unlink 3769 (time 1788874079.46 total 30.03 last 121.81)
total: 4500 /unlink in 35.05 seconds: 128.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 700 (time 1788874098.66 total 10.01 last 69.95)
 - open/close 1415 (time 1788874108.67 total 20.01 last 71.47)
 - open/close 2168 (time 1788874118.68 total 30.02 last 75.24)
 - open/close 3034 (time 1788874128.68 total 40.02 last 86.56)
 - open/close 3760 (time 1788874138.69 total 50.03 last 72.53)
total: 4500 open/close in 58.54 seconds: 76.86 ops/second
 - /unlink 1551 (time 1788874162.13 total 10.00 last 155.04)
 - /unlink 2936 (time 1788874172.17 total 20.04 last 137.95)
 - /unlink 4167 (time 1788874182.17 total 30.05 last 123.07)
total: 4500 /unlink in 32.40 seconds: 138.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 767 (time 1788874201.21 total 10.00 last 76.69)
 - open/close 1540 (time 1788874211.22 total 20.01 last 77.26)
 - open/close 2353 (time 1788874221.23 total 30.02 last 81.21)
 - open/close 3145 (time 1788874231.24 total 40.03 last 79.10)
 - open/close 3856 (time 1788874241.25 total 50.04 last 71.05)
total: 4500 open/close in 57.25 seconds: 78.60 ops/second
 - /unlink 1376 (time 1788874262.92 total 10.00 last 137.58)
 - /unlink 2621 (time 1788874272.92 total 20.00 last 124.49)
 - /unlink 3714 (time 1788874282.92 total 30.00 last 109.27)
total: 4500 /unlink in 35.91 seconds: 125.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 683 (time 1788874303.35 total 10.02 last 68.18)
 - open/close 1436 (time 1788874313.36 total 20.02 last 75.25)
 - open/close 2258 (time 1788874323.36 total 30.03 last 82.16)
 - open/close 3096 (time 1788874333.36 total 40.03 last 83.78)
 - open/close 3852 (time 1788874343.37 total 50.04 last 75.57)
total: 4500 open/close in 58.29 seconds: 77.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1275 (time 1788874365.71 total 10.01 last 127.41)
 - /unlink 2895 (time 1788874375.72 total 20.01 last 161.98)
 - /unlink 4293 (time 1788874385.72 total 30.01 last 139.74)
total: 4500 /unlink in 31.67 seconds: 142.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 714 (time 1788874403.05 total 10.02 last 71.28)
 - open/close 1500 (time 1788874413.06 total 20.02 last 78.56)
 - open/close 2267 (time 1788874423.06 total 30.03 last 76.66)
 - open/close 2975 (time 1788874433.06 total 40.03 last 70.78)
 - open/close 3673 (time 1788874443.08 total 50.04 last 69.72)
 - open/close 4237 (time 1788874453.08 total 60.04 last 56.39)
total: 4500 open/close in 65.50 seconds: 68.70 ops/second
 - /unlink 1515 (time 1788874474.17 total 10.00 last 151.46)
 - /unlink 3152 (time 1788874484.18 total 20.01 last 163.61)
total: 4500 /unlink in 29.70 seconds: 151.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 759 (time 1788874508.46 total 10.01 last 75.81)
 - open/close 1466 (time 1788874518.49 total 20.04 last 70.48)
 - open/close 2216 (time 1788874528.49 total 30.05 last 74.97)
 - open/close 2989 (time 1788874538.50 total 40.05 last 77.28)
 - open/close 3821 (time 1788874548.50 total 50.05 last 83.20)
total: 4500 open/close in 58.32 seconds: 77.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1489 (time 1788874570.79 total 10.01 last 148.79)
 - /unlink 2867 (time 1788874580.79 total 20.01 last 137.72)
 - /unlink 4450 (time 1788874590.80 total 30.02 last 158.19)
total: 4500 /unlink in 30.63 seconds: 146.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 618 (time 1788874606.16 total 10.03 last 61.61)
 - open/close 1195 (time 1788874616.16 total 20.04 last 57.66)
 - open/close 2027 (time 1788874626.17 total 30.04 last 83.16)
 - open/close 2766 (time 1788874636.19 total 40.07 last 73.72)
 - open/close 3509 (time 1788874646.20 total 50.07 last 74.25)
 - open/close 4291 (time 1788874656.22 total 60.09 last 78.05)
total: 4500 open/close in 62.58 seconds: 71.91 ops/second
 - /unlink 1564 (time 1788874673.05 total 10.01 last 156.24)
 - /unlink 2815 (time 1788874683.06 total 20.01 last 125.07)
 - /unlink 4132 (time 1788874693.06 total 30.01 last 131.67)
total: 4500 /unlink in 33.06 seconds: 136.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 723 (time 1788874712.36 total 10.01 last 72.23)
 - open/close 1613 (time 1788874722.37 total 20.02 last 88.91)
 - open/close 2383 (time 1788874732.39 total 30.04 last 76.81)
 - open/close 3195 (time 1788874742.41 total 40.07 last 81.01)
 - open/close 3985 (time 1788874752.42 total 50.07 last 79.00)
total: 4500 open/close in 56.05 seconds: 80.28 ops/second
 - /unlink 1533 (time 1788874772.20 total 10.00 last 153.24)
 - /unlink 3068 (time 1788874782.21 total 20.01 last 153.46)
 - /unlink 4441 (time 1788874792.21 total 30.01 last 137.29)
total: 4500 /unlink in 30.58 seconds: 147.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 800 (time 1788874806.71 total 10.00 last 79.96)
 - open/close 1636 (time 1788874816.73 total 20.02 last 83.46)
 - open/close 2446 (time 1788874826.74 total 30.03 last 80.93)
 - open/close 3315 (time 1788874836.74 total 40.03 last 86.88)
 - open/close 4140 (time 1788874846.76 total 50.05 last 82.35)
total: 4500 open/close in 54.76 seconds: 82.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1474 (time 1788874865.83 total 10.00 last 147.39)
 - /unlink 2960 (time 1788874875.83 total 20.00 last 148.59)
total: 4500 /unlink in 29.81 seconds: 150.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 882 (time 1788874899.36 total 10.01 last 88.08)
 - open/close 1761 (time 1788874909.37 total 20.02 last 87.85)
 - open/close 2506 (time 1788874919.40 total 30.06 last 74.22)
 - open/close 3226 (time 1788874929.41 total 40.06 last 71.98)
 - open/close 3876 (time 1788874939.42 total 50.08 last 64.90)
 - open/close 4355 (time 1788874949.42 total 60.08 last 47.88)
total: 4500 open/close in 61.98 seconds: 72.61 ops/second
 - /unlink 1282 (time 1788874965.89 total 10.00 last 128.20)
 - /unlink 2846 (time 1788874975.89 total 20.00 last 156.37)
 - /unlink 3984 (time 1788874985.90 total 30.01 last 113.74)
total: 4500 /unlink in 33.72 seconds: 133.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 692 (time 1788875004.71 total 10.01 last 69.16)
 - open/close 1484 (time 1788875014.71 total 20.01 last 79.18)
 - open/close 2292 (time 1788875024.72 total 30.01 last 80.76)
 - open/close 3039 (time 1788875034.74 total 40.03 last 74.55)
 - open/close 3818 (time 1788875044.74 total 50.04 last 77.87)
total: 4500 open/close in 58.28 seconds: 77.22 ops/second
 - /unlink 1440 (time 1788875067.95 total 10.00 last 144.00)
 - /unlink 2876 (time 1788875077.96 total 20.00 last 143.58)
 - /unlink 4203 (time 1788875087.96 total 30.00 last 132.65)
total: 4500 /unlink in 32.04 seconds: 140.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 704 (time 1788875107.56 total 10.01 last 70.32)
 - open/close 1492 (time 1788875117.56 total 20.01 last 78.78)
 - open/close 2359 (time 1788875127.56 total 30.01 last 86.70)
 - open/close 3170 (time 1788875137.57 total 40.02 last 81.04)
 - open/close 3874 (time 1788875147.57 total 50.02 last 70.39)
total: 4500 open/close in 58.01 seconds: 77.58 ops/second
 - /unlink 1647 (time 1788875169.83 total 10.01 last 164.58)
 - /unlink 3188 (time 1788875179.84 total 20.02 last 153.95)
total: 4500 /unlink in 29.28 seconds: 153.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 775 (time 1788875203.73 total 10.03 last 77.30)
 - open/close 1551 (time 1788875213.73 total 20.03 last 77.57)
 - open/close 2357 (time 1788875223.75 total 30.04 last 80.51)
 - open/close 3192 (time 1788875233.76 total 40.06 last 83.36)
 - open/close 3984 (time 1788875243.77 total 50.07 last 79.11)
total: 4500 open/close in 55.20 seconds: 81.53 ops/second
 - /unlink 1527 (time 1788875263.72 total 10.00 last 152.68)
 - /unlink 2929 (time 1788875273.72 total 20.00 last 140.17)
 - /unlink 4472 (time 1788875283.72 total 30.01 last 154.26)
total: 4500 /unlink in 30.21 seconds: 148.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 652 (time 1788875298.51 total 10.01 last 65.15)
 - open/close 1473 (time 1788875308.53 total 20.03 last 81.95)
 - open/close 2312 (time 1788875318.53 total 30.03 last 83.89)
 - open/close 3275 (time 1788875328.54 total 40.03 last 96.22)
 - open/close 4116 (time 1788875338.54 total 50.04 last 84.07)
total: 4500 open/close in 55.74 seconds: 80.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1451 (time 1788875359.38 total 10.00 last 145.09)
 - /unlink 2656 (time 1788875369.39 total 20.01 last 120.42)
 - /unlink 4396 (time 1788875379.40 total 30.02 last 173.84)
total: 4500 /unlink in 30.82 seconds: 146.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 630 (time 1788875395.41 total 10.02 last 62.88)
 - open/close 1405 (time 1788875405.43 total 20.03 last 77.42)
 - open/close 2203 (time 1788875415.43 total 30.03 last 79.80)
 - open/close 3079 (time 1788875425.43 total 40.04 last 87.55)
 - open/close 3681 (time 1788875435.46 total 50.06 last 60.03)
 - open/close 4340 (time 1788875445.47 total 60.07 last 65.86)
total: 4500 open/close in 62.30 seconds: 72.23 ops/second
 - /unlink 1418 (time 1788875463.07 total 10.00 last 141.79)
 - /unlink 2845 (time 1788875473.07 total 20.00 last 142.67)
 - /unlink 4166 (time 1788875483.08 total 30.01 last 132.07)
total: 4500 /unlink in 32.92 seconds: 136.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 724 (time 1788875501.78 total 10.01 last 72.35)
 - open/close 1499 (time 1788875511.78 total 20.01 last 77.45)
 - open/close 2286 (time 1788875521.79 total 30.02 last 78.67)
 - open/close 2925 (time 1788875531.79 total 40.02 last 63.90)
 - open/close 3721 (time 1788875541.79 total 50.02 last 79.57)
 - open/close 4498 (time 1788875551.80 total 60.03 last 77.64)
total: 4500 open/close in 60.04 seconds: 74.95 ops/second
 - /unlink 1736 (time 1788875566.27 total 10.00 last 173.59)
 - /unlink 3402 (time 1788875576.28 total 20.01 last 166.52)
total: 4500 /unlink in 27.58 seconds: 163.16 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95263 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg150-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg150-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server
stop mds service on oleg150-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server
stop mds service on oleg150-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg150-server
unloading modules via unload_modules_local on: 'oleg150-server'
oleg150-server: oleg150-server.virtnet: executing unload_modules_local
oleg150-server: kmemleak disabled
modules unloaded.