== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:59:12 (1789311552)
Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg441-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg441-server
oleg441-server: oleg441-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg441-server'
oleg441-server: oleg441-server.virtnet: executing load_modules_local
oleg441-server: kmemleak disabled
oleg441-server: Loading modules from /home/green/git/lustre-release/lustre
oleg441-server: detected 4 online CPUs by sysfs
oleg441-server: MODOPTS_LIBCFS=
oleg441-server: Force libcfs to create 2 CPU partitions
oleg441-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg441-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all
pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg441-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all
pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg441-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all
pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg441-client.virtnet:  -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 685 (time 1789311768.68 total 10.00 last 68.48)
 - open/close 1325 (time 1789311778.68 total 20.01 last 63.97)
 - open/close 2043 (time 1789311788.68 total 30.01 last 71.77)
 - open/close 2673 (time 1789311798.69 total 40.02 last 62.94)
 - open/close 3407 (time 1789311808.70 total 50.03 last 73.33)
 - open/close 4078 (time 1789311818.71 total 60.03 last 67.08)
total: 4500 open/close in 65.85 seconds: 68.34 ops/second
 - /unlink 1191 (time 1789311839.12 total 10.01 last 119.04)
 - /unlink 2424 (time 1789311849.12 total 20.01 last 123.24)
 - /unlink 3589 (time 1789311859.13 total 30.02 last 116.45)
total: 4500 /unlink in 37.97 seconds: 118.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 707 (time 1789311881.70 total 10.01 last 70.65)
 - open/close 1390 (time 1789311891.73 total 20.04 last 68.10)
 - open/close 2120 (time 1789311901.74 total 30.05 last 72.92)
 - open/close 2783 (time 1789311911.74 total 40.05 last 66.27)
 - open/close 3504 (time 1789311921.75 total 50.06 last 72.07)
 - open/close 4153 (time 1789311931.75 total 60.06 last 64.88)
total: 4500 open/close in 65.38 seconds: 68.82 ops/second
 - /unlink 1198 (time 1789311952.76 total 10.00 last 119.77)
 - /unlink 2313 (time 1789311962.76 total 20.00 last 111.50)
 - /unlink 3517 (time 1789311972.76 total 30.01 last 120.37)
total: 4500 /unlink in 38.05 seconds: 118.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 685 (time 1789311995.88 total 10.02 last 68.37)
 - open/close 1366 (time 1789312005.89 total 20.02 last 68.07)
 - open/close 2051 (time 1789312015.90 total 30.04 last 68.41)
 - open/close 2679 (time 1789312025.91 total 40.04 last 62.75)
 - open/close 3214 (time 1789312035.92 total 50.05 last 53.46)
 - open/close 3805 (time 1789312045.92 total 60.06 last 59.05)
 - open/close 4479 (time 1789312055.93 total 70.06 last 67.37)
total: 4500 open/close in 70.35 seconds: 63.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1254 (time 1789312070.49 total 10.00 last 125.38)
 - /unlink 2533 (time 1789312080.49 total 20.01 last 127.84)
 - /unlink 3802 (time 1789312090.49 total 30.01 last 126.88)
total: 4500 /unlink in 36.24 seconds: 124.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 694 (time 1789312112.08 total 10.00 last 69.40)
 - open/close 1336 (time 1789312122.10 total 20.02 last 64.07)
 - open/close 1904 (time 1789312132.12 total 30.04 last 56.69)
 - open/close 2505 (time 1789312142.13 total 40.04 last 60.08)
 - open/close 3160 (time 1789312152.13 total 50.05 last 65.47)
 - open/close 3880 (time 1789312162.14 total 60.06 last 71.92)
total: 4500 open/close in 68.84 seconds: 65.37 ops/second
 - /unlink 1393 (time 1789312184.98 total 10.00 last 139.24)
 - /unlink 2620 (time 1789312194.98 total 20.01 last 122.61)
 - /unlink 3987 (time 1789312204.98 total 30.01 last 136.70)
total: 4500 /unlink in 34.12 seconds: 131.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 687 (time 1789312223.51 total 10.01 last 68.65)
 - open/close 1387 (time 1789312233.51 total 20.01 last 69.99)
 - open/close 2094 (time 1789312243.53 total 30.03 last 70.56)
 - open/close 2828 (time 1789312253.53 total 40.03 last 73.36)
 - open/close 3486 (time 1789312263.54 total 50.05 last 65.72)
 - open/close 4153 (time 1789312273.56 total 60.06 last 66.63)
total: 4500 open/close in 65.16 seconds: 69.06 ops/second
 - /unlink 1327 (time 1789312292.85 total 10.00 last 132.70)
 - /unlink 2664 (time 1789312302.85 total 20.00 last 133.66)
 - /unlink 3823 (time 1789312312.86 total 30.01 last 115.84)
total: 4500 /unlink in 36.60 seconds: 122.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 581 (time 1789312335.21 total 10.00 last 58.10)
 - open/close 1137 (time 1789312345.25 total 20.04 last 55.38)
 - open/close 1683 (time 1789312355.26 total 30.05 last 54.56)
 - open/close 2229 (time 1789312365.26 total 40.05 last 54.57)
 - open/close 2820 (time 1789312375.26 total 50.05 last 59.09)
 - open/close 3405 (time 1789312385.27 total 60.06 last 58.47)
 - open/close 3956 (time 1789312395.27 total 70.06 last 55.07)
total: 4500 open/close in 79.16 seconds: 56.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1164 (time 1789312419.36 total 10.00 last 116.40)
 - /unlink 2418 (time 1789312429.37 total 20.01 last 125.33)
 - /unlink 3649 (time 1789312439.38 total 30.02 last 122.98)
total: 4500 /unlink in 37.13 seconds: 121.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 668 (time 1789312461.48 total 10.01 last 66.74)
 - open/close 1360 (time 1789312471.48 total 20.01 last 69.20)
 - open/close 2021 (time 1789312481.49 total 30.02 last 66.06)
 - open/close 2692 (time 1789312491.50 total 40.02 last 67.04)
 - open/close 3369 (time 1789312501.50 total 50.03 last 67.68)
 - open/close 4085 (time 1789312511.51 total 60.03 last 71.55)
total: 4500 open/close in 66.28 seconds: 67.89 ops/second
 - /unlink 1390 (time 1789312531.82 total 10.00 last 138.94)
 - /unlink 2702 (time 1789312541.82 total 20.01 last 131.17)
 - /unlink 4117 (time 1789312551.82 total 30.01 last 141.49)
total: 4500 /unlink in 32.76 seconds: 137.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 703 (time 1789312569.17 total 10.02 last 70.14)
 - open/close 1408 (time 1789312579.19 total 20.04 last 70.36)
 - open/close 2128 (time 1789312589.19 total 30.05 last 71.94)
 - open/close 2869 (time 1789312599.21 total 40.06 last 74.00)
 - open/close 3605 (time 1789312609.22 total 50.08 last 73.50)
 - open/close 4315 (time 1789312619.24 total 60.09 last 70.89)
total: 4500 open/close in 62.33 seconds: 72.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1445 (time 1789312634.76 total 10.00 last 144.45)
 - /unlink 2781 (time 1789312644.76 total 20.01 last 133.55)
 - /unlink 4111 (time 1789312654.77 total 30.01 last 132.91)
total: 4500 /unlink in 32.99 seconds: 136.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 663 (time 1789312672.60 total 10.02 last 66.19)
 - open/close 1301 (time 1789312682.60 total 20.02 last 63.79)
 - open/close 2002 (time 1789312692.63 total 30.04 last 69.95)
 - open/close 2619 (time 1789312702.63 total 40.05 last 61.66)
 - open/close 3274 (time 1789312712.64 total 50.05 last 65.48)
 - open/close 3938 (time 1789312722.64 total 60.05 last 66.38)
total: 4500 open/close in 67.88 seconds: 66.30 ops/second
 - /unlink 1295 (time 1789312744.14 total 10.00 last 129.47)
 - /unlink 2551 (time 1789312754.14 total 20.01 last 125.57)
 - /unlink 3957 (time 1789312764.14 total 30.01 last 140.57)
total: 4500 /unlink in 34.15 seconds: 131.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 701 (time 1789312782.46 total 10.01 last 70.04)
 - open/close 1385 (time 1789312792.47 total 20.01 last 68.36)
 - open/close 2112 (time 1789312802.47 total 30.02 last 72.67)
 - open/close 2797 (time 1789312812.48 total 40.03 last 68.43)
 - open/close 3537 (time 1789312822.49 total 50.03 last 73.96)
 - open/close 4276 (time 1789312832.49 total 60.04 last 73.84)
total: 4500 open/close in 62.97 seconds: 71.46 ops/second
 - /unlink 1482 (time 1789312848.72 total 10.00 last 148.14)
 - /unlink 2920 (time 1789312858.72 total 20.00 last 143.79)
 - /unlink 4410 (time 1789312868.73 total 30.01 last 148.87)
total: 4500 /unlink in 30.71 seconds: 146.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 713 (time 1789312882.74 total 10.01 last 71.21)
 - open/close 1382 (time 1789312892.75 total 20.02 last 66.86)
 - open/close 2074 (time 1789312902.77 total 30.04 last 69.08)
 - open/close 2808 (time 1789312912.77 total 40.04 last 73.36)
 - open/close 3544 (time 1789312922.78 total 50.05 last 73.54)
 - open/close 4304 (time 1789312932.78 total 60.05 last 75.97)
total: 4500 open/close in 62.54 seconds: 71.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1454 (time 1789312948.82 total 10.00 last 145.37)
 - /unlink 3019 (time 1789312958.83 total 20.01 last 156.38)
 - /unlink 4488 (time 1789312968.84 total 30.01 last 146.84)
total: 4500 /unlink in 30.09 seconds: 149.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 771 (time 1789312982.53 total 10.00 last 77.09)
 - open/close 1561 (time 1789312992.54 total 20.01 last 78.92)
 - open/close 2300 (time 1789313002.54 total 30.02 last 73.86)
 - open/close 3010 (time 1789313012.55 total 40.03 last 70.94)
 - open/close 3670 (time 1789313022.56 total 50.04 last 65.94)
 - open/close 4417 (time 1789313032.56 total 60.04 last 74.68)
total: 4500 open/close in 61.07 seconds: 73.68 ops/second
 - /unlink 1332 (time 1789313047.36 total 10.01 last 133.10)
 - /unlink 2679 (time 1789313057.36 total 20.01 last 134.69)
 - /unlink 4125 (time 1789313067.37 total 30.02 last 144.46)
total: 4500 /unlink in 32.74 seconds: 137.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 773 (time 1789313084.46 total 10.01 last 77.24)
 - open/close 1565 (time 1789313094.46 total 20.01 last 79.20)
 - open/close 2388 (time 1789313104.47 total 30.02 last 82.22)
 - open/close 3179 (time 1789313114.47 total 40.02 last 79.07)
 - open/close 3956 (time 1789313124.48 total 50.03 last 77.61)
total: 4500 open/close in 56.71 seconds: 79.35 ops/second
 - /unlink 1459 (time 1789313144.34 total 10.00 last 145.90)
 - /unlink 2949 (time 1789313154.34 total 20.00 last 148.98)
 - /unlink 4393 (time 1789313164.34 total 30.00 last 144.40)
total: 4500 /unlink in 30.68 seconds: 146.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 795 (time 1789313178.25 total 10.00 last 79.50)
 - open/close 1644 (time 1789313188.26 total 20.01 last 84.84)
 - open/close 2578 (time 1789313198.27 total 30.01 last 93.33)
 - open/close 3420 (time 1789313208.27 total 40.02 last 84.16)
 - open/close 4292 (time 1789313218.28 total 50.03 last 87.10)
total: 4500 open/close in 52.43 seconds: 85.83 ops/second
 - /unlink 1684 (time 1789313233.92 total 10.01 last 168.31)
 - /unlink 3507 (time 1789313243.92 total 20.01 last 182.29)
total: 4500 /unlink in 26.70 seconds: 168.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 811 (time 1789313264.17 total 10.01 last 81.04)
 - open/close 1711 (time 1789313274.19 total 20.02 last 89.90)
 - open/close 2574 (time 1789313284.19 total 30.03 last 86.22)
 - open/close 3479 (time 1789313294.20 total 40.03 last 90.48)
 - open/close 4414 (time 1789313304.20 total 50.04 last 93.42)
total: 4500 open/close in 51.09 seconds: 88.08 ops/second
 - /unlink 1754 (time 1789313318.16 total 10.00 last 175.36)
 - /unlink 3619 (time 1789313328.16 total 20.00 last 186.45)
total: 4500 /unlink in 24.41 seconds: 184.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 1108 (time 1789313344.91 total 10.01 last 110.72)
 - open/close 2212 (time 1789313354.91 total 20.01 last 110.40)
 - open/close 3291 (time 1789313364.92 total 30.01 last 107.83)
total: 4500 open/close in 40.00 seconds: 112.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 2500 (time 1789313387.04 total 10.00 last 249.95)
total: 4500 /unlink in 18.38 seconds: 244.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 1191 (time 1789313407.75 total 10.00 last 119.04)
 - open/close 2452 (time 1789313417.75 total 20.01 last 126.10)
 - open/close 3720 (time 1789313427.75 total 30.01 last 126.78)
total: 4500 open/close in 36.83 seconds: 122.18 ops/second
 - /unlink 2120 (time 1789313446.77 total 10.00 last 211.99)
 - /unlink 4314 (time 1789313456.78 total 20.00 last 219.34)
total: 4500 /unlink in 20.98 seconds: 214.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 981 (time 1789313470.19 total 10.00 last 98.05)
 - open/close 1867 (time 1789313480.19 total 20.01 last 88.58)
 - open/close 2938 (time 1789313490.20 total 30.01 last 107.05)
 - open/close 3992 (time 1789313500.21 total 40.02 last 105.31)
total: 4500 open/close in 45.34 seconds: 99.25 ops/second
 - /unlink 2107 (time 1789313518.08 total 10.00 last 210.66)
 - /unlink 4282 (time 1789313528.08 total 20.00 last 217.46)
total: 4500 /unlink in 20.94 seconds: 214.90 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95007 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg441-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg441-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server
stop mds service on oleg441-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server
stop mds service on oleg441-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server
unloading modules via unload_modules_local on: 'oleg441-server'
oleg441-server: oleg441-server.virtnet: executing unload_modules_local
oleg441-server: kmemleak disabled
modules unloaded.