== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:21:49 (1789302109)
Stopping clients: oleg341-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg341-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg341-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg341-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg341-server
oleg341-server: oleg341-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg341-server'
oleg341-server: oleg341-server.virtnet: executing load_modules_local
oleg341-server: kmemleak disabled
oleg341-server: Loading modules from /home/green/git/lustre-release/lustre
oleg341-server: detected 4 online CPUs by sysfs
oleg341-server: MODOPTS_LIBCFS=
oleg341-server: Force libcfs to create 2 CPU partitions
oleg341-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg341-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all
pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg341-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all
pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg341-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all
pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg341-client.virtnet:  -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 638 (time 1789302329.99 total 10.02 last 63.67)
 - open/close 1344 (time 1789302340.01 total 20.04 last 70.45)
 - open/close 2031 (time 1789302350.02 total 30.05 last 68.63)
 - open/close 2649 (time 1789302360.03 total 40.06 last 61.74)
 - open/close 3352 (time 1789302370.03 total 50.06 last 70.29)
 - open/close 4035 (time 1789302380.05 total 60.08 last 68.21)
total: 4500 open/close in 67.36 seconds: 66.81 ops/second
 - /unlink 1347 (time 1789302402.01 total 10.01 last 134.56)
 - /unlink 2502 (time 1789302412.01 total 20.01 last 115.46)
 - /unlink 3645 (time 1789302422.01 total 30.01 last 114.30)
total: 4500 /unlink in 36.82 seconds: 122.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 724 (time 1789302443.82 total 10.00 last 72.37)
 - open/close 1419 (time 1789302453.82 total 20.01 last 69.47)
 - open/close 2101 (time 1789302463.83 total 30.02 last 68.11)
 - open/close 2804 (time 1789302473.84 total 40.02 last 70.28)
 - open/close 3576 (time 1789302483.85 total 50.04 last 77.08)
 - open/close 4305 (time 1789302493.85 total 60.04 last 72.90)
total: 4500 open/close in 62.74 seconds: 71.73 ops/second
 - /unlink 1268 (time 1789302511.42 total 10.00 last 126.75)
 - /unlink 2601 (time 1789302521.43 total 20.01 last 133.22)
 - /unlink 3827 (time 1789302531.43 total 30.01 last 122.55)
total: 4500 /unlink in 34.94 seconds: 128.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 685 (time 1789302551.36 total 10.00 last 68.47)
 - open/close 1308 (time 1789302561.36 total 20.01 last 62.28)
 - open/close 1961 (time 1789302571.37 total 30.02 last 65.25)
 - open/close 2632 (time 1789302581.38 total 40.03 last 67.02)
 - open/close 3316 (time 1789302591.38 total 50.03 last 68.39)
 - open/close 3973 (time 1789302601.40 total 60.05 last 65.55)
total: 4500 open/close in 67.68 seconds: 66.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1292 (time 1789302623.67 total 10.01 last 129.10)
 - /unlink 2623 (time 1789302633.70 total 20.04 last 132.69)
 - /unlink 3835 (time 1789302643.70 total 30.04 last 121.18)
total: 4500 /unlink in 34.79 seconds: 129.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 627 (time 1789302662.91 total 10.01 last 62.61)
 - open/close 1265 (time 1789302672.92 total 20.02 last 63.75)
 - open/close 1849 (time 1789302682.93 total 30.03 last 58.34)
 - open/close 2495 (time 1789302692.94 total 40.05 last 64.52)
 - open/close 3190 (time 1789302702.95 total 50.05 last 69.46)
 - open/close 3889 (time 1789302712.97 total 60.07 last 69.75)
total: 4500 open/close in 69.47 seconds: 64.78 ops/second
 - /unlink 1208 (time 1789302738.41 total 10.02 last 120.54)
 - /unlink 2377 (time 1789302748.41 total 20.03 last 116.85)
 - /unlink 3649 (time 1789302758.41 total 30.03 last 127.16)
total: 4500 /unlink in 37.43 seconds: 120.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 624 (time 1789302781.34 total 10.00 last 62.37)
 - open/close 1222 (time 1789302791.35 total 20.02 last 59.69)
 - open/close 1789 (time 1789302801.37 total 30.04 last 56.59)
 - open/close 2310 (time 1789302811.39 total 40.06 last 51.99)
 - open/close 2984 (time 1789302821.40 total 50.07 last 67.37)
 - open/close 3736 (time 1789302831.41 total 60.08 last 75.13)
 - open/close 4386 (time 1789302841.43 total 70.10 last 64.88)
total: 4500 open/close in 71.85 seconds: 62.63 ops/second
 - /unlink 1250 (time 1789302857.70 total 10.01 last 124.88)
 - /unlink 2494 (time 1789302867.70 total 20.01 last 124.36)
 - /unlink 3849 (time 1789302877.70 total 30.02 last 135.43)
total: 4500 /unlink in 35.37 seconds: 127.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 540 (time 1789302899.02 total 10.02 last 53.89)
 - open/close 1102 (time 1789302909.03 total 20.03 last 56.14)
 - open/close 1738 (time 1789302919.04 total 30.04 last 63.52)
 - open/close 2262 (time 1789302929.05 total 40.06 last 52.33)
 - open/close 2871 (time 1789302939.05 total 50.06 last 60.90)
 - open/close 3558 (time 1789302949.05 total 60.06 last 68.68)
 - open/close 4244 (time 1789302959.06 total 70.07 last 68.54)
total: 4500 open/close in 73.93 seconds: 60.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1238 (time 1789302977.38 total 10.00 last 123.75)
 - /unlink 2421 (time 1789302987.38 total 20.01 last 118.28)
 - /unlink 3616 (time 1789302997.39 total 30.02 last 119.36)
total: 4500 /unlink in 36.64 seconds: 122.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 664 (time 1789303019.47 total 10.01 last 66.36)
 - open/close 1368 (time 1789303029.47 total 20.01 last 70.37)
 - open/close 2097 (time 1789303039.48 total 30.02 last 72.85)
 - open/close 2820 (time 1789303049.49 total 40.03 last 72.24)
 - open/close 3532 (time 1789303059.49 total 50.03 last 71.19)
 - open/close 4174 (time 1789303069.50 total 60.03 last 64.15)
total: 4500 open/close in 64.08 seconds: 70.23 ops/second
 - /unlink 1172 (time 1789303088.34 total 10.00 last 117.20)
 - /unlink 2338 (time 1789303098.34 total 20.00 last 116.55)
 - /unlink 3498 (time 1789303108.34 total 30.01 last 115.99)
total: 4500 /unlink in 39.37 seconds: 114.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 728 (time 1789303133.65 total 10.00 last 72.78)
 - open/close 1465 (time 1789303143.70 total 20.05 last 73.36)
 - open/close 2179 (time 1789303153.70 total 30.05 last 71.36)
 - open/close 2879 (time 1789303163.73 total 40.08 last 69.82)
 - open/close 3736 (time 1789303173.74 total 50.09 last 85.64)
 - open/close 4496 (time 1789303183.74 total 60.09 last 75.97)
total: 4500 open/close in 60.15 seconds: 74.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1292 (time 1789303198.15 total 10.01 last 129.05)
 - /unlink 2661 (time 1789303208.15 total 20.02 last 136.86)
 - /unlink 3977 (time 1789303218.16 total 30.03 last 131.45)
total: 4500 /unlink in 34.19 seconds: 131.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 752 (time 1789303236.71 total 10.01 last 75.11)
 - open/close 1514 (time 1789303246.72 total 20.01 last 76.19)
 - open/close 2126 (time 1789303256.72 total 30.02 last 61.17)
 - open/close 2669 (time 1789303266.74 total 40.04 last 54.18)
 - open/close 3396 (time 1789303276.75 total 50.05 last 72.63)
 - open/close 4096 (time 1789303286.76 total 60.06 last 69.94)
total: 4500 open/close in 67.00 seconds: 67.16 ops/second
 - /unlink 1294 (time 1789303307.92 total 10.00 last 129.35)
 - /unlink 2177 (time 1789303317.92 total 20.01 last 88.29)
 - /unlink 3357 (time 1789303327.92 total 30.01 last 117.94)
total: 4500 /unlink in 39.10 seconds: 115.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 624 (time 1789303352.52 total 10.01 last 62.31)
 - open/close 1171 (time 1789303362.52 total 20.02 last 54.69)
 - open/close 1801 (time 1789303372.54 total 30.03 last 62.91)
 - open/close 2377 (time 1789303382.55 total 40.05 last 57.51)
 - open/close 2900 (time 1789303392.56 total 50.06 last 52.25)
 - open/close 3461 (time 1789303402.57 total 60.06 last 56.07)
 - open/close 4019 (time 1789303412.57 total 70.07 last 55.76)
total: 4500 open/close in 78.15 seconds: 57.58 ops/second
 - /unlink 1108 (time 1789303435.60 total 10.02 last 110.59)
 - /unlink 2201 (time 1789303445.61 total 20.03 last 109.17)
 - /unlink 3451 (time 1789303455.61 total 30.04 last 124.94)
total: 4500 /unlink in 38.33 seconds: 117.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 591 (time 1789303479.27 total 10.01 last 59.05)
 - open/close 1256 (time 1789303489.28 total 20.02 last 66.44)
 - open/close 2006 (time 1789303499.28 total 30.02 last 74.97)
 - open/close 2577 (time 1789303509.28 total 40.02 last 57.08)
 - open/close 3279 (time 1789303519.29 total 50.03 last 70.17)
 - open/close 3922 (time 1789303529.30 total 60.04 last 64.26)
total: 4500 open/close in 68.55 seconds: 65.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1186 (time 1789303553.08 total 10.00 last 118.57)
 - /unlink 2290 (time 1789303563.08 total 20.00 last 110.38)
 - /unlink 3543 (time 1789303573.09 total 30.01 last 125.27)
total: 4500 /unlink in 37.13 seconds: 121.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 714 (time 1789303595.02 total 10.00 last 71.37)
 - open/close 1441 (time 1789303605.02 total 20.01 last 72.66)
 - open/close 2039 (time 1789303615.03 total 30.02 last 59.77)
 - open/close 2720 (time 1789303625.04 total 40.03 last 68.02)
 - open/close 3332 (time 1789303635.05 total 50.04 last 61.14)
 - open/close 3971 (time 1789303645.05 total 60.04 last 63.89)
total: 4500 open/close in 68.31 seconds: 65.87 ops/second
 - /unlink 1273 (time 1789303668.31 total 10.00 last 127.27)
 - /unlink 2521 (time 1789303678.31 total 20.01 last 124.76)
 - /unlink 3768 (time 1789303688.31 total 30.01 last 124.70)
total: 4500 /unlink in 36.26 seconds: 124.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 616 (time 1789303709.52 total 10.01 last 61.53)
 - open/close 1316 (time 1789303719.53 total 20.01 last 69.99)
 - open/close 1811 (time 1789303729.53 total 30.01 last 49.50)
 - open/close 2351 (time 1789303739.53 total 40.02 last 53.96)
 - open/close 2982 (time 1789303749.54 total 50.03 last 63.06)
 - open/close 3661 (time 1789303759.55 total 60.04 last 67.85)
 - open/close 4318 (time 1789303769.55 total 70.04 last 65.66)
total: 4500 open/close in 72.71 seconds: 61.89 ops/second
 - /unlink 1280 (time 1789303786.62 total 10.01 last 127.93)
 - /unlink 2445 (time 1789303796.63 total 20.01 last 116.44)
 - /unlink 3635 (time 1789303806.63 total 30.01 last 119.00)
total: 4500 /unlink in 37.48 seconds: 120.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 668 (time 1789303828.89 total 10.02 last 66.70)
 - open/close 1308 (time 1789303838.89 total 20.02 last 63.98)
 - open/close 1999 (time 1789303848.89 total 30.02 last 69.10)
 - open/close 2698 (time 1789303858.90 total 40.03 last 69.83)
 - open/close 3386 (time 1789303868.91 total 50.04 last 68.72)
 - open/close 4143 (time 1789303878.91 total 60.04 last 75.69)
total: 4500 open/close in 65.02 seconds: 69.21 ops/second
 - /unlink 1233 (time 1789303898.22 total 10.00 last 123.28)
 - /unlink 2522 (time 1789303908.22 total 20.01 last 128.85)
 - /unlink 3793 (time 1789303918.23 total 30.01 last 127.06)
total: 4500 /unlink in 35.18 seconds: 127.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 726 (time 1789303938.03 total 10.00 last 72.58)
 - open/close 1419 (time 1789303948.03 total 20.00 last 69.30)
 - open/close 2101 (time 1789303958.03 total 30.00 last 68.19)
 - open/close 2844 (time 1789303968.04 total 40.02 last 74.21)
 - open/close 3611 (time 1789303978.04 total 50.02 last 76.69)
 - open/close 4309 (time 1789303988.05 total 60.03 last 69.74)
total: 4500 open/close in 62.77 seconds: 71.69 ops/second
 - /unlink 1281 (time 1789304005.79 total 10.00 last 128.10)
 - /unlink 2556 (time 1789304015.79 total 20.00 last 127.48)
 - /unlink 3837 (time 1789304025.80 total 30.01 last 128.03)
total: 4500 /unlink in 35.83 seconds: 125.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 705 (time 1789304046.03 total 10.01 last 70.46)
 - open/close 1388 (time 1789304056.03 total 20.01 last 68.25)
 - open/close 2123 (time 1789304066.04 total 30.02 last 73.48)
 - open/close 2813 (time 1789304076.05 total 40.02 last 68.94)
 - open/close 3448 (time 1789304086.05 total 50.03 last 63.47)
 - open/close 4037 (time 1789304096.06 total 60.04 last 58.83)
total: 4500 open/close in 67.70 seconds: 66.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 993 (time 1789304120.20 total 10.02 last 99.15)
 - /unlink 2170 (time 1789304130.20 total 20.02 last 117.68)
 - /unlink 3313 (time 1789304140.20 total 30.02 last 114.26)
total: 4500 /unlink in 39.80 seconds: 113.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 713 (time 1789304165.51 total 10.01 last 71.26)
 - open/close 1348 (time 1789304175.51 total 20.01 last 63.48)
 - open/close 1944 (time 1789304185.52 total 30.01 last 59.57)
 - open/close 2671 (time 1789304195.52 total 40.01 last 72.70)
 - open/close 3345 (time 1789304205.53 total 50.03 last 67.31)
 - open/close 4062 (time 1789304215.54 total 60.04 last 71.61)
total: 4500 open/close in 66.57 seconds: 67.60 ops/second
 - /unlink 1382 (time 1789304237.05 total 10.01 last 138.07)
 - /unlink 2505 (time 1789304247.06 total 20.02 last 112.20)
 - /unlink 3684 (time 1789304257.06 total 30.02 last 117.89)
total: 4500 /unlink in 37.45 seconds: 120.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 673 (time 1789304280.12 total 10.01 last 67.26)
 - open/close 1244 (time 1789304290.15 total 20.03 last 56.96)
 - open/close 1808 (time 1789304300.19 total 30.07 last 56.17)
 - open/close 2416 (time 1789304310.20 total 40.09 last 60.70)
 - open/close 3041 (time 1789304320.21 total 50.09 last 62.47)
 - open/close 3681 (time 1789304330.21 total 60.09 last 64.00)
 - open/close 4463 (time 1789304340.21 total 70.10 last 78.17)
total: 4500 open/close in 70.63 seconds: 63.72 ops/second
 - /unlink 1175 (time 1789304355.02 total 10.00 last 117.47)
 - /unlink 2438 (time 1789304365.02 total 20.00 last 126.30)
 - /unlink 3815 (time 1789304375.03 total 30.00 last 137.68)
total: 4500 /unlink in 35.62 seconds: 126.34 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95297 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg341-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg341-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server
stop mds service on oleg341-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server
stop mds service on oleg341-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg341-server
unloading modules via unload_modules_local on: 'oleg341-server'
oleg341-server: oleg341-server.virtnet: executing unload_modules_local
oleg341-server: kmemleak disabled
modules unloaded.