== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:10:23 (1789600223)
Stopping clients: oleg655-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg655-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg655-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg655-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg655-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg655-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg655-server
oleg655-server: oleg655-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg655-server'
oleg655-server: oleg655-server.virtnet: executing load_modules_local
oleg655-server: kmemleak disabled
oleg655-server: Loading modules from /home/green/git/lustre-release/lustre
oleg655-server: detected 4 online CPUs by sysfs
oleg655-server: MODOPTS_LIBCFS=
oleg655-server: Force libcfs to create 2 CPU partitions
oleg655-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg655-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all
pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg655-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all
pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg655-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all
pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg655-client.virtnet:  -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 709 (time 1789600431.01 total 10.00 last 70.89)
 - open/close 1467 (time 1789600441.02 total 20.01 last 75.75)
 - open/close 2169 (time 1789600451.02 total 30.01 last 70.19)
 - open/close 2882 (time 1789600461.02 total 40.01 last 71.29)
 - open/close 3616 (time 1789600471.02 total 50.01 last 73.40)
 - open/close 4310 (time 1789600481.03 total 60.02 last 69.35)
total: 4500 open/close in 63.06 seconds: 71.36 ops/second
 - /unlink 1385 (time 1789600498.55 total 10.02 last 138.26)
 - /unlink 2674 (time 1789600508.56 total 20.02 last 128.85)
 - /unlink 4058 (time 1789600518.56 total 30.03 last 138.35)
total: 4500 /unlink in 33.88 seconds: 132.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 686 (time 1789600537.63 total 10.01 last 68.53)
 - open/close 1368 (time 1789600547.65 total 20.03 last 68.05)
 - open/close 2049 (time 1789600557.65 total 30.03 last 68.10)
 - open/close 2819 (time 1789600567.66 total 40.04 last 76.96)
 - open/close 3586 (time 1789600577.66 total 50.05 last 76.64)
 - open/close 4284 (time 1789600587.67 total 60.05 last 69.74)
total: 4500 open/close in 62.94 seconds: 71.50 ops/second
 - /unlink 1456 (time 1789600604.34 total 10.01 last 145.53)
 - /unlink 2849 (time 1789600614.34 total 20.01 last 139.27)
 - /unlink 4259 (time 1789600624.35 total 30.01 last 140.99)
total: 4500 /unlink in 31.75 seconds: 141.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 793 (time 1789600640.84 total 10.01 last 79.23)
 - open/close 1505 (time 1789600650.86 total 20.03 last 71.06)
 - open/close 2226 (time 1789600660.87 total 30.04 last 72.05)
 - open/close 2880 (time 1789600670.87 total 40.04 last 65.36)
 - open/close 3543 (time 1789600680.88 total 50.05 last 66.25)
 - open/close 4265 (time 1789600690.89 total 60.06 last 72.16)
total: 4500 open/close in 62.56 seconds: 71.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1335 (time 1789600707.55 total 10.00 last 133.46)
 - /unlink 2716 (time 1789600717.55 total 20.00 last 138.07)
 - /unlink 4132 (time 1789600727.55 total 30.01 last 141.59)
total: 4500 /unlink in 32.68 seconds: 137.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 735 (time 1789600744.37 total 10.01 last 73.41)
 - open/close 1445 (time 1789600754.38 total 20.02 last 70.93)
 - open/close 2091 (time 1789600764.38 total 30.03 last 64.58)
 - open/close 2736 (time 1789600774.40 total 40.04 last 64.41)
 - open/close 3474 (time 1789600784.41 total 50.06 last 73.69)
 - open/close 4127 (time 1789600794.42 total 60.06 last 65.24)
total: 4500 open/close in 64.64 seconds: 69.62 ops/second
 - /unlink 1229 (time 1789600814.08 total 10.03 last 122.59)
 - /unlink 2495 (time 1789600824.09 total 20.04 last 126.45)
 - /unlink 3954 (time 1789600834.09 total 30.04 last 145.89)
total: 4500 /unlink in 34.03 seconds: 132.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 704 (time 1789600853.11 total 10.01 last 70.33)
 - open/close 1367 (time 1789600863.11 total 20.01 last 66.29)
 - open/close 2013 (time 1789600873.11 total 30.01 last 64.58)
 - open/close 2691 (time 1789600883.12 total 40.03 last 67.73)
 - open/close 3537 (time 1789600893.12 total 50.03 last 84.59)
 - open/close 4212 (time 1789600903.13 total 60.04 last 67.44)
total: 4500 open/close in 64.30 seconds: 69.99 ops/second
 - /unlink 1365 (time 1789600922.14 total 10.01 last 136.38)
 - /unlink 2719 (time 1789600932.14 total 20.01 last 135.36)
 - /unlink 4083 (time 1789600942.15 total 30.02 last 136.30)
total: 4500 /unlink in 33.16 seconds: 135.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 754 (time 1789600959.72 total 10.02 last 75.26)
 - open/close 1494 (time 1789600969.72 total 20.02 last 73.97)
 - open/close 2218 (time 1789600979.72 total 30.03 last 72.37)
 - open/close 2917 (time 1789600989.72 total 40.03 last 69.88)
 - open/close 3685 (time 1789600999.73 total 50.04 last 76.74)
 - open/close 4465 (time 1789601009.75 total 60.05 last 77.87)
total: 4500 open/close in 60.54 seconds: 74.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1382 (time 1789601024.61 total 10.00 last 138.17)
 - /unlink 2797 (time 1789601034.61 total 20.01 last 141.45)
 - /unlink 4168 (time 1789601044.61 total 30.01 last 137.10)
total: 4500 /unlink in 32.36 seconds: 139.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 712 (time 1789601061.16 total 10.01 last 71.11)
 - open/close 1402 (time 1789601071.17 total 20.02 last 68.95)
 - open/close 2152 (time 1789601081.17 total 30.02 last 74.98)
 - open/close 2836 (time 1789601091.19 total 40.04 last 68.31)
 - open/close 3571 (time 1789601101.20 total 50.05 last 73.41)
 - open/close 4244 (time 1789601111.21 total 60.06 last 67.25)
total: 4500 open/close in 64.15 seconds: 70.15 ops/second
 - /unlink 1209 (time 1789601130.59 total 10.00 last 120.85)
 - /unlink 2428 (time 1789601140.59 total 20.01 last 121.85)
 - /unlink 3630 (time 1789601150.59 total 30.01 last 120.18)
total: 4500 /unlink in 37.00 seconds: 121.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 754 (time 1789601172.20 total 10.01 last 75.35)
 - open/close 1498 (time 1789601182.20 total 20.01 last 74.36)
 - open/close 2203 (time 1789601192.20 total 30.02 last 70.48)
 - open/close 2972 (time 1789601202.21 total 40.02 last 76.84)
 - open/close 3732 (time 1789601212.22 total 50.03 last 75.93)
total: 4500 open/close in 59.97 seconds: 75.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1555 (time 1789601235.78 total 10.01 last 155.39)
 - /unlink 3065 (time 1789601245.78 total 20.01 last 150.94)
total: 4500 /unlink in 29.40 seconds: 153.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 773 (time 1789601269.37 total 10.02 last 77.13)
 - open/close 1600 (time 1789601279.37 total 20.02 last 82.69)
 - open/close 2302 (time 1789601289.37 total 30.03 last 70.15)
 - open/close 3082 (time 1789601299.38 total 40.03 last 77.98)
 - open/close 3786 (time 1789601309.39 total 50.04 last 70.33)
 - open/close 4278 (time 1789601319.40 total 60.06 last 49.12)
total: 4500 open/close in 64.07 seconds: 70.23 ops/second
 - /unlink 1103 (time 1789601339.67 total 10.00 last 110.29)
 - /unlink 2504 (time 1789601349.67 total 20.00 last 140.10)
 - /unlink 3927 (time 1789601359.67 total 30.01 last 142.19)
total: 4500 /unlink in 33.81 seconds: 133.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 748 (time 1789601377.60 total 10.01 last 74.72)
 - open/close 1500 (time 1789601387.62 total 20.02 last 75.10)
 - open/close 2260 (time 1789601397.62 total 30.03 last 75.94)
 - open/close 3101 (time 1789601407.62 total 40.03 last 84.09)
 - open/close 3858 (time 1789601417.63 total 50.04 last 75.63)
total: 4500 open/close in 57.76 seconds: 77.91 ops/second
 - /unlink 1329 (time 1789601439.72 total 10.00 last 132.86)
 - /unlink 2750 (time 1789601449.72 total 20.00 last 142.08)
 - /unlink 4170 (time 1789601459.73 total 30.01 last 141.94)
total: 4500 /unlink in 32.69 seconds: 137.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 776 (time 1789601476.85 total 10.01 last 77.52)
 - open/close 1539 (time 1789601486.85 total 20.01 last 76.29)
 - open/close 2269 (time 1789601496.87 total 30.02 last 72.91)
 - open/close 2974 (time 1789601506.87 total 40.03 last 70.47)
 - open/close 3608 (time 1789601516.88 total 50.03 last 63.37)
 - open/close 4264 (time 1789601526.89 total 60.05 last 65.49)
total: 4500 open/close in 63.40 seconds: 70.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1329 (time 1789601544.06 total 10.00 last 132.85)
 - /unlink 2685 (time 1789601554.06 total 20.00 last 135.59)
 - /unlink 4050 (time 1789601564.06 total 30.01 last 136.43)
total: 4500 /unlink in 33.17 seconds: 135.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 761 (time 1789601582.29 total 10.01 last 76.02)
 - open/close 1483 (time 1789601592.29 total 20.01 last 72.19)
 - open/close 2321 (time 1789601602.30 total 30.02 last 83.73)
 - open/close 3035 (time 1789601612.30 total 40.03 last 71.36)
 - open/close 3752 (time 1789601622.32 total 50.04 last 71.58)
 - open/close 4487 (time 1789601632.32 total 60.04 last 73.50)
total: 4500 open/close in 60.16 seconds: 74.80 ops/second
 - /unlink 1448 (time 1789601647.09 total 10.00 last 144.74)
 - /unlink 2727 (time 1789601657.10 total 20.02 last 127.75)
 - /unlink 4125 (time 1789601667.11 total 30.02 last 139.78)
total: 4500 /unlink in 32.90 seconds: 136.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 687 (time 1789601685.75 total 10.00 last 68.70)
 - open/close 1441 (time 1789601695.75 total 20.01 last 75.34)
 - open/close 2146 (time 1789601705.76 total 30.02 last 70.43)
 - open/close 2880 (time 1789601715.77 total 40.03 last 73.34)
 - open/close 3568 (time 1789601725.77 total 50.03 last 68.77)
 - open/close 4261 (time 1789601735.79 total 60.04 last 69.23)
total: 4500 open/close in 63.53 seconds: 70.83 ops/second
 - /unlink 1244 (time 1789601754.54 total 10.00 last 124.37)
 - /unlink 2490 (time 1789601764.54 total 20.01 last 124.51)
 - /unlink 3725 (time 1789601774.54 total 30.01 last 123.49)
total: 4500 /unlink in 36.18 seconds: 124.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 715 (time 1789601795.64 total 10.01 last 71.44)
 - open/close 1429 (time 1789601805.64 total 20.01 last 71.39)
 - open/close 2153 (time 1789601815.65 total 30.02 last 72.33)
 - open/close 2880 (time 1789601825.65 total 40.02 last 72.70)
 - open/close 3589 (time 1789601835.65 total 50.02 last 70.90)
 - open/close 4362 (time 1789601845.65 total 60.02 last 77.26)
total: 4500 open/close in 62.23 seconds: 72.31 ops/second
 - /unlink 1504 (time 1789601862.27 total 10.01 last 150.30)
 - /unlink 2952 (time 1789601872.28 total 20.02 last 144.67)
 - /unlink 4388 (time 1789601882.28 total 30.02 last 143.55)
total: 4500 /unlink in 30.98 seconds: 145.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 694 (time 1789601897.87 total 10.01 last 69.35)
 - open/close 1433 (time 1789601907.88 total 20.01 last 73.87)
 - open/close 2199 (time 1789601917.88 total 30.01 last 76.57)
 - open/close 2902 (time 1789601927.89 total 40.02 last 70.24)
 - open/close 3644 (time 1789601937.90 total 50.03 last 74.15)
 - open/close 4318 (time 1789601947.90 total 60.03 last 67.40)
total: 4500 open/close in 62.66 seconds: 71.82 ops/second
 - /unlink 1195 (time 1789601965.11 total 10.00 last 119.46)
 - /unlink 2671 (time 1789601975.12 total 20.01 last 147.54)
 - /unlink 3984 (time 1789601985.12 total 30.01 last 131.22)
total: 4500 /unlink in 33.51 seconds: 134.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 526 (time 1789602003.81 total 10.02 last 52.52)
 - open/close 1214 (time 1789602013.81 total 20.02 last 68.79)
 - open/close 1851 (time 1789602023.81 total 30.02 last 63.66)
 - open/close 2578 (time 1789602033.82 total 40.03 last 72.65)
 - open/close 3289 (time 1789602043.82 total 50.03 last 71.08)
 - open/close 4002 (time 1789602053.84 total 60.04 last 71.22)
total: 4500 open/close in 66.89 seconds: 67.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1501 (time 1789602074.42 total 10.00 last 150.03)
 - /unlink 2814 (time 1789602084.42 total 20.00 last 131.30)
 - /unlink 4049 (time 1789602094.42 total 30.01 last 123.47)
total: 4500 /unlink in 33.47 seconds: 134.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 760 (time 1789602113.20 total 10.01 last 75.95)
 - open/close 1463 (time 1789602123.21 total 20.01 last 70.28)
 - open/close 2240 (time 1789602133.22 total 30.03 last 77.57)
 - open/close 3043 (time 1789602143.23 total 40.03 last 80.27)
 - open/close 3792 (time 1789602153.24 total 50.04 last 74.82)
total: 4500 open/close in 59.52 seconds: 75.60 ops/second
 - /unlink 1371 (time 1789602176.94 total 10.00 last 137.05)
 - /unlink 2575 (time 1789602186.94 total 20.01 last 120.36)
 - /unlink 4101 (time 1789602196.94 total 30.01 last 152.60)
total: 4500 /unlink in 33.30 seconds: 135.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 745 (time 1789602216.14 total 10.01 last 74.39)
 - open/close 1425 (time 1789602226.16 total 20.03 last 67.91)
 - open/close 2127 (time 1789602236.18 total 30.05 last 70.04)
 - open/close 2840 (time 1789602246.19 total 40.06 last 71.21)
 - open/close 3604 (time 1789602256.21 total 50.08 last 76.28)
 - open/close 4324 (time 1789602266.24 total 60.11 last 71.79)
total: 4500 open/close in 62.90 seconds: 71.54 ops/second
 - /unlink 1349 (time 1789602284.19 total 10.01 last 134.82)
 - /unlink 2626 (time 1789602294.20 total 20.02 last 127.53)
 - /unlink 3965 (time 1789602304.20 total 30.02 last 133.87)
total: 4500 /unlink in 34.56 seconds: 130.21 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95270 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg655-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg655-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg655-server
stop mds service on oleg655-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg655-server
stop mds service on oleg655-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg655-server
unloading modules via unload_modules_local on: 'oleg655-server'
oleg655-server: oleg655-server.virtnet: executing unload_modules_local
oleg655-server: kmemleak disabled
modules unloaded.