== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:05:12 (1789999512)
Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg344-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg344-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg344-server
oleg344-server: oleg344-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg344-server'
oleg344-server: oleg344-server.virtnet: executing load_modules_local
oleg344-server: kmemleak disabled
oleg344-server: Loading modules from /home/green/git/lustre-release/lustre
oleg344-server: detected 4 online CPUs by sysfs
oleg344-server: MODOPTS_LIBCFS=
oleg344-server: Force libcfs to create 2 CPU partitions
oleg344-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg344-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all
pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg344-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all
pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg344-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all
pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg344-client.virtnet:  -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 717 (time 1789999705.96 total 10.00 last 71.69)
 - open/close 1503 (time 1789999715.96 total 20.00 last 78.57)
 - open/close 2255 (time 1789999725.97 total 30.02 last 75.12)
 - open/close 3053 (time 1789999735.97 total 40.02 last 79.78)
 - open/close 3805 (time 1789999745.98 total 50.03 last 75.12)
 - open/close 4429 (time 1789999756.01 total 60.05 last 62.25)
total: 4500 open/close in 61.28 seconds: 73.43 ops/second
 - /unlink 1231 (time 1789999773.23 total 10.00 last 123.05)
 - /unlink 2465 (time 1789999783.24 total 20.01 last 123.35)
 - /unlink 3775 (time 1789999793.24 total 30.01 last 130.94)
total: 4500 /unlink in 35.47 seconds: 126.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 750 (time 1789999812.64 total 10.00 last 74.96)
 - open/close 1454 (time 1789999822.65 total 20.01 last 70.38)
 - open/close 2179 (time 1789999832.66 total 30.02 last 72.40)
 - open/close 2836 (time 1789999842.68 total 40.04 last 65.60)
 - open/close 3532 (time 1789999852.69 total 50.05 last 69.53)
 - open/close 4209 (time 1789999862.69 total 60.05 last 67.70)
total: 4500 open/close in 63.69 seconds: 70.65 ops/second
 - /unlink 1288 (time 1789999880.63 total 10.00 last 128.77)
 - /unlink 2578 (time 1789999890.63 total 20.00 last 128.99)
 - /unlink 3865 (time 1789999900.63 total 30.00 last 128.70)
total: 4500 /unlink in 34.72 seconds: 129.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 725 (time 1789999919.97 total 10.00 last 72.49)
 - open/close 1393 (time 1789999929.98 total 20.01 last 66.76)
 - open/close 2125 (time 1789999939.99 total 30.01 last 73.16)
 - open/close 2887 (time 1789999950.00 total 40.03 last 76.10)
 - open/close 3622 (time 1789999960.00 total 50.03 last 73.46)
 - open/close 4332 (time 1789999970.01 total 60.04 last 70.97)
total: 4500 open/close in 62.71 seconds: 71.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1340 (time 1789999987.41 total 10.01 last 133.93)
 - /unlink 2692 (time 1789999997.42 total 20.02 last 135.00)
 - /unlink 3845 (time 1790000007.44 total 30.03 last 115.14)
total: 4500 /unlink in 34.86 seconds: 129.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 697 (time 1790000027.08 total 10.00 last 69.68)
 - open/close 1444 (time 1790000037.08 total 20.01 last 74.67)
 - open/close 2142 (time 1790000047.09 total 30.02 last 69.72)
 - open/close 2933 (time 1790000057.10 total 40.03 last 79.04)
 - open/close 3618 (time 1790000067.11 total 50.04 last 68.41)
 - open/close 4320 (time 1790000077.13 total 60.05 last 70.10)
total: 4500 open/close in 62.74 seconds: 71.72 ops/second
 - /unlink 1426 (time 1790000094.86 total 10.00 last 142.54)
 - /unlink 2712 (time 1790000104.86 total 20.01 last 128.54)
 - /unlink 4096 (time 1790000114.86 total 30.01 last 138.39)
total: 4500 /unlink in 33.06 seconds: 136.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 743 (time 1790000132.92 total 10.01 last 74.24)
 - open/close 1430 (time 1790000142.93 total 20.02 last 68.63)
 - open/close 2067 (time 1790000152.94 total 30.02 last 63.68)
 - open/close 2757 (time 1790000162.96 total 40.05 last 68.81)
 - open/close 3436 (time 1790000172.98 total 50.06 last 67.80)
 - open/close 4102 (time 1790000182.99 total 60.07 last 66.52)
total: 4500 open/close in 67.76 seconds: 66.41 ops/second
 - /unlink 915 (time 1790000205.87 total 10.01 last 91.44)
 - /unlink 2106 (time 1790000215.88 total 20.01 last 119.05)
 - /unlink 3381 (time 1790000225.88 total 30.01 last 127.47)
 - /unlink 4305 (time 1790000235.88 total 40.02 last 92.36)
total: 4500 /unlink in 41.97 seconds: 107.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 419 (time 1790000254.92 total 10.00 last 41.89)
 - open/close 981 (time 1790000264.93 total 20.02 last 56.13)
 - open/close 1753 (time 1790000274.94 total 30.02 last 77.15)
 - open/close 2446 (time 1790000284.94 total 40.02 last 69.30)
 - open/close 3118 (time 1790000294.94 total 50.03 last 67.16)
 - open/close 3897 (time 1790000304.95 total 60.03 last 77.86)
total: 4500 open/close in 68.68 seconds: 65.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1194 (time 1790000328.26 total 10.00 last 119.36)
 - /unlink 2408 (time 1790000338.27 total 20.01 last 121.27)
 - /unlink 3537 (time 1790000348.28 total 30.02 last 112.82)
total: 4500 /unlink in 39.08 seconds: 115.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 640 (time 1790000373.64 total 10.01 last 63.94)
 - open/close 1336 (time 1790000383.64 total 20.01 last 69.58)
 - open/close 1994 (time 1790000393.65 total 30.02 last 65.74)
 - open/close 2682 (time 1790000403.66 total 40.02 last 68.77)
 - open/close 3356 (time 1790000413.66 total 50.03 last 67.34)
 - open/close 4062 (time 1790000423.67 total 60.04 last 70.56)
total: 4500 open/close in 66.48 seconds: 67.69 ops/second
 - /unlink 1372 (time 1790000444.07 total 10.00 last 137.14)
 - /unlink 2709 (time 1790000454.08 total 20.01 last 133.60)
 - /unlink 4049 (time 1790000464.08 total 30.01 last 133.97)
total: 4500 /unlink in 33.45 seconds: 134.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 667 (time 1790000483.52 total 10.01 last 66.62)
 - open/close 1299 (time 1790000493.53 total 20.02 last 63.13)
 - open/close 1914 (time 1790000503.54 total 30.03 last 61.45)
 - open/close 2581 (time 1790000513.54 total 40.04 last 66.66)
 - open/close 3306 (time 1790000523.55 total 50.04 last 72.46)
 - open/close 4038 (time 1790000533.55 total 60.05 last 73.17)
total: 4500 open/close in 66.41 seconds: 67.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1231 (time 1790000554.32 total 10.00 last 123.04)
 - /unlink 2394 (time 1790000564.32 total 20.01 last 116.29)
 - /unlink 3702 (time 1790000574.33 total 30.01 last 130.69)
total: 4500 /unlink in 36.61 seconds: 122.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 683 (time 1790000595.46 total 10.01 last 68.24)
 - open/close 1290 (time 1790000605.47 total 20.02 last 60.65)
 - open/close 1825 (time 1790000615.49 total 30.04 last 53.36)
 - open/close 2448 (time 1790000625.51 total 40.06 last 62.22)
 - open/close 3162 (time 1790000635.51 total 50.06 last 71.37)
 - open/close 3869 (time 1790000645.52 total 60.07 last 70.66)
total: 4500 open/close in 68.72 seconds: 65.48 ops/second
 - /unlink 1288 (time 1790000668.75 total 10.00 last 128.79)
 - /unlink 2575 (time 1790000678.75 total 20.00 last 128.66)
 - /unlink 3890 (time 1790000688.75 total 30.01 last 131.46)
total: 4500 /unlink in 35.33 seconds: 127.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 747 (time 1790000709.07 total 10.02 last 74.57)
 - open/close 1477 (time 1790000719.07 total 20.02 last 72.97)
 - open/close 2192 (time 1790000729.08 total 30.04 last 71.39)
 - open/close 2878 (time 1790000739.09 total 40.04 last 68.54)
 - open/close 3565 (time 1790000749.10 total 50.05 last 68.68)
 - open/close 4229 (time 1790000759.10 total 60.05 last 66.35)
total: 4500 open/close in 64.62 seconds: 69.64 ops/second
 - /unlink 1332 (time 1790000778.13 total 10.01 last 133.08)
 - /unlink 2546 (time 1790000788.15 total 20.02 last 121.21)
 - /unlink 3773 (time 1790000798.15 total 30.03 last 122.67)
total: 4500 /unlink in 35.62 seconds: 126.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 673 (time 1790000818.36 total 10.01 last 67.23)
 - open/close 1363 (time 1790000828.37 total 20.01 last 68.98)
 - open/close 1981 (time 1790000838.38 total 30.03 last 61.71)
 - open/close 2701 (time 1790000848.39 total 40.03 last 71.96)
 - open/close 3359 (time 1790000858.39 total 50.04 last 65.77)
 - open/close 4045 (time 1790000868.39 total 60.04 last 68.59)
total: 4500 open/close in 66.63 seconds: 67.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1121 (time 1790000891.15 total 10.01 last 111.99)
 - /unlink 2371 (time 1790000901.15 total 20.01 last 124.99)
 - /unlink 3458 (time 1790000911.15 total 30.02 last 108.64)
total: 4500 /unlink in 38.82 seconds: 115.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 672 (time 1790000935.38 total 10.01 last 67.15)
 - open/close 1401 (time 1790000945.39 total 20.02 last 72.79)
 - open/close 2120 (time 1790000955.40 total 30.03 last 71.83)
 - open/close 2857 (time 1790000965.42 total 40.05 last 73.59)
 - open/close 3579 (time 1790000975.43 total 50.06 last 72.13)
 - open/close 4293 (time 1790000985.45 total 60.08 last 71.27)
total: 4500 open/close in 63.10 seconds: 71.32 ops/second
 - /unlink 1349 (time 1790001002.58 total 10.01 last 134.75)
 - /unlink 2680 (time 1790001012.58 total 20.01 last 133.06)
 - /unlink 4055 (time 1790001022.58 total 30.02 last 137.46)
total: 4500 /unlink in 32.65 seconds: 137.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 634 (time 1790001039.93 total 10.01 last 63.34)
 - open/close 1377 (time 1790001049.95 total 20.02 last 74.21)
 - open/close 2122 (time 1790001059.96 total 30.03 last 74.42)
 - open/close 2842 (time 1790001069.96 total 40.03 last 71.99)
 - open/close 3525 (time 1790001079.97 total 50.05 last 68.21)
 - open/close 4226 (time 1790001089.97 total 60.05 last 70.10)
total: 4500 open/close in 63.95 seconds: 70.36 ops/second
 - /unlink 1323 (time 1790001109.21 total 10.00 last 132.24)
 - /unlink 2565 (time 1790001119.22 total 20.01 last 124.14)
 - /unlink 3806 (time 1790001129.22 total 30.01 last 124.09)
total: 4500 /unlink in 38.03 seconds: 118.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 609 (time 1790001152.58 total 10.01 last 60.86)
 - open/close 1207 (time 1790001162.60 total 20.03 last 59.68)
 - open/close 1924 (time 1790001172.61 total 30.03 last 71.66)
 - open/close 2646 (time 1790001182.61 total 40.04 last 72.15)
 - open/close 3421 (time 1790001192.63 total 50.05 last 77.39)
 - open/close 4158 (time 1790001202.63 total 60.06 last 73.69)
total: 4500 open/close in 64.55 seconds: 69.71 ops/second
 - /unlink 1303 (time 1790001221.36 total 10.01 last 130.22)
 - /unlink 2602 (time 1790001231.37 total 20.01 last 129.85)
 - /unlink 3880 (time 1790001241.38 total 30.02 last 127.70)
total: 4500 /unlink in 34.68 seconds: 129.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 680 (time 1790001259.95 total 10.02 last 67.88)
 - open/close 1351 (time 1790001269.96 total 20.03 last 67.05)
 - open/close 2052 (time 1790001279.96 total 30.03 last 70.09)
 - open/close 2705 (time 1790001289.97 total 40.04 last 65.23)
 - open/close 3433 (time 1790001299.98 total 50.05 last 72.72)
 - open/close 4150 (time 1790001310.00 total 60.06 last 71.61)
total: 4500 open/close in 65.70 seconds: 68.49 ops/second
 - /unlink 1118 (time 1790001330.53 total 10.00 last 111.77)
 - /unlink 1972 (time 1790001340.54 total 20.01 last 85.31)
 - /unlink 3061 (time 1790001350.54 total 30.02 last 108.83)
 - /unlink 4344 (time 1790001360.55 total 40.03 last 128.18)
total: 4500 /unlink in 41.36 seconds: 108.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 698 (time 1790001376.27 total 10.00 last 69.77)
 - open/close 1396 (time 1790001386.27 total 20.01 last 69.76)
 - open/close 2076 (time 1790001396.28 total 30.02 last 67.93)
 - open/close 2750 (time 1790001406.29 total 40.03 last 67.35)
 - open/close 3459 (time 1790001416.30 total 50.04 last 70.85)
 - open/close 4138 (time 1790001426.30 total 60.04 last 67.88)
total: 4500 open/close in 65.52 seconds: 68.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1273 (time 1790001446.61 total 10.00 last 127.26)
 - /unlink 2404 (time 1790001456.61 total 20.01 last 113.06)
 - /unlink 3743 (time 1790001466.61 total 30.01 last 133.90)
total: 4500 /unlink in 35.72 seconds: 125.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 657 (time 1790001486.92 total 10.01 last 65.66)
 - open/close 1352 (time 1790001496.92 total 20.01 last 69.45)
 - open/close 2088 (time 1790001506.93 total 30.02 last 73.58)
 - open/close 2778 (time 1790001516.94 total 40.03 last 68.93)
 - open/close 3436 (time 1790001526.94 total 50.03 last 65.77)
 - open/close 4082 (time 1790001536.96 total 60.05 last 64.49)
total: 4500 open/close in 66.34 seconds: 67.83 ops/second
 - /unlink 1256 (time 1790001558.17 total 10.00 last 125.59)
 - /unlink 2569 (time 1790001568.17 total 20.01 last 131.24)
 - /unlink 3865 (time 1790001578.17 total 30.01 last 129.59)
total: 4500 /unlink in 35.57 seconds: 126.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 654 (time 1790001597.94 total 10.01 last 65.32)
 - open/close 1328 (time 1790001607.95 total 20.02 last 67.34)
 - open/close 1917 (time 1790001617.95 total 30.02 last 58.89)
 - open/close 2618 (time 1790001627.96 total 40.03 last 70.05)
 - open/close 3342 (time 1790001637.98 total 50.05 last 72.26)
 - open/close 4054 (time 1790001647.98 total 60.05 last 71.20)
total: 4500 open/close in 65.80 seconds: 68.38 ops/second
 - /unlink 1298 (time 1790001668.40 total 10.00 last 129.75)
 - /unlink 2493 (time 1790001678.40 total 20.00 last 119.50)
 - /unlink 3766 (time 1790001688.40 total 30.01 last 127.29)
total: 4500 /unlink in 36.64 seconds: 122.83 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11390: 95293 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg344-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg344-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server
stop mds service on oleg344-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server
stop mds service on oleg344-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg344-server
unloading modules via unload_modules_local on: 'oleg344-server'
oleg344-server: oleg344-server.virtnet: executing unload_modules_local
oleg344-server: kmemleak disabled
modules unloaded.