== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:05:38 (1788473138)
Stopping clients: oleg123-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg123-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg123-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg123-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg123-server
oleg123-server: oleg123-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg123-server'
oleg123-server: oleg123-server.virtnet: executing load_modules_local
oleg123-server: kmemleak disabled
oleg123-server: Loading modules from /home/green/git/lustre-release/lustre
oleg123-server: detected 4 online CPUs by sysfs
oleg123-server: MODOPTS_LIBCFS=
oleg123-server: Force libcfs to create 2 CPU partitions
oleg123-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg123-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all
pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg123-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all
pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg123-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all
pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg123-client.virtnet:  -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 635 (time 1788473371.86 total 10.01 last 63.42)
 - open/close 1377 (time 1788473381.88 total 20.03 last 74.09)
 - open/close 2202 (time 1788473391.89 total 30.04 last 82.38)
 - open/close 2938 (time 1788473401.90 total 40.04 last 73.58)
 - open/close 3703 (time 1788473411.90 total 50.05 last 76.45)
 - open/close 4363 (time 1788473421.92 total 60.07 last 65.87)
total: 4500 open/close in 62.11 seconds: 72.45 ops/second
 - /unlink 1202 (time 1788473438.68 total 10.00 last 120.17)
 - /unlink 2552 (time 1788473448.68 total 20.00 last 134.98)
 - /unlink 3730 (time 1788473458.69 total 30.01 last 117.71)
total: 4500 /unlink in 36.70 seconds: 122.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 779 (time 1788473480.57 total 10.01 last 77.84)
 - open/close 1497 (time 1788473490.59 total 20.02 last 71.71)
 - open/close 2229 (time 1788473500.60 total 30.03 last 73.14)
 - open/close 2922 (time 1788473510.61 total 40.04 last 69.20)
 - open/close 3676 (time 1788473520.61 total 50.04 last 75.39)
 - open/close 4289 (time 1788473530.62 total 60.05 last 61.24)
total: 4500 open/close in 63.16 seconds: 71.24 ops/second
 - /unlink 1279 (time 1788473549.01 total 10.00 last 127.84)
 - /unlink 2638 (time 1788473559.01 total 20.01 last 135.82)
 - /unlink 3935 (time 1788473569.02 total 30.02 last 129.63)
total: 4500 /unlink in 34.12 seconds: 131.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 710 (time 1788473588.75 total 10.01 last 70.96)
 - open/close 1442 (time 1788473598.79 total 20.05 last 72.90)
 - open/close 2165 (time 1788473608.80 total 30.06 last 72.23)
 - open/close 2979 (time 1788473618.81 total 40.07 last 81.32)
 - open/close 3683 (time 1788473628.84 total 50.09 last 70.22)
 - open/close 4412 (time 1788473638.84 total 60.09 last 72.88)
total: 4500 open/close in 61.26 seconds: 73.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1481 (time 1788473654.23 total 10.00 last 148.03)
 - /unlink 2618 (time 1788473664.23 total 20.01 last 113.69)
 - /unlink 3825 (time 1788473674.23 total 30.01 last 120.66)
total: 4500 /unlink in 35.25 seconds: 127.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 842 (time 1788473694.80 total 10.00 last 84.19)
 - open/close 1465 (time 1788473704.81 total 20.00 last 62.29)
 - open/close 2123 (time 1788473714.81 total 30.01 last 65.78)
 - open/close 2788 (time 1788473724.81 total 40.01 last 66.48)
 - open/close 3656 (time 1788473734.82 total 50.02 last 86.73)
 - open/close 4436 (time 1788473744.84 total 60.04 last 77.81)
total: 4500 open/close in 60.80 seconds: 74.01 ops/second
 - /unlink 1421 (time 1788473759.92 total 10.00 last 142.03)
 - /unlink 2765 (time 1788473769.93 total 20.01 last 134.33)
 - /unlink 3812 (time 1788473779.94 total 30.02 last 104.61)
total: 4500 /unlink in 39.30 seconds: 114.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 744 (time 1788473804.84 total 10.03 last 74.21)
 - open/close 1516 (time 1788473814.85 total 20.03 last 77.15)
 - open/close 2319 (time 1788473824.87 total 30.05 last 80.18)
 - open/close 3077 (time 1788473834.88 total 40.06 last 75.71)
 - open/close 3825 (time 1788473844.88 total 50.06 last 74.75)
total: 4500 open/close in 59.92 seconds: 75.10 ops/second
 - /unlink 1293 (time 1788473869.97 total 10.01 last 129.22)
 - /unlink 2648 (time 1788473879.97 total 20.01 last 135.50)
 - /unlink 4190 (time 1788473889.97 total 30.01 last 154.18)
total: 4500 /unlink in 32.42 seconds: 138.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 706 (time 1788473909.01 total 10.01 last 70.53)
 - open/close 1426 (time 1788473919.03 total 20.03 last 71.84)
 - open/close 2234 (time 1788473929.04 total 30.04 last 80.72)
 - open/close 2997 (time 1788473939.07 total 40.07 last 76.12)
 - open/close 3753 (time 1788473949.10 total 50.10 last 75.36)
 - open/close 4441 (time 1788473959.12 total 60.11 last 68.68)
total: 4500 open/close in 60.98 seconds: 73.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1250 (time 1788473975.64 total 10.00 last 124.99)
 - /unlink 2781 (time 1788473985.65 total 20.01 last 152.93)
 - /unlink 4194 (time 1788473995.66 total 30.02 last 141.25)
total: 4500 /unlink in 32.39 seconds: 138.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 723 (time 1788474012.75 total 10.00 last 72.27)
 - open/close 1495 (time 1788474022.76 total 20.01 last 77.14)
 - open/close 2264 (time 1788474032.77 total 30.03 last 76.77)
 - open/close 2969 (time 1788474042.79 total 40.04 last 70.40)
 - open/close 3576 (time 1788474052.79 total 50.05 last 60.66)
 - open/close 4141 (time 1788474062.80 total 60.05 last 56.48)
total: 4500 open/close in 65.46 seconds: 68.75 ops/second
 - /unlink 1261 (time 1788474083.62 total 10.00 last 126.05)
 - /unlink 2517 (time 1788474093.63 total 20.01 last 125.47)
 - /unlink 3653 (time 1788474103.64 total 30.02 last 113.58)
total: 4500 /unlink in 37.64 seconds: 119.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 752 (time 1788474126.67 total 10.02 last 75.03)
 - open/close 1253 (time 1788474136.71 total 20.06 last 49.90)
 - open/close 1774 (time 1788474146.75 total 30.10 last 51.91)
 - open/close 2389 (time 1788474156.76 total 40.11 last 61.45)
 - open/close 3075 (time 1788474166.76 total 50.11 last 68.55)
 - open/close 3889 (time 1788474176.77 total 60.12 last 81.38)
total: 4500 open/close in 68.53 seconds: 65.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1236 (time 1788474199.80 total 10.00 last 123.57)
 - /unlink 2640 (time 1788474209.81 total 20.01 last 140.36)
 - /unlink 4153 (time 1788474219.82 total 30.02 last 151.15)
total: 4500 /unlink in 33.11 seconds: 135.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 689 (time 1788474238.38 total 10.00 last 68.87)
 - open/close 1365 (time 1788474248.38 total 20.01 last 67.57)
 - open/close 2161 (time 1788474258.39 total 30.02 last 79.53)
 - open/close 2851 (time 1788474268.40 total 40.02 last 68.98)
 - open/close 3522 (time 1788474278.41 total 50.03 last 67.04)
 - open/close 4154 (time 1788474288.41 total 60.03 last 63.18)
total: 4500 open/close in 66.61 seconds: 67.56 ops/second
 - /unlink 1469 (time 1788474310.30 total 10.00 last 146.88)
 - /unlink 2851 (time 1788474320.30 total 20.00 last 138.18)
 - /unlink 4114 (time 1788474330.31 total 30.01 last 126.27)
total: 4500 /unlink in 33.45 seconds: 134.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 616 (time 1788474348.82 total 10.00 last 61.59)
 - open/close 1239 (time 1788474358.83 total 20.00 last 62.28)
 - open/close 1816 (time 1788474368.84 total 30.01 last 57.65)
 - open/close 2350 (time 1788474378.84 total 40.02 last 53.37)
 - open/close 2971 (time 1788474388.86 total 50.04 last 62.00)
 - open/close 3653 (time 1788474398.87 total 60.04 last 68.14)
 - open/close 4239 (time 1788474408.87 total 70.05 last 58.58)
total: 4500 open/close in 74.54 seconds: 60.37 ops/second
 - /unlink 1293 (time 1788474429.91 total 10.01 last 129.19)
 - /unlink 2550 (time 1788474439.91 total 20.01 last 125.64)
 - /unlink 3841 (time 1788474449.92 total 30.02 last 129.04)
total: 4500 /unlink in 35.48 seconds: 126.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 601 (time 1788474471.52 total 10.02 last 59.98)
 - open/close 1219 (time 1788474481.53 total 20.03 last 61.76)
 - open/close 1904 (time 1788474491.54 total 30.04 last 68.43)
 - open/close 2596 (time 1788474501.54 total 40.04 last 69.18)
 - open/close 3286 (time 1788474511.55 total 50.05 last 68.94)
 - open/close 3975 (time 1788474521.55 total 60.05 last 68.88)
total: 4500 open/close in 68.03 seconds: 66.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1283 (time 1788474545.12 total 10.00 last 128.26)
 - /unlink 2409 (time 1788474555.13 total 20.01 last 112.49)
 - /unlink 3434 (time 1788474565.13 total 30.02 last 102.47)
 - /unlink 4472 (time 1788474575.13 total 40.02 last 103.79)
total: 4500 /unlink in 40.39 seconds: 111.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 678 (time 1788474590.85 total 10.00 last 67.78)
 - open/close 1371 (time 1788474600.86 total 20.01 last 69.27)
 - open/close 1998 (time 1788474610.86 total 30.01 last 62.70)
 - open/close 2609 (time 1788474620.86 total 40.01 last 61.07)
 - open/close 3176 (time 1788474630.87 total 50.02 last 56.68)
 - open/close 3717 (time 1788474640.88 total 60.03 last 54.04)
 - open/close 4358 (time 1788474650.89 total 70.04 last 64.04)
total: 4500 open/close in 72.34 seconds: 62.20 ops/second
 - /unlink 1248 (time 1788474669.38 total 10.01 last 124.73)
 - /unlink 2523 (time 1788474679.38 total 20.01 last 127.49)
 - /unlink 3576 (time 1788474689.39 total 30.01 last 105.25)
total: 4500 /unlink in 38.85 seconds: 115.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 502 (time 1788474714.52 total 10.01 last 50.14)
 - open/close 997 (time 1788474724.53 total 20.02 last 49.46)
 - open/close 1519 (time 1788474734.55 total 30.04 last 52.09)
 - open/close 2002 (time 1788474744.59 total 40.08 last 48.12)
 - open/close 2494 (time 1788474754.67 total 50.16 last 48.83)
 - open/close 3021 (time 1788474764.67 total 60.16 last 52.67)
 - open/close 3564 (time 1788474774.68 total 70.17 last 54.23)
 - open/close 4193 (time 1788474784.69 total 80.18 last 62.85)
total: 4500 open/close in 85.75 seconds: 52.48 ops/second
 - /unlink 1267 (time 1788474805.83 total 10.00 last 126.67)
 - /unlink 2337 (time 1788474815.83 total 20.00 last 106.99)
 - /unlink 3538 (time 1788474825.83 total 30.01 last 120.06)
total: 4500 /unlink in 38.38 seconds: 117.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 627 (time 1788474850.35 total 10.01 last 62.63)
 - open/close 1303 (time 1788474860.37 total 20.03 last 67.50)
 - open/close 1879 (time 1788474870.39 total 30.05 last 57.48)
 - open/close 2619 (time 1788474880.39 total 40.05 last 73.97)
 - open/close 3422 (time 1788474890.43 total 50.08 last 80.03)
 - open/close 4216 (time 1788474900.43 total 60.09 last 79.34)
total: 4500 open/close in 63.77 seconds: 70.57 ops/second
 - /unlink 1342 (time 1788474918.43 total 10.00 last 134.15)
 - /unlink 2593 (time 1788474928.43 total 20.01 last 125.03)
 - /unlink 3910 (time 1788474938.43 total 30.01 last 131.69)
total: 4500 /unlink in 34.66 seconds: 129.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 660 (time 1788474957.79 total 10.00 last 65.99)
 - open/close 1275 (time 1788474967.80 total 20.01 last 61.43)
 - open/close 1961 (time 1788474977.80 total 30.01 last 68.59)
 - open/close 2565 (time 1788474987.82 total 40.03 last 60.31)
 - open/close 3227 (time 1788474997.83 total 50.04 last 66.14)
 - open/close 3978 (time 1788475007.85 total 60.05 last 74.97)
total: 4500 open/close in 67.63 seconds: 66.54 ops/second
 - /unlink 1215 (time 1788475030.30 total 10.00 last 121.48)
 - /unlink 2429 (time 1788475040.30 total 20.00 last 121.37)
 - /unlink 3564 (time 1788475050.30 total 30.00 last 113.49)
total: 4500 /unlink in 38.82 seconds: 115.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 664 (time 1788475074.54 total 10.00 last 66.40)
 - open/close 1379 (time 1788475084.57 total 20.03 last 71.31)
 - open/close 2138 (time 1788475094.57 total 30.03 last 75.87)
 - open/close 2928 (time 1788475104.57 total 40.03 last 79.00)
 - open/close 3719 (time 1788475114.58 total 50.04 last 79.01)
total: 4500 open/close in 59.35 seconds: 75.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1420 (time 1788475138.71 total 10.00 last 141.97)
 - /unlink 2634 (time 1788475148.71 total 20.01 last 121.35)
 - /unlink 3981 (time 1788475158.72 total 30.01 last 134.66)
total: 4500 /unlink in 33.92 seconds: 132.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 747 (time 1788475177.67 total 10.00 last 74.67)
 - open/close 1466 (time 1788475187.68 total 20.01 last 71.88)
 - open/close 2087 (time 1788475197.68 total 30.02 last 62.05)
 - open/close 2696 (time 1788475207.69 total 40.02 last 60.85)
 - open/close 3194 (time 1788475217.73 total 50.06 last 49.62)
 - open/close 3786 (time 1788475227.73 total 60.06 last 59.20)
 - open/close 4336 (time 1788475237.74 total 70.07 last 54.95)
total: 4500 open/close in 72.72 seconds: 61.88 ops/second
 - /unlink 1244 (time 1788475254.68 total 10.00 last 124.35)
 - /unlink 2408 (time 1788475264.68 total 20.01 last 116.34)
 - /unlink 3709 (time 1788475274.68 total 30.01 last 130.06)
total: 4500 /unlink in 37.40 seconds: 120.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 503 (time 1788475298.78 total 10.01 last 50.26)
 - open/close 1218 (time 1788475308.79 total 20.01 last 71.47)
 - open/close 1882 (time 1788475318.80 total 30.02 last 66.35)
 - open/close 2520 (time 1788475328.80 total 40.03 last 63.74)
 - open/close 3062 (time 1788475338.82 total 50.04 last 54.13)
 - open/close 3531 (time 1788475348.85 total 60.07 last 46.76)
 - open/close 4066 (time 1788475358.85 total 70.07 last 53.50)
total: 4500 open/close in 74.99 seconds: 60.01 ops/second
 - /unlink 1232 (time 1788475378.88 total 10.00 last 123.20)
 - /unlink 2257 (time 1788475388.89 total 20.01 last 102.36)
 - /unlink 3251 (time 1788475398.90 total 30.02 last 99.31)
 - /unlink 4261 (time 1788475408.90 total 40.03 last 100.95)
total: 4500 /unlink in 42.44 seconds: 106.03 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95375 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg123-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg123-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server
stop mds service on oleg123-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server
stop mds service on oleg123-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg123-server
unloading modules via unload_modules_local on: 'oleg123-server'
oleg123-server: oleg123-server.virtnet: executing unload_modules_local
oleg123-server: kmemleak disabled
modules unloaded.