== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:52:58 (1789483978)
Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg251-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg251-server
oleg251-server: oleg251-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg251-server'
oleg251-server: oleg251-server.virtnet: executing load_modules_local
oleg251-server: kmemleak disabled
oleg251-server: Loading modules from /home/green/git/lustre-release/lustre
oleg251-server: detected 4 online CPUs by sysfs
oleg251-server: MODOPTS_LIBCFS=
oleg251-server: Force libcfs to create 2 CPU partitions
oleg251-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg251-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg251-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg251-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all
pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg251-client.virtnet:  -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 519 (time 1789484199.36 total 10.01 last 51.87)
 - open/close 1151 (time 1789484209.36 total 20.01 last 63.15)
 - open/close 1895 (time 1789484219.37 total 30.02 last 74.35)
 - open/close 2743 (time 1789484229.38 total 40.03 last 84.74)
 - open/close 3345 (time 1789484239.38 total 50.03 last 60.17)
 - open/close 4025 (time 1789484249.39 total 60.04 last 67.96)
total: 4500 open/close in 68.11 seconds: 66.07 ops/second
 - /unlink 1368 (time 1789484273.89 total 10.02 last 136.50)
 - /unlink 2641 (time 1789484283.90 total 20.02 last 127.27)
 - /unlink 3918 (time 1789484293.90 total 30.03 last 127.62)
total: 4500 /unlink in 34.17 seconds: 131.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 722 (time 1789484312.65 total 10.01 last 72.11)
 - open/close 1450 (time 1789484322.65 total 20.02 last 72.75)
 - open/close 2203 (time 1789484332.66 total 30.03 last 75.25)
 - open/close 2935 (time 1789484342.68 total 40.04 last 73.08)
 - open/close 3599 (time 1789484352.70 total 50.06 last 66.26)
 - open/close 4370 (time 1789484362.71 total 60.08 last 77.00)
total: 4500 open/close in 62.17 seconds: 72.38 ops/second
 - /unlink 1254 (time 1789484378.69 total 10.00 last 125.34)
 - /unlink 2360 (time 1789484388.70 total 20.01 last 110.57)
 - /unlink 3781 (time 1789484398.70 total 30.01 last 142.08)
total: 4500 /unlink in 34.62 seconds: 129.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 852 (time 1789484418.45 total 10.00 last 85.17)
 - open/close 1569 (time 1789484428.46 total 20.01 last 71.63)
 - open/close 2181 (time 1789484438.47 total 30.02 last 61.18)
 - open/close 2955 (time 1789484448.48 total 40.03 last 77.30)
 - open/close 3791 (time 1789484458.50 total 50.05 last 83.45)
total: 4500 open/close in 59.99 seconds: 75.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1664 (time 1789484482.55 total 10.01 last 166.31)
 - /unlink 3154 (time 1789484492.56 total 20.02 last 148.84)
total: 4500 /unlink in 29.80 seconds: 151.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 754 (time 1789484517.16 total 10.00 last 75.39)
 - open/close 1461 (time 1789484527.17 total 20.02 last 70.58)
 - open/close 2148 (time 1789484537.18 total 30.02 last 68.68)
 - open/close 2882 (time 1789484547.18 total 40.03 last 73.36)
 - open/close 3714 (time 1789484557.19 total 50.03 last 83.15)
total: 4500 open/close in 59.58 seconds: 75.52 ops/second
 - /unlink 1719 (time 1789484581.49 total 10.00 last 171.84)
 - /unlink 3172 (time 1789484591.50 total 20.01 last 145.16)
total: 4500 /unlink in 29.82 seconds: 150.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 805 (time 1789484616.72 total 10.00 last 80.49)
 - open/close 1452 (time 1789484626.73 total 20.01 last 64.61)
 - open/close 2216 (time 1789484636.73 total 30.02 last 76.39)
 - open/close 2997 (time 1789484646.75 total 40.03 last 77.99)
 - open/close 3735 (time 1789484656.76 total 50.04 last 73.73)
total: 4500 open/close in 59.41 seconds: 75.75 ops/second
 - /unlink 1595 (time 1789484680.25 total 10.02 last 159.22)
 - /unlink 3285 (time 1789484690.26 total 20.02 last 168.93)
total: 4500 /unlink in 27.96 seconds: 160.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 771 (time 1789484712.96 total 10.01 last 77.04)
 - open/close 1511 (time 1789484722.97 total 20.01 last 73.96)
 - open/close 2105 (time 1789484732.98 total 30.02 last 59.36)
 - open/close 2635 (time 1789484742.98 total 40.02 last 52.98)
 - open/close 3233 (time 1789484753.01 total 50.05 last 59.65)
 - open/close 4032 (time 1789484763.01 total 60.05 last 79.90)
total: 4500 open/close in 67.07 seconds: 67.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1355 (time 1789484785.95 total 10.01 last 135.37)
 - /unlink 2319 (time 1789484795.95 total 20.01 last 96.37)
 - /unlink 3194 (time 1789484805.96 total 30.02 last 87.43)
 - /unlink 4214 (time 1789484815.97 total 40.03 last 101.96)
total: 4500 /unlink in 42.67 seconds: 105.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 643 (time 1789484834.60 total 10.02 last 64.16)
 - open/close 1485 (time 1789484844.61 total 20.03 last 84.14)
 - open/close 2204 (time 1789484854.62 total 30.04 last 71.79)
 - open/close 3004 (time 1789484864.63 total 40.05 last 79.95)
 - open/close 3699 (time 1789484874.64 total 50.06 last 69.45)
 - open/close 4492 (time 1789484884.64 total 60.06 last 79.26)
total: 4500 open/close in 60.23 seconds: 74.72 ops/second
 - /unlink 1453 (time 1789484900.36 total 10.00 last 145.25)
 - /unlink 2685 (time 1789484910.36 total 20.01 last 123.17)
 - /unlink 3853 (time 1789484920.36 total 30.01 last 116.80)
total: 4500 /unlink in 34.70 seconds: 129.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 696 (time 1789484939.65 total 10.00 last 69.59)
 - open/close 1351 (time 1789484949.65 total 20.01 last 65.46)
 - open/close 1955 (time 1789484959.65 total 30.01 last 60.39)
 - open/close 2484 (time 1789484969.66 total 40.01 last 52.89)
 - open/close 3176 (time 1789484979.66 total 50.01 last 69.17)
 - open/close 3959 (time 1789484989.66 total 60.02 last 78.28)
total: 4500 open/close in 66.90 seconds: 67.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1500 (time 1789485011.88 total 10.01 last 149.88)
 - /unlink 2852 (time 1789485021.88 total 20.01 last 135.20)
 - /unlink 4282 (time 1789485031.88 total 30.01 last 142.98)
total: 4500 /unlink in 31.87 seconds: 141.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 655 (time 1789485049.24 total 10.00 last 65.49)
 - open/close 1425 (time 1789485059.24 total 20.00 last 76.99)
 - open/close 2221 (time 1789485069.26 total 30.02 last 79.49)
 - open/close 2969 (time 1789485079.27 total 40.02 last 74.73)
 - open/close 3764 (time 1789485089.27 total 50.03 last 79.45)
 - open/close 4455 (time 1789485099.28 total 60.04 last 69.07)
total: 4500 open/close in 60.94 seconds: 73.84 ops/second
 - /unlink 1371 (time 1789485115.52 total 10.00 last 137.05)
 - /unlink 2970 (time 1789485125.53 total 20.01 last 159.83)
 - /unlink 4208 (time 1789485135.53 total 30.01 last 123.79)
total: 4500 /unlink in 32.05 seconds: 140.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 630 (time 1789485152.68 total 10.01 last 62.95)
 - open/close 1237 (time 1789485162.69 total 20.02 last 60.63)
 - open/close 1966 (time 1789485172.70 total 30.03 last 72.80)
 - open/close 2613 (time 1789485182.71 total 40.04 last 64.65)
 - open/close 3422 (time 1789485192.72 total 50.05 last 80.81)
 - open/close 4251 (time 1789485202.72 total 60.06 last 82.87)
total: 4500 open/close in 63.73 seconds: 70.61 ops/second
 - /unlink 849 (time 1789485225.51 total 10.03 last 84.68)
 - /unlink 1867 (time 1789485235.52 total 20.03 last 101.76)
 - /unlink 2597 (time 1789485245.52 total 30.04 last 72.94)
 - /unlink 3374 (time 1789485255.53 total 40.04 last 77.67)
 - /unlink 4330 (time 1789485265.53 total 50.05 last 95.54)
total: 4500 /unlink in 51.95 seconds: 86.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 645 (time 1789485283.70 total 10.01 last 64.44)
 - open/close 1317 (time 1789485293.71 total 20.02 last 67.12)
 - open/close 2072 (time 1789485303.71 total 30.02 last 75.49)
 - open/close 2727 (time 1789485313.72 total 40.03 last 65.45)
 - open/close 3368 (time 1789485323.73 total 50.05 last 64.01)
 - open/close 4056 (time 1789485333.74 total 60.05 last 68.77)
total: 4500 open/close in 65.87 seconds: 68.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1446 (time 1789485354.44 total 10.01 last 144.51)
 - /unlink 2983 (time 1789485364.45 total 20.01 last 153.65)
 - /unlink 4389 (time 1789485374.45 total 30.01 last 140.54)
total: 4500 /unlink in 30.95 seconds: 145.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 666 (time 1789485390.20 total 10.01 last 66.55)
 - open/close 1402 (time 1789485400.22 total 20.04 last 73.40)
 - open/close 2198 (time 1789485410.22 total 30.04 last 79.59)
 - open/close 2958 (time 1789485420.22 total 40.04 last 76.00)
 - open/close 3663 (time 1789485430.23 total 50.04 last 70.46)
 - open/close 4444 (time 1789485440.24 total 60.05 last 78.03)
total: 4500 open/close in 60.82 seconds: 73.99 ops/second
 - /unlink 1243 (time 1789485455.92 total 10.01 last 124.17)
 - /unlink 2421 (time 1789485465.92 total 20.01 last 117.77)
 - /unlink 3608 (time 1789485475.93 total 30.03 last 118.56)
total: 4500 /unlink in 37.52 seconds: 119.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 624 (time 1789485499.59 total 10.00 last 62.37)
 - open/close 1448 (time 1789485509.60 total 20.01 last 82.37)
 - open/close 2157 (time 1789485519.61 total 30.02 last 70.81)
 - open/close 2700 (time 1789485529.61 total 40.02 last 54.29)
 - open/close 3359 (time 1789485539.63 total 50.04 last 65.80)
 - open/close 4022 (time 1789485549.63 total 60.04 last 66.26)
total: 4500 open/close in 66.80 seconds: 67.36 ops/second
 - /unlink 1158 (time 1789485572.64 total 10.00 last 115.76)
 - /unlink 2364 (time 1789485582.65 total 20.01 last 120.50)
 - /unlink 3609 (time 1789485592.66 total 30.02 last 124.39)
total: 4500 /unlink in 37.11 seconds: 121.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 682 (time 1789485614.99 total 10.02 last 68.08)
 - open/close 1440 (time 1789485625.00 total 20.02 last 75.76)
 - open/close 2185 (time 1789485635.00 total 30.03 last 74.48)
 - open/close 2965 (time 1789485645.01 total 40.03 last 77.97)
 - open/close 3699 (time 1789485655.02 total 50.04 last 73.34)
 - open/close 4460 (time 1789485665.03 total 60.05 last 75.97)
total: 4500 open/close in 60.51 seconds: 74.37 ops/second
 - /unlink 1571 (time 1789485679.40 total 10.01 last 157.01)
 - /unlink 2842 (time 1789485689.40 total 20.01 last 127.04)
 - /unlink 3781 (time 1789485699.41 total 30.02 last 93.84)
total: 4500 /unlink in 37.49 seconds: 120.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 681 (time 1789485723.74 total 10.01 last 68.06)
 - open/close 1325 (time 1789485733.76 total 20.02 last 64.31)
 - open/close 2049 (time 1789485743.76 total 30.02 last 72.37)
 - open/close 2711 (time 1789485753.79 total 40.05 last 66.01)
 - open/close 3401 (time 1789485763.80 total 50.06 last 68.97)
 - open/close 4191 (time 1789485773.81 total 60.07 last 78.89)
total: 4500 open/close in 64.26 seconds: 70.03 ops/second
 - /unlink 1341 (time 1789485792.42 total 10.00 last 134.08)
 - /unlink 2577 (time 1789485802.42 total 20.00 last 123.59)
 - /unlink 3724 (time 1789485812.43 total 30.01 last 114.66)
total: 4500 /unlink in 36.36 seconds: 123.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 669 (time 1789485834.66 total 10.01 last 66.82)
 - open/close 1250 (time 1789485844.66 total 20.01 last 58.08)
 - open/close 2006 (time 1789485854.67 total 30.02 last 75.53)
 - open/close 2792 (time 1789485864.67 total 40.02 last 78.60)
 - open/close 3522 (time 1789485874.68 total 50.03 last 72.94)
 - open/close 4243 (time 1789485884.69 total 60.04 last 72.05)
total: 4500 open/close in 63.55 seconds: 70.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1507 (time 1789485902.30 total 10.00 last 150.67)
 - /unlink 2754 (time 1789485912.30 total 20.00 last 124.69)
 - /unlink 4143 (time 1789485922.30 total 30.01 last 138.86)
total: 4500 /unlink in 32.83 seconds: 137.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 743 (time 1789485939.91 total 10.00 last 74.28)
 - open/close 1455 (time 1789485949.92 total 20.01 last 71.15)
 - open/close 2179 (time 1789485959.93 total 30.02 last 72.36)
 - open/close 2985 (time 1789485969.93 total 40.02 last 80.57)
 - open/close 3693 (time 1789485979.94 total 50.03 last 70.72)
 - open/close 4411 (time 1789485989.95 total 60.04 last 71.74)
total: 4500 open/close in 61.32 seconds: 73.39 ops/second
 - /unlink 1316 (time 1789486006.46 total 10.00 last 131.59)
 - /unlink 2712 (time 1789486016.46 total 20.00 last 139.58)
 - /unlink 4253 (time 1789486026.46 total 30.00 last 154.07)
total: 4500 /unlink in 31.67 seconds: 142.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 717 (time 1789486043.03 total 10.00 last 71.68)
 - open/close 1338 (time 1789486053.04 total 20.01 last 62.07)
 - open/close 1922 (time 1789486063.04 total 30.02 last 58.35)
 - open/close 2514 (time 1789486073.06 total 40.03 last 59.10)
 - open/close 3199 (time 1789486083.07 total 50.04 last 68.41)
 - open/close 3832 (time 1789486093.08 total 60.05 last 63.27)
total: 4500 open/close in 69.22 seconds: 65.01 ops/second
 - /unlink 1475 (time 1789486117.42 total 10.03 last 147.06)
 - /unlink 2851 (time 1789486127.42 total 20.03 last 137.56)
 - /unlink 4287 (time 1789486137.42 total 30.03 last 143.59)
total: 4500 /unlink in 31.74 seconds: 141.78 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11357: 95191 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg251-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg251-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server
stop mds service on oleg251-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server
stop mds service on oleg251-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server
unloading modules via unload_modules_local on: 'oleg251-server'
oleg251-server: oleg251-server.virtnet: executing unload_modules_local
oleg251-server: kmemleak disabled
modules unloaded.