== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:59:28 (1789286368)
Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg457-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg457-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg457-server
oleg457-server: oleg457-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg457-server'
oleg457-server: oleg457-server.virtnet: executing load_modules_local
oleg457-server: kmemleak disabled
oleg457-server: Loading modules from /home/green/git/lustre-release/lustre
oleg457-server: detected 4 online CPUs by sysfs
oleg457-server: MODOPTS_LIBCFS=
oleg457-server: Force libcfs to create 2 CPU partitions
oleg457-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg457-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all
pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg457-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all
pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg457-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all
pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg457-client.virtnet:  -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 562 (time 1789286587.88 total 10.02 last 56.08)
 - open/close 1223 (time 1789286597.89 total 20.02 last 66.09)
 - open/close 1947 (time 1789286607.89 total 30.02 last 72.40)
 - open/close 2626 (time 1789286617.89 total 40.03 last 67.88)
 - open/close 3361 (time 1789286627.91 total 50.04 last 73.39)
 - open/close 3892 (time 1789286637.91 total 60.05 last 53.07)
 - open/close 4480 (time 1789286647.91 total 70.05 last 58.78)
total: 4500 open/close in 70.35 seconds: 63.97 ops/second
 - /unlink 1354 (time 1789286663.16 total 10.00 last 135.39)
 - /unlink 2659 (time 1789286673.17 total 20.01 last 130.40)
 - /unlink 3992 (time 1789286683.17 total 30.01 last 133.28)
total: 4500 /unlink in 33.60 seconds: 133.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 767 (time 1789286702.14 total 10.00 last 76.69)
 - open/close 1403 (time 1789286712.15 total 20.01 last 63.55)
 - open/close 2063 (time 1789286722.16 total 30.02 last 65.94)
 - open/close 2735 (time 1789286732.17 total 40.02 last 67.17)
 - open/close 3338 (time 1789286742.18 total 50.04 last 60.22)
 - open/close 3965 (time 1789286752.18 total 60.04 last 62.67)
total: 4500 open/close in 67.61 seconds: 66.56 ops/second
 - /unlink 1242 (time 1789286774.59 total 10.00 last 124.18)
 - /unlink 2567 (time 1789286784.59 total 20.00 last 132.47)
 - /unlink 3784 (time 1789286794.59 total 30.00 last 121.69)
total: 4500 /unlink in 36.50 seconds: 123.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 608 (time 1789286817.41 total 10.01 last 60.74)
 - open/close 1196 (time 1789286827.42 total 20.02 last 58.77)
 - open/close 1822 (time 1789286837.42 total 30.02 last 62.58)
 - open/close 2529 (time 1789286847.42 total 40.02 last 70.69)
 - open/close 3231 (time 1789286857.43 total 50.03 last 70.14)
 - open/close 3854 (time 1789286867.43 total 60.03 last 62.29)
total: 4500 open/close in 69.78 seconds: 64.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1213 (time 1789286892.74 total 10.01 last 121.23)
 - /unlink 2286 (time 1789286902.74 total 20.01 last 107.27)
 - /unlink 3414 (time 1789286912.74 total 30.01 last 112.78)
total: 4500 /unlink in 38.87 seconds: 115.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 634 (time 1789286937.15 total 10.00 last 63.38)
 - open/close 1391 (time 1789286947.16 total 20.02 last 75.60)
 - open/close 2064 (time 1789286957.17 total 30.02 last 67.25)
 - open/close 2714 (time 1789286967.17 total 40.03 last 64.99)
 - open/close 3526 (time 1789286977.18 total 50.04 last 81.13)
 - open/close 4221 (time 1789286987.18 total 60.04 last 69.48)
total: 4500 open/close in 64.71 seconds: 69.55 ops/second
 - /unlink 1291 (time 1789287007.52 total 10.01 last 129.02)
 - /unlink 2600 (time 1789287017.54 total 20.02 last 130.74)
 - /unlink 3831 (time 1789287027.54 total 30.02 last 123.10)
total: 4500 /unlink in 36.75 seconds: 122.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 555 (time 1789287049.66 total 10.00 last 55.49)
 - open/close 1074 (time 1789287059.67 total 20.01 last 51.86)
 - open/close 1731 (time 1789287069.67 total 30.02 last 65.66)
 - open/close 2296 (time 1789287079.69 total 40.03 last 56.42)
 - open/close 2907 (time 1789287089.69 total 50.04 last 61.07)
 - open/close 3503 (time 1789287099.69 total 60.04 last 59.59)
 - open/close 4103 (time 1789287109.70 total 70.05 last 59.94)
total: 4500 open/close in 76.15 seconds: 59.09 ops/second
 - /unlink 1193 (time 1789287130.40 total 10.00 last 119.28)
 - /unlink 2147 (time 1789287140.41 total 20.00 last 95.38)
 - /unlink 3374 (time 1789287150.41 total 30.01 last 122.62)
total: 4500 /unlink in 39.32 seconds: 114.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 627 (time 1789287175.05 total 10.01 last 62.65)
 - open/close 1296 (time 1789287185.06 total 20.01 last 66.88)
 - open/close 1997 (time 1789287195.06 total 30.01 last 70.10)
 - open/close 2649 (time 1789287205.07 total 40.02 last 65.13)
 - open/close 3256 (time 1789287215.07 total 50.02 last 60.68)
 - open/close 3833 (time 1789287225.07 total 60.03 last 57.70)
 - open/close 4418 (time 1789287235.09 total 70.04 last 58.39)
total: 4500 open/close in 71.50 seconds: 62.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1157 (time 1789287251.84 total 10.01 last 115.60)
 - /unlink 2325 (time 1789287261.84 total 20.01 last 116.76)
 - /unlink 3575 (time 1789287271.85 total 30.02 last 124.90)
total: 4500 /unlink in 38.46 seconds: 117.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 610 (time 1789287296.72 total 10.00 last 60.99)
 - open/close 1243 (time 1789287306.73 total 20.01 last 63.26)
 - open/close 1846 (time 1789287316.77 total 30.05 last 60.03)
 - open/close 2433 (time 1789287326.78 total 40.06 last 58.67)
 - open/close 3066 (time 1789287336.78 total 50.06 last 63.26)
 - open/close 3770 (time 1789287346.80 total 60.08 last 70.30)
 - open/close 4433 (time 1789287356.80 total 70.08 last 66.30)
total: 4500 open/close in 71.47 seconds: 62.96 ops/second
 - /unlink 1052 (time 1789287373.65 total 10.01 last 105.11)
 - /unlink 2093 (time 1789287383.65 total 20.01 last 104.06)
 - /unlink 3296 (time 1789287393.66 total 30.02 last 120.25)
total: 4500 /unlink in 39.77 seconds: 113.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 393 (time 1789287419.56 total 10.01 last 39.25)
 - open/close 859 (time 1789287429.56 total 20.01 last 46.60)
 - open/close 1403 (time 1789287439.57 total 30.02 last 54.37)
 - open/close 1925 (time 1789287449.57 total 40.02 last 52.18)
 - open/close 2489 (time 1789287459.59 total 50.04 last 56.28)
 - open/close 3012 (time 1789287469.60 total 60.05 last 52.27)
 - open/close 3595 (time 1789287479.61 total 70.06 last 58.23)
 - open/close 4195 (time 1789287489.62 total 80.07 last 59.94)
total: 4500 open/close in 85.83 seconds: 52.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1181 (time 1789287510.58 total 10.00 last 118.09)
 - /unlink 2305 (time 1789287520.59 total 20.01 last 112.25)
 - /unlink 3628 (time 1789287530.59 total 30.02 last 132.28)
total: 4500 /unlink in 36.99 seconds: 121.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 749 (time 1789287552.50 total 10.00 last 74.87)
 - open/close 1347 (time 1789287562.51 total 20.01 last 59.74)
 - open/close 1953 (time 1789287572.52 total 30.03 last 60.52)
 - open/close 2552 (time 1789287582.52 total 40.03 last 59.90)
 - open/close 3171 (time 1789287592.54 total 50.05 last 61.77)
 - open/close 3786 (time 1789287602.57 total 60.07 last 61.34)
 - open/close 4322 (time 1789287612.57 total 70.08 last 53.57)
total: 4500 open/close in 72.84 seconds: 61.78 ops/second
 - /unlink 1043 (time 1789287631.16 total 10.00 last 104.28)
 - /unlink 2133 (time 1789287641.17 total 20.01 last 108.94)
 - /unlink 3285 (time 1789287651.17 total 30.01 last 115.19)
 - /unlink 4333 (time 1789287661.18 total 40.02 last 104.71)
total: 4500 /unlink in 41.79 seconds: 107.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 548 (time 1789287679.37 total 10.01 last 54.77)
 - open/close 1088 (time 1789287689.40 total 20.03 last 53.87)
 - open/close 1727 (time 1789287699.41 total 30.04 last 63.82)
 - open/close 2413 (time 1789287709.42 total 40.05 last 68.56)
 - open/close 3025 (time 1789287719.44 total 50.07 last 61.08)
 - open/close 3598 (time 1789287729.45 total 60.08 last 57.21)
 - open/close 4278 (time 1789287739.47 total 70.10 last 67.92)
total: 4500 open/close in 72.81 seconds: 61.81 ops/second
 - /unlink 1218 (time 1789287757.20 total 10.00 last 121.79)
 - /unlink 2303 (time 1789287767.21 total 20.01 last 108.43)
 - /unlink 3566 (time 1789287777.21 total 30.01 last 126.29)
total: 4500 /unlink in 38.59 seconds: 116.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 561 (time 1789287801.36 total 10.00 last 56.08)
 - open/close 1239 (time 1789287811.36 total 20.01 last 67.78)
 - open/close 1895 (time 1789287821.37 total 30.02 last 65.53)
 - open/close 2578 (time 1789287831.38 total 40.03 last 68.22)
 - open/close 3222 (time 1789287841.39 total 50.03 last 64.36)
 - open/close 3752 (time 1789287851.41 total 60.05 last 52.91)
 - open/close 4300 (time 1789287861.42 total 70.06 last 54.75)
total: 4500 open/close in 73.44 seconds: 61.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1023 (time 1789287880.38 total 10.00 last 102.29)
 - /unlink 1837 (time 1789287890.40 total 20.01 last 81.30)
 - /unlink 2810 (time 1789287900.41 total 30.02 last 97.21)
 - /unlink 3818 (time 1789287910.43 total 40.05 last 100.52)
total: 4500 /unlink in 47.08 seconds: 95.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 548 (time 1789287932.69 total 10.00 last 54.80)
 - open/close 1187 (time 1789287942.71 total 20.02 last 63.75)
 - open/close 1800 (time 1789287952.72 total 30.03 last 61.27)
 - open/close 2371 (time 1789287962.72 total 40.04 last 57.06)
 - open/close 2983 (time 1789287972.73 total 50.04 last 61.17)
 - open/close 3591 (time 1789287982.76 total 60.07 last 60.61)
 - open/close 4215 (time 1789287992.77 total 70.08 last 62.37)
total: 4500 open/close in 74.77 seconds: 60.18 ops/second
 - /unlink 1089 (time 1789288013.08 total 10.00 last 108.90)
 - /unlink 1999 (time 1789288023.10 total 20.02 last 90.82)
 - /unlink 3140 (time 1789288033.11 total 30.03 last 114.03)
 - /unlink 4147 (time 1789288043.11 total 40.03 last 100.68)
total: 4500 /unlink in 43.53 seconds: 103.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 558 (time 1789288063.05 total 10.01 last 55.77)
 - open/close 1087 (time 1789288073.09 total 20.05 last 52.69)
 - open/close 1650 (time 1789288083.10 total 30.06 last 56.24)
 - open/close 2244 (time 1789288093.12 total 40.07 last 59.34)
 - open/close 2819 (time 1789288103.13 total 50.08 last 57.40)
 - open/close 3408 (time 1789288113.14 total 60.10 last 58.83)
 - open/close 4052 (time 1789288123.15 total 70.10 last 64.37)
total: 4500 open/close in 76.28 seconds: 59.00 ops/second
 - /unlink 1051 (time 1789288144.62 total 10.00 last 105.08)
 - /unlink 2241 (time 1789288154.63 total 20.01 last 118.92)
 - /unlink 3563 (time 1789288164.64 total 30.02 last 132.09)
total: 4500 /unlink in 37.54 seconds: 119.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 548 (time 1789288187.73 total 10.03 last 54.65)
 - open/close 1085 (time 1789288197.74 total 20.03 last 53.68)
 - open/close 1657 (time 1789288207.75 total 30.04 last 57.13)
 - open/close 2214 (time 1789288217.75 total 40.05 last 55.67)
 - open/close 2846 (time 1789288227.76 total 50.05 last 63.19)
 - open/close 3439 (time 1789288237.76 total 60.06 last 59.26)
 - open/close 4018 (time 1789288247.77 total 70.06 last 57.88)
total: 4500 open/close in 78.56 seconds: 57.28 ops/second
 - /unlink 1253 (time 1789288271.28 total 10.01 last 125.12)
 - /unlink 2512 (time 1789288281.28 total 20.02 last 125.87)
 - /unlink 3730 (time 1789288291.28 total 30.02 last 121.78)
total: 4500 /unlink in 36.97 seconds: 121.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 625 (time 1789288313.44 total 10.01 last 62.43)
 - open/close 1337 (time 1789288323.45 total 20.02 last 71.14)
 - open/close 1961 (time 1789288333.45 total 30.02 last 62.38)
 - open/close 2696 (time 1789288343.46 total 40.03 last 73.43)
 - open/close 3302 (time 1789288353.47 total 50.04 last 60.55)
 - open/close 3963 (time 1789288363.48 total 60.05 last 66.02)
total: 4500 open/close in 68.52 seconds: 65.68 ops/second
 - /unlink 1090 (time 1789288387.92 total 10.01 last 108.92)
 - /unlink 2023 (time 1789288397.92 total 20.01 last 93.25)
 - /unlink 3221 (time 1789288407.94 total 30.03 last 119.63)
 - /unlink 4292 (time 1789288417.94 total 40.04 last 107.00)
total: 4500 /unlink in 42.24 seconds: 106.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 612 (time 1789288435.94 total 10.02 last 61.08)
 - open/close 1296 (time 1789288445.95 total 20.03 last 68.33)
 - open/close 1939 (time 1789288455.96 total 30.04 last 64.24)
 - open/close 2584 (time 1789288465.97 total 40.05 last 64.46)
 - open/close 3220 (time 1789288475.98 total 50.06 last 63.52)
 - open/close 3752 (time 1789288486.00 total 60.07 last 53.11)
 - open/close 4409 (time 1789288496.00 total 70.08 last 65.70)
total: 4500 open/close in 71.52 seconds: 62.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1364 (time 1789288512.94 total 10.01 last 136.30)
 - /unlink 2640 (time 1789288522.95 total 20.01 last 127.54)
 - /unlink 3866 (time 1789288532.95 total 30.02 last 122.56)
total: 4500 /unlink in 36.22 seconds: 124.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 586 (time 1789288554.84 total 10.00 last 58.57)
 - open/close 1280 (time 1789288564.85 total 20.02 last 69.31)
 - open/close 1930 (time 1789288574.85 total 30.02 last 64.98)
 - open/close 2527 (time 1789288584.86 total 40.03 last 59.66)
 - open/close 3187 (time 1789288594.89 total 50.06 last 65.80)
 - open/close 3888 (time 1789288604.89 total 60.06 last 70.09)
 - open/close 4478 (time 1789288614.91 total 70.07 last 58.92)
total: 4500 open/close in 70.45 seconds: 63.87 ops/second
 - /unlink 1094 (time 1789288630.32 total 10.00 last 109.38)
 - /unlink 2075 (time 1789288640.32 total 20.01 last 98.04)
 - /unlink 3101 (time 1789288650.33 total 30.02 last 102.52)
 - /unlink 4279 (time 1789288660.34 total 40.02 last 117.74)
total: 4500 /unlink in 42.17 seconds: 106.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 651 (time 1789288678.04 total 10.01 last 65.06)
 - open/close 1321 (time 1789288688.05 total 20.02 last 66.93)
 - open/close 2038 (time 1789288698.05 total 30.02 last 71.65)
 - open/close 2745 (time 1789288708.07 total 40.04 last 70.60)
 - open/close 3441 (time 1789288718.07 total 50.04 last 69.58)
 - open/close 4076 (time 1789288728.08 total 60.05 last 63.45)
total: 4500 open/close in 66.38 seconds: 67.79 ops/second
 - /unlink 1091 (time 1789288749.37 total 10.00 last 109.07)
 - /unlink 2226 (time 1789288759.38 total 20.01 last 113.41)
 - /unlink 3294 (time 1789288769.38 total 30.01 last 106.77)
 - /unlink 4297 (time 1789288779.39 total 40.02 last 100.28)
total: 4500 /unlink in 42.14 seconds: 106.79 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95315 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg457-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg457-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server
stop mds service on oleg457-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server
stop mds service on oleg457-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg457-server
unloading modules via unload_modules_local on: 'oleg457-server'
oleg457-server: oleg457-server.virtnet: executing unload_modules_local
oleg457-server: kmemleak disabled
modules unloaded.