== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:24:48 (1790781888)
Stopping clients: oleg436-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg436-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg436-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg436-server
oleg436-server: oleg436-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg436-server'
oleg436-server: oleg436-server.virtnet: executing load_modules_local
oleg436-server: kmemleak disabled
oleg436-server: Loading modules from /home/green/git/lustre-release/lustre
oleg436-server: detected 4 online CPUs by sysfs
oleg436-server: MODOPTS_LIBCFS=
oleg436-server: Force libcfs to create 2 CPU partitions
oleg436-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg436-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all
pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg436-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all
pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg436-client.virtnet:  -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
Registered 1 changelog users: 'cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 587 (time 1790782045.07 total 10.00 last 58.69)
 - open/close 1226 (time 1790782055.08 total 20.01 last 63.87)
 - open/close 1844 (time 1790782065.09 total 30.02 last 61.71)
 - open/close 2417 (time 1790782075.10 total 40.03 last 57.25)
 - open/close 2999 (time 1790782085.12 total 50.05 last 58.10)
 - open/close 3574 (time 1790782095.12 total 60.05 last 57.46)
 - open/close 4182 (time 1790782105.16 total 70.10 last 60.55)
total: 4500 open/close in 75.34 seconds: 59.73 ops/second
 - /unlink 1190 (time 1790782125.62 total 10.03 last 118.60)
 - /unlink 2298 (time 1790782135.63 total 20.04 last 110.74)
 - /unlink 3410 (time 1790782145.63 total 30.04 last 111.16)
total: 4500 /unlink in 39.73 seconds: 113.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 622 (time 1790782170.92 total 10.00 last 62.19)
 - open/close 1237 (time 1790782180.93 total 20.01 last 61.42)
 - open/close 1834 (time 1790782190.94 total 30.02 last 59.68)
 - open/close 2430 (time 1790782200.96 total 40.04 last 59.48)
 - open/close 2929 (time 1790782210.96 total 50.04 last 49.90)
 - open/close 3556 (time 1790782220.96 total 60.04 last 62.66)
 - open/close 4136 (time 1790782230.97 total 70.05 last 57.98)
total: 4500 open/close in 77.19 seconds: 58.30 ops/second
 - /unlink 920 (time 1790782253.42 total 10.02 last 91.81)
 - /unlink 1939 (time 1790782263.43 total 20.04 last 101.73)
 - /unlink 2761 (time 1790782273.46 total 30.06 last 82.01)
 - /unlink 3675 (time 1790782283.47 total 40.08 last 91.25)
total: 4500 /unlink in 47.18 seconds: 95.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 617 (time 1790782306.01 total 10.01 last 61.62)
 - open/close 1225 (time 1790782316.04 total 20.04 last 60.63)
 - open/close 1768 (time 1790782326.04 total 30.04 last 54.28)
 - open/close 2293 (time 1790782336.05 total 40.05 last 52.49)
 - open/close 2855 (time 1790782346.05 total 50.05 last 56.19)
 - open/close 3468 (time 1790782356.06 total 60.07 last 61.19)
 - open/close 4130 (time 1790782366.07 total 70.07 last 66.15)
total: 4500 open/close in 76.13 seconds: 59.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1092 (time 1790782387.12 total 10.00 last 109.17)
 - /unlink 2219 (time 1790782397.12 total 20.01 last 112.63)
 - /unlink 3318 (time 1790782407.13 total 30.01 last 109.86)
total: 4500 /unlink in 39.97 seconds: 112.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 526 (time 1790782432.88 total 10.01 last 52.53)
 - open/close 1091 (time 1790782442.89 total 20.03 last 56.43)
 - open/close 1566 (time 1790782452.89 total 30.03 last 47.48)
 - open/close 2100 (time 1790782462.91 total 40.05 last 53.29)
 - open/close 2683 (time 1790782472.94 total 50.07 last 58.17)
 - open/close 3269 (time 1790782482.95 total 60.09 last 58.51)
 - open/close 3913 (time 1790782492.97 total 70.11 last 64.29)
total: 4500 open/close in 79.68 seconds: 56.48 ops/second
 - /unlink 1181 (time 1790782517.15 total 10.00 last 118.07)
 - /unlink 2276 (time 1790782527.16 total 20.01 last 109.44)
 - /unlink 3345 (time 1790782537.18 total 30.03 last 106.63)
total: 4500 /unlink in 39.68 seconds: 113.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 606 (time 1790782562.31 total 10.00 last 60.59)
 - open/close 1224 (time 1790782572.33 total 20.02 last 61.69)
 - open/close 1877 (time 1790782582.34 total 30.03 last 65.23)
 - open/close 2475 (time 1790782592.36 total 40.05 last 59.69)
 - open/close 2988 (time 1790782602.38 total 50.07 last 51.21)
 - open/close 3588 (time 1790782612.38 total 60.07 last 59.98)
 - open/close 4130 (time 1790782622.39 total 70.07 last 54.17)
total: 4500 open/close in 76.82 seconds: 58.58 ops/second
 - /unlink 1061 (time 1790782644.25 total 10.01 last 106.04)
 - /unlink 2129 (time 1790782654.26 total 20.02 last 106.69)
 - /unlink 3205 (time 1790782664.27 total 30.02 last 107.57)
 - /unlink 4321 (time 1790782674.27 total 40.02 last 111.54)
total: 4500 /unlink in 41.61 seconds: 108.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 491 (time 1790782691.12 total 10.02 last 48.98)
 - open/close 1087 (time 1790782701.12 total 20.03 last 59.58)
 - open/close 1666 (time 1790782711.12 total 30.03 last 57.89)
 - open/close 2261 (time 1790782721.13 total 40.04 last 59.44)
 - open/close 2855 (time 1790782731.14 total 50.05 last 59.35)
 - open/close 3424 (time 1790782741.17 total 60.07 last 56.76)
 - open/close 4029 (time 1790782751.17 total 70.07 last 60.50)
total: 4500 open/close in 77.36 seconds: 58.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1106 (time 1790782774.42 total 10.01 last 110.47)
 - /unlink 2182 (time 1790782784.43 total 20.02 last 107.52)
 - /unlink 3242 (time 1790782794.43 total 30.03 last 105.91)
 - /unlink 4299 (time 1790782804.44 total 40.03 last 105.67)
total: 4500 /unlink in 42.05 seconds: 107.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 628 (time 1790782821.74 total 10.01 last 62.75)
 - open/close 1262 (time 1790782831.74 total 20.01 last 63.40)
 - open/close 1884 (time 1790782841.74 total 30.01 last 62.20)
 - open/close 2482 (time 1790782851.75 total 40.02 last 59.72)
 - open/close 3007 (time 1790782861.75 total 50.02 last 52.50)
 - open/close 3525 (time 1790782871.75 total 60.02 last 51.79)
 - open/close 4004 (time 1790782881.81 total 70.08 last 47.61)
 - open/close 4471 (time 1790782891.82 total 80.09 last 46.68)
total: 4500 open/close in 80.59 seconds: 55.84 ops/second
 - /unlink 992 (time 1790782909.43 total 10.00 last 99.16)
 - /unlink 2117 (time 1790782919.43 total 20.00 last 112.50)
 - /unlink 3263 (time 1790782929.44 total 30.01 last 114.54)
 - /unlink 4404 (time 1790782939.45 total 40.02 last 113.99)
total: 4500 /unlink in 40.99 seconds: 109.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 584 (time 1790782955.89 total 10.01 last 58.33)
 - open/close 1163 (time 1790782965.89 total 20.02 last 57.87)
 - open/close 1726 (time 1790782975.90 total 30.03 last 56.25)
 - open/close 2273 (time 1790782985.91 total 40.04 last 54.64)
 - open/close 2869 (time 1790782995.92 total 50.04 last 59.57)
 - open/close 3514 (time 1790783005.92 total 60.05 last 64.46)
 - open/close 4165 (time 1790783015.93 total 70.05 last 65.06)
total: 4500 open/close in 75.96 seconds: 59.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 994 (time 1790783037.56 total 10.02 last 99.19)
 - /unlink 2019 (time 1790783047.56 total 20.03 last 102.44)
 - /unlink 2943 (time 1790783057.58 total 30.04 last 92.27)
 - /unlink 3891 (time 1790783067.58 total 40.05 last 94.74)
total: 4500 /unlink in 46.25 seconds: 97.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 622 (time 1790783089.46 total 10.00 last 62.17)
 - open/close 1200 (time 1790783099.51 total 20.05 last 57.54)
 - open/close 1877 (time 1790783109.52 total 30.06 last 67.64)
 - open/close 2476 (time 1790783119.52 total 40.06 last 59.88)
 - open/close 3038 (time 1790783129.54 total 50.08 last 56.11)
 - open/close 3671 (time 1790783139.54 total 60.08 last 63.29)
 - open/close 4221 (time 1790783149.54 total 70.08 last 54.99)
total: 4500 open/close in 76.25 seconds: 59.02 ops/second
 - /unlink 1166 (time 1790783171.05 total 10.00 last 116.59)
 - /unlink 2298 (time 1790783181.05 total 20.00 last 113.19)
 - /unlink 3310 (time 1790783191.05 total 30.00 last 101.17)
 - /unlink 4442 (time 1790783201.06 total 40.01 last 113.16)
total: 4500 /unlink in 40.66 seconds: 110.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 517 (time 1790783218.43 total 10.01 last 51.67)
 - open/close 1095 (time 1790783228.43 total 20.01 last 57.78)
 - open/close 1719 (time 1790783238.44 total 30.02 last 62.34)
 - open/close 2363 (time 1790783248.45 total 40.02 last 64.38)
 - open/close 2972 (time 1790783258.45 total 50.03 last 60.86)
 - open/close 3571 (time 1790783268.47 total 60.04 last 59.82)
 - open/close 4186 (time 1790783278.48 total 70.06 last 61.42)
total: 4500 open/close in 74.70 seconds: 60.24 ops/second
 - /unlink 1102 (time 1790783298.18 total 10.01 last 110.12)
 - /unlink 2151 (time 1790783308.18 total 20.01 last 104.85)
 - /unlink 3183 (time 1790783318.19 total 30.02 last 103.13)
 - /unlink 4133 (time 1790783328.19 total 40.02 last 94.96)
total: 4500 /unlink in 44.45 seconds: 101.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 519 (time 1790783350.62 total 10.03 last 51.74)
 - open/close 1042 (time 1790783360.65 total 20.06 last 52.16)
 - open/close 1580 (time 1790783370.66 total 30.06 last 53.76)
 - open/close 2135 (time 1790783380.67 total 40.08 last 55.43)
 - open/close 2730 (time 1790783390.68 total 50.09 last 59.43)
 - open/close 3329 (time 1790783400.72 total 60.12 last 59.70)
 - open/close 3892 (time 1790783410.72 total 70.12 last 56.28)
total: 4500 open/close in 79.71 seconds: 56.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1035 (time 1790783436.03 total 10.00 last 103.45)
 - /unlink 2105 (time 1790783446.04 total 20.01 last 106.98)
 - /unlink 3206 (time 1790783456.06 total 30.03 last 109.84)
 - /unlink 4337 (time 1790783466.06 total 40.03 last 113.10)
total: 4500 /unlink in 41.56 seconds: 108.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 574 (time 1790783482.97 total 10.01 last 57.35)
 - open/close 1173 (time 1790783492.99 total 20.02 last 59.80)
 - open/close 1735 (time 1790783503.00 total 30.03 last 56.14)
 - open/close 2324 (time 1790783513.01 total 40.04 last 58.86)
 - open/close 2966 (time 1790783523.01 total 50.04 last 64.18)
 - open/close 3559 (time 1790783533.01 total 60.05 last 59.29)
 - open/close 4157 (time 1790783543.02 total 70.05 last 59.77)
total: 4500 open/close in 76.02 seconds: 59.19 ops/second
 - /unlink 1188 (time 1790783564.61 total 10.00 last 118.75)
 - /unlink 2171 (time 1790783574.62 total 20.01 last 98.21)
 - /unlink 3334 (time 1790783584.62 total 30.02 last 116.26)
 - /unlink 4412 (time 1790783594.62 total 40.02 last 107.76)
total: 4500 /unlink in 41.09 seconds: 109.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 579 (time 1790783610.97 total 10.01 last 57.81)
 - open/close 1176 (time 1790783620.97 total 20.02 last 59.68)
 - open/close 1743 (time 1790783630.98 total 30.03 last 56.63)
 - open/close 2329 (time 1790783640.99 total 40.04 last 58.54)
 - open/close 2932 (time 1790783651.00 total 50.04 last 60.28)
 - open/close 3539 (time 1790783661.01 total 60.06 last 60.61)
 - open/close 4142 (time 1790783671.02 total 70.07 last 60.25)
total: 4500 open/close in 75.30 seconds: 59.76 ops/second
 - /unlink 1114 (time 1790783691.13 total 10.01 last 111.30)
 - /unlink 2281 (time 1790783701.13 total 20.01 last 116.64)
 - /unlink 3383 (time 1790783711.14 total 30.02 last 110.16)
total: 4500 /unlink in 39.57 seconds: 113.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 568 (time 1790783735.43 total 10.01 last 56.75)
 - open/close 1191 (time 1790783745.43 total 20.01 last 62.27)
 - open/close 1754 (time 1790783755.44 total 30.02 last 56.25)
 - open/close 2342 (time 1790783765.44 total 40.02 last 58.80)
 - open/close 2941 (time 1790783775.45 total 50.03 last 59.86)
 - open/close 3541 (time 1790783785.48 total 60.06 last 59.81)
 - open/close 4135 (time 1790783795.50 total 70.09 last 59.27)
total: 4500 open/close in 75.95 seconds: 59.25 ops/second
 - /unlink 1155 (time 1790783815.66 total 10.00 last 115.50)
 - /unlink 2268 (time 1790783825.66 total 20.00 last 111.27)
 - /unlink 3458 (time 1790783835.68 total 30.02 last 118.77)
total: 4500 /unlink in 39.68 seconds: 113.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 561 (time 1790783860.19 total 10.02 last 56.01)
 - open/close 1107 (time 1790783870.20 total 20.03 last 54.53)
 - open/close 1730 (time 1790783880.21 total 30.03 last 62.29)
 - open/close 2339 (time 1790783890.21 total 40.03 last 60.88)
 - open/close 2922 (time 1790783900.22 total 50.04 last 58.24)
 - open/close 3542 (time 1790783910.23 total 60.05 last 61.93)
 - open/close 4158 (time 1790783920.25 total 70.08 last 61.47)
total: 4500 open/close in 76.00 seconds: 59.21 ops/second
 - /unlink 1129 (time 1790783941.74 total 10.00 last 112.89)
 - /unlink 2234 (time 1790783951.75 total 20.00 last 110.48)
 - /unlink 3375 (time 1790783961.75 total 30.00 last 114.09)
 - /unlink 4450 (time 1790783971.75 total 40.01 last 107.44)
total: 4500 /unlink in 40.59 seconds: 110.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 628 (time 1790783988.09 total 10.00 last 62.78)
 - open/close 1207 (time 1790783998.10 total 20.01 last 57.87)
 - open/close 1833 (time 1790784008.10 total 30.01 last 62.58)
 - open/close 2456 (time 1790784018.10 total 40.01 last 62.28)
 - open/close 3075 (time 1790784028.11 total 50.02 last 61.89)
 - open/close 3721 (time 1790784038.12 total 60.03 last 64.50)
 - open/close 4297 (time 1790784048.14 total 70.06 last 57.47)
total: 4500 open/close in 73.34 seconds: 61.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1050 (time 1790784065.88 total 10.00 last 104.97)
 - /unlink 2025 (time 1790784075.88 total 20.00 last 97.48)
 - /unlink 3068 (time 1790784085.90 total 30.02 last 104.13)
 - /unlink 4217 (time 1790784095.91 total 40.03 last 114.74)
total: 4500 /unlink in 43.02 seconds: 104.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 439 (time 1790784114.47 total 10.02 last 43.81)
 - open/close 1047 (time 1790784124.48 total 20.03 last 60.74)
 - open/close 1678 (time 1790784134.48 total 30.03 last 63.07)
 - open/close 2262 (time 1790784144.49 total 40.04 last 58.33)
 - open/close 2863 (time 1790784154.49 total 50.04 last 60.09)
 - open/close 3458 (time 1790784164.53 total 60.08 last 59.30)
 - open/close 4072 (time 1790784174.54 total 70.09 last 61.33)
total: 4500 open/close in 77.84 seconds: 57.81 ops/second
 - /unlink 1143 (time 1790784197.57 total 10.01 last 114.23)
 - /unlink 2094 (time 1790784207.58 total 20.01 last 95.02)
 - /unlink 3060 (time 1790784217.58 total 30.02 last 96.58)
 - /unlink 4090 (time 1790784227.59 total 40.02 last 102.95)
total: 4500 /unlink in 43.81 seconds: 102.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 578 (time 1790784246.91 total 10.01 last 57.76)
 - open/close 1162 (time 1790784256.92 total 20.02 last 58.33)
 - open/close 1820 (time 1790784266.93 total 30.03 last 65.75)
 - open/close 2390 (time 1790784276.93 total 40.03 last 56.98)
 - open/close 3022 (time 1790784286.94 total 50.03 last 63.18)
 - open/close 3663 (time 1790784296.94 total 60.03 last 64.09)
 - open/close 4300 (time 1790784306.94 total 70.04 last 63.65)
total: 4500 open/close in 73.06 seconds: 61.60 ops/second
 - /unlink 1230 (time 1790784325.37 total 10.00 last 123.00)
 - /unlink 2421 (time 1790784335.38 total 20.01 last 119.01)
 - /unlink 3383 (time 1790784345.38 total 30.01 last 96.16)
 - /unlink 4416 (time 1790784355.39 total 40.01 last 103.29)
total: 4500 /unlink in 41.25 seconds: 109.09 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11358: 75824 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg436-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg436-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server
stop mds service on oleg436-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server
unloading modules via unload_modules_local on: 'oleg436-server'
oleg436-server: oleg436-server.virtnet: executing unload_modules_local
oleg436-server: kmemleak disabled
modules unloaded.