== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:06:35 (1789301195)
Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg237-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg237-server
oleg237-server: oleg237-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing load_modules_local
oleg237-server: kmemleak disabled
oleg237-server: Loading modules from /home/green/git/lustre-release/lustre
oleg237-server: detected 4 online CPUs by sysfs
oleg237-server: MODOPTS_LIBCFS=
oleg237-server: Force libcfs to create 2 CPU partitions
oleg237-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg237-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg237-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg237-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg237-client.virtnet:  -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 767 (time 1789301411.42 total 10.01 last 76.60)
 - open/close 1470 (time 1789301421.43 total 20.02 last 70.24)
 - open/close 2278 (time 1789301431.44 total 30.03 last 80.74)
 - open/close 2960 (time 1789301441.45 total 40.04 last 68.12)
 - open/close 3701 (time 1789301451.46 total 50.05 last 74.06)
 - open/close 4416 (time 1789301461.47 total 60.06 last 71.41)
total: 4500 open/close in 61.75 seconds: 72.87 ops/second
 - /unlink 1232 (time 1789301478.87 total 10.00 last 123.20)
 - /unlink 2588 (time 1789301488.87 total 20.00 last 135.55)
 - /unlink 3740 (time 1789301498.88 total 30.02 last 115.04)
total: 4500 /unlink in 36.89 seconds: 121.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 665 (time 1789301520.69 total 10.01 last 66.45)
 - open/close 1384 (time 1789301530.69 total 20.02 last 71.84)
 - open/close 2058 (time 1789301540.69 total 30.02 last 67.40)
 - open/close 2715 (time 1789301550.70 total 40.02 last 65.66)
 - open/close 3389 (time 1789301560.72 total 50.05 last 67.24)
 - open/close 4060 (time 1789301570.73 total 60.05 last 67.05)
total: 4500 open/close in 66.65 seconds: 67.52 ops/second
 - /unlink 1318 (time 1789301592.73 total 10.00 last 131.79)
 - /unlink 2715 (time 1789301602.74 total 20.01 last 139.61)
 - /unlink 3921 (time 1789301612.75 total 30.02 last 120.49)
total: 4500 /unlink in 34.31 seconds: 131.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 660 (time 1789301632.05 total 10.01 last 65.93)
 - open/close 1300 (time 1789301642.07 total 20.04 last 63.84)
 - open/close 1837 (time 1789301652.09 total 30.06 last 53.58)
 - open/close 2491 (time 1789301662.10 total 40.06 last 65.38)
 - open/close 3251 (time 1789301672.10 total 50.07 last 75.94)
 - open/close 3880 (time 1789301682.11 total 60.07 last 62.88)
total: 4500 open/close in 68.25 seconds: 65.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1608 (time 1789301705.73 total 10.00 last 160.75)
 - /unlink 3017 (time 1789301715.73 total 20.01 last 140.80)
 - /unlink 4139 (time 1789301725.74 total 30.01 last 112.16)
total: 4500 /unlink in 32.96 seconds: 136.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 620 (time 1789301744.48 total 10.00 last 61.99)
 - open/close 1310 (time 1789301754.49 total 20.01 last 68.93)
 - open/close 2017 (time 1789301764.50 total 30.02 last 70.65)
 - open/close 2703 (time 1789301774.51 total 40.03 last 68.56)
 - open/close 3364 (time 1789301784.51 total 50.03 last 66.06)
 - open/close 3966 (time 1789301794.54 total 60.06 last 60.03)
total: 4500 open/close in 68.91 seconds: 65.30 ops/second
 - /unlink 1107 (time 1789301819.02 total 10.01 last 110.59)
 - /unlink 2154 (time 1789301829.02 total 20.01 last 104.70)
 - /unlink 3310 (time 1789301839.03 total 30.02 last 115.52)
total: 4500 /unlink in 39.87 seconds: 112.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 583 (time 1789301864.33 total 10.01 last 58.22)
 - open/close 1362 (time 1789301874.34 total 20.02 last 77.83)
 - open/close 2099 (time 1789301884.35 total 30.03 last 73.66)
 - open/close 2841 (time 1789301894.36 total 40.04 last 74.13)
 - open/close 3528 (time 1789301904.38 total 50.06 last 68.56)
 - open/close 4174 (time 1789301914.38 total 60.06 last 64.57)
total: 4500 open/close in 64.72 seconds: 69.53 ops/second
 - /unlink 1391 (time 1789301933.93 total 10.00 last 139.09)
 - /unlink 2536 (time 1789301943.93 total 20.00 last 114.48)
 - /unlink 3845 (time 1789301953.94 total 30.01 last 130.83)
total: 4500 /unlink in 35.08 seconds: 128.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 594 (time 1789301975.26 total 10.00 last 59.38)
 - open/close 1206 (time 1789301985.27 total 20.01 last 61.15)
 - open/close 1793 (time 1789301995.29 total 30.03 last 58.62)
 - open/close 2410 (time 1789302005.30 total 40.04 last 61.64)
 - open/close 2994 (time 1789302015.30 total 50.04 last 58.36)
 - open/close 3677 (time 1789302025.31 total 60.05 last 68.25)
 - open/close 4409 (time 1789302035.31 total 70.05 last 73.20)
total: 4500 open/close in 71.29 seconds: 63.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1294 (time 1789302051.32 total 10.00 last 129.36)
 - /unlink 2345 (time 1789302061.32 total 20.00 last 105.09)
 - /unlink 3607 (time 1789302071.33 total 30.01 last 126.07)
total: 4500 /unlink in 36.67 seconds: 122.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 705 (time 1789302094.62 total 10.01 last 70.45)
 - open/close 1357 (time 1789302104.63 total 20.02 last 65.13)
 - open/close 2059 (time 1789302114.65 total 30.04 last 70.05)
 - open/close 2752 (time 1789302124.65 total 40.04 last 69.27)
 - open/close 3354 (time 1789302134.65 total 50.05 last 60.18)
 - open/close 4073 (time 1789302144.66 total 60.05 last 71.87)
total: 4500 open/close in 65.90 seconds: 68.29 ops/second
 - /unlink 1168 (time 1789302166.40 total 10.02 last 116.53)
 - /unlink 2292 (time 1789302176.41 total 20.03 last 112.29)
 - /unlink 3507 (time 1789302186.42 total 30.04 last 121.44)
total: 4500 /unlink in 37.48 seconds: 120.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 698 (time 1789302209.36 total 10.01 last 69.76)
 - open/close 1411 (time 1789302219.38 total 20.03 last 71.16)
 - open/close 2057 (time 1789302229.39 total 30.03 last 64.56)
 - open/close 2824 (time 1789302239.39 total 40.04 last 76.66)
 - open/close 3476 (time 1789302249.40 total 50.05 last 65.12)
 - open/close 4276 (time 1789302259.43 total 60.08 last 79.76)
total: 4500 open/close in 63.21 seconds: 71.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1467 (time 1789302276.77 total 10.01 last 146.62)
 - /unlink 2903 (time 1789302286.77 total 20.01 last 143.52)
 - /unlink 4251 (time 1789302296.79 total 30.03 last 134.56)
total: 4500 /unlink in 32.32 seconds: 139.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 575 (time 1789302313.95 total 10.03 last 57.34)
 - open/close 1251 (time 1789302323.96 total 20.03 last 67.56)
 - open/close 1936 (time 1789302333.98 total 30.06 last 68.34)
 - open/close 2577 (time 1789302343.98 total 40.06 last 64.10)
 - open/close 3380 (time 1789302353.99 total 50.07 last 80.21)
 - open/close 4137 (time 1789302364.00 total 60.07 last 75.68)
total: 4500 open/close in 65.23 seconds: 68.98 ops/second
 - /unlink 1549 (time 1789302383.69 total 10.00 last 154.84)
 - /unlink 3047 (time 1789302393.69 total 20.01 last 149.73)
total: 4500 /unlink in 29.17 seconds: 154.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 752 (time 1789302417.91 total 10.03 last 74.99)
 - open/close 1506 (time 1789302427.92 total 20.03 last 75.38)
 - open/close 2218 (time 1789302437.92 total 30.03 last 71.18)
 - open/close 2986 (time 1789302447.92 total 40.04 last 76.76)
 - open/close 3705 (time 1789302457.93 total 50.05 last 71.83)
 - open/close 4271 (time 1789302467.94 total 60.05 last 56.57)
total: 4500 open/close in 62.78 seconds: 71.68 ops/second
 - /unlink 1583 (time 1789302485.05 total 10.00 last 158.25)
 - /unlink 3086 (time 1789302495.05 total 20.00 last 150.30)
total: 4500 /unlink in 29.40 seconds: 153.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 901 (time 1789302518.94 total 10.02 last 89.93)
 - open/close 1560 (time 1789302528.94 total 20.02 last 65.87)
 - open/close 2226 (time 1789302538.94 total 30.03 last 66.57)
 - open/close 2932 (time 1789302548.97 total 40.06 last 70.39)
 - open/close 3500 (time 1789302558.97 total 50.06 last 56.80)
 - open/close 4184 (time 1789302568.98 total 60.06 last 68.37)
total: 4500 open/close in 64.69 seconds: 69.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1268 (time 1789302588.43 total 10.00 last 126.77)
 - /unlink 2605 (time 1789302598.44 total 20.01 last 133.61)
 - /unlink 3991 (time 1789302608.44 total 30.01 last 138.56)
total: 4500 /unlink in 33.62 seconds: 133.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 889 (time 1789302626.73 total 10.00 last 88.89)
 - open/close 1745 (time 1789302636.75 total 20.02 last 85.47)
 - open/close 2478 (time 1789302646.75 total 30.02 last 73.30)
 - open/close 3261 (time 1789302656.76 total 40.03 last 78.23)
 - open/close 3875 (time 1789302666.76 total 50.03 last 61.40)
total: 4500 open/close in 57.84 seconds: 77.80 ops/second
 - /unlink 1321 (time 1789302689.36 total 10.00 last 132.09)
 - /unlink 2585 (time 1789302699.36 total 20.00 last 126.36)
 - /unlink 3768 (time 1789302709.37 total 30.01 last 118.28)
total: 4500 /unlink in 34.80 seconds: 129.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 867 (time 1789302729.20 total 10.01 last 86.62)
 - open/close 1670 (time 1789302739.20 total 20.02 last 80.25)
 - open/close 2412 (time 1789302749.21 total 30.02 last 74.18)
 - open/close 3131 (time 1789302759.22 total 40.03 last 71.80)
 - open/close 3933 (time 1789302769.23 total 50.04 last 80.15)
total: 4500 open/close in 57.45 seconds: 78.34 ops/second
 - /unlink 1187 (time 1789302790.79 total 10.01 last 118.61)
 - /unlink 2305 (time 1789302800.80 total 20.01 last 111.76)
 - /unlink 3536 (time 1789302810.80 total 30.02 last 123.04)
total: 4500 /unlink in 37.27 seconds: 120.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 798 (time 1789302833.83 total 10.00 last 79.80)
 - open/close 1611 (time 1789302843.84 total 20.02 last 81.18)
 - open/close 2355 (time 1789302853.86 total 30.03 last 74.30)
 - open/close 3107 (time 1789302863.86 total 40.04 last 75.15)
 - open/close 3897 (time 1789302873.86 total 50.04 last 78.99)
total: 4500 open/close in 57.70 seconds: 77.99 ops/second
 - /unlink 1417 (time 1789302896.37 total 10.00 last 141.64)
 - /unlink 2842 (time 1789302906.38 total 20.01 last 142.41)
 - /unlink 4371 (time 1789302916.39 total 30.02 last 152.74)
total: 4500 /unlink in 31.45 seconds: 143.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 722 (time 1789302933.88 total 10.01 last 72.13)
 - open/close 1526 (time 1789302943.90 total 20.02 last 80.28)
 - open/close 2211 (time 1789302953.91 total 30.03 last 68.43)
 - open/close 2957 (time 1789302963.91 total 40.04 last 74.56)
 - open/close 3814 (time 1789302973.92 total 50.05 last 85.64)
total: 4500 open/close in 58.41 seconds: 77.04 ops/second
 - /unlink 1366 (time 1789302996.81 total 10.00 last 136.55)
 - /unlink 2732 (time 1789303006.81 total 20.00 last 136.59)
 - /unlink 3926 (time 1789303016.81 total 30.00 last 119.40)
total: 4500 /unlink in 34.52 seconds: 130.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 586 (time 1789303037.64 total 10.02 last 58.46)
 - open/close 1301 (time 1789303047.64 total 20.03 last 71.49)
 - open/close 2125 (time 1789303057.65 total 30.03 last 82.34)
 - open/close 2797 (time 1789303067.66 total 40.05 last 67.10)
 - open/close 3379 (time 1789303077.67 total 50.05 last 58.17)
 - open/close 4060 (time 1789303087.67 total 60.05 last 68.10)
total: 4500 open/close in 65.74 seconds: 68.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1328 (time 1789303108.12 total 10.00 last 132.75)
 - /unlink 2653 (time 1789303118.13 total 20.01 last 132.45)
 - /unlink 4104 (time 1789303128.13 total 30.01 last 145.06)
total: 4500 /unlink in 33.17 seconds: 135.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 731 (time 1789303146.02 total 10.01 last 73.05)
 - open/close 1465 (time 1789303156.02 total 20.01 last 73.38)
 - open/close 2142 (time 1789303166.03 total 30.02 last 67.62)
 - open/close 2728 (time 1789303176.03 total 40.02 last 58.60)
 - open/close 3407 (time 1789303186.04 total 50.03 last 67.87)
 - open/close 4095 (time 1789303196.06 total 60.05 last 68.67)
total: 4500 open/close in 66.32 seconds: 67.85 ops/second
 - /unlink 1190 (time 1789303217.91 total 10.01 last 118.94)
 - /unlink 2486 (time 1789303227.91 total 20.01 last 129.57)
 - /unlink 3811 (time 1789303237.92 total 30.01 last 132.43)
total: 4500 /unlink in 35.43 seconds: 126.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 630 (time 1789303257.66 total 10.01 last 62.95)
 - open/close 1371 (time 1789303267.66 total 20.01 last 74.09)
 - open/close 2013 (time 1789303277.68 total 30.02 last 64.09)
 - open/close 2592 (time 1789303287.70 total 40.05 last 57.78)
 - open/close 3113 (time 1789303297.70 total 50.05 last 52.07)
 - open/close 3829 (time 1789303307.71 total 60.06 last 71.53)
 - open/close 4496 (time 1789303317.72 total 70.06 last 66.68)
total: 4500 open/close in 70.10 seconds: 64.20 ops/second
 - /unlink 1411 (time 1789303332.25 total 10.00 last 141.06)
 - /unlink 2934 (time 1789303342.25 total 20.01 last 152.23)
 - /unlink 4194 (time 1789303352.26 total 30.01 last 125.93)
total: 4500 /unlink in 32.81 seconds: 137.14 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95239 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg237-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg237-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server
stop mds service on oleg237-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server
stop mds service on oleg237-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg237-server
unloading modules via unload_modules_local on: 'oleg237-server'
oleg237-server: oleg237-server.virtnet: executing unload_modules_local
oleg237-server: kmemleak disabled
modules unloaded.