== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:09:26 (1784189366)
Stopping clients: oleg446-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg446-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg446-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg446-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg446-server
oleg446-server: oleg446-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg446-server'
oleg446-server: oleg446-server.virtnet: executing load_modules_local
oleg446-server: Loading modules from /home/green/git/lustre-release/lustre
oleg446-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg446-server: detected 4 online CPUs by sysfs
oleg446-server: MODOPTS_LIBCFS=
oleg446-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg446-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg446-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg446-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg446-client.virtnet:  -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 484 (time 1784189596.07 total 10.01 last 48.36)
 - open/close 890 (time 1784189606.09 total 20.03 last 40.51)
 - open/close 1412 (time 1784189616.10 total 30.04 last 52.15)
 - open/close 1913 (time 1784189626.10 total 40.04 last 50.09)
 - open/close 2496 (time 1784189636.11 total 50.04 last 58.29)
 - open/close 3005 (time 1784189646.11 total 60.05 last 50.87)
 - open/close 3565 (time 1784189656.13 total 70.06 last 55.92)
 - open/close 4129 (time 1784189666.14 total 80.08 last 56.33)
total: 4500 open/close in 87.56 seconds: 51.39 ops/second
 - /unlink 1160 (time 1784189689.78 total 10.00 last 115.95)
 - /unlink 2272 (time 1784189699.78 total 20.01 last 111.14)
 - /unlink 3326 (time 1784189709.78 total 30.01 last 105.39)
 - /unlink 4297 (time 1784189719.79 total 40.01 last 97.09)
total: 4500 /unlink in 42.23 seconds: 106.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 523 (time 1784189738.60 total 10.04 last 52.11)
 - open/close 1107 (time 1784189748.64 total 20.08 last 58.14)
 - open/close 1707 (time 1784189758.66 total 30.09 last 59.92)
 - open/close 2246 (time 1784189768.66 total 40.10 last 53.90)
 - open/close 2839 (time 1784189778.66 total 50.10 last 59.27)
 - open/close 3433 (time 1784189788.68 total 60.12 last 59.28)
 - open/close 4014 (time 1784189798.70 total 70.14 last 57.97)
total: 4500 open/close in 77.70 seconds: 57.92 ops/second
 - /unlink 995 (time 1784189821.98 total 10.00 last 99.45)
 - /unlink 1912 (time 1784189831.99 total 20.01 last 91.62)
 - /unlink 2757 (time 1784189841.99 total 30.02 last 84.46)
 - /unlink 3538 (time 1784189852.01 total 40.03 last 78.01)
 - /unlink 4424 (time 1784189862.01 total 50.04 last 88.53)
total: 4500 /unlink in 51.36 seconds: 87.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 554 (time 1784189878.33 total 10.01 last 55.34)
 - open/close 986 (time 1784189888.34 total 20.02 last 43.15)
 - open/close 1330 (time 1784189898.35 total 30.02 last 34.39)
 - open/close 1772 (time 1784189908.36 total 40.04 last 44.12)
 - open/close 2203 (time 1784189918.38 total 50.06 last 43.04)
 - open/close 2653 (time 1784189928.38 total 60.06 last 44.97)
 - open/close 3267 (time 1784189938.39 total 70.07 last 61.35)
 - open/close 3883 (time 1784189948.39 total 80.07 last 61.59)
 - open/close 4413 (time 1784189958.39 total 90.07 last 53.00)
total: 4500 open/close in 91.82 seconds: 49.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 992 (time 1784189975.35 total 10.00 last 99.17)
 - /unlink 2159 (time 1784189985.35 total 20.00 last 116.69)
 - /unlink 3231 (time 1784189995.36 total 30.01 last 107.12)
 - /unlink 4279 (time 1784190005.36 total 40.02 last 104.75)
total: 4500 /unlink in 41.84 seconds: 107.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 617 (time 1784190022.79 total 10.01 last 61.67)
 - open/close 1218 (time 1784190032.80 total 20.01 last 60.05)
 - open/close 1768 (time 1784190042.81 total 30.02 last 54.94)
 - open/close 2359 (time 1784190052.81 total 40.03 last 59.09)
 - open/close 2933 (time 1784190062.83 total 50.04 last 57.31)
 - open/close 3523 (time 1784190072.84 total 60.05 last 58.94)
 - open/close 4099 (time 1784190082.85 total 70.06 last 57.55)
total: 4500 open/close in 77.71 seconds: 57.91 ops/second
 - /unlink 1114 (time 1784190105.43 total 10.01 last 111.26)
 - /unlink 2234 (time 1784190115.43 total 20.01 last 111.97)
 - /unlink 3281 (time 1784190125.44 total 30.02 last 104.61)
 - /unlink 4273 (time 1784190135.45 total 40.04 last 99.04)
total: 4500 /unlink in 42.30 seconds: 106.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 641 (time 1784190153.09 total 10.01 last 64.03)
 - open/close 1151 (time 1784190163.11 total 20.03 last 50.93)
 - open/close 1737 (time 1784190173.12 total 30.04 last 58.49)
 - open/close 2337 (time 1784190183.13 total 40.05 last 59.99)
 - open/close 2925 (time 1784190193.15 total 50.07 last 58.65)
 - open/close 3535 (time 1784190203.17 total 60.09 last 60.87)
 - open/close 4104 (time 1784190213.17 total 70.10 last 56.89)
total: 4500 open/close in 75.91 seconds: 59.28 ops/second
 - /unlink 1172 (time 1784190233.99 total 10.00 last 117.16)
 - /unlink 2289 (time 1784190244.00 total 20.01 last 111.61)
 - /unlink 3419 (time 1784190254.00 total 30.01 last 112.99)
 - /unlink 4456 (time 1784190264.01 total 40.02 last 103.62)
total: 4500 /unlink in 40.73 seconds: 110.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 602 (time 1784190280.62 total 10.01 last 60.13)
 - open/close 1166 (time 1784190290.63 total 20.02 last 56.35)
 - open/close 1816 (time 1784190300.63 total 30.02 last 64.98)
 - open/close 2532 (time 1784190310.65 total 40.04 last 71.47)
 - open/close 3139 (time 1784190320.65 total 50.04 last 60.70)
 - open/close 3696 (time 1784190330.66 total 60.05 last 55.66)
 - open/close 4365 (time 1784190340.67 total 70.06 last 66.82)
total: 4500 open/close in 71.96 seconds: 62.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1048 (time 1784190357.16 total 10.01 last 104.71)
 - /unlink 2177 (time 1784190367.17 total 20.02 last 112.79)
 - /unlink 3232 (time 1784190377.18 total 30.02 last 105.44)
 - /unlink 4301 (time 1784190387.19 total 40.03 last 106.80)
total: 4500 /unlink in 42.62 seconds: 105.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 619 (time 1784190404.81 total 10.01 last 61.86)
 - open/close 1276 (time 1784190414.82 total 20.02 last 65.63)
 - open/close 1947 (time 1784190424.83 total 30.02 last 67.07)
 - open/close 2601 (time 1784190434.84 total 40.03 last 65.35)
 - open/close 3185 (time 1784190444.84 total 50.03 last 58.37)
 - open/close 3784 (time 1784190454.86 total 60.06 last 59.76)
 - open/close 4274 (time 1784190464.87 total 70.07 last 48.95)
total: 4500 open/close in 74.76 seconds: 60.19 ops/second
 - /unlink 1081 (time 1784190484.79 total 10.00 last 108.07)
 - /unlink 2118 (time 1784190494.79 total 20.01 last 103.66)
 - /unlink 3257 (time 1784190504.80 total 30.02 last 113.75)
 - /unlink 4406 (time 1784190514.81 total 40.02 last 114.83)
total: 4500 /unlink in 41.02 seconds: 109.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 537 (time 1784190530.65 total 10.00 last 53.70)
 - open/close 1179 (time 1784190540.65 total 20.00 last 64.17)
 - open/close 1780 (time 1784190550.65 total 30.00 last 60.10)
 - open/close 2284 (time 1784190560.66 total 40.02 last 50.33)
 - open/close 2926 (time 1784190570.67 total 50.03 last 64.14)
 - open/close 3498 (time 1784190580.68 total 60.03 last 57.17)
 - open/close 4142 (time 1784190590.68 total 70.04 last 64.36)
total: 4500 open/close in 75.88 seconds: 59.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1200 (time 1784190611.28 total 10.01 last 119.89)
 - /unlink 2384 (time 1784190621.28 total 20.01 last 118.36)
 - /unlink 3555 (time 1784190631.29 total 30.02 last 117.00)
total: 4500 /unlink in 37.78 seconds: 119.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 633 (time 1784190654.07 total 10.00 last 63.30)
 - open/close 1335 (time 1784190664.10 total 20.02 last 70.03)
 - open/close 1990 (time 1784190674.12 total 30.05 last 65.35)
 - open/close 2621 (time 1784190684.13 total 40.05 last 63.07)
 - open/close 3312 (time 1784190694.14 total 50.06 last 69.03)
 - open/close 3968 (time 1784190704.14 total 60.07 last 65.56)
total: 4500 open/close in 67.92 seconds: 66.25 ops/second
 - /unlink 1160 (time 1784190726.98 total 10.00 last 115.94)
 - /unlink 2253 (time 1784190736.99 total 20.02 last 109.18)
 - /unlink 3374 (time 1784190746.99 total 30.02 last 112.04)
total: 4500 /unlink in 39.86 seconds: 112.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 608 (time 1784190772.54 total 10.00 last 60.79)
 - open/close 1250 (time 1784190782.55 total 20.01 last 64.15)
 - open/close 1849 (time 1784190792.56 total 30.02 last 59.85)
 - open/close 2535 (time 1784190802.56 total 40.02 last 68.58)
 - open/close 3199 (time 1784190812.56 total 50.03 last 66.38)
 - open/close 3848 (time 1784190822.57 total 60.03 last 64.88)
 - open/close 4446 (time 1784190832.58 total 70.04 last 59.72)
total: 4500 open/close in 70.95 seconds: 63.43 ops/second
 - /unlink 1173 (time 1784190849.21 total 10.00 last 117.27)
 - /unlink 2190 (time 1784190859.22 total 20.01 last 101.65)
 - /unlink 3428 (time 1784190869.22 total 30.01 last 123.76)
total: 4500 /unlink in 39.83 seconds: 112.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 603 (time 1784190894.85 total 10.01 last 60.26)
 - open/close 1184 (time 1784190904.86 total 20.02 last 58.01)
 - open/close 1797 (time 1784190914.88 total 30.04 last 61.20)
 - open/close 2484 (time 1784190924.88 total 40.04 last 68.67)
 - open/close 3091 (time 1784190934.90 total 50.07 last 60.57)
 - open/close 3747 (time 1784190944.92 total 60.08 last 65.52)
 - open/close 4275 (time 1784190954.93 total 70.09 last 52.73)
total: 4500 open/close in 73.36 seconds: 61.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1093 (time 1784190973.10 total 10.00 last 109.27)
 - /unlink 2105 (time 1784190983.10 total 20.00 last 101.19)
 - /unlink 3185 (time 1784190993.12 total 30.03 last 107.77)
 - /unlink 4244 (time 1784191003.13 total 40.04 last 105.76)
total: 4500 /unlink in 42.07 seconds: 106.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 598 (time 1784191020.86 total 10.01 last 59.73)
 - open/close 1248 (time 1784191030.86 total 20.01 last 64.99)
 - open/close 1806 (time 1784191040.87 total 30.02 last 55.76)
 - open/close 2354 (time 1784191050.88 total 40.03 last 54.73)
 - open/close 3007 (time 1784191060.90 total 50.05 last 65.21)
 - open/close 3613 (time 1784191070.90 total 60.06 last 60.56)
 - open/close 4305 (time 1784191080.91 total 70.06 last 69.17)
total: 4500 open/close in 72.72 seconds: 61.88 ops/second
 - /unlink 1224 (time 1784191097.95 total 10.01 last 122.33)
 - /unlink 2409 (time 1784191107.95 total 20.01 last 118.42)
 - /unlink 3605 (time 1784191117.97 total 30.03 last 119.44)
total: 4500 /unlink in 37.28 seconds: 120.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 588 (time 1784191140.70 total 10.00 last 58.79)
 - open/close 1128 (time 1784191150.72 total 20.03 last 53.88)
 - open/close 1725 (time 1784191160.73 total 30.03 last 59.66)
 - open/close 2429 (time 1784191170.73 total 40.04 last 70.37)
 - open/close 3113 (time 1784191180.75 total 50.05 last 68.31)
 - open/close 3742 (time 1784191190.75 total 60.06 last 62.87)
 - open/close 4384 (time 1784191200.76 total 70.07 last 64.12)
total: 4500 open/close in 71.61 seconds: 62.84 ops/second
 - /unlink 1208 (time 1784191216.41 total 10.02 last 120.59)
 - /unlink 2162 (time 1784191226.41 total 20.02 last 95.37)
 - /unlink 3352 (time 1784191236.42 total 30.03 last 118.89)
total: 4500 /unlink in 39.93 seconds: 112.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 603 (time 1784191261.24 total 10.00 last 60.30)
 - open/close 1235 (time 1784191271.25 total 20.01 last 63.15)
 - open/close 1756 (time 1784191281.25 total 30.01 last 52.09)
 - open/close 2368 (time 1784191291.27 total 40.03 last 61.11)
 - open/close 2968 (time 1784191301.28 total 50.04 last 59.91)
 - open/close 3591 (time 1784191311.29 total 60.05 last 62.26)
 - open/close 4193 (time 1784191321.30 total 70.06 last 60.13)
total: 4500 open/close in 76.17 seconds: 59.08 ops/second
 - /unlink 1065 (time 1784191343.72 total 10.00 last 106.45)
 - /unlink 2233 (time 1784191353.72 total 20.01 last 116.79)
 - /unlink 3252 (time 1784191363.73 total 30.01 last 101.84)
 - /unlink 4269 (time 1784191373.73 total 40.01 last 101.69)
total: 4500 /unlink in 42.45 seconds: 106.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 571 (time 1784191391.97 total 10.01 last 57.06)
 - open/close 1176 (time 1784191401.98 total 20.02 last 60.44)
 - open/close 1845 (time 1784191411.99 total 30.02 last 66.88)
 - open/close 2489 (time 1784191422.00 total 40.03 last 64.35)
 - open/close 3083 (time 1784191432.00 total 50.04 last 59.36)
 - open/close 3738 (time 1784191442.03 total 60.06 last 65.36)
 - open/close 4438 (time 1784191452.03 total 70.07 last 69.95)
total: 4500 open/close in 71.06 seconds: 63.32 ops/second
 - /unlink 1121 (time 1784191467.73 total 10.00 last 112.08)
 - /unlink 2250 (time 1784191477.73 total 20.00 last 112.87)
 - /unlink 3391 (time 1784191487.74 total 30.01 last 114.07)
 - /unlink 4497 (time 1784191497.74 total 40.02 last 110.51)
total: 4500 /unlink in 40.03 seconds: 112.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 645 (time 1784191512.63 total 10.00 last 64.48)
 - open/close 1232 (time 1784191522.63 total 20.01 last 58.66)
 - open/close 1837 (time 1784191532.65 total 30.03 last 60.37)
 - open/close 2491 (time 1784191542.66 total 40.04 last 65.33)
 - open/close 3114 (time 1784191552.68 total 50.06 last 62.20)
 - open/close 3745 (time 1784191562.69 total 60.07 last 63.05)
 - open/close 4399 (time 1784191572.70 total 70.07 last 65.34)
total: 4500 open/close in 71.48 seconds: 62.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1177 (time 1784191588.87 total 10.00 last 117.68)
 - /unlink 2293 (time 1784191598.88 total 20.01 last 111.54)
 - /unlink 3478 (time 1784191608.88 total 30.01 last 118.48)
total: 4500 /unlink in 38.21 seconds: 117.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 630 (time 1784191632.37 total 10.01 last 62.96)
 - open/close 1178 (time 1784191642.38 total 20.02 last 54.72)
 - open/close 1869 (time 1784191652.39 total 30.03 last 69.04)
 - open/close 2522 (time 1784191662.40 total 40.04 last 65.25)
 - open/close 3060 (time 1784191672.41 total 50.05 last 53.72)
 - open/close 3624 (time 1784191682.42 total 60.06 last 56.35)
 - open/close 4172 (time 1784191692.43 total 70.07 last 54.73)
total: 4500 open/close in 75.72 seconds: 59.43 ops/second
 - /unlink 1112 (time 1784191713.29 total 10.01 last 111.08)
 - /unlink 2150 (time 1784191723.29 total 20.02 last 103.73)
 - /unlink 3364 (time 1784191733.30 total 30.03 last 121.28)
total: 4500 /unlink in 39.81 seconds: 113.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 557 (time 1784191758.15 total 10.00 last 55.69)
 - open/close 1173 (time 1784191768.16 total 20.01 last 61.55)
 - open/close 1771 (time 1784191778.16 total 30.01 last 59.79)
 - open/close 2401 (time 1784191788.17 total 40.02 last 62.92)
 - open/close 3023 (time 1784191798.18 total 50.03 last 62.16)
 - open/close 3638 (time 1784191808.19 total 60.04 last 61.41)
 - open/close 4302 (time 1784191818.20 total 70.05 last 66.39)
total: 4500 open/close in 73.51 seconds: 61.22 ops/second
 - /unlink 1161 (time 1784191835.89 total 10.02 last 115.92)
 - /unlink 2310 (time 1784191845.89 total 20.02 last 114.86)
 - /unlink 3259 (time 1784191855.89 total 30.02 last 94.86)
 - /unlink 4388 (time 1784191865.90 total 40.03 last 112.79)
total: 4500 /unlink in 41.25 seconds: 109.08 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95345 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg446-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg446-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server
stop mds service on oleg446-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server
stop mds service on oleg446-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg446-server

unloading modules via unload_modules_local on: 'oleg446-server'
oleg446-server: oleg446-server.virtnet: executing unload_modules_local
oleg446-server: 
modules unloaded.