== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:26:26 (1790799986)
Stopping clients: oleg415-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg415-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg415-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg415-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg415-server
oleg415-server: oleg415-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing load_modules_local
oleg415-server: kmemleak disabled
oleg415-server: Loading modules from /home/green/git/lustre-release/lustre
oleg415-server: detected 4 online CPUs by sysfs
oleg415-server: MODOPTS_LIBCFS=
oleg415-server: Force libcfs to create 2 CPU partitions
oleg415-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg415-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg415-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg415-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 719 (time 1790800202.88 total 10.01 last 71.86)
 - open/close 1400 (time 1790800212.88 total 20.01 last 68.08)
 - open/close 1998 (time 1790800222.90 total 30.02 last 59.73)
 - open/close 2506 (time 1790800232.92 total 40.04 last 50.69)
 - open/close 3107 (time 1790800242.92 total 50.05 last 60.08)
 - open/close 3797 (time 1790800252.93 total 60.05 last 68.95)
total: 4500 open/close in 69.48 seconds: 64.76 ops/second
 - /unlink 1285 (time 1790800276.30 total 10.00 last 128.49)
 - /unlink 2710 (time 1790800286.30 total 20.01 last 142.40)
 - /unlink 4022 (time 1790800296.31 total 30.01 last 131.17)
total: 4500 /unlink in 33.90 seconds: 132.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 726 (time 1790800314.97 total 10.00 last 72.58)
 - open/close 1372 (time 1790800324.98 total 20.01 last 64.57)
 - open/close 2011 (time 1790800334.98 total 30.02 last 63.85)
 - open/close 2677 (time 1790800344.99 total 40.02 last 66.60)
 - open/close 3303 (time 1790800354.99 total 50.02 last 62.59)
 - open/close 4002 (time 1790800365.00 total 60.03 last 69.81)
total: 4500 open/close in 66.85 seconds: 67.31 ops/second
 - /unlink 1258 (time 1790800386.39 total 10.01 last 125.72)
 - /unlink 2445 (time 1790800396.39 total 20.01 last 118.66)
 - /unlink 3779 (time 1790800406.39 total 30.01 last 133.38)
total: 4500 /unlink in 35.65 seconds: 126.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 676 (time 1790800426.99 total 10.01 last 67.55)
 - open/close 1355 (time 1790800436.99 total 20.01 last 67.88)
 - open/close 2058 (time 1790800447.00 total 30.02 last 70.21)
 - open/close 2697 (time 1790800457.00 total 40.02 last 63.90)
 - open/close 3367 (time 1790800467.01 total 50.03 last 66.95)
 - open/close 4030 (time 1790800477.02 total 60.05 last 66.20)
total: 4500 open/close in 66.40 seconds: 67.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1311 (time 1790800497.96 total 10.00 last 131.04)
 - /unlink 2514 (time 1790800507.96 total 20.01 last 120.26)
 - /unlink 3854 (time 1790800517.96 total 30.01 last 133.99)
total: 4500 /unlink in 35.34 seconds: 127.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 622 (time 1790800538.33 total 10.02 last 62.07)
 - open/close 1254 (time 1790800548.33 total 20.02 last 63.19)
 - open/close 1907 (time 1790800558.34 total 30.03 last 65.24)
 - open/close 2601 (time 1790800568.35 total 40.04 last 69.36)
 - open/close 3221 (time 1790800578.36 total 50.05 last 61.91)
 - open/close 3917 (time 1790800588.37 total 60.06 last 69.56)
total: 4500 open/close in 68.60 seconds: 65.59 ops/second
 - /unlink 1277 (time 1790800611.64 total 10.00 last 127.65)
 - /unlink 2445 (time 1790800621.65 total 20.01 last 116.74)
 - /unlink 3605 (time 1790800631.65 total 30.01 last 115.93)
total: 4500 /unlink in 37.35 seconds: 120.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 658 (time 1790800654.31 total 10.00 last 65.79)
 - open/close 1263 (time 1790800664.32 total 20.01 last 60.44)
 - open/close 1838 (time 1790800674.33 total 30.02 last 57.43)
 - open/close 2416 (time 1790800684.35 total 40.04 last 57.72)
 - open/close 2975 (time 1790800694.35 total 50.04 last 55.87)
 - open/close 3476 (time 1790800704.36 total 60.05 last 50.05)
 - open/close 3964 (time 1790800714.38 total 70.07 last 48.73)
total: 4500 open/close in 79.84 seconds: 56.36 ops/second
 - /unlink 1155 (time 1790800739.25 total 10.00 last 115.49)
 - /unlink 2379 (time 1790800749.25 total 20.01 last 122.32)
 - /unlink 3518 (time 1790800759.26 total 30.01 last 113.86)
total: 4500 /unlink in 39.17 seconds: 114.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 656 (time 1790800783.61 total 10.01 last 65.53)
 - open/close 1284 (time 1790800793.63 total 20.02 last 62.71)
 - open/close 1933 (time 1790800803.65 total 30.04 last 64.79)
 - open/close 2577 (time 1790800813.65 total 40.05 last 64.36)
 - open/close 3177 (time 1790800823.66 total 50.06 last 59.96)
 - open/close 3697 (time 1790800833.67 total 60.07 last 51.95)
 - open/close 4307 (time 1790800843.67 total 70.07 last 60.98)
total: 4500 open/close in 73.46 seconds: 61.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1220 (time 1790800861.98 total 10.01 last 121.90)
 - /unlink 2425 (time 1790800871.98 total 20.01 last 120.46)
 - /unlink 3703 (time 1790800881.99 total 30.01 last 127.77)
total: 4500 /unlink in 37.19 seconds: 120.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 690 (time 1790800904.85 total 10.02 last 68.90)
 - open/close 1299 (time 1790800914.86 total 20.02 last 60.85)
 - open/close 1867 (time 1790800924.87 total 30.03 last 56.74)
 - open/close 2557 (time 1790800934.87 total 40.04 last 68.98)
 - open/close 3205 (time 1790800944.88 total 50.04 last 64.75)
 - open/close 3847 (time 1790800954.88 total 60.05 last 64.19)
 - open/close 4454 (time 1790800964.89 total 70.06 last 60.63)
total: 4500 open/close in 70.64 seconds: 63.71 ops/second
 - /unlink 1312 (time 1790800981.74 total 10.01 last 131.10)
 - /unlink 2538 (time 1790800991.74 total 20.01 last 122.56)
 - /unlink 3737 (time 1790801001.75 total 30.02 last 119.82)
total: 4500 /unlink in 37.09 seconds: 121.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 539 (time 1790801025.29 total 10.01 last 53.86)
 - open/close 1167 (time 1790801035.30 total 20.01 last 62.78)
 - open/close 1765 (time 1790801045.30 total 30.02 last 59.75)
 - open/close 2405 (time 1790801055.31 total 40.02 last 63.98)
 - open/close 3101 (time 1790801065.31 total 50.03 last 69.55)
 - open/close 3746 (time 1790801075.34 total 60.05 last 64.36)
 - open/close 4358 (time 1790801085.35 total 70.07 last 61.10)
total: 4500 open/close in 72.12 seconds: 62.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 910 (time 1790801102.73 total 10.00 last 91.00)
 - /unlink 2050 (time 1790801112.73 total 20.00 last 113.94)
 - /unlink 2960 (time 1790801122.74 total 30.01 last 90.97)
 - /unlink 3864 (time 1790801132.76 total 40.03 last 90.17)
total: 4500 /unlink in 46.21 seconds: 97.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 614 (time 1790801154.24 total 10.00 last 61.37)
 - open/close 1280 (time 1790801164.26 total 20.02 last 66.49)
 - open/close 1941 (time 1790801174.26 total 30.02 last 66.09)
 - open/close 2425 (time 1790801184.26 total 40.03 last 48.38)
 - open/close 2833 (time 1790801194.29 total 50.05 last 40.68)
 - open/close 3313 (time 1790801204.29 total 60.06 last 47.99)
 - open/close 3806 (time 1790801214.30 total 70.07 last 49.26)
 - open/close 4340 (time 1790801224.30 total 80.07 last 53.40)
total: 4500 open/close in 82.80 seconds: 54.35 ops/second
 - /unlink 1202 (time 1790801241.70 total 10.00 last 120.16)
 - /unlink 2309 (time 1790801251.70 total 20.00 last 110.69)
 - /unlink 3409 (time 1790801261.71 total 30.01 last 109.94)
total: 4500 /unlink in 38.10 seconds: 118.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 469 (time 1790801287.57 total 10.01 last 46.85)
 - open/close 1095 (time 1790801297.60 total 20.04 last 62.42)
 - open/close 1735 (time 1790801307.63 total 30.07 last 63.82)
 - open/close 2404 (time 1790801317.63 total 40.07 last 66.86)
 - open/close 3023 (time 1790801327.63 total 50.08 last 61.89)
 - open/close 3654 (time 1790801337.64 total 60.08 last 63.09)
 - open/close 4316 (time 1790801347.65 total 70.09 last 66.09)
total: 4500 open/close in 72.72 seconds: 61.88 ops/second
 - /unlink 1265 (time 1790801365.37 total 10.01 last 126.39)
 - /unlink 2506 (time 1790801375.37 total 20.01 last 124.04)
 - /unlink 3921 (time 1790801385.39 total 30.03 last 141.30)
total: 4500 /unlink in 33.89 seconds: 132.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 657 (time 1790801404.76 total 10.00 last 65.69)
 - open/close 1306 (time 1790801414.78 total 20.01 last 64.82)
 - open/close 1967 (time 1790801424.78 total 30.02 last 66.04)
 - open/close 2604 (time 1790801434.80 total 40.04 last 63.61)
 - open/close 3282 (time 1790801444.80 total 50.04 last 67.80)
 - open/close 3943 (time 1790801454.81 total 60.05 last 66.02)
total: 4500 open/close in 68.24 seconds: 65.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1180 (time 1790801476.68 total 10.00 last 117.96)
 - /unlink 2052 (time 1790801486.69 total 20.01 last 87.12)
 - /unlink 2775 (time 1790801496.70 total 30.03 last 72.19)
 - /unlink 3625 (time 1790801506.70 total 40.03 last 85.00)
total: 4500 /unlink in 49.31 seconds: 91.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 560 (time 1790801532.01 total 10.01 last 55.97)
 - open/close 1220 (time 1790801542.03 total 20.03 last 65.86)
 - open/close 1817 (time 1790801552.05 total 30.05 last 59.56)
 - open/close 2392 (time 1790801562.08 total 40.08 last 57.32)
 - open/close 3006 (time 1790801572.08 total 50.08 last 61.40)
 - open/close 3676 (time 1790801582.09 total 60.09 last 66.98)
 - open/close 4297 (time 1790801592.09 total 70.09 last 62.09)
total: 4500 open/close in 73.27 seconds: 61.41 ops/second
 - /unlink 876 (time 1790801610.31 total 10.00 last 87.58)
 - /unlink 1796 (time 1790801620.31 total 20.01 last 91.94)
 - /unlink 2872 (time 1790801630.32 total 30.01 last 107.56)
 - /unlink 3986 (time 1790801640.32 total 40.01 last 111.39)
total: 4500 /unlink in 45.53 seconds: 98.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 604 (time 1790801661.25 total 10.03 last 60.22)
 - open/close 1069 (time 1790801671.26 total 20.04 last 46.47)
 - open/close 1596 (time 1790801681.26 total 30.04 last 52.68)
 - open/close 2108 (time 1790801691.27 total 40.05 last 51.15)
 - open/close 2703 (time 1790801701.28 total 50.06 last 59.47)
 - open/close 3386 (time 1790801711.28 total 60.06 last 68.25)
 - open/close 4093 (time 1790801721.29 total 70.07 last 70.68)
total: 4500 open/close in 75.87 seconds: 59.31 ops/second
 - /unlink 1230 (time 1790801741.46 total 10.01 last 122.93)
 - /unlink 2436 (time 1790801751.46 total 20.01 last 120.60)
 - /unlink 3709 (time 1790801761.46 total 30.01 last 127.28)
total: 4500 /unlink in 36.92 seconds: 121.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 696 (time 1790801784.05 total 10.01 last 69.55)
 - open/close 1328 (time 1790801794.06 total 20.02 last 63.10)
 - open/close 2001 (time 1790801804.08 total 30.04 last 67.22)
 - open/close 2722 (time 1790801814.08 total 40.04 last 72.07)
 - open/close 3383 (time 1790801824.09 total 50.05 last 66.02)
 - open/close 3974 (time 1790801834.10 total 60.06 last 59.08)
total: 4500 open/close in 69.10 seconds: 65.12 ops/second
 - /unlink 902 (time 1790801858.91 total 10.00 last 90.16)
 - /unlink 1728 (time 1790801868.92 total 20.01 last 82.58)
 - /unlink 2801 (time 1790801878.92 total 30.01 last 107.22)
 - /unlink 3731 (time 1790801888.93 total 40.02 last 92.93)
total: 4500 /unlink in 47.11 seconds: 95.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 609 (time 1790801911.38 total 10.01 last 60.83)
 - open/close 1199 (time 1790801921.38 total 20.01 last 58.99)
 - open/close 1821 (time 1790801931.38 total 30.01 last 62.20)
 - open/close 2400 (time 1790801941.39 total 40.02 last 57.84)
 - open/close 3009 (time 1790801951.41 total 50.04 last 60.79)
 - open/close 3535 (time 1790801961.43 total 60.06 last 52.49)
 - open/close 3929 (time 1790801971.43 total 70.06 last 39.39)
 - open/close 4424 (time 1790801981.43 total 80.07 last 49.49)
total: 4500 open/close in 81.14 seconds: 55.46 ops/second
 - /unlink 1120 (time 1790801997.40 total 10.00 last 111.97)
 - /unlink 2277 (time 1790802007.40 total 20.01 last 115.63)
 - /unlink 3412 (time 1790802017.41 total 30.01 last 113.45)
total: 4500 /unlink in 38.56 seconds: 116.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 641 (time 1790802041.30 total 10.00 last 64.09)
 - open/close 1103 (time 1790802051.31 total 20.01 last 46.16)
 - open/close 1533 (time 1790802061.34 total 30.05 last 42.85)
 - open/close 1922 (time 1790802071.37 total 40.07 last 38.81)
 - open/close 2505 (time 1790802081.37 total 50.07 last 58.29)
 - open/close 3074 (time 1790802091.37 total 60.07 last 56.89)
 - open/close 3600 (time 1790802101.40 total 70.10 last 52.47)
 - open/close 4203 (time 1790802111.41 total 80.11 last 60.23)
total: 4500 open/close in 84.35 seconds: 53.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 961 (time 1790802132.17 total 10.00 last 96.06)
 - /unlink 2120 (time 1790802142.18 total 20.01 last 115.84)
 - /unlink 3362 (time 1790802152.19 total 30.02 last 124.06)
total: 4500 /unlink in 38.85 seconds: 115.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 492 (time 1790802176.93 total 10.02 last 49.12)
 - open/close 951 (time 1790802186.93 total 20.02 last 45.90)
 - open/close 1528 (time 1790802196.94 total 30.02 last 57.66)
 - open/close 2188 (time 1790802206.94 total 40.03 last 65.98)
 - open/close 2739 (time 1790802216.95 total 50.03 last 55.06)
 - open/close 3224 (time 1790802226.95 total 60.04 last 48.49)
 - open/close 3608 (time 1790802237.00 total 70.08 last 38.22)
 - open/close 4176 (time 1790802247.02 total 80.11 last 56.65)
total: 4500 open/close in 86.35 seconds: 52.11 ops/second
 - /unlink 1154 (time 1790802268.47 total 10.01 last 115.31)
 - /unlink 2329 (time 1790802278.47 total 20.01 last 117.45)
 - /unlink 3534 (time 1790802288.47 total 30.01 last 120.49)
total: 4500 /unlink in 38.99 seconds: 115.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 516 (time 1790802313.59 total 10.02 last 51.52)
 - open/close 1191 (time 1790802323.60 total 20.02 last 67.45)
 - open/close 1826 (time 1790802333.61 total 30.03 last 63.44)
 - open/close 2363 (time 1790802343.62 total 40.04 last 53.64)
 - open/close 2998 (time 1790802353.62 total 50.05 last 63.49)
 - open/close 3608 (time 1790802363.63 total 60.05 last 60.98)
 - open/close 4242 (time 1790802373.64 total 70.06 last 63.31)
total: 4500 open/close in 73.98 seconds: 60.83 ops/second
 - /unlink 1100 (time 1790802393.17 total 10.00 last 109.96)
 - /unlink 2288 (time 1790802403.18 total 20.01 last 118.76)
 - /unlink 3367 (time 1790802413.18 total 30.01 last 107.90)
total: 4500 /unlink in 39.02 seconds: 115.33 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11358: 95564 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg415-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg415-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server
stop mds service on oleg415-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server
stop mds service on oleg415-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg415-server
unloading modules via unload_modules_local on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing unload_modules_local
oleg415-server: kmemleak disabled
modules unloaded.