== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:16:48 (1788848208)
Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg339-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server
oleg339-server: oleg339-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg339-server'
oleg339-server: oleg339-server.virtnet: executing load_modules_local
oleg339-server: kmemleak disabled
oleg339-server: Loading modules from /home/green/git/lustre-release/lustre
oleg339-server: detected 4 online CPUs by sysfs
oleg339-server: MODOPTS_LIBCFS=
oleg339-server: Force libcfs to create 2 CPU partitions
oleg339-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg339-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg339-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg339-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg339-client.virtnet:  -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 725 (time 1788848424.58 total 10.01 last 72.46)
 - open/close 1334 (time 1788848434.62 total 20.05 last 60.61)
 - open/close 2094 (time 1788848444.63 total 30.06 last 75.97)
 - open/close 2750 (time 1788848454.65 total 40.08 last 65.44)
 - open/close 3435 (time 1788848464.66 total 50.09 last 68.45)
 - open/close 4014 (time 1788848474.66 total 60.09 last 57.90)
total: 4500 open/close in 68.63 seconds: 65.56 ops/second
 - /unlink 1151 (time 1788848499.30 total 10.01 last 114.98)
 - /unlink 2033 (time 1788848509.30 total 20.02 last 88.12)
 - /unlink 3218 (time 1788848519.32 total 30.03 last 118.34)
 - /unlink 4385 (time 1788848529.32 total 40.04 last 116.63)
total: 4500 /unlink in 41.18 seconds: 109.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 636 (time 1788848546.68 total 10.00 last 63.59)
 - open/close 1205 (time 1788848556.69 total 20.01 last 56.84)
 - open/close 1866 (time 1788848566.69 total 30.02 last 66.07)
 - open/close 2434 (time 1788848576.70 total 40.02 last 56.77)
 - open/close 2980 (time 1788848586.72 total 50.05 last 54.46)
 - open/close 3579 (time 1788848596.73 total 60.06 last 59.85)
 - open/close 4165 (time 1788848606.73 total 70.06 last 58.59)
total: 4500 open/close in 75.98 seconds: 59.22 ops/second
 - /unlink 992 (time 1788848629.33 total 10.00 last 99.18)
 - /unlink 1951 (time 1788848639.34 total 20.01 last 95.81)
 - /unlink 3075 (time 1788848649.34 total 30.02 last 112.33)
 - /unlink 4263 (time 1788848659.35 total 40.02 last 118.76)
total: 4500 /unlink in 42.19 seconds: 106.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 603 (time 1788848677.13 total 10.00 last 60.29)
 - open/close 1234 (time 1788848687.13 total 20.01 last 63.07)
 - open/close 1887 (time 1788848697.13 total 30.01 last 65.28)
 - open/close 2564 (time 1788848707.14 total 40.01 last 67.68)
 - open/close 3294 (time 1788848717.15 total 50.02 last 72.94)
 - open/close 3849 (time 1788848727.17 total 60.04 last 55.38)
 - open/close 4408 (time 1788848737.18 total 70.05 last 55.85)
total: 4500 open/close in 71.50 seconds: 62.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1157 (time 1788848753.90 total 10.00 last 115.70)
 - /unlink 2222 (time 1788848763.91 total 20.00 last 106.46)
 - /unlink 3326 (time 1788848773.91 total 30.01 last 110.37)
total: 4500 /unlink in 39.16 seconds: 114.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 621 (time 1788848798.40 total 10.01 last 62.03)
 - open/close 1247 (time 1788848808.41 total 20.02 last 62.52)
 - open/close 1931 (time 1788848818.42 total 30.03 last 68.37)
 - open/close 2456 (time 1788848828.44 total 40.05 last 52.39)
 - open/close 3168 (time 1788848838.44 total 50.05 last 71.19)
 - open/close 3788 (time 1788848848.46 total 60.07 last 61.87)
 - open/close 4397 (time 1788848858.47 total 70.08 last 60.86)
total: 4500 open/close in 72.08 seconds: 62.43 ops/second
 - /unlink 1199 (time 1788848875.93 total 10.01 last 119.84)
 - /unlink 2242 (time 1788848885.94 total 20.01 last 104.25)
 - /unlink 3371 (time 1788848895.94 total 30.01 last 112.84)
 - /unlink 4462 (time 1788848905.95 total 40.02 last 109.05)
total: 4500 /unlink in 40.35 seconds: 111.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 644 (time 1788848921.59 total 10.02 last 64.30)
 - open/close 1337 (time 1788848931.60 total 20.02 last 69.25)
 - open/close 1998 (time 1788848941.61 total 30.04 last 66.00)
 - open/close 2648 (time 1788848951.62 total 40.04 last 64.97)
 - open/close 3271 (time 1788848961.63 total 50.06 last 62.22)
 - open/close 3900 (time 1788848971.64 total 60.07 last 62.83)
total: 4500 open/close in 68.62 seconds: 65.58 ops/second
 - /unlink 1230 (time 1788848995.47 total 10.00 last 122.97)
 - /unlink 2421 (time 1788849005.48 total 20.01 last 118.97)
 - /unlink 3589 (time 1788849015.49 total 30.02 last 116.73)
total: 4500 /unlink in 37.59 seconds: 119.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 675 (time 1788849038.10 total 10.02 last 67.37)
 - open/close 1294 (time 1788849048.13 total 20.05 last 61.73)
 - open/close 1995 (time 1788849058.13 total 30.06 last 70.04)
 - open/close 2709 (time 1788849068.14 total 40.06 last 71.38)
 - open/close 3386 (time 1788849078.15 total 50.07 last 67.61)
 - open/close 4009 (time 1788849088.16 total 60.08 last 62.23)
total: 4500 open/close in 67.20 seconds: 66.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1220 (time 1788849110.40 total 10.00 last 121.99)
 - /unlink 2349 (time 1788849120.41 total 20.01 last 112.84)
 - /unlink 3589 (time 1788849130.41 total 30.01 last 123.95)
total: 4500 /unlink in 37.40 seconds: 120.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 682 (time 1788849153.87 total 10.00 last 68.19)
 - open/close 1304 (time 1788849163.87 total 20.00 last 62.20)
 - open/close 1871 (time 1788849173.88 total 30.01 last 56.67)
 - open/close 2485 (time 1788849183.88 total 40.01 last 61.39)
 - open/close 3133 (time 1788849193.88 total 50.01 last 64.77)
 - open/close 3740 (time 1788849203.89 total 60.02 last 60.66)
 - open/close 4449 (time 1788849213.90 total 70.03 last 70.85)
total: 4500 open/close in 70.78 seconds: 63.58 ops/second
 - /unlink 1280 (time 1788849229.50 total 10.01 last 127.92)
 - /unlink 2487 (time 1788849239.51 total 20.01 last 120.65)
 - /unlink 3674 (time 1788849249.51 total 30.01 last 118.67)
total: 4500 /unlink in 36.38 seconds: 123.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 586 (time 1788849270.49 total 10.02 last 58.47)
 - open/close 1158 (time 1788849280.51 total 20.04 last 57.10)
 - open/close 1828 (time 1788849290.52 total 30.05 last 66.91)
 - open/close 2518 (time 1788849300.52 total 40.06 last 68.99)
 - open/close 3146 (time 1788849310.54 total 50.07 last 62.71)
 - open/close 3817 (time 1788849320.56 total 60.09 last 66.97)
 - open/close 4479 (time 1788849330.56 total 70.09 last 66.19)
total: 4500 open/close in 70.37 seconds: 63.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1219 (time 1788849345.29 total 10.00 last 121.84)
 - /unlink 2439 (time 1788849355.29 total 20.01 last 121.96)
 - /unlink 3709 (time 1788849365.30 total 30.02 last 126.89)
total: 4500 /unlink in 36.74 seconds: 122.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 748 (time 1788849386.33 total 10.01 last 74.75)
 - open/close 1472 (time 1788849396.33 total 20.01 last 72.37)
 - open/close 2206 (time 1788849406.33 total 30.01 last 73.40)
 - open/close 2937 (time 1788849416.33 total 40.01 last 73.09)
 - open/close 3616 (time 1788849426.33 total 50.01 last 67.90)
 - open/close 4313 (time 1788849436.36 total 60.04 last 69.54)
total: 4500 open/close in 62.73 seconds: 71.73 ops/second
 - /unlink 1348 (time 1788849453.18 total 10.01 last 134.72)
 - /unlink 2570 (time 1788849463.18 total 20.01 last 122.18)
 - /unlink 3817 (time 1788849473.18 total 30.01 last 124.69)
total: 4500 /unlink in 35.66 seconds: 126.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 668 (time 1788849492.90 total 10.01 last 66.75)
 - open/close 1355 (time 1788849502.91 total 20.01 last 68.67)
 - open/close 2044 (time 1788849512.91 total 30.01 last 68.88)
 - open/close 2751 (time 1788849522.91 total 40.01 last 70.70)
 - open/close 3460 (time 1788849532.91 total 50.02 last 70.87)
 - open/close 4170 (time 1788849542.92 total 60.02 last 70.97)
total: 4500 open/close in 64.68 seconds: 69.58 ops/second
 - /unlink 1402 (time 1788849561.79 total 10.01 last 140.12)
 - /unlink 2734 (time 1788849571.79 total 20.01 last 133.17)
 - /unlink 4137 (time 1788849581.79 total 30.01 last 140.25)
total: 4500 /unlink in 32.63 seconds: 137.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 756 (time 1788849598.42 total 10.02 last 75.46)
 - open/close 1463 (time 1788849608.42 total 20.03 last 70.65)
 - open/close 2185 (time 1788849618.43 total 30.04 last 72.12)
 - open/close 2955 (time 1788849628.44 total 40.04 last 76.97)
 - open/close 3746 (time 1788849638.45 total 50.05 last 78.99)
 - open/close 4497 (time 1788849648.45 total 60.06 last 75.10)
total: 4500 open/close in 60.08 seconds: 74.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1441 (time 1788849662.38 total 10.01 last 144.02)
 - /unlink 2808 (time 1788849672.38 total 20.01 last 136.68)
 - /unlink 4226 (time 1788849682.38 total 30.01 last 141.79)
total: 4500 /unlink in 31.96 seconds: 140.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 726 (time 1788849698.88 total 10.01 last 72.55)
 - open/close 1465 (time 1788849708.89 total 20.02 last 73.82)
 - open/close 2216 (time 1788849718.89 total 30.02 last 75.10)
 - open/close 2944 (time 1788849728.90 total 40.03 last 72.73)
 - open/close 3665 (time 1788849738.91 total 50.03 last 72.07)
 - open/close 4462 (time 1788849748.91 total 60.03 last 79.68)
total: 4500 open/close in 60.50 seconds: 74.38 ops/second
 - /unlink 1486 (time 1788849763.12 total 10.00 last 148.60)
 - /unlink 3022 (time 1788849773.13 total 20.01 last 153.46)
total: 4500 /unlink in 29.34 seconds: 153.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 851 (time 1788849796.24 total 10.02 last 84.97)
 - open/close 1678 (time 1788849806.25 total 20.02 last 82.67)
 - open/close 2468 (time 1788849816.25 total 30.02 last 79.00)
 - open/close 3301 (time 1788849826.25 total 40.02 last 83.27)
 - open/close 4170 (time 1788849836.25 total 50.02 last 86.89)
total: 4500 open/close in 53.95 seconds: 83.41 ops/second
 - /unlink 1450 (time 1788849853.38 total 10.00 last 144.96)
 - /unlink 2963 (time 1788849863.38 total 20.00 last 151.30)
 - /unlink 4428 (time 1788849873.38 total 30.01 last 146.45)
total: 4500 /unlink in 30.49 seconds: 147.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 794 (time 1788849887.63 total 10.01 last 79.30)
 - open/close 1586 (time 1788849897.64 total 20.02 last 79.14)
 - open/close 2388 (time 1788849907.64 total 30.02 last 80.18)
 - open/close 3208 (time 1788849917.65 total 40.03 last 81.96)
 - open/close 4021 (time 1788849927.65 total 50.04 last 81.23)
total: 4500 open/close in 55.87 seconds: 80.54 ops/second
 - /unlink 1539 (time 1788849946.76 total 10.00 last 153.84)
 - /unlink 3038 (time 1788849956.76 total 20.01 last 149.84)
total: 4500 /unlink in 29.79 seconds: 151.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 802 (time 1788849980.21 total 10.01 last 80.14)
 - open/close 1673 (time 1788849990.22 total 20.01 last 87.07)
 - open/close 2537 (time 1788850000.22 total 30.01 last 86.40)
 - open/close 3380 (time 1788850010.23 total 40.02 last 84.19)
 - open/close 4269 (time 1788850020.23 total 50.03 last 88.88)
total: 4500 open/close in 52.46 seconds: 85.78 ops/second
 - /unlink 1584 (time 1788850035.64 total 10.00 last 158.37)
 - /unlink 3474 (time 1788850045.64 total 20.00 last 188.98)
total: 4500 /unlink in 25.62 seconds: 175.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 970 (time 1788850064.62 total 10.01 last 96.92)
 - open/close 1901 (time 1788850074.62 total 20.01 last 93.07)
 - open/close 2786 (time 1788850084.62 total 30.01 last 88.48)
 - open/close 3701 (time 1788850094.63 total 40.02 last 91.45)
total: 4500 open/close in 48.44 seconds: 92.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1994 (time 1788850115.68 total 10.00 last 199.33)
 - /unlink 3872 (time 1788850125.69 total 20.01 last 187.75)
total: 4500 /unlink in 23.34 seconds: 192.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 1050 (time 1788850141.53 total 10.02 last 104.83)
 - open/close 2043 (time 1788850151.53 total 20.02 last 99.25)
 - open/close 3045 (time 1788850161.54 total 30.03 last 100.14)
 - open/close 3933 (time 1788850171.55 total 40.04 last 88.71)
total: 4500 open/close in 46.14 seconds: 97.53 ops/second
 - /unlink 2175 (time 1788850190.28 total 10.00 last 217.43)
 - /unlink 4240 (time 1788850200.28 total 20.01 last 206.44)
total: 4500 /unlink in 21.30 seconds: 211.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 1157 (time 1788850214.09 total 10.01 last 115.59)
 - open/close 2353 (time 1788850224.10 total 20.02 last 119.50)
 - open/close 3374 (time 1788850234.11 total 30.03 last 102.02)
total: 4500 open/close in 39.93 seconds: 112.70 ops/second
 - /unlink 2349 (time 1788850256.29 total 10.00 last 234.83)
total: 4500 /unlink in 19.65 seconds: 228.96 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95328 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg339-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg339-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server
stop mds service on oleg339-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server
stop mds service on oleg339-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server
unloading modules via unload_modules_local on: 'oleg339-server'
oleg339-server: oleg339-server.virtnet: executing unload_modules_local
oleg339-server: kmemleak disabled
modules unloaded.