== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:14:18 (1789028058)
Stopping clients: oleg157-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg157-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg157-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg157-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg157-server
oleg157-server: oleg157-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg157-server'
oleg157-server: oleg157-server.virtnet: executing load_modules_local
oleg157-server: kmemleak disabled
oleg157-server: Loading modules from /home/green/git/lustre-release/lustre
oleg157-server: detected 4 online CPUs by sysfs
oleg157-server: MODOPTS_LIBCFS=
oleg157-server: Force libcfs to create 2 CPU partitions
oleg157-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg157-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg157-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg157-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg157-client.virtnet:  -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 681 (time 1789028284.80 total 10.01 last 68.06)
 - open/close 1443 (time 1789028294.81 total 20.01 last 76.17)
 - open/close 2155 (time 1789028304.83 total 30.03 last 71.04)
 - open/close 2880 (time 1789028314.83 total 40.04 last 72.47)
 - open/close 3621 (time 1789028324.84 total 50.04 last 74.04)
 - open/close 4366 (time 1789028334.84 total 60.05 last 74.49)
total: 4500 open/close in 61.82 seconds: 72.79 ops/second
 - /unlink 1306 (time 1789028351.52 total 10.00 last 130.59)
 - /unlink 2589 (time 1789028361.53 total 20.01 last 128.22)
 - /unlink 4042 (time 1789028371.55 total 30.02 last 145.05)
total: 4500 /unlink in 33.41 seconds: 134.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 697 (time 1789028390.30 total 10.00 last 69.69)
 - open/close 1325 (time 1789028400.31 total 20.01 last 62.75)
 - open/close 1820 (time 1789028410.35 total 30.04 last 49.33)
 - open/close 2390 (time 1789028420.36 total 40.05 last 56.94)
 - open/close 3042 (time 1789028430.37 total 50.06 last 65.14)
 - open/close 3657 (time 1789028440.37 total 60.07 last 61.47)
 - open/close 4429 (time 1789028450.38 total 70.08 last 77.11)
total: 4500 open/close in 71.54 seconds: 62.90 ops/second
 - /unlink 1301 (time 1789028468.14 total 10.01 last 130.01)
 - /unlink 2504 (time 1789028478.15 total 20.01 last 120.26)
 - /unlink 3723 (time 1789028488.15 total 30.02 last 121.82)
total: 4500 /unlink in 36.17 seconds: 124.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 692 (time 1789028510.12 total 10.00 last 69.18)
 - open/close 1409 (time 1789028520.13 total 20.01 last 71.66)
 - open/close 2031 (time 1789028530.13 total 30.01 last 62.19)
 - open/close 2747 (time 1789028540.13 total 40.01 last 71.58)
 - open/close 3530 (time 1789028550.14 total 50.02 last 78.24)
 - open/close 4256 (time 1789028560.14 total 60.02 last 72.60)
total: 4500 open/close in 63.07 seconds: 71.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1326 (time 1789028577.58 total 10.00 last 132.60)
 - /unlink 2547 (time 1789028587.58 total 20.00 last 122.10)
 - /unlink 3364 (time 1789028597.59 total 30.01 last 81.61)
 - /unlink 4386 (time 1789028607.59 total 40.02 last 102.16)
total: 4500 /unlink in 40.98 seconds: 109.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 741 (time 1789028624.72 total 10.02 last 73.96)
 - open/close 1366 (time 1789028634.73 total 20.03 last 62.43)
 - open/close 1971 (time 1789028644.75 total 30.05 last 60.38)
 - open/close 2555 (time 1789028654.76 total 40.06 last 58.31)
 - open/close 3133 (time 1789028664.76 total 50.07 last 57.78)
 - open/close 3698 (time 1789028674.78 total 60.08 last 56.41)
 - open/close 4254 (time 1789028684.79 total 70.10 last 55.53)
total: 4500 open/close in 76.56 seconds: 58.78 ops/second
 - /unlink 1358 (time 1789028706.52 total 10.00 last 135.78)
 - /unlink 2591 (time 1789028716.54 total 20.02 last 123.11)
 - /unlink 3884 (time 1789028726.54 total 30.02 last 129.28)
total: 4500 /unlink in 34.52 seconds: 130.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 688 (time 1789028746.45 total 10.03 last 68.57)
 - open/close 1304 (time 1789028756.46 total 20.05 last 61.52)
 - open/close 2070 (time 1789028766.46 total 30.05 last 76.60)
 - open/close 2734 (time 1789028776.49 total 40.08 last 66.20)
 - open/close 3451 (time 1789028786.50 total 50.08 last 71.65)
 - open/close 4176 (time 1789028796.51 total 60.09 last 72.43)
total: 4500 open/close in 64.48 seconds: 69.79 ops/second
 - /unlink 1256 (time 1789028816.56 total 10.01 last 125.41)
 - /unlink 2201 (time 1789028826.56 total 20.02 last 94.47)
 - /unlink 3546 (time 1789028836.57 total 30.02 last 134.48)
total: 4500 /unlink in 37.51 seconds: 119.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 602 (time 1789028859.92 total 10.00 last 60.20)
 - open/close 1245 (time 1789028869.93 total 20.01 last 64.26)
 - open/close 1549 (time 1789028879.93 total 30.01 last 30.39)
 - open/close 1993 (time 1789028889.94 total 40.01 last 44.39)
 - open/close 2562 (time 1789028899.96 total 50.04 last 56.77)
 - open/close 3349 (time 1789028909.96 total 60.04 last 78.67)
 - open/close 4001 (time 1789028919.98 total 70.06 last 65.07)
total: 4500 open/close in 76.32 seconds: 58.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1193 (time 1789028941.03 total 10.00 last 119.25)
 - /unlink 2660 (time 1789028951.03 total 20.01 last 146.67)
 - /unlink 4117 (time 1789028961.03 total 30.01 last 145.63)
total: 4500 /unlink in 32.47 seconds: 138.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 593 (time 1789028978.58 total 10.00 last 59.28)
 - open/close 1173 (time 1789028988.64 total 20.06 last 57.69)
 - open/close 1767 (time 1789028998.66 total 30.08 last 59.28)
 - open/close 2278 (time 1789029008.66 total 40.08 last 51.08)
 - open/close 2871 (time 1789029018.67 total 50.09 last 59.24)
 - open/close 3606 (time 1789029028.69 total 60.11 last 73.35)
 - open/close 4357 (time 1789029038.71 total 70.13 last 74.99)
total: 4500 open/close in 71.93 seconds: 62.56 ops/second
 - /unlink 1166 (time 1789029055.40 total 10.00 last 116.58)
 - /unlink 2489 (time 1789029065.40 total 20.01 last 132.24)
 - /unlink 3890 (time 1789029075.41 total 30.01 last 139.99)
total: 4500 /unlink in 35.45 seconds: 126.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 661 (time 1789029096.27 total 10.02 last 65.99)
 - open/close 1349 (time 1789029106.28 total 20.03 last 68.73)
 - open/close 1918 (time 1789029116.29 total 30.03 last 56.88)
 - open/close 2615 (time 1789029126.31 total 40.05 last 69.57)
 - open/close 3244 (time 1789029136.35 total 50.10 last 62.60)
 - open/close 3728 (time 1789029146.37 total 60.11 last 48.32)
 - open/close 4422 (time 1789029156.37 total 70.12 last 69.36)
total: 4500 open/close in 71.89 seconds: 62.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1108 (time 1789029173.94 total 10.01 last 110.71)
 - /unlink 2125 (time 1789029183.94 total 20.01 last 101.65)
 - /unlink 3253 (time 1789029193.94 total 30.01 last 112.78)
total: 4500 /unlink in 38.27 seconds: 117.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 688 (time 1789029218.31 total 10.01 last 68.70)
 - open/close 1286 (time 1789029228.33 total 20.03 last 59.70)
 - open/close 1899 (time 1789029238.36 total 30.06 last 61.14)
 - open/close 2491 (time 1789029248.36 total 40.06 last 59.18)
 - open/close 3168 (time 1789029258.37 total 50.07 last 67.65)
 - open/close 3848 (time 1789029268.37 total 60.07 last 67.99)
 - open/close 4487 (time 1789029278.37 total 70.07 last 63.88)
total: 4500 open/close in 70.24 seconds: 64.07 ops/second
 - /unlink 1088 (time 1789029294.30 total 10.01 last 108.74)
 - /unlink 2322 (time 1789029304.31 total 20.02 last 123.26)
 - /unlink 3494 (time 1789029314.31 total 30.02 last 117.19)
total: 4500 /unlink in 37.91 seconds: 118.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 670 (time 1789029338.62 total 10.01 last 66.91)
 - open/close 1333 (time 1789029348.63 total 20.02 last 66.26)
 - open/close 2102 (time 1789029358.63 total 30.02 last 76.87)
 - open/close 2641 (time 1789029368.65 total 40.04 last 53.82)
 - open/close 3177 (time 1789029378.66 total 50.05 last 53.55)
 - open/close 3634 (time 1789029388.68 total 60.07 last 45.60)
 - open/close 4245 (time 1789029398.69 total 70.08 last 61.03)
total: 4500 open/close in 73.62 seconds: 61.12 ops/second
 - /unlink 1383 (time 1789029417.50 total 10.01 last 138.21)
 - /unlink 2609 (time 1789029427.51 total 20.02 last 122.49)
 - /unlink 3768 (time 1789029437.52 total 30.02 last 115.87)
total: 4500 /unlink in 36.59 seconds: 122.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 608 (time 1789029460.40 total 10.00 last 60.77)
 - open/close 1100 (time 1789029470.43 total 20.04 last 49.03)
 - open/close 1733 (time 1789029480.46 total 30.07 last 63.11)
 - open/close 2295 (time 1789029490.48 total 40.08 last 56.13)
 - open/close 3108 (time 1789029500.48 total 50.09 last 81.26)
 - open/close 3767 (time 1789029510.48 total 60.09 last 65.89)
total: 4500 open/close in 69.79 seconds: 64.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1347 (time 1789029534.73 total 10.02 last 134.48)
 - /unlink 2600 (time 1789029544.76 total 20.05 last 124.90)
 - /unlink 3935 (time 1789029554.76 total 30.05 last 133.47)
total: 4500 /unlink in 34.25 seconds: 131.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 754 (time 1789029574.92 total 10.01 last 75.36)
 - open/close 1604 (time 1789029584.92 total 20.01 last 84.99)
 - open/close 2470 (time 1789029594.93 total 30.01 last 86.56)
 - open/close 3253 (time 1789029604.94 total 40.03 last 78.16)
 - open/close 3940 (time 1789029614.95 total 50.03 last 68.67)
total: 4500 open/close in 58.36 seconds: 77.11 ops/second
 - /unlink 1154 (time 1789029639.19 total 10.01 last 115.32)
 - /unlink 2482 (time 1789029649.20 total 20.01 last 132.75)
 - /unlink 3904 (time 1789029659.20 total 30.01 last 142.19)
total: 4500 /unlink in 35.97 seconds: 125.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 632 (time 1789029681.31 total 10.02 last 63.10)
 - open/close 1285 (time 1789029691.32 total 20.03 last 65.21)
 - open/close 2035 (time 1789029701.33 total 30.04 last 74.94)
 - open/close 2837 (time 1789029711.33 total 40.04 last 80.18)
 - open/close 3543 (time 1789029721.33 total 50.04 last 70.59)
 - open/close 4355 (time 1789029731.35 total 60.06 last 81.04)
total: 4500 open/close in 62.17 seconds: 72.39 ops/second
 - /unlink 1423 (time 1789029747.63 total 10.00 last 142.29)
 - /unlink 2831 (time 1789029757.63 total 20.00 last 140.78)
 - /unlink 4080 (time 1789029767.63 total 30.01 last 124.81)
total: 4500 /unlink in 34.25 seconds: 131.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 640 (time 1789029788.77 total 10.01 last 63.96)
 - open/close 1297 (time 1789029798.80 total 20.03 last 65.54)
 - open/close 1819 (time 1789029808.80 total 30.04 last 52.17)
 - open/close 2395 (time 1789029818.81 total 40.05 last 57.55)
 - open/close 3119 (time 1789029828.83 total 50.06 last 72.28)
 - open/close 3797 (time 1789029838.83 total 60.06 last 67.80)
total: 4500 open/close in 68.54 seconds: 65.66 ops/second
 - /unlink 1522 (time 1789029862.14 total 10.00 last 152.18)
 - /unlink 2719 (time 1789029872.15 total 20.00 last 119.67)
 - /unlink 4142 (time 1789029882.15 total 30.01 last 142.27)
total: 4500 /unlink in 33.09 seconds: 136.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 745 (time 1789029900.57 total 10.00 last 74.47)
 - open/close 1361 (time 1789029910.57 total 20.01 last 61.59)
 - open/close 1865 (time 1789029920.58 total 30.01 last 50.37)
 - open/close 2502 (time 1789029930.59 total 40.02 last 63.64)
 - open/close 3086 (time 1789029940.59 total 50.02 last 58.39)
 - open/close 3770 (time 1789029950.59 total 60.03 last 68.37)
 - open/close 4361 (time 1789029960.62 total 70.05 last 58.94)
total: 4500 open/close in 72.20 seconds: 62.32 ops/second
 - /unlink 1083 (time 1789029977.99 total 10.00 last 108.28)
 - /unlink 2213 (time 1789029987.99 total 20.00 last 113.00)
 - /unlink 3598 (time 1789029997.99 total 30.01 last 138.46)
total: 4500 /unlink in 35.57 seconds: 126.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 798 (time 1789030017.51 total 10.02 last 79.66)
 - open/close 1596 (time 1789030027.51 total 20.02 last 79.79)
 - open/close 2345 (time 1789030037.53 total 30.03 last 74.80)
 - open/close 2966 (time 1789030047.54 total 40.05 last 62.01)
 - open/close 3550 (time 1789030057.55 total 50.06 last 58.34)
 - open/close 4222 (time 1789030067.56 total 60.06 last 67.17)
total: 4500 open/close in 64.84 seconds: 69.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1337 (time 1789030089.23 total 10.00 last 133.64)
 - /unlink 2408 (time 1789030099.23 total 20.01 last 107.07)
 - /unlink 3532 (time 1789030109.26 total 30.04 last 112.04)
total: 4500 /unlink in 38.46 seconds: 117.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 909 (time 1789030132.36 total 10.00 last 90.86)
 - open/close 1574 (time 1789030142.37 total 20.01 last 66.46)
 - open/close 2270 (time 1789030152.38 total 30.03 last 69.49)
 - open/close 2921 (time 1789030162.39 total 40.04 last 65.03)
 - open/close 3652 (time 1789030172.40 total 50.04 last 73.04)
 - open/close 4272 (time 1789030182.42 total 60.06 last 61.91)
total: 4500 open/close in 63.60 seconds: 70.76 ops/second
 - /unlink 1553 (time 1789030200.50 total 10.01 last 155.13)
 - /unlink 2776 (time 1789030210.51 total 20.02 last 122.22)
 - /unlink 4059 (time 1789030220.51 total 30.02 last 128.28)
total: 4500 /unlink in 33.71 seconds: 133.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 614 (time 1789030241.20 total 10.01 last 61.37)
 - open/close 1339 (time 1789030251.21 total 20.01 last 72.46)
 - open/close 2119 (time 1789030261.21 total 30.01 last 77.98)
 - open/close 2980 (time 1789030271.22 total 40.02 last 86.01)
 - open/close 3724 (time 1789030281.23 total 50.03 last 74.34)
 - open/close 4416 (time 1789030291.23 total 60.04 last 69.18)
total: 4500 open/close in 60.96 seconds: 73.82 ops/second
 - /unlink 1369 (time 1789030307.61 total 10.01 last 136.83)
 - /unlink 2633 (time 1789030317.62 total 20.01 last 126.34)
 - /unlink 3575 (time 1789030327.62 total 30.01 last 94.18)
total: 4500 /unlink in 38.24 seconds: 117.69 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95340 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg157-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg157-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server
stop mds service on oleg157-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server
stop mds service on oleg157-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg157-server
unloading modules via unload_modules_local on: 'oleg157-server'
oleg157-server: oleg157-server.virtnet: executing unload_modules_local
oleg157-server: kmemleak disabled
modules unloaded.