== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:08:11 (1789780091)
Stopping clients: oleg342-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg342-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg342-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg342-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg342-server
oleg342-server: oleg342-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg342-server'
oleg342-server: oleg342-server.virtnet: executing load_modules_local
oleg342-server: kmemleak disabled
oleg342-server: Loading modules from /home/green/git/lustre-release/lustre
oleg342-server: detected 4 online CPUs by sysfs
oleg342-server: MODOPTS_LIBCFS=
oleg342-server: Force libcfs to create 2 CPU partitions
oleg342-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg342-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg342-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg342-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg342-client.virtnet:  -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 680 (time 1789780281.81 total 10.00 last 67.99)
 - open/close 1352 (time 1789780291.81 total 20.01 last 67.16)
 - open/close 1953 (time 1789780301.83 total 30.02 last 60.03)
 - open/close 2508 (time 1789780311.85 total 40.04 last 55.38)
 - open/close 3124 (time 1789780321.86 total 50.05 last 61.53)
 - open/close 3716 (time 1789780331.87 total 60.06 last 59.14)
 - open/close 4269 (time 1789780341.87 total 70.06 last 55.30)
total: 4500 open/close in 73.99 seconds: 60.82 ops/second
 - /unlink 1170 (time 1789780361.48 total 10.00 last 116.99)
 - /unlink 2279 (time 1789780371.48 total 20.00 last 110.88)
 - /unlink 3299 (time 1789780381.49 total 30.01 last 101.91)
total: 4500 /unlink in 39.32 seconds: 114.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 726 (time 1789780406.37 total 10.01 last 72.50)
 - open/close 1463 (time 1789780416.37 total 20.02 last 73.67)
 - open/close 2114 (time 1789780426.37 total 30.02 last 65.08)
 - open/close 2768 (time 1789780436.38 total 40.03 last 65.37)
 - open/close 3463 (time 1789780446.39 total 50.03 last 69.44)
 - open/close 4111 (time 1789780456.39 total 60.04 last 64.77)
total: 4500 open/close in 65.44 seconds: 68.76 ops/second
 - /unlink 1234 (time 1789780476.74 total 10.00 last 123.34)
 - /unlink 2585 (time 1789780486.74 total 20.01 last 135.03)
 - /unlink 3916 (time 1789780496.78 total 30.04 last 132.64)
total: 4500 /unlink in 34.37 seconds: 130.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 696 (time 1789780515.53 total 10.02 last 69.45)
 - open/close 1362 (time 1789780525.54 total 20.03 last 66.53)
 - open/close 2036 (time 1789780535.55 total 30.04 last 67.38)
 - open/close 2692 (time 1789780545.56 total 40.04 last 65.55)
 - open/close 3368 (time 1789780555.57 total 50.06 last 67.50)
 - open/close 4071 (time 1789780565.59 total 60.07 last 70.18)
total: 4500 open/close in 66.78 seconds: 67.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1274 (time 1789780587.34 total 10.01 last 127.30)
 - /unlink 2636 (time 1789780597.34 total 20.01 last 136.13)
 - /unlink 3854 (time 1789780607.35 total 30.02 last 121.73)
total: 4500 /unlink in 34.54 seconds: 130.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 693 (time 1789780626.57 total 10.01 last 69.22)
 - open/close 1351 (time 1789780636.58 total 20.02 last 65.72)
 - open/close 1954 (time 1789780646.59 total 30.03 last 60.25)
 - open/close 2664 (time 1789780656.60 total 40.04 last 70.94)
 - open/close 3345 (time 1789780666.61 total 50.05 last 68.01)
 - open/close 3955 (time 1789780676.62 total 60.06 last 60.95)
total: 4500 open/close in 67.74 seconds: 66.43 ops/second
 - /unlink 1267 (time 1789780699.37 total 10.00 last 126.64)
 - /unlink 2466 (time 1789780709.39 total 20.02 last 119.77)
 - /unlink 3703 (time 1789780719.39 total 30.02 last 123.63)
total: 4500 /unlink in 36.55 seconds: 123.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 661 (time 1789780741.38 total 10.03 last 65.90)
 - open/close 1330 (time 1789780751.39 total 20.04 last 66.85)
 - open/close 2061 (time 1789780761.40 total 30.04 last 73.07)
 - open/close 2764 (time 1789780771.40 total 40.05 last 70.26)
 - open/close 3428 (time 1789780781.41 total 50.05 last 66.36)
 - open/close 4089 (time 1789780791.44 total 60.09 last 65.88)
total: 4500 open/close in 67.83 seconds: 66.34 ops/second
 - /unlink 1246 (time 1789780814.71 total 10.01 last 124.50)
 - /unlink 2608 (time 1789780824.72 total 20.01 last 136.14)
 - /unlink 3916 (time 1789780834.73 total 30.02 last 130.66)
total: 4500 /unlink in 34.86 seconds: 129.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 710 (time 1789780854.19 total 10.03 last 70.76)
 - open/close 1343 (time 1789780864.21 total 20.05 last 63.20)
 - open/close 2000 (time 1789780874.22 total 30.06 last 65.62)
 - open/close 2743 (time 1789780884.22 total 40.06 last 74.28)
 - open/close 3454 (time 1789780894.22 total 50.07 last 71.09)
 - open/close 4157 (time 1789780904.23 total 60.07 last 70.26)
total: 4500 open/close in 64.73 seconds: 69.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1404 (time 1789780923.05 total 10.00 last 140.39)
 - /unlink 2597 (time 1789780933.06 total 20.00 last 119.29)
 - /unlink 3610 (time 1789780943.06 total 30.00 last 101.27)
total: 4500 /unlink in 36.94 seconds: 121.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 665 (time 1789780964.73 total 10.00 last 66.47)
 - open/close 1349 (time 1789780974.75 total 20.02 last 68.27)
 - open/close 2048 (time 1789780984.76 total 30.03 last 69.82)
 - open/close 2753 (time 1789780994.77 total 40.04 last 70.43)
 - open/close 3375 (time 1789781004.78 total 50.06 last 62.11)
 - open/close 4023 (time 1789781014.80 total 60.08 last 64.66)
total: 4500 open/close in 67.52 seconds: 66.65 ops/second
 - /unlink 1192 (time 1789781036.98 total 10.00 last 119.19)
 - /unlink 2515 (time 1789781046.99 total 20.01 last 132.14)
 - /unlink 3568 (time 1789781057.01 total 30.03 last 105.09)
total: 4500 /unlink in 37.57 seconds: 119.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 663 (time 1789781079.53 total 10.00 last 66.28)
 - open/close 1373 (time 1789781089.54 total 20.01 last 70.96)
 - open/close 2118 (time 1789781099.55 total 30.02 last 74.44)
 - open/close 2865 (time 1789781109.60 total 40.07 last 74.31)
 - open/close 3464 (time 1789781119.61 total 50.08 last 59.85)
 - open/close 4060 (time 1789781129.62 total 60.09 last 59.51)
total: 4500 open/close in 66.60 seconds: 67.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1253 (time 1789781150.66 total 10.02 last 125.11)
 - /unlink 2174 (time 1789781160.66 total 20.02 last 92.08)
 - /unlink 3249 (time 1789781170.67 total 30.02 last 107.45)
 - /unlink 4382 (time 1789781180.67 total 40.03 last 113.26)
total: 4500 /unlink in 40.91 seconds: 109.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 718 (time 1789781195.59 total 10.01 last 71.73)
 - open/close 1391 (time 1789781205.61 total 20.03 last 67.17)
 - open/close 2081 (time 1789781215.63 total 30.04 last 68.90)
 - open/close 2764 (time 1789781225.63 total 40.05 last 68.26)
 - open/close 3489 (time 1789781235.64 total 50.05 last 72.48)
 - open/close 4026 (time 1789781245.65 total 60.07 last 53.61)
total: 4500 open/close in 67.37 seconds: 66.80 ops/second
 - /unlink 1377 (time 1789781267.83 total 10.00 last 137.64)
 - /unlink 2633 (time 1789781277.83 total 20.01 last 125.56)
 - /unlink 3958 (time 1789781287.83 total 30.01 last 132.50)
total: 4500 /unlink in 34.88 seconds: 129.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 565 (time 1789781309.01 total 10.03 last 56.35)
 - open/close 1256 (time 1789781319.02 total 20.04 last 69.03)
 - open/close 1950 (time 1789781329.04 total 30.05 last 69.29)
 - open/close 2630 (time 1789781339.05 total 40.06 last 67.93)
 - open/close 3325 (time 1789781349.07 total 50.08 last 69.39)
 - open/close 3951 (time 1789781359.07 total 60.09 last 62.55)
total: 4500 open/close in 67.95 seconds: 66.23 ops/second
 - /unlink 1269 (time 1789781381.74 total 10.00 last 126.87)
 - /unlink 2560 (time 1789781391.75 total 20.01 last 129.03)
 - /unlink 3855 (time 1789781401.75 total 30.01 last 129.50)
total: 4500 /unlink in 35.99 seconds: 125.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 656 (time 1789781422.87 total 10.01 last 65.54)
 - open/close 1286 (time 1789781432.87 total 20.02 last 62.96)
 - open/close 1911 (time 1789781442.88 total 30.02 last 62.45)
 - open/close 2531 (time 1789781452.88 total 40.02 last 61.99)
 - open/close 3150 (time 1789781462.90 total 50.04 last 61.81)
 - open/close 3832 (time 1789781472.96 total 60.10 last 67.81)
 - open/close 4455 (time 1789781482.96 total 70.10 last 62.29)
total: 4500 open/close in 70.87 seconds: 63.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1263 (time 1789781498.46 total 10.00 last 126.24)
 - /unlink 2398 (time 1789781508.47 total 20.01 last 113.41)
 - /unlink 3633 (time 1789781518.47 total 30.02 last 123.47)
total: 4500 /unlink in 38.26 seconds: 117.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 667 (time 1789781543.56 total 10.01 last 66.65)
 - open/close 1289 (time 1789781553.57 total 20.01 last 62.16)
 - open/close 1826 (time 1789781563.58 total 30.02 last 53.65)
 - open/close 2285 (time 1789781573.59 total 40.04 last 45.84)
 - open/close 2847 (time 1789781583.61 total 50.06 last 56.08)
 - open/close 3447 (time 1789781593.62 total 60.06 last 59.97)
 - open/close 4063 (time 1789781603.63 total 70.07 last 61.53)
total: 4500 open/close in 76.75 seconds: 58.63 ops/second
 - /unlink 1264 (time 1789781624.76 total 10.00 last 126.37)
 - /unlink 2539 (time 1789781634.76 total 20.01 last 127.42)
 - /unlink 3837 (time 1789781644.77 total 30.01 last 129.78)
total: 4500 /unlink in 35.38 seconds: 127.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 607 (time 1789781664.48 total 10.00 last 60.69)
 - open/close 1198 (time 1789781674.49 total 20.01 last 59.08)
 - open/close 1869 (time 1789781684.50 total 30.02 last 66.98)
 - open/close 2578 (time 1789781694.51 total 40.03 last 70.87)
 - open/close 3219 (time 1789781704.52 total 50.03 last 64.06)
 - open/close 3907 (time 1789781714.53 total 60.05 last 68.72)
total: 4500 open/close in 68.48 seconds: 65.72 ops/second
 - /unlink 1256 (time 1789781737.58 total 10.00 last 125.54)
 - /unlink 2599 (time 1789781747.59 total 20.01 last 134.22)
 - /unlink 3883 (time 1789781757.59 total 30.01 last 128.39)
total: 4500 /unlink in 34.35 seconds: 131.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 702 (time 1789781776.73 total 10.01 last 70.16)
 - open/close 1343 (time 1789781786.73 total 20.01 last 64.08)
 - open/close 2051 (time 1789781796.74 total 30.01 last 70.75)
 - open/close 2733 (time 1789781806.75 total 40.03 last 68.07)
 - open/close 3381 (time 1789781816.75 total 50.03 last 64.80)
 - open/close 4053 (time 1789781826.78 total 60.06 last 67.05)
total: 4500 open/close in 66.43 seconds: 67.75 ops/second
 - /unlink 1339 (time 1789781847.53 total 10.00 last 133.85)
 - /unlink 2567 (time 1789781857.53 total 20.01 last 122.76)
 - /unlink 3723 (time 1789781867.54 total 30.01 last 115.56)
total: 4500 /unlink in 35.15 seconds: 128.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 725 (time 1789781886.92 total 10.00 last 72.50)
 - open/close 1445 (time 1789781896.93 total 20.01 last 71.95)
 - open/close 2184 (time 1789781906.94 total 30.02 last 73.84)
 - open/close 2827 (time 1789781916.96 total 40.03 last 64.18)
 - open/close 3521 (time 1789781926.96 total 50.04 last 69.37)
 - open/close 4099 (time 1789781936.96 total 60.04 last 57.79)
total: 4500 open/close in 66.90 seconds: 67.26 ops/second
 - /unlink 1320 (time 1789781958.45 total 10.00 last 131.98)
 - /unlink 2594 (time 1789781968.45 total 20.00 last 127.37)
 - /unlink 3917 (time 1789781978.45 total 30.01 last 132.28)
total: 4500 /unlink in 34.71 seconds: 129.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 660 (time 1789781998.14 total 10.00 last 66.00)
 - open/close 1369 (time 1789782008.15 total 20.01 last 70.80)
 - open/close 2039 (time 1789782018.16 total 30.02 last 66.96)
 - open/close 2800 (time 1789782028.16 total 40.02 last 76.08)
 - open/close 3487 (time 1789782038.16 total 50.02 last 68.68)
 - open/close 4165 (time 1789782048.16 total 60.03 last 67.79)
total: 4500 open/close in 64.76 seconds: 69.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1176 (time 1789782067.99 total 10.00 last 117.57)
 - /unlink 2062 (time 1789782078.00 total 20.01 last 88.55)
 - /unlink 3102 (time 1789782088.00 total 30.01 last 103.97)
 - /unlink 4363 (time 1789782098.01 total 40.02 last 126.04)
total: 4500 /unlink in 41.04 seconds: 109.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 674 (time 1789782114.59 total 10.02 last 67.28)
 - open/close 1394 (time 1789782124.61 total 20.03 last 71.88)
 - open/close 2103 (time 1789782134.63 total 30.05 last 70.76)
 - open/close 2763 (time 1789782144.64 total 40.06 last 65.94)
 - open/close 3406 (time 1789782154.65 total 50.08 last 64.21)
 - open/close 4039 (time 1789782164.66 total 60.08 last 63.26)
total: 4500 open/close in 66.98 seconds: 67.18 ops/second
 - /unlink 1208 (time 1789782186.81 total 10.00 last 120.78)
 - /unlink 2262 (time 1789782196.82 total 20.01 last 105.26)
 - /unlink 3455 (time 1789782206.86 total 30.05 last 118.88)
 - /unlink 4471 (time 1789782216.87 total 40.06 last 101.45)
total: 4500 /unlink in 40.41 seconds: 111.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 623 (time 1789782232.06 total 10.00 last 62.28)
 - open/close 1241 (time 1789782242.06 total 20.01 last 61.79)
 - open/close 1892 (time 1789782252.10 total 30.04 last 64.85)
 - open/close 2577 (time 1789782262.10 total 40.04 last 68.50)
 - open/close 3177 (time 1789782272.12 total 50.06 last 59.91)
 - open/close 3817 (time 1789782282.13 total 60.07 last 63.93)
 - open/close 4475 (time 1789782292.13 total 70.07 last 65.79)
total: 4500 open/close in 70.53 seconds: 63.80 ops/second
 - /unlink 1198 (time 1789782307.41 total 10.01 last 119.70)
 - /unlink 2294 (time 1789782317.42 total 20.01 last 109.56)
 - /unlink 3464 (time 1789782327.42 total 30.01 last 117.00)
total: 4500 /unlink in 39.50 seconds: 113.92 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95156 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg342-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg342-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server
stop mds service on oleg342-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server
stop mds service on oleg342-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg342-server
unloading modules via unload_modules_local on: 'oleg342-server'
oleg342-server: oleg342-server.virtnet: executing unload_modules_local
oleg342-server: kmemleak disabled
modules unloaded.