== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:10:30 (1781244630)
Stopping clients: oleg320-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg320-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg320-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg320-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg320-server
oleg320-server: oleg320-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg320-server'
oleg320-server: oleg320-server.virtnet: executing load_modules_local
oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg320-server: Loading modules from /home/green/git/lustre-release/lustre
oleg320-server: detected 4 online CPUs by sysfs
oleg320-server: MODOPTS_LIBCFS=
oleg320-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg320-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all
pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg320-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all
pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg320-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all
pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg320-client.virtnet:  -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 607 (time 1781244836.12 total 10.00 last 60.68)
 - open/close 1206 (time 1781244846.12 total 20.01 last 59.87)
 - open/close 1811 (time 1781244856.13 total 30.01 last 60.48)
 - open/close 2531 (time 1781244866.15 total 40.04 last 71.81)
 - open/close 3259 (time 1781244876.17 total 50.05 last 72.71)
 - open/close 3993 (time 1781244886.17 total 60.06 last 73.35)
total: 4500 open/close in 67.90 seconds: 66.28 ops/second
 - /unlink 1220 (time 1781244908.42 total 10.00 last 122.00)
 - /unlink 2513 (time 1781244918.42 total 20.00 last 129.30)
 - /unlink 3773 (time 1781244928.42 total 30.00 last 125.99)
total: 4500 /unlink in 35.81 seconds: 125.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 614 (time 1781244949.17 total 10.01 last 61.36)
 - open/close 1235 (time 1781244959.18 total 20.02 last 62.03)
 - open/close 1903 (time 1781244969.21 total 30.04 last 66.64)
 - open/close 2571 (time 1781244979.21 total 40.05 last 66.75)
 - open/close 3224 (time 1781244989.22 total 50.05 last 65.27)
 - open/close 3853 (time 1781244999.23 total 60.06 last 62.86)
total: 4500 open/close in 69.38 seconds: 64.86 ops/second
 - /unlink 1332 (time 1781245023.08 total 10.00 last 133.18)
 - /unlink 2564 (time 1781245033.09 total 20.01 last 123.15)
 - /unlink 3866 (time 1781245043.09 total 30.01 last 130.18)
total: 4500 /unlink in 35.07 seconds: 128.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 663 (time 1781245062.62 total 10.01 last 66.24)
 - open/close 1358 (time 1781245072.62 total 20.01 last 69.46)
 - open/close 2006 (time 1781245082.63 total 30.02 last 64.78)
 - open/close 2680 (time 1781245092.63 total 40.02 last 67.40)
 - open/close 3376 (time 1781245102.63 total 50.02 last 69.58)
 - open/close 4013 (time 1781245112.64 total 60.03 last 63.65)
total: 4500 open/close in 67.88 seconds: 66.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1271 (time 1781245135.41 total 10.00 last 127.04)
 - /unlink 2370 (time 1781245145.42 total 20.01 last 109.86)
 - /unlink 3628 (time 1781245155.42 total 30.01 last 125.78)
total: 4500 /unlink in 36.76 seconds: 122.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 671 (time 1781245176.14 total 10.02 last 66.95)
 - open/close 1243 (time 1781245186.14 total 20.03 last 57.16)
 - open/close 1894 (time 1781245196.15 total 30.04 last 65.04)
 - open/close 2564 (time 1781245206.16 total 40.04 last 66.97)
 - open/close 3290 (time 1781245216.16 total 50.04 last 72.59)
 - open/close 4010 (time 1781245226.17 total 60.05 last 71.93)
total: 4500 open/close in 66.93 seconds: 67.24 ops/second
 - /unlink 1436 (time 1781245247.26 total 10.00 last 143.54)
 - /unlink 2694 (time 1781245257.27 total 20.01 last 125.74)
 - /unlink 4115 (time 1781245267.27 total 30.01 last 142.05)
total: 4500 /unlink in 33.31 seconds: 135.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 708 (time 1781245285.37 total 10.01 last 70.72)
 - open/close 1324 (time 1781245295.37 total 20.02 last 61.55)
 - open/close 1757 (time 1781245305.41 total 30.05 last 43.16)
 - open/close 2132 (time 1781245315.42 total 40.07 last 37.44)
 - open/close 2597 (time 1781245325.45 total 50.09 last 46.39)
 - open/close 3151 (time 1781245335.46 total 60.10 last 55.33)
 - open/close 3789 (time 1781245345.48 total 70.12 last 63.69)
 - open/close 4431 (time 1781245355.49 total 80.13 last 64.12)
total: 4500 open/close in 81.12 seconds: 55.47 ops/second
 - /unlink 1300 (time 1781245370.45 total 10.00 last 129.95)
 - /unlink 2386 (time 1781245380.45 total 20.01 last 108.58)
 - /unlink 3697 (time 1781245390.45 total 30.01 last 131.07)
total: 4500 /unlink in 36.13 seconds: 124.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 725 (time 1781245411.22 total 10.00 last 72.49)
 - open/close 1538 (time 1781245421.23 total 20.00 last 81.28)
 - open/close 2229 (time 1781245431.25 total 30.03 last 68.92)
 - open/close 2857 (time 1781245441.26 total 40.04 last 62.76)
 - open/close 3566 (time 1781245451.27 total 50.05 last 70.80)
 - open/close 4278 (time 1781245461.29 total 60.07 last 71.08)
total: 4500 open/close in 63.03 seconds: 71.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1350 (time 1781245478.16 total 10.00 last 134.96)
 - /unlink 2371 (time 1781245488.16 total 20.01 last 102.06)
 - /unlink 3078 (time 1781245498.16 total 30.01 last 70.67)
 - /unlink 4016 (time 1781245508.17 total 40.01 last 93.78)
total: 4500 /unlink in 43.86 seconds: 102.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 726 (time 1781245527.12 total 10.02 last 72.48)
 - open/close 1418 (time 1781245537.15 total 20.05 last 68.99)
 - open/close 2052 (time 1781245547.15 total 30.05 last 63.37)
 - open/close 2637 (time 1781245557.15 total 40.05 last 58.49)
 - open/close 3126 (time 1781245567.16 total 50.06 last 48.88)
 - open/close 3756 (time 1781245577.18 total 60.08 last 62.86)
total: 4500 open/close in 70.03 seconds: 64.26 ops/second
 - /unlink 1399 (time 1781245600.71 total 10.00 last 139.88)
 - /unlink 2743 (time 1781245610.71 total 20.01 last 134.34)
 - /unlink 4155 (time 1781245620.71 total 30.01 last 141.20)
total: 4500 /unlink in 32.42 seconds: 138.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 813 (time 1781245636.58 total 10.01 last 81.25)
 - open/close 1613 (time 1781245646.58 total 20.01 last 79.98)
 - open/close 2370 (time 1781245656.59 total 30.02 last 75.66)
 - open/close 3122 (time 1781245666.60 total 40.02 last 75.15)
 - open/close 3901 (time 1781245676.60 total 50.03 last 77.87)
total: 4500 open/close in 58.27 seconds: 77.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1439 (time 1781245698.42 total 10.01 last 143.79)
 - /unlink 2818 (time 1781245708.43 total 20.01 last 137.86)
 - /unlink 4256 (time 1781245718.43 total 30.01 last 143.75)
total: 4500 /unlink in 31.78 seconds: 141.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 536 (time 1781245734.12 total 10.03 last 53.46)
 - open/close 1067 (time 1781245744.12 total 20.03 last 53.06)
 - open/close 1560 (time 1781245754.13 total 30.04 last 49.27)
 - open/close 2183 (time 1781245764.16 total 40.07 last 62.09)
 - open/close 2789 (time 1781245774.17 total 50.08 last 60.54)
 - open/close 3408 (time 1781245784.22 total 60.13 last 61.61)
 - open/close 4047 (time 1781245794.30 total 70.21 last 63.40)
total: 4500 open/close in 78.62 seconds: 57.24 ops/second
 - /unlink 1203 (time 1781245817.76 total 10.00 last 120.27)
 - /unlink 2404 (time 1781245827.77 total 20.01 last 120.01)
 - /unlink 3604 (time 1781245837.77 total 30.01 last 119.98)
total: 4500 /unlink in 38.04 seconds: 118.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 665 (time 1781245861.03 total 10.01 last 66.44)
 - open/close 1301 (time 1781245871.03 total 20.01 last 63.58)
 - open/close 1944 (time 1781245881.04 total 30.02 last 64.27)
 - open/close 2632 (time 1781245891.05 total 40.02 last 68.75)
 - open/close 3379 (time 1781245901.06 total 50.03 last 74.63)
 - open/close 4039 (time 1781245911.06 total 60.03 last 65.99)
total: 4500 open/close in 67.00 seconds: 67.16 ops/second
 - /unlink 1306 (time 1781245932.32 total 10.00 last 130.60)
 - /unlink 2567 (time 1781245942.32 total 20.00 last 126.06)
 - /unlink 3880 (time 1781245952.33 total 30.01 last 131.18)
total: 4500 /unlink in 35.56 seconds: 126.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 641 (time 1781245972.47 total 10.01 last 64.03)
 - open/close 1258 (time 1781245982.49 total 20.03 last 61.57)
 - open/close 1865 (time 1781245992.50 total 30.04 last 60.67)
 - open/close 2559 (time 1781246002.50 total 40.04 last 69.37)
 - open/close 3253 (time 1781246012.51 total 50.05 last 69.31)
 - open/close 3890 (time 1781246022.52 total 60.06 last 63.66)
total: 4500 open/close in 68.30 seconds: 65.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1361 (time 1781246045.35 total 10.01 last 135.94)
 - /unlink 2585 (time 1781246055.35 total 20.02 last 122.33)
 - /unlink 3844 (time 1781246065.36 total 30.03 last 125.80)
total: 4500 /unlink in 35.98 seconds: 125.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 527 (time 1781246087.28 total 10.01 last 52.63)
 - open/close 928 (time 1781246097.28 total 20.02 last 40.08)
 - open/close 1322 (time 1781246107.30 total 30.04 last 39.32)
 - open/close 1833 (time 1781246117.31 total 40.04 last 51.09)
 - open/close 2424 (time 1781246127.32 total 50.05 last 59.03)
 - open/close 2975 (time 1781246137.32 total 60.05 last 55.09)
 - open/close 3626 (time 1781246147.33 total 70.06 last 65.05)
 - open/close 4247 (time 1781246157.37 total 80.10 last 61.85)
total: 4500 open/close in 84.32 seconds: 53.37 ops/second
 - /unlink 1119 (time 1781246177.79 total 10.01 last 111.84)
 - /unlink 2162 (time 1781246187.80 total 20.01 last 104.21)
 - /unlink 3339 (time 1781246197.81 total 30.02 last 117.63)
 - /unlink 4303 (time 1781246207.81 total 40.03 last 96.35)
total: 4500 /unlink in 41.78 seconds: 107.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 636 (time 1781246226.04 total 10.03 last 63.43)
 - open/close 1212 (time 1781246236.05 total 20.03 last 57.56)
 - open/close 1904 (time 1781246246.07 total 30.06 last 69.03)
 - open/close 2553 (time 1781246256.08 total 40.07 last 64.84)
 - open/close 3214 (time 1781246266.09 total 50.07 last 66.08)
 - open/close 3931 (time 1781246276.09 total 60.08 last 71.66)
total: 4500 open/close in 68.16 seconds: 66.02 ops/second
 - /unlink 1293 (time 1781246298.01 total 10.00 last 129.26)
 - /unlink 2533 (time 1781246308.02 total 20.01 last 123.89)
 - /unlink 3649 (time 1781246318.02 total 30.01 last 111.58)
total: 4500 /unlink in 38.21 seconds: 117.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 660 (time 1781246342.32 total 10.01 last 65.91)
 - open/close 1349 (time 1781246352.33 total 20.02 last 68.83)
 - open/close 1970 (time 1781246362.33 total 30.03 last 62.06)
 - open/close 2571 (time 1781246372.34 total 40.04 last 60.04)
 - open/close 3002 (time 1781246382.37 total 50.06 last 43.00)
 - open/close 3366 (time 1781246392.39 total 60.08 last 36.33)
 - open/close 3913 (time 1781246402.39 total 70.09 last 54.68)
 - open/close 4430 (time 1781246412.40 total 80.10 last 51.64)
total: 4500 open/close in 81.56 seconds: 55.17 ops/second
 - /unlink 1235 (time 1781246429.31 total 10.00 last 123.46)
 - /unlink 2464 (time 1781246439.32 total 20.01 last 122.86)
 - /unlink 3697 (time 1781246449.32 total 30.01 last 123.24)
total: 4500 /unlink in 36.38 seconds: 123.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 724 (time 1781246470.81 total 10.01 last 72.33)
 - open/close 1368 (time 1781246480.84 total 20.04 last 64.18)
 - open/close 1984 (time 1781246490.85 total 30.04 last 61.59)
 - open/close 2585 (time 1781246500.86 total 40.06 last 60.00)
 - open/close 3195 (time 1781246510.89 total 50.08 last 60.86)
 - open/close 3887 (time 1781246520.89 total 60.09 last 69.14)
total: 4500 open/close in 68.63 seconds: 65.57 ops/second
 - /unlink 1265 (time 1781246543.55 total 10.01 last 126.44)
 - /unlink 2453 (time 1781246553.55 total 20.01 last 118.80)
 - /unlink 3365 (time 1781246563.55 total 30.01 last 91.18)
 - /unlink 4360 (time 1781246573.56 total 40.01 last 99.48)
total: 4500 /unlink in 41.46 seconds: 108.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 469 (time 1781246592.01 total 10.01 last 46.83)
 - open/close 944 (time 1781246602.02 total 20.03 last 47.43)
 - open/close 1496 (time 1781246612.04 total 30.05 last 55.09)
 - open/close 1903 (time 1781246622.06 total 40.07 last 40.62)
 - open/close 2299 (time 1781246632.07 total 50.08 last 39.54)
 - open/close 2840 (time 1781246642.08 total 60.09 last 54.08)
 - open/close 3449 (time 1781246652.08 total 70.09 last 60.89)
 - open/close 4065 (time 1781246662.08 total 80.09 last 61.58)
total: 4500 open/close in 86.64 seconds: 51.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1191 (time 1781246683.01 total 10.00 last 119.10)
 - /unlink 2392 (time 1781246693.01 total 20.00 last 120.05)
 - /unlink 3669 (time 1781246703.01 total 30.01 last 127.64)
total: 4500 /unlink in 37.12 seconds: 121.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 671 (time 1781246725.15 total 10.02 last 66.95)
 - open/close 1303 (time 1781246735.16 total 20.03 last 63.13)
 - open/close 1952 (time 1781246745.16 total 30.04 last 64.87)
 - open/close 2677 (time 1781246755.17 total 40.04 last 72.47)
 - open/close 3263 (time 1781246765.19 total 50.06 last 58.47)
 - open/close 3891 (time 1781246775.19 total 60.07 last 62.78)
 - open/close 4496 (time 1781246785.20 total 70.07 last 60.49)
total: 4500 open/close in 70.11 seconds: 64.19 ops/second
 - /unlink 1208 (time 1781246800.43 total 10.02 last 120.51)
 - /unlink 2246 (time 1781246810.43 total 20.03 last 103.77)
 - /unlink 3408 (time 1781246820.44 total 30.03 last 116.11)
 - /unlink 4333 (time 1781246830.44 total 40.04 last 92.45)
total: 4500 /unlink in 41.88 seconds: 107.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 614 (time 1781246847.51 total 10.01 last 61.34)
 - open/close 1246 (time 1781246857.52 total 20.02 last 63.16)
 - open/close 1850 (time 1781246867.52 total 30.02 last 60.38)
 - open/close 2358 (time 1781246877.53 total 40.03 last 50.74)
 - open/close 2983 (time 1781246887.54 total 50.04 last 62.46)
 - open/close 3619 (time 1781246897.54 total 60.04 last 63.58)
 - open/close 4328 (time 1781246907.55 total 70.05 last 70.83)
total: 4500 open/close in 72.38 seconds: 62.17 ops/second
 - /unlink 1208 (time 1781246924.20 total 10.01 last 120.68)
 - /unlink 2369 (time 1781246934.21 total 20.02 last 116.00)
 - /unlink 3648 (time 1781246944.21 total 30.03 last 127.80)
total: 4500 /unlink in 37.13 seconds: 121.20 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11195: 94003 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg320-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg320-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server
stop mds service on oleg320-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server
stop mds service on oleg320-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg320-server

unloading modules via unload_modules_local on: 'oleg320-server'
oleg320-server: oleg320-server.virtnet: executing unload_modules_local
oleg320-server: 
modules unloaded.