== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:05:41 (1787886341)
Stopping clients: oleg320-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg320-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg320-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg320-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg320-server
oleg320-server: oleg320-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg320-server'
oleg320-server: oleg320-server.virtnet: executing load_modules_local
oleg320-server: Loading modules from /home/green/git/lustre-release/lustre
oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg320-server: detected 4 online CPUs by sysfs
oleg320-server: MODOPTS_LIBCFS=
oleg320-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg320-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all
pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg320-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all
pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg320-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all
pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg320-client.virtnet:  -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 574 (time 1787886533.78 total 10.01 last 57.36)
 - open/close 1194 (time 1787886543.80 total 20.02 last 61.92)
 - open/close 1769 (time 1787886553.80 total 30.02 last 57.47)
 - open/close 2349 (time 1787886563.82 total 40.04 last 57.91)
 - open/close 2969 (time 1787886573.83 total 50.05 last 61.92)
 - open/close 3693 (time 1787886583.83 total 60.06 last 72.38)
 - open/close 4342 (time 1787886593.85 total 70.08 last 64.77)
total: 4500 open/close in 72.56 seconds: 62.02 ops/second
 - /unlink 1229 (time 1787886610.59 total 10.00 last 122.84)
 - /unlink 2537 (time 1787886620.60 total 20.01 last 130.74)
 - /unlink 3868 (time 1787886630.60 total 30.01 last 133.05)
total: 4500 /unlink in 34.90 seconds: 128.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 718 (time 1787886650.41 total 10.01 last 71.76)
 - open/close 1406 (time 1787886660.41 total 20.01 last 68.76)
 - open/close 2139 (time 1787886670.42 total 30.02 last 73.26)
 - open/close 2842 (time 1787886680.42 total 40.02 last 70.28)
 - open/close 3552 (time 1787886690.43 total 50.03 last 70.95)
 - open/close 4275 (time 1787886700.46 total 60.06 last 72.04)
total: 4500 open/close in 63.04 seconds: 71.39 ops/second
 - /unlink 1283 (time 1787886718.07 total 10.01 last 128.20)
 - /unlink 2547 (time 1787886728.07 total 20.01 last 126.34)
 - /unlink 3706 (time 1787886738.07 total 30.02 last 115.85)
total: 4500 /unlink in 36.81 seconds: 122.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 702 (time 1787886760.07 total 10.00 last 70.20)
 - open/close 1361 (time 1787886770.09 total 20.02 last 65.77)
 - open/close 2079 (time 1787886780.10 total 30.03 last 71.73)
 - open/close 2825 (time 1787886790.11 total 40.04 last 74.50)
 - open/close 3510 (time 1787886800.13 total 50.06 last 68.41)
 - open/close 4197 (time 1787886810.14 total 60.07 last 68.60)
total: 4500 open/close in 64.24 seconds: 70.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1390 (time 1787886828.61 total 10.00 last 138.97)
 - /unlink 2745 (time 1787886838.61 total 20.00 last 135.47)
 - /unlink 4131 (time 1787886848.61 total 30.01 last 138.57)
total: 4500 /unlink in 32.55 seconds: 138.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 704 (time 1787886865.73 total 10.01 last 70.33)
 - open/close 1448 (time 1787886875.74 total 20.02 last 74.34)
 - open/close 2218 (time 1787886885.74 total 30.02 last 76.96)
 - open/close 2921 (time 1787886895.75 total 40.03 last 70.26)
 - open/close 3633 (time 1787886905.78 total 50.06 last 70.98)
 - open/close 4291 (time 1787886915.79 total 60.07 last 65.75)
total: 4500 open/close in 63.42 seconds: 70.95 ops/second
 - /unlink 1218 (time 1787886933.75 total 10.00 last 121.79)
 - /unlink 2442 (time 1787886943.75 total 20.00 last 122.37)
 - /unlink 3660 (time 1787886953.75 total 30.00 last 121.78)
total: 4500 /unlink in 37.13 seconds: 121.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 659 (time 1787886975.30 total 10.00 last 65.87)
 - open/close 1298 (time 1787886985.30 total 20.01 last 63.89)
 - open/close 1985 (time 1787886995.32 total 30.02 last 68.61)
 - open/close 2728 (time 1787887005.32 total 40.02 last 74.29)
 - open/close 3439 (time 1787887015.34 total 50.05 last 70.92)
 - open/close 4094 (time 1787887025.36 total 60.06 last 65.40)
total: 4500 open/close in 65.84 seconds: 68.35 ops/second
 - /unlink 1254 (time 1787887045.43 total 10.00 last 125.38)
 - /unlink 2398 (time 1787887055.43 total 20.01 last 114.34)
 - /unlink 3730 (time 1787887065.44 total 30.01 last 133.13)
total: 4500 /unlink in 35.33 seconds: 127.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 619 (time 1787887085.11 total 10.01 last 61.85)
 - open/close 1314 (time 1787887095.12 total 20.02 last 69.42)
 - open/close 2067 (time 1787887105.13 total 30.03 last 75.24)
 - open/close 2777 (time 1787887115.13 total 40.03 last 70.99)
 - open/close 3449 (time 1787887125.14 total 50.04 last 67.13)
 - open/close 4170 (time 1787887135.16 total 60.05 last 72.00)
total: 4500 open/close in 64.39 seconds: 69.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1435 (time 1787887154.02 total 10.00 last 143.49)
 - /unlink 2730 (time 1787887164.02 total 20.00 last 129.49)
 - /unlink 3790 (time 1787887174.02 total 30.00 last 105.98)
total: 4500 /unlink in 36.15 seconds: 124.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 740 (time 1787887194.46 total 10.01 last 73.94)
 - open/close 1448 (time 1787887204.46 total 20.01 last 70.80)
 - open/close 2223 (time 1787887214.47 total 30.02 last 77.39)
 - open/close 2889 (time 1787887224.47 total 40.02 last 66.59)
 - open/close 3535 (time 1787887234.48 total 50.03 last 64.57)
 - open/close 4184 (time 1787887244.49 total 60.03 last 64.87)
total: 4500 open/close in 64.98 seconds: 69.25 ops/second
 - /unlink 1341 (time 1787887264.88 total 10.01 last 133.99)
 - /unlink 2796 (time 1787887274.88 total 20.01 last 145.49)
 - /unlink 4120 (time 1787887284.88 total 30.01 last 132.36)
total: 4500 /unlink in 33.10 seconds: 135.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 707 (time 1787887302.29 total 10.02 last 70.56)
 - open/close 1389 (time 1787887312.30 total 20.02 last 68.18)
 - open/close 2098 (time 1787887322.31 total 30.04 last 70.80)
 - open/close 2835 (time 1787887332.32 total 40.05 last 73.63)
 - open/close 3545 (time 1787887342.33 total 50.06 last 70.95)
 - open/close 4256 (time 1787887352.34 total 60.06 last 71.05)
total: 4500 open/close in 63.39 seconds: 70.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1447 (time 1787887369.98 total 10.01 last 144.58)
 - /unlink 2786 (time 1787887380.01 total 20.04 last 133.49)
 - /unlink 4215 (time 1787887390.01 total 30.05 last 142.80)
total: 4500 /unlink in 32.16 seconds: 139.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 698 (time 1787887407.27 total 10.01 last 69.74)
 - open/close 1361 (time 1787887417.27 total 20.01 last 66.27)
 - open/close 2078 (time 1787887427.29 total 30.02 last 71.62)
 - open/close 2812 (time 1787887437.29 total 40.03 last 73.38)
 - open/close 3506 (time 1787887447.30 total 50.03 last 69.35)
 - open/close 4179 (time 1787887457.30 total 60.04 last 67.26)
total: 4500 open/close in 64.46 seconds: 69.81 ops/second
 - /unlink 1312 (time 1787887476.52 total 10.00 last 131.16)
 - /unlink 2637 (time 1787887486.52 total 20.01 last 132.46)
 - /unlink 3963 (time 1787887496.53 total 30.01 last 132.54)
total: 4500 /unlink in 33.73 seconds: 133.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 686 (time 1787887515.23 total 10.01 last 68.51)
 - open/close 1300 (time 1787887525.23 total 20.02 last 61.35)
 - open/close 2027 (time 1787887535.24 total 30.02 last 72.68)
 - open/close 2742 (time 1787887545.24 total 40.03 last 71.47)
 - open/close 3437 (time 1787887555.24 total 50.03 last 69.49)
 - open/close 4186 (time 1787887565.25 total 60.04 last 74.86)
total: 4500 open/close in 64.29 seconds: 70.00 ops/second
 - /unlink 1424 (time 1787887583.35 total 10.00 last 142.36)
 - /unlink 2845 (time 1787887593.35 total 20.00 last 142.07)
 - /unlink 4216 (time 1787887603.35 total 30.01 last 137.09)
total: 4500 /unlink in 32.45 seconds: 138.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 673 (time 1787887620.49 total 10.01 last 67.26)
 - open/close 1347 (time 1787887630.49 total 20.01 last 67.35)
 - open/close 2004 (time 1787887640.51 total 30.03 last 65.61)
 - open/close 2675 (time 1787887650.52 total 40.04 last 66.99)
 - open/close 3334 (time 1787887660.53 total 50.05 last 65.88)
 - open/close 4034 (time 1787887670.54 total 60.06 last 69.93)
total: 4500 open/close in 66.73 seconds: 67.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1410 (time 1787887691.58 total 10.00 last 141.00)
 - /unlink 2711 (time 1787887701.59 total 20.00 last 130.05)
 - /unlink 3921 (time 1787887711.59 total 30.01 last 120.99)
total: 4500 /unlink in 33.97 seconds: 132.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 665 (time 1787887731.08 total 10.01 last 66.40)
 - open/close 1324 (time 1787887741.08 total 20.02 last 65.89)
 - open/close 2035 (time 1787887751.08 total 30.02 last 71.10)
 - open/close 2694 (time 1787887761.08 total 40.02 last 65.89)
 - open/close 3376 (time 1787887771.10 total 50.04 last 68.07)
 - open/close 4095 (time 1787887781.11 total 60.04 last 71.86)
total: 4500 open/close in 65.73 seconds: 68.46 ops/second
 - /unlink 1454 (time 1787887801.04 total 10.00 last 145.34)
 - /unlink 2826 (time 1787887811.04 total 20.01 last 137.17)
 - /unlink 4095 (time 1787887821.05 total 30.01 last 126.85)
total: 4500 /unlink in 33.53 seconds: 134.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 675 (time 1787887839.48 total 10.02 last 67.37)
 - open/close 1354 (time 1787887849.49 total 20.02 last 67.86)
 - open/close 2091 (time 1787887859.50 total 30.03 last 73.65)
 - open/close 2818 (time 1787887869.52 total 40.05 last 72.53)
 - open/close 3492 (time 1787887879.52 total 50.06 last 67.39)
 - open/close 4160 (time 1787887889.53 total 60.06 last 66.77)
total: 4500 open/close in 64.85 seconds: 69.39 ops/second
 - /unlink 1391 (time 1787887909.26 total 10.00 last 139.07)
 - /unlink 2754 (time 1787887919.26 total 20.00 last 136.27)
 - /unlink 4168 (time 1787887929.26 total 30.01 last 141.32)
total: 4500 /unlink in 32.51 seconds: 138.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 727 (time 1787887946.11 total 10.01 last 72.62)
 - open/close 1438 (time 1787887956.12 total 20.03 last 70.99)
 - open/close 2186 (time 1787887966.12 total 30.03 last 74.79)
 - open/close 2862 (time 1787887976.13 total 40.04 last 67.55)
 - open/close 3570 (time 1787887986.13 total 50.04 last 70.77)
 - open/close 4310 (time 1787887996.15 total 60.05 last 73.91)
total: 4500 open/close in 62.75 seconds: 71.71 ops/second
 - /unlink 1443 (time 1787888013.41 total 10.00 last 144.23)
 - /unlink 2856 (time 1787888023.41 total 20.01 last 141.27)
 - /unlink 4210 (time 1787888033.41 total 30.01 last 135.36)
total: 4500 /unlink in 32.25 seconds: 139.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 665 (time 1787888050.15 total 10.00 last 66.50)
 - open/close 1320 (time 1787888060.15 total 20.00 last 65.50)
 - open/close 1997 (time 1787888070.17 total 30.02 last 67.58)
 - open/close 2709 (time 1787888080.18 total 40.03 last 71.11)
 - open/close 3452 (time 1787888090.18 total 50.04 last 74.27)
 - open/close 4195 (time 1787888100.20 total 60.06 last 74.15)
total: 4500 open/close in 63.81 seconds: 70.52 ops/second
 - /unlink 1373 (time 1787888118.21 total 10.01 last 137.18)
 - /unlink 2716 (time 1787888128.21 total 20.01 last 134.25)
 - /unlink 4047 (time 1787888138.23 total 30.03 last 132.83)
total: 4500 /unlink in 33.52 seconds: 134.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 772 (time 1787888156.72 total 10.01 last 77.11)
 - open/close 1491 (time 1787888166.73 total 20.03 last 71.80)
 - open/close 2232 (time 1787888176.74 total 30.03 last 74.06)
 - open/close 2920 (time 1787888186.74 total 40.04 last 68.74)
 - open/close 3623 (time 1787888196.75 total 50.04 last 70.28)
 - open/close 4329 (time 1787888206.76 total 60.05 last 70.55)
total: 4500 open/close in 62.40 seconds: 72.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1358 (time 1787888223.56 total 10.01 last 135.69)
 - /unlink 2641 (time 1787888233.57 total 20.03 last 128.06)
 - /unlink 3779 (time 1787888243.58 total 30.03 last 113.74)
total: 4500 /unlink in 36.66 seconds: 122.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 739 (time 1787888266.03 total 10.00 last 73.89)
 - open/close 1493 (time 1787888276.03 total 20.00 last 75.39)
 - open/close 2219 (time 1787888286.03 total 30.01 last 72.55)
 - open/close 2973 (time 1787888296.06 total 40.03 last 75.22)
 - open/close 3611 (time 1787888306.06 total 50.04 last 63.79)
 - open/close 4233 (time 1787888316.08 total 60.05 last 62.08)
total: 4500 open/close in 64.79 seconds: 69.45 ops/second
 - /unlink 1209 (time 1787888335.72 total 10.00 last 120.86)
 - /unlink 2446 (time 1787888345.72 total 20.01 last 123.65)
 - /unlink 3597 (time 1787888355.73 total 30.01 last 115.03)
total: 4500 /unlink in 37.17 seconds: 121.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 683 (time 1787888377.97 total 10.00 last 68.30)
 - open/close 1335 (time 1787888387.99 total 20.01 last 65.12)
 - open/close 2018 (time 1787888398.00 total 30.02 last 68.23)
 - open/close 2672 (time 1787888408.00 total 40.03 last 65.39)
 - open/close 3353 (time 1787888418.03 total 50.05 last 67.91)
 - open/close 4002 (time 1787888428.05 total 60.07 last 64.77)
total: 4500 open/close in 67.98 seconds: 66.19 ops/second
 - /unlink 1398 (time 1787888450.71 total 10.01 last 139.67)
 - /unlink 2585 (time 1787888460.72 total 20.01 last 118.70)
 - /unlink 3931 (time 1787888470.72 total 30.01 last 134.53)
total: 4500 /unlink in 34.19 seconds: 131.62 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11375: 95318 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg320-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg320-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server
stop mds service on oleg320-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server
stop mds service on oleg320-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg320-server

unloading modules via unload_modules_local on: 'oleg320-server'
oleg320-server: oleg320-server.virtnet: executing unload_modules_local
oleg320-server: 
modules unloaded.