== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:39:02 (1787845142)
Stopping clients: oleg618-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg618-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg618-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg618-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg618-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg618-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg618-server
oleg618-server: oleg618-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg618-server'
oleg618-server: oleg618-server.virtnet: executing load_modules_local
oleg618-server: Loading modules from /home/green/git/lustre-release/lustre
oleg618-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg618-server: detected 4 online CPUs by sysfs
oleg618-server: MODOPTS_LIBCFS=
oleg618-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg618-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all
pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg618-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all
pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg618-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg618-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all
pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg618-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg618-client.virtnet:  -o user_xattr,flock 192.168.206.118@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 769 (time 1787845334.59 total 10.01 last 76.83)
 - open/close 1441 (time 1787845344.60 total 20.01 last 67.19)
 - open/close 2122 (time 1787845354.60 total 30.02 last 68.06)
 - open/close 2799 (time 1787845364.61 total 40.02 last 67.64)
 - open/close 3448 (time 1787845374.62 total 50.04 last 64.82)
 - open/close 3845 (time 1787845384.63 total 60.04 last 39.68)
 - open/close 4318 (time 1787845394.64 total 70.06 last 47.24)
total: 4500 open/close in 73.29 seconds: 61.40 ops/second
 - /unlink 1070 (time 1787845412.67 total 10.00 last 106.98)
 - /unlink 2388 (time 1787845422.69 total 20.01 last 131.65)
 - /unlink 3848 (time 1787845432.69 total 30.01 last 145.99)
total: 4500 /unlink in 34.66 seconds: 129.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 720 (time 1787845451.47 total 10.00 last 71.97)
 - open/close 1444 (time 1787845461.48 total 20.02 last 72.29)
 - open/close 2228 (time 1787845471.49 total 30.03 last 78.35)
 - open/close 3006 (time 1787845481.49 total 40.03 last 77.76)
 - open/close 3775 (time 1787845491.50 total 50.04 last 76.82)
 - open/close 4475 (time 1787845501.51 total 60.04 last 69.98)
total: 4500 open/close in 60.46 seconds: 74.43 ops/second
 - /unlink 1415 (time 1787845515.94 total 10.00 last 141.45)
 - /unlink 2817 (time 1787845525.94 total 20.01 last 140.13)
 - /unlink 4139 (time 1787845535.95 total 30.02 last 132.10)
total: 4500 /unlink in 32.89 seconds: 136.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 692 (time 1787845553.13 total 10.00 last 69.19)
 - open/close 1388 (time 1787845563.14 total 20.01 last 69.52)
 - open/close 2137 (time 1787845573.15 total 30.02 last 74.86)
 - open/close 2882 (time 1787845583.15 total 40.02 last 74.46)
 - open/close 3586 (time 1787845593.16 total 50.03 last 70.35)
 - open/close 4286 (time 1787845603.16 total 60.04 last 69.96)
total: 4500 open/close in 62.95 seconds: 71.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1431 (time 1787845620.27 total 10.01 last 143.02)
 - /unlink 2803 (time 1787845630.27 total 20.01 last 137.18)
 - /unlink 4196 (time 1787845640.27 total 30.01 last 139.26)
total: 4500 /unlink in 32.43 seconds: 138.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 709 (time 1787845656.75 total 10.01 last 70.85)
 - open/close 1457 (time 1787845666.76 total 20.01 last 74.76)
 - open/close 2187 (time 1787845676.77 total 30.02 last 72.96)
 - open/close 2881 (time 1787845686.77 total 40.03 last 69.35)
 - open/close 3617 (time 1787845696.78 total 50.03 last 73.56)
 - open/close 4360 (time 1787845706.79 total 60.04 last 74.25)
total: 4500 open/close in 61.95 seconds: 72.64 ops/second
 - /unlink 1261 (time 1787845722.70 total 10.00 last 126.07)
 - /unlink 2399 (time 1787845732.72 total 20.02 last 113.65)
 - /unlink 3647 (time 1787845742.72 total 30.02 last 124.75)
total: 4500 /unlink in 37.88 seconds: 118.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 704 (time 1787845765.30 total 10.00 last 70.37)
 - open/close 1276 (time 1787845775.30 total 20.01 last 57.19)
 - open/close 2009 (time 1787845785.31 total 30.01 last 73.23)
 - open/close 2775 (time 1787845795.33 total 40.03 last 76.46)
 - open/close 3370 (time 1787845805.35 total 50.05 last 59.39)
 - open/close 4021 (time 1787845815.36 total 60.07 last 65.01)
total: 4500 open/close in 66.76 seconds: 67.41 ops/second
 - /unlink 1239 (time 1787845836.98 total 10.01 last 123.80)
 - /unlink 2618 (time 1787845846.98 total 20.01 last 137.84)
 - /unlink 4084 (time 1787845856.99 total 30.01 last 146.58)
total: 4500 /unlink in 33.07 seconds: 136.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 683 (time 1787845875.08 total 10.01 last 68.25)
 - open/close 1418 (time 1787845885.10 total 20.04 last 73.29)
 - open/close 2126 (time 1787845895.12 total 30.05 last 70.67)
 - open/close 2803 (time 1787845905.12 total 40.06 last 67.69)
 - open/close 3386 (time 1787845915.13 total 50.06 last 58.26)
 - open/close 4143 (time 1787845925.14 total 60.07 last 75.64)
total: 4500 open/close in 66.12 seconds: 68.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1208 (time 1787845945.99 total 10.01 last 120.67)
 - /unlink 2578 (time 1787845956.00 total 20.02 last 136.92)
 - /unlink 3943 (time 1787845966.00 total 30.02 last 136.42)
total: 4500 /unlink in 34.31 seconds: 131.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 671 (time 1787845985.55 total 10.01 last 67.03)
 - open/close 1428 (time 1787845995.55 total 20.01 last 75.70)
 - open/close 2125 (time 1787846005.56 total 30.02 last 69.67)
 - open/close 2819 (time 1787846015.56 total 40.02 last 69.37)
 - open/close 3484 (time 1787846025.57 total 50.02 last 66.47)
 - open/close 4177 (time 1787846035.58 total 60.03 last 69.23)
total: 4500 open/close in 64.68 seconds: 69.57 ops/second
 - /unlink 1312 (time 1787846055.87 total 10.00 last 131.19)
 - /unlink 2574 (time 1787846065.88 total 20.00 last 126.17)
 - /unlink 3904 (time 1787846075.88 total 30.00 last 133.00)
total: 4500 /unlink in 35.02 seconds: 128.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 581 (time 1787846096.15 total 10.00 last 58.08)
 - open/close 1209 (time 1787846106.15 total 20.01 last 62.78)
 - open/close 1931 (time 1787846116.16 total 30.01 last 72.14)
 - open/close 2549 (time 1787846126.16 total 40.02 last 61.76)
 - open/close 3156 (time 1787846136.19 total 50.05 last 60.55)
 - open/close 3834 (time 1787846146.19 total 60.05 last 67.75)
total: 4500 open/close in 69.72 seconds: 64.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1356 (time 1787846171.00 total 10.00 last 135.59)
 - /unlink 2539 (time 1787846181.01 total 20.00 last 118.25)
 - /unlink 3665 (time 1787846191.01 total 30.01 last 112.58)
total: 4500 /unlink in 37.64 seconds: 119.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 663 (time 1787846215.09 total 10.01 last 66.23)
 - open/close 1316 (time 1787846225.09 total 20.01 last 65.30)
 - open/close 2029 (time 1787846235.10 total 30.02 last 71.25)
 - open/close 2767 (time 1787846245.11 total 40.03 last 73.73)
 - open/close 3427 (time 1787846255.11 total 50.03 last 65.99)
 - open/close 4076 (time 1787846265.11 total 60.03 last 64.88)
total: 4500 open/close in 65.93 seconds: 68.26 ops/second
 - /unlink 1304 (time 1787846286.41 total 10.01 last 130.29)
 - /unlink 2489 (time 1787846296.42 total 20.01 last 118.47)
 - /unlink 3776 (time 1787846306.42 total 30.02 last 128.64)
total: 4500 /unlink in 35.93 seconds: 125.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 710 (time 1787846328.75 total 10.01 last 70.93)
 - open/close 1420 (time 1787846338.75 total 20.02 last 70.95)
 - open/close 2144 (time 1787846348.76 total 30.03 last 72.33)
 - open/close 2839 (time 1787846358.77 total 40.03 last 69.48)
 - open/close 3555 (time 1787846368.77 total 50.03 last 71.59)
 - open/close 4264 (time 1787846378.77 total 60.04 last 70.85)
total: 4500 open/close in 63.17 seconds: 71.24 ops/second
 - /unlink 1269 (time 1787846396.58 total 10.00 last 126.87)
 - /unlink 2551 (time 1787846406.58 total 20.00 last 128.17)
 - /unlink 3959 (time 1787846416.59 total 30.01 last 140.66)
total: 4500 /unlink in 34.09 seconds: 131.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 654 (time 1787846435.41 total 10.00 last 65.38)
 - open/close 1294 (time 1787846445.44 total 20.03 last 63.83)
 - open/close 1968 (time 1787846455.46 total 30.05 last 67.23)
 - open/close 2663 (time 1787846465.47 total 40.06 last 69.47)
 - open/close 3340 (time 1787846475.48 total 50.06 last 67.66)
 - open/close 4080 (time 1787846485.48 total 60.07 last 73.94)
total: 4500 open/close in 65.08 seconds: 69.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1464 (time 1787846505.33 total 10.00 last 146.33)
 - /unlink 2687 (time 1787846515.33 total 20.01 last 122.28)
 - /unlink 4020 (time 1787846525.34 total 30.01 last 133.22)
total: 4500 /unlink in 33.13 seconds: 135.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 772 (time 1787846543.59 total 10.01 last 77.10)
 - open/close 1498 (time 1787846553.59 total 20.01 last 72.60)
 - open/close 2210 (time 1787846563.60 total 30.02 last 71.15)
 - open/close 2951 (time 1787846573.63 total 40.06 last 73.84)
 - open/close 3715 (time 1787846583.64 total 50.06 last 76.36)
 - open/close 4442 (time 1787846593.65 total 60.07 last 72.61)
total: 4500 open/close in 60.81 seconds: 74.00 ops/second
 - /unlink 1318 (time 1787846608.87 total 10.00 last 131.78)
 - /unlink 2784 (time 1787846618.87 total 20.01 last 146.52)
 - /unlink 4026 (time 1787846628.88 total 30.01 last 124.12)
total: 4500 /unlink in 34.28 seconds: 131.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 563 (time 1787846648.47 total 10.00 last 56.30)
 - open/close 1132 (time 1787846658.48 total 20.01 last 56.88)
 - open/close 1673 (time 1787846668.48 total 30.01 last 54.06)
 - open/close 2272 (time 1787846678.52 total 40.05 last 59.67)
 - open/close 2980 (time 1787846688.53 total 50.06 last 70.75)
 - open/close 3736 (time 1787846698.53 total 60.06 last 75.56)
 - open/close 4475 (time 1787846708.55 total 70.08 last 73.75)
total: 4500 open/close in 70.51 seconds: 63.82 ops/second
 - /unlink 1305 (time 1787846723.30 total 10.00 last 130.44)
 - /unlink 2729 (time 1787846733.31 total 20.01 last 142.26)
 - /unlink 4120 (time 1787846743.32 total 30.03 last 138.92)
total: 4500 /unlink in 33.41 seconds: 134.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 684 (time 1787846762.49 total 10.01 last 68.36)
 - open/close 1417 (time 1787846772.49 total 20.01 last 73.26)
 - open/close 1944 (time 1787846782.49 total 30.01 last 52.69)
 - open/close 2604 (time 1787846792.50 total 40.02 last 65.94)
 - open/close 3218 (time 1787846802.50 total 50.02 last 61.40)
 - open/close 3859 (time 1787846812.51 total 60.03 last 64.07)
total: 4500 open/close in 69.58 seconds: 64.67 ops/second
 - /unlink 966 (time 1787846836.37 total 10.01 last 96.54)
 - /unlink 2137 (time 1787846846.37 total 20.01 last 117.09)
 - /unlink 3282 (time 1787846856.38 total 30.01 last 114.42)
 - /unlink 4453 (time 1787846866.38 total 40.02 last 117.03)
total: 4500 /unlink in 40.40 seconds: 111.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 854 (time 1787846881.47 total 10.01 last 85.31)
 - open/close 1642 (time 1787846891.48 total 20.01 last 78.78)
 - open/close 2352 (time 1787846901.48 total 30.02 last 70.98)
 - open/close 3149 (time 1787846911.48 total 40.02 last 79.70)
 - open/close 3961 (time 1787846921.48 total 50.02 last 81.18)
total: 4500 open/close in 57.51 seconds: 78.25 ops/second
 - /unlink 1385 (time 1787846943.02 total 10.01 last 138.42)
 - /unlink 2649 (time 1787846953.03 total 20.01 last 126.31)
 - /unlink 4073 (time 1787846963.03 total 30.02 last 142.34)
total: 4500 /unlink in 33.74 seconds: 133.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 742 (time 1787846981.94 total 10.01 last 74.15)
 - open/close 1515 (time 1787846991.97 total 20.03 last 77.10)
 - open/close 2228 (time 1787847001.97 total 30.04 last 71.28)
 - open/close 2869 (time 1787847011.97 total 40.04 last 64.06)
 - open/close 3519 (time 1787847021.97 total 50.04 last 65.00)
 - open/close 4133 (time 1787847031.98 total 60.05 last 61.37)
total: 4500 open/close in 66.63 seconds: 67.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1385 (time 1787847054.09 total 10.00 last 138.46)
 - /unlink 2809 (time 1787847064.09 total 20.00 last 142.38)
 - /unlink 4120 (time 1787847074.10 total 30.01 last 131.03)
total: 4500 /unlink in 33.10 seconds: 135.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 737 (time 1787847093.19 total 10.00 last 73.67)
 - open/close 1398 (time 1787847103.20 total 20.01 last 66.04)
 - open/close 2099 (time 1787847113.20 total 30.01 last 70.09)
 - open/close 2776 (time 1787847123.20 total 40.02 last 67.69)
 - open/close 3276 (time 1787847133.21 total 50.02 last 49.99)
 - open/close 3694 (time 1787847143.25 total 60.07 last 41.60)
 - open/close 4389 (time 1787847153.25 total 70.07 last 69.50)
total: 4500 open/close in 71.86 seconds: 62.62 ops/second
 - /unlink 1070 (time 1787847169.87 total 10.00 last 106.96)
 - /unlink 2520 (time 1787847179.87 total 20.01 last 144.94)
 - /unlink 3936 (time 1787847189.88 total 30.01 last 141.56)
total: 4500 /unlink in 33.50 seconds: 134.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 723 (time 1787847208.08 total 10.01 last 72.23)
 - open/close 1429 (time 1787847218.09 total 20.02 last 70.52)
 - open/close 2098 (time 1787847228.09 total 30.02 last 66.88)
 - open/close 2865 (time 1787847238.10 total 40.03 last 76.67)
 - open/close 3627 (time 1787847248.11 total 50.04 last 76.09)
 - open/close 4405 (time 1787847258.12 total 60.05 last 77.75)
total: 4500 open/close in 61.44 seconds: 73.24 ops/second
 - /unlink 1471 (time 1787847274.17 total 10.00 last 147.03)
 - /unlink 2797 (time 1787847284.18 total 20.02 last 132.45)
 - /unlink 3988 (time 1787847294.18 total 30.02 last 119.05)
total: 4500 /unlink in 34.58 seconds: 130.13 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11375: 95229 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg618-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg618-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg618-server
stop mds service on oleg618-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg618-server
stop mds service on oleg618-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg618-server

unloading modules via unload_modules_local on: 'oleg618-server'
oleg618-server: oleg618-server.virtnet: executing unload_modules_local
oleg618-server: 
modules unloaded.