== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:45:37 (1786761937)
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg225-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
oleg225-server: oleg225-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg225-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 647 (time 1786762141.17 total 10.00 last 64.69)
 - open/close 1474 (time 1786762151.18 total 20.01 last 82.63)
 - open/close 2123 (time 1786762161.18 total 30.01 last 64.88)
 - open/close 2947 (time 1786762171.18 total 40.01 last 82.40)
 - open/close 3696 (time 1786762181.18 total 50.02 last 74.88)
 - open/close 4474 (time 1786762191.19 total 60.02 last 77.75)
total: 4500 open/close in 60.41 seconds: 74.49 ops/second
 - /unlink 1568 (time 1786762206.47 total 10.00 last 156.74)
 - /unlink 2979 (time 1786762216.48 total 20.01 last 141.07)
total: 4500 /unlink in 29.98 seconds: 150.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 660 (time 1786762241.37 total 10.01 last 65.94)
 - open/close 1419 (time 1786762251.37 total 20.01 last 75.90)
 - open/close 2202 (time 1786762261.40 total 30.04 last 78.04)
 - open/close 2958 (time 1786762271.40 total 40.04 last 75.59)
 - open/close 3671 (time 1786762281.40 total 50.04 last 71.30)
 - open/close 4360 (time 1786762291.41 total 60.05 last 68.86)
total: 4500 open/close in 62.00 seconds: 72.58 ops/second
 - /unlink 1478 (time 1786762309.05 total 10.01 last 147.70)
 - /unlink 2909 (time 1786762319.06 total 20.01 last 143.04)
 - /unlink 4448 (time 1786762329.06 total 30.01 last 153.89)
total: 4500 /unlink in 30.69 seconds: 146.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 776 (time 1786762344.48 total 10.02 last 77.45)
 - open/close 1486 (time 1786762354.48 total 20.02 last 70.99)
 - open/close 2068 (time 1786762364.49 total 30.04 last 58.09)
 - open/close 2663 (time 1786762374.53 total 40.07 last 59.31)
 - open/close 3413 (time 1786762384.53 total 50.07 last 74.98)
 - open/close 4232 (time 1786762394.54 total 60.08 last 81.86)
total: 4500 open/close in 63.40 seconds: 70.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1198 (time 1786762413.51 total 10.00 last 119.78)
 - /unlink 2665 (time 1786762423.52 total 20.01 last 146.58)
 - /unlink 4081 (time 1786762433.52 total 30.01 last 141.53)
total: 4500 /unlink in 32.97 seconds: 136.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 720 (time 1786762451.36 total 10.00 last 71.98)
 - open/close 1484 (time 1786762461.38 total 20.01 last 76.31)
 - open/close 2180 (time 1786762471.38 total 30.02 last 69.54)
 - open/close 2962 (time 1786762481.39 total 40.03 last 78.16)
 - open/close 3635 (time 1786762491.39 total 50.03 last 67.27)
 - open/close 4460 (time 1786762501.40 total 60.03 last 82.48)
total: 4500 open/close in 60.59 seconds: 74.27 ops/second
 - /unlink 1366 (time 1786762516.17 total 10.00 last 136.56)
 - /unlink 2328 (time 1786762526.19 total 20.02 last 96.08)
 - /unlink 3530 (time 1786762536.19 total 30.02 last 120.15)
total: 4500 /unlink in 36.54 seconds: 123.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 575 (time 1786762558.11 total 10.01 last 57.44)
 - open/close 1243 (time 1786762568.12 total 20.02 last 66.74)
 - open/close 1877 (time 1786762578.12 total 30.02 last 63.40)
 - open/close 2484 (time 1786762588.13 total 40.03 last 60.62)
 - open/close 3279 (time 1786762598.14 total 50.04 last 79.44)
 - open/close 4069 (time 1786762608.15 total 60.05 last 78.91)
total: 4500 open/close in 66.68 seconds: 67.48 ops/second
 - /unlink 1347 (time 1786762629.64 total 10.00 last 134.65)
 - /unlink 2637 (time 1786762639.64 total 20.01 last 128.95)
 - /unlink 4125 (time 1786762649.65 total 30.01 last 148.75)
total: 4500 /unlink in 33.26 seconds: 135.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 663 (time 1786762668.79 total 10.00 last 66.28)
 - open/close 1425 (time 1786762678.79 total 20.00 last 76.20)
 - open/close 2021 (time 1786762688.81 total 30.02 last 59.52)
 - open/close 2774 (time 1786762698.83 total 40.04 last 75.14)
 - open/close 3522 (time 1786762708.83 total 50.04 last 74.78)
 - open/close 4397 (time 1786762718.84 total 60.05 last 87.43)
total: 4500 open/close in 61.16 seconds: 73.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1409 (time 1786762734.08 total 10.00 last 140.87)
 - /unlink 2651 (time 1786762744.09 total 20.01 last 124.16)
 - /unlink 4000 (time 1786762754.09 total 30.01 last 134.86)
total: 4500 /unlink in 34.74 seconds: 129.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 597 (time 1786762775.85 total 10.00 last 59.68)
 - open/close 1253 (time 1786762785.85 total 20.01 last 65.57)
 - open/close 2050 (time 1786762795.86 total 30.02 last 79.64)
 - open/close 2786 (time 1786762805.86 total 40.02 last 73.59)
 - open/close 3297 (time 1786762815.86 total 50.02 last 51.10)
 - open/close 3957 (time 1786762825.86 total 60.02 last 66.00)
total: 4500 open/close in 67.58 seconds: 66.59 ops/second
 - /unlink 1547 (time 1786762848.50 total 10.00 last 154.67)
 - /unlink 2921 (time 1786762858.50 total 20.00 last 137.36)
 - /unlink 4193 (time 1786762868.51 total 30.01 last 127.16)
total: 4500 /unlink in 33.08 seconds: 136.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 667 (time 1786762887.54 total 10.02 last 66.57)
 - open/close 1402 (time 1786762897.55 total 20.03 last 73.45)
 - open/close 2100 (time 1786762907.55 total 30.03 last 69.80)
 - open/close 2908 (time 1786762917.55 total 40.03 last 80.75)
 - open/close 3635 (time 1786762927.55 total 50.04 last 72.68)
 - open/close 4367 (time 1786762937.56 total 60.04 last 73.16)
total: 4500 open/close in 61.63 seconds: 73.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1433 (time 1786762954.06 total 10.00 last 143.28)
 - /unlink 2812 (time 1786762964.08 total 20.02 last 137.67)
 - /unlink 4223 (time 1786762974.10 total 30.04 last 140.75)
total: 4500 /unlink in 32.47 seconds: 138.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 808 (time 1786762991.65 total 10.01 last 80.71)
 - open/close 1483 (time 1786763001.66 total 20.02 last 67.45)
 - open/close 2133 (time 1786763011.68 total 30.04 last 64.84)
 - open/close 2851 (time 1786763021.73 total 40.09 last 71.46)
 - open/close 3491 (time 1786763031.74 total 50.10 last 63.94)
 - open/close 4235 (time 1786763041.75 total 60.11 last 74.33)
total: 4500 open/close in 63.77 seconds: 70.57 ops/second
 - /unlink 1289 (time 1786763061.16 total 10.01 last 128.78)
 - /unlink 2402 (time 1786763071.17 total 20.02 last 111.14)
 - /unlink 3754 (time 1786763081.18 total 30.03 last 135.11)
total: 4500 /unlink in 35.40 seconds: 127.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 701 (time 1786763101.48 total 10.01 last 70.00)
 - open/close 1279 (time 1786763111.49 total 20.02 last 57.77)
 - open/close 1906 (time 1786763121.49 total 30.02 last 62.67)
 - open/close 2527 (time 1786763131.51 total 40.04 last 62.00)
 - open/close 3213 (time 1786763141.51 total 50.05 last 68.57)
 - open/close 4029 (time 1786763151.53 total 60.06 last 81.49)
total: 4500 open/close in 66.03 seconds: 68.15 ops/second
 - /unlink 1149 (time 1786763173.17 total 10.01 last 114.82)
 - /unlink 2099 (time 1786763183.19 total 20.03 last 94.78)
 - /unlink 2993 (time 1786763193.19 total 30.03 last 89.39)
 - /unlink 4185 (time 1786763203.19 total 40.03 last 119.18)
total: 4500 /unlink in 42.16 seconds: 106.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 670 (time 1786763220.48 total 10.00 last 66.99)
 - open/close 1273 (time 1786763230.49 total 20.01 last 60.26)
 - open/close 2000 (time 1786763240.49 total 30.01 last 72.67)
 - open/close 2761 (time 1786763250.50 total 40.02 last 76.03)
 - open/close 3475 (time 1786763260.51 total 50.02 last 71.39)
 - open/close 4145 (time 1786763270.51 total 60.03 last 66.99)
total: 4500 open/close in 66.75 seconds: 67.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1268 (time 1786763294.21 total 10.00 last 126.75)
 - /unlink 2617 (time 1786763304.22 total 20.01 last 134.76)
 - /unlink 3653 (time 1786763314.22 total 30.02 last 103.57)
total: 4500 /unlink in 36.70 seconds: 122.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 689 (time 1786763336.27 total 10.03 last 68.72)
 - open/close 1250 (time 1786763346.28 total 20.03 last 56.07)
 - open/close 1873 (time 1786763356.29 total 30.04 last 62.23)
 - open/close 2625 (time 1786763366.30 total 40.05 last 75.14)
 - open/close 3295 (time 1786763376.30 total 50.06 last 66.95)
 - open/close 4006 (time 1786763386.31 total 60.06 last 71.09)
total: 4500 open/close in 68.79 seconds: 65.42 ops/second
 - /unlink 1008 (time 1786763410.15 total 10.00 last 100.80)
 - /unlink 2238 (time 1786763420.15 total 20.00 last 122.98)
 - /unlink 3520 (time 1786763430.15 total 30.00 last 128.17)
total: 4500 /unlink in 37.92 seconds: 118.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 694 (time 1786763453.60 total 10.01 last 69.32)
 - open/close 1468 (time 1786763463.61 total 20.02 last 77.31)
 - open/close 2139 (time 1786763473.62 total 30.03 last 67.07)
 - open/close 2799 (time 1786763483.63 total 40.04 last 65.95)
 - open/close 3462 (time 1786763493.64 total 50.05 last 66.23)
 - open/close 4216 (time 1786763503.64 total 60.05 last 75.38)
total: 4500 open/close in 64.23 seconds: 70.06 ops/second
 - /unlink 1166 (time 1786763523.07 total 10.00 last 116.55)
 - /unlink 2473 (time 1786763533.07 total 20.01 last 130.67)
 - /unlink 3656 (time 1786763543.08 total 30.01 last 118.25)
total: 4500 /unlink in 38.20 seconds: 117.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 707 (time 1786763567.65 total 10.01 last 70.61)
 - open/close 1409 (time 1786763577.67 total 20.03 last 70.09)
 - open/close 2117 (time 1786763587.68 total 30.04 last 70.70)
 - open/close 2698 (time 1786763597.68 total 40.05 last 58.09)
 - open/close 3350 (time 1786763607.69 total 50.05 last 65.17)
 - open/close 4079 (time 1786763617.71 total 60.07 last 72.77)
total: 4500 open/close in 66.61 seconds: 67.55 ops/second
 - /unlink 1392 (time 1786763639.19 total 10.00 last 139.17)
 - /unlink 2744 (time 1786763649.19 total 20.01 last 135.16)
 - /unlink 4179 (time 1786763659.19 total 30.01 last 143.46)
total: 4500 /unlink in 32.68 seconds: 137.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 647 (time 1786763677.35 total 10.00 last 64.67)
 - open/close 1404 (time 1786763687.35 total 20.01 last 75.66)
 - open/close 2236 (time 1786763697.35 total 30.01 last 83.19)
 - open/close 2959 (time 1786763707.36 total 40.01 last 72.27)
 - open/close 3572 (time 1786763717.36 total 50.02 last 61.27)
 - open/close 4160 (time 1786763727.38 total 60.03 last 58.73)
total: 4500 open/close in 66.60 seconds: 67.57 ops/second
 - /unlink 1331 (time 1786763749.29 total 10.00 last 133.09)
 - /unlink 2688 (time 1786763759.29 total 20.00 last 135.70)
 - /unlink 4291 (time 1786763769.29 total 30.01 last 160.23)
total: 4500 /unlink in 31.49 seconds: 142.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 768 (time 1786763785.00 total 10.01 last 76.73)
 - open/close 1416 (time 1786763795.00 total 20.01 last 64.78)
 - open/close 2039 (time 1786763805.01 total 30.02 last 62.23)
 - open/close 2806 (time 1786763815.02 total 40.03 last 76.63)
 - open/close 3454 (time 1786763825.05 total 50.06 last 64.62)
 - open/close 4050 (time 1786763835.05 total 60.07 last 59.58)
total: 4500 open/close in 67.42 seconds: 66.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1068 (time 1786763858.78 total 10.00 last 106.75)
 - /unlink 2381 (time 1786763868.78 total 20.01 last 131.28)
 - /unlink 3907 (time 1786763878.78 total 30.01 last 152.52)
total: 4500 /unlink in 34.57 seconds: 130.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 743 (time 1786763898.67 total 10.02 last 74.17)
 - open/close 1442 (time 1786763908.67 total 20.03 last 69.84)
 - open/close 2145 (time 1786763918.68 total 30.03 last 70.27)
 - open/close 2821 (time 1786763928.68 total 40.04 last 67.56)
 - open/close 3590 (time 1786763938.69 total 50.04 last 76.89)
 - open/close 4224 (time 1786763948.69 total 60.04 last 63.37)
total: 4500 open/close in 63.64 seconds: 70.71 ops/second
 - /unlink 1360 (time 1786763967.15 total 10.00 last 135.98)
 - /unlink 2547 (time 1786763977.15 total 20.01 last 118.65)
 - /unlink 4083 (time 1786763987.16 total 30.01 last 153.52)
total: 4500 /unlink in 33.18 seconds: 135.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 723 (time 1786764005.63 total 10.00 last 72.30)
 - open/close 1479 (time 1786764015.63 total 20.00 last 75.58)
 - open/close 2276 (time 1786764025.64 total 30.01 last 79.63)
 - open/close 3035 (time 1786764035.64 total 40.02 last 75.87)
 - open/close 3721 (time 1786764045.64 total 50.02 last 68.60)
 - open/close 4470 (time 1786764055.65 total 60.02 last 74.85)
total: 4500 open/close in 60.59 seconds: 74.26 ops/second
 - /unlink 1347 (time 1786764070.90 total 10.01 last 134.55)
 - /unlink 2414 (time 1786764080.90 total 20.01 last 106.68)
 - /unlink 3948 (time 1786764090.91 total 30.02 last 153.36)
total: 4500 /unlink in 34.32 seconds: 131.13 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95331 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.