== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:26:47 (1779715607)
Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg132-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg132-server
oleg132-server: oleg132-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing load_modules_local
oleg132-server: Loading modules from /home/green/git/lustre-release/lustre
oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg132-server: detected 4 online CPUs by sysfs
oleg132-server: MODOPTS_LIBCFS=
oleg132-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg132-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg132-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg132-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all
pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg132-client.virtnet:  -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 385 (time 1779715900.92 total 10.01 last 38.46)
 - open/close 770 (time 1779715910.93 total 20.02 last 38.45)
 - open/close 1163 (time 1779715920.93 total 30.03 last 39.28)
 - open/close 1580 (time 1779715930.94 total 40.04 last 41.65)
 - open/close 2090 (time 1779715940.95 total 50.04 last 50.99)
 - open/close 2615 (time 1779715950.95 total 60.05 last 52.48)
 - open/close 3112 (time 1779715960.96 total 70.05 last 49.66)
 - open/close 3667 (time 1779715970.97 total 80.06 last 55.45)
 - open/close 4103 (time 1779715980.98 total 90.07 last 43.55)
total: 4500 open/close in 98.04 seconds: 45.90 ops/second
 - /unlink 939 (time 1779716005.80 total 10.01 last 93.84)
 - /unlink 1786 (time 1779716015.80 total 20.01 last 84.69)
 - /unlink 2670 (time 1779716025.81 total 30.02 last 88.31)
 - /unlink 3613 (time 1779716035.81 total 40.02 last 94.29)
total: 4500 /unlink in 49.32 seconds: 91.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 521 (time 1779716062.92 total 10.02 last 52.00)
 - open/close 1016 (time 1779716072.93 total 20.03 last 49.46)
 - open/close 1491 (time 1779716082.93 total 30.03 last 47.49)
 - open/close 1926 (time 1779716092.95 total 40.04 last 43.45)
 - open/close 2377 (time 1779716102.97 total 50.06 last 45.01)
 - open/close 2869 (time 1779716112.97 total 60.06 last 49.18)
 - open/close 3363 (time 1779716122.98 total 70.07 last 49.36)
 - open/close 3875 (time 1779716132.99 total 80.09 last 51.11)
 - open/close 4459 (time 1779716143.00 total 90.10 last 58.33)
total: 4500 open/close in 90.62 seconds: 49.66 ops/second
 - /unlink 928 (time 1779716160.61 total 10.01 last 92.74)
 - /unlink 1868 (time 1779716170.63 total 20.03 last 93.80)
 - /unlink 2847 (time 1779716180.64 total 30.04 last 97.82)
 - /unlink 3809 (time 1779716190.65 total 40.05 last 96.11)
total: 4500 /unlink in 47.84 seconds: 94.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 392 (time 1779716216.21 total 10.01 last 39.17)
 - open/close 809 (time 1779716226.24 total 20.04 last 41.57)
 - open/close 1282 (time 1779716236.25 total 30.05 last 47.25)
 - open/close 1775 (time 1779716246.28 total 40.07 last 49.18)
 - open/close 2287 (time 1779716256.29 total 50.08 last 51.14)
 - open/close 2822 (time 1779716266.29 total 60.08 last 53.50)
 - open/close 3345 (time 1779716276.30 total 70.09 last 52.25)
 - open/close 3798 (time 1779716286.32 total 80.12 last 45.19)
 - open/close 4227 (time 1779716296.33 total 90.12 last 42.89)
total: 4500 open/close in 95.22 seconds: 47.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 976 (time 1779716318.98 total 10.01 last 97.53)
 - /unlink 1998 (time 1779716329.00 total 20.02 last 102.09)
 - /unlink 2835 (time 1779716339.00 total 30.02 last 83.67)
 - /unlink 3678 (time 1779716349.00 total 40.02 last 84.29)
total: 4500 /unlink in 49.75 seconds: 90.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 608 (time 1779716376.23 total 10.03 last 60.65)
 - open/close 1195 (time 1779716386.24 total 20.03 last 58.68)
 - open/close 1640 (time 1779716396.24 total 30.03 last 44.50)
 - open/close 2059 (time 1779716406.26 total 40.05 last 41.79)
 - open/close 2577 (time 1779716416.29 total 50.08 last 51.68)
 - open/close 3054 (time 1779716426.31 total 60.11 last 47.56)
 - open/close 3516 (time 1779716436.35 total 70.14 last 46.06)
 - open/close 3933 (time 1779716446.38 total 80.17 last 41.57)
 - open/close 4449 (time 1779716456.39 total 90.18 last 51.53)
total: 4500 open/close in 91.40 seconds: 49.23 ops/second
 - /unlink 978 (time 1779716474.97 total 10.00 last 97.76)
 - /unlink 1793 (time 1779716484.97 total 20.00 last 81.50)
 - /unlink 2773 (time 1779716494.98 total 30.01 last 97.95)
 - /unlink 3618 (time 1779716504.98 total 40.01 last 84.46)
total: 4500 /unlink in 49.91 seconds: 90.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 443 (time 1779716532.49 total 10.00 last 44.29)
 - open/close 999 (time 1779716542.50 total 20.01 last 55.54)
 - open/close 1547 (time 1779716552.53 total 30.04 last 54.67)
 - open/close 2065 (time 1779716562.54 total 40.05 last 51.75)
 - open/close 2588 (time 1779716572.54 total 50.05 last 52.28)
 - open/close 3041 (time 1779716582.55 total 60.06 last 45.25)
 - open/close 3519 (time 1779716592.58 total 70.10 last 47.63)
 - open/close 3956 (time 1779716602.59 total 80.10 last 43.66)
 - open/close 4465 (time 1779716612.61 total 90.12 last 50.81)
total: 4500 open/close in 90.93 seconds: 49.49 ops/second
 - /unlink 885 (time 1779716631.08 total 10.00 last 88.48)
 - /unlink 1695 (time 1779716641.09 total 20.01 last 80.92)
 - /unlink 2511 (time 1779716651.09 total 30.01 last 81.59)
 - /unlink 3300 (time 1779716661.11 total 40.03 last 78.77)
total: 4500 /unlink in 47.40 seconds: 94.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 1641 (time 1779716680.88 total 10.00 last 164.10)
 - open/close 3780 (time 1779716690.88 total 20.00 last 213.86)
total: 4500 open/close in 22.23 seconds: 202.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
total: 4500 /unlink in 6.74 seconds: 667.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 3072 (time 1779716712.96 total 10.01 last 307.03)
total: 4500 open/close in 15.07 seconds: 298.67 ops/second
total: 4500 /unlink in 7.73 seconds: 581.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 3030 (time 1779716738.92 total 10.00 last 302.90)
total: 4500 open/close in 14.74 seconds: 305.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
total: 4500 /unlink in 8.53 seconds: 527.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 2847 (time 1779716765.54 total 10.00 last 284.65)
total: 4500 open/close in 16.82 seconds: 267.60 ops/second
 - /unlink 4261 (time 1779716784.41 total 10.00 last 426.01)
total: 4500 /unlink in 10.57 seconds: 425.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 3009 (time 1779716796.87 total 10.00 last 300.89)
total: 4500 open/close in 15.06 seconds: 298.80 ops/second
total: 4500 /unlink in 7.75 seconds: 580.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 2976 (time 1779716822.90 total 10.00 last 297.50)
total: 4500 open/close in 15.13 seconds: 297.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
total: 4500 /unlink in 6.27 seconds: 718.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 4024 (time 1779716847.28 total 10.00 last 402.40)
total: 4500 open/close in 11.30 seconds: 398.37 ops/second
total: 4500 /unlink in 6.69 seconds: 672.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 4290 (time 1779716868.25 total 10.00 last 428.93)
total: 4500 open/close in 10.52 seconds: 427.84 ops/second
total: 4500 /unlink in 5.34 seconds: 842.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
 - open/close 3981 (time 1779716886.95 total 10.00 last 398.05)
total: 4500 open/close in 11.38 seconds: 395.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #246998
total: 4500 /unlink in 6.05 seconds: 743.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
total: 4500 open/close in 9.89 seconds: 455.18 ops/second
total: 4500 /unlink in 5.10 seconds: 881.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
total: 4500 open/close in 9.66 seconds: 466.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
total: 4500 /unlink in 4.95 seconds: 909.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
total: 4500 open/close in 9.72 seconds: 463.07 ops/second
total: 4500 /unlink in 5.02 seconds: 895.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
total: 4500 open/close in 9.14 seconds: 492.35 ops/second
total: 4500 /unlink in 5.36 seconds: 839.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #324998
total: 4500 open/close in 9.47 seconds: 475.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #337998
total: 4500 /unlink in 5.41 seconds: 832.34 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 94413 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg132-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg132-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server
stop mds service on oleg132-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server
stop mds service on oleg132-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server

unloading modules via unload_modules_local on: 'oleg132-server'
oleg132-server: oleg132-server.virtnet: executing unload_modules_local
oleg132-server: 
modules unloaded.