== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:13:47 (1786418027)
Stopping clients: oleg317-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg317-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg317-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg317-server
oleg317-server: oleg317-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg317-server'
oleg317-server: oleg317-server.virtnet: executing load_modules_local
oleg317-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg317-server: Loading modules from /home/green/git/lustre-release/lustre
oleg317-server: detected 4 online CPUs by sysfs
oleg317-server: MODOPTS_LIBCFS=
oleg317-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg317-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all
pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg317-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all
pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg317-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all
pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg317-client.virtnet:  -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 756 (time 1786418229.27 total 10.00 last 75.57)
 - open/close 1504 (time 1786418239.28 total 20.01 last 74.78)
 - open/close 2256 (time 1786418249.30 total 30.03 last 75.03)
 - open/close 3034 (time 1786418259.31 total 40.04 last 77.73)
 - open/close 3762 (time 1786418269.31 total 50.04 last 72.80)
total: 4500 open/close in 59.71 seconds: 75.36 ops/second
 - /unlink 1354 (time 1786418294.01 total 10.01 last 135.22)
 - /unlink 2734 (time 1786418304.03 total 20.03 last 137.76)
 - /unlink 4034 (time 1786418314.06 total 30.07 last 129.53)
total: 4500 /unlink in 33.60 seconds: 133.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 744 (time 1786418332.93 total 10.05 last 74.07)
 - open/close 1491 (time 1786418342.93 total 20.05 last 74.69)
 - open/close 2170 (time 1786418352.93 total 30.05 last 67.88)
 - open/close 2896 (time 1786418362.94 total 40.06 last 72.54)
 - open/close 3691 (time 1786418372.94 total 50.06 last 79.48)
 - open/close 4445 (time 1786418382.98 total 60.10 last 75.10)
total: 4500 open/close in 60.93 seconds: 73.86 ops/second
 - /unlink 1313 (time 1786418399.89 total 10.00 last 131.26)
 - /unlink 2717 (time 1786418409.89 total 20.00 last 140.38)
 - /unlink 4076 (time 1786418419.89 total 30.01 last 135.88)
total: 4500 /unlink in 33.56 seconds: 134.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 757 (time 1786418438.16 total 10.01 last 75.64)
 - open/close 1469 (time 1786418448.17 total 20.02 last 71.11)
 - open/close 2288 (time 1786418458.20 total 30.04 last 81.71)
 - open/close 3043 (time 1786418468.20 total 40.05 last 75.47)
 - open/close 3856 (time 1786418478.21 total 50.06 last 81.25)
total: 4500 open/close in 59.79 seconds: 75.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1334 (time 1786418502.03 total 10.00 last 133.33)
 - /unlink 2444 (time 1786418512.03 total 20.01 last 110.98)
 - /unlink 3703 (time 1786418522.03 total 30.01 last 125.87)
total: 4500 /unlink in 35.96 seconds: 125.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 772 (time 1786418543.36 total 10.01 last 77.16)
 - open/close 1541 (time 1786418553.36 total 20.01 last 76.89)
 - open/close 2272 (time 1786418563.36 total 30.01 last 73.09)
 - open/close 2997 (time 1786418573.37 total 40.02 last 72.41)
 - open/close 3747 (time 1786418583.38 total 50.02 last 74.97)
total: 4500 open/close in 59.48 seconds: 75.66 ops/second
 - /unlink 1229 (time 1786418607.80 total 10.10 last 121.64)
 - /unlink 2488 (time 1786418617.81 total 20.11 last 125.84)
 - /unlink 3599 (time 1786418627.81 total 30.11 last 111.05)
total: 4500 /unlink in 37.61 seconds: 119.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 689 (time 1786418650.45 total 10.04 last 68.62)
 - open/close 1406 (time 1786418660.45 total 20.04 last 71.68)
 - open/close 2190 (time 1786418670.45 total 30.05 last 78.38)
 - open/close 2948 (time 1786418680.46 total 40.05 last 75.75)
 - open/close 3645 (time 1786418690.46 total 50.06 last 69.69)
 - open/close 4404 (time 1786418700.47 total 60.06 last 75.86)
total: 4500 open/close in 61.27 seconds: 73.44 ops/second
 - /unlink 1381 (time 1786418716.37 total 10.01 last 138.02)
 - /unlink 2598 (time 1786418726.37 total 20.01 last 121.69)
 - /unlink 3846 (time 1786418736.38 total 30.01 last 124.74)
total: 4500 /unlink in 36.09 seconds: 124.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 760 (time 1786418757.74 total 10.00 last 75.96)
 - open/close 1372 (time 1786418767.74 total 20.01 last 61.15)
 - open/close 2092 (time 1786418777.79 total 30.05 last 71.70)
 - open/close 2718 (time 1786418787.80 total 40.07 last 62.53)
 - open/close 3460 (time 1786418797.81 total 50.07 last 74.14)
 - open/close 4142 (time 1786418807.81 total 60.08 last 68.15)
total: 4500 open/close in 65.61 seconds: 68.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1176 (time 1786418829.10 total 10.01 last 117.54)
 - /unlink 2482 (time 1786418839.10 total 20.01 last 130.54)
 - /unlink 3747 (time 1786418849.10 total 30.01 last 126.48)
total: 4500 /unlink in 36.79 seconds: 122.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 789 (time 1786418870.62 total 10.00 last 78.86)
 - open/close 1562 (time 1786418880.62 total 20.01 last 77.26)
 - open/close 2352 (time 1786418890.62 total 30.01 last 78.98)
 - open/close 3094 (time 1786418900.63 total 40.02 last 74.14)
 - open/close 3728 (time 1786418910.64 total 50.03 last 63.36)
 - open/close 4445 (time 1786418920.64 total 60.03 last 71.69)
total: 4500 open/close in 60.84 seconds: 73.97 ops/second
 - /unlink 1398 (time 1786418936.18 total 10.00 last 139.80)
 - /unlink 2659 (time 1786418946.19 total 20.01 last 125.92)
 - /unlink 3839 (time 1786418956.20 total 30.02 last 117.90)
total: 4500 /unlink in 35.41 seconds: 127.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 781 (time 1786418976.02 total 10.00 last 78.09)
 - open/close 1514 (time 1786418986.03 total 20.00 last 73.29)
 - open/close 2275 (time 1786418996.03 total 30.00 last 76.10)
 - open/close 3044 (time 1786419006.04 total 40.02 last 76.81)
 - open/close 3806 (time 1786419016.04 total 50.02 last 76.18)
 - open/close 4441 (time 1786419026.05 total 60.03 last 63.42)
total: 4500 open/close in 60.87 seconds: 73.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1339 (time 1786419041.79 total 10.01 last 133.82)
 - /unlink 2581 (time 1786419051.79 total 20.01 last 124.16)
 - /unlink 3704 (time 1786419061.79 total 30.02 last 112.23)
total: 4500 /unlink in 37.68 seconds: 119.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 696 (time 1786419085.06 total 10.00 last 69.58)
 - open/close 1393 (time 1786419095.08 total 20.02 last 69.58)
 - open/close 2029 (time 1786419105.09 total 30.03 last 63.54)
 - open/close 2686 (time 1786419115.12 total 40.06 last 65.53)
 - open/close 3371 (time 1786419125.12 total 50.06 last 68.47)
 - open/close 4082 (time 1786419135.13 total 60.07 last 71.05)
total: 4500 open/close in 65.31 seconds: 68.90 ops/second
 - /unlink 1401 (time 1786419154.61 total 10.00 last 140.09)
 - /unlink 2691 (time 1786419164.61 total 20.00 last 128.99)
 - /unlink 3909 (time 1786419174.61 total 30.00 last 121.79)
total: 4500 /unlink in 34.68 seconds: 129.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 714 (time 1786419194.45 total 10.05 last 71.05)
 - open/close 1430 (time 1786419204.49 total 20.09 last 71.29)
 - open/close 2139 (time 1786419214.50 total 30.10 last 70.85)
 - open/close 2882 (time 1786419224.50 total 40.10 last 74.28)
 - open/close 3614 (time 1786419234.54 total 50.14 last 72.93)
 - open/close 4356 (time 1786419244.54 total 60.14 last 74.20)
total: 4500 open/close in 62.04 seconds: 72.53 ops/second
 - /unlink 1403 (time 1786419261.37 total 10.00 last 140.27)
 - /unlink 2712 (time 1786419271.37 total 20.01 last 130.85)
 - /unlink 4075 (time 1786419281.38 total 30.01 last 136.23)
total: 4500 /unlink in 34.48 seconds: 130.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 780 (time 1786419300.93 total 10.01 last 77.92)
 - open/close 1487 (time 1786419310.93 total 20.02 last 70.65)
 - open/close 2231 (time 1786419320.94 total 30.02 last 74.36)
 - open/close 2988 (time 1786419330.94 total 40.03 last 75.67)
 - open/close 3780 (time 1786419340.96 total 50.04 last 79.07)
total: 4500 open/close in 59.80 seconds: 75.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1280 (time 1786419365.46 total 10.00 last 127.95)
 - /unlink 2470 (time 1786419375.46 total 20.01 last 118.97)
 - /unlink 3706 (time 1786419385.48 total 30.02 last 123.39)
total: 4500 /unlink in 37.24 seconds: 120.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 752 (time 1786419408.09 total 10.15 last 74.12)
 - open/close 1451 (time 1786419418.09 total 20.15 last 69.89)
 - open/close 2171 (time 1786419428.10 total 30.16 last 71.92)
 - open/close 2925 (time 1786419438.11 total 40.16 last 75.36)
 - open/close 3668 (time 1786419448.11 total 50.16 last 74.29)
 - open/close 4356 (time 1786419458.13 total 60.19 last 68.65)
total: 4500 open/close in 62.15 seconds: 72.41 ops/second
 - /unlink 1256 (time 1786419474.05 total 10.00 last 125.60)
 - /unlink 2641 (time 1786419484.05 total 20.00 last 138.44)
 - /unlink 3983 (time 1786419494.06 total 30.01 last 134.10)
total: 4500 /unlink in 34.49 seconds: 130.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 720 (time 1786419513.03 total 10.01 last 71.95)
 - open/close 1456 (time 1786419523.05 total 20.02 last 73.52)
 - open/close 2198 (time 1786419533.05 total 30.03 last 74.14)
 - open/close 2922 (time 1786419543.07 total 40.04 last 72.30)
 - open/close 3681 (time 1786419553.07 total 50.04 last 75.88)
 - open/close 4428 (time 1786419563.08 total 60.06 last 74.59)
total: 4500 open/close in 60.98 seconds: 73.80 ops/second
 - /unlink 1250 (time 1786419578.69 total 10.00 last 124.97)
 - /unlink 2520 (time 1786419588.69 total 20.01 last 126.96)
 - /unlink 3786 (time 1786419598.71 total 30.02 last 126.41)
total: 4500 /unlink in 35.57 seconds: 126.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 779 (time 1786419618.29 total 10.00 last 77.87)
 - open/close 1559 (time 1786419628.29 total 20.01 last 77.98)
 - open/close 2281 (time 1786419638.30 total 30.01 last 72.14)
 - open/close 3020 (time 1786419648.30 total 40.02 last 73.89)
 - open/close 3765 (time 1786419658.30 total 50.02 last 74.45)
total: 4500 open/close in 58.91 seconds: 76.39 ops/second
 - /unlink 1379 (time 1786419681.60 total 10.00 last 137.84)
 - /unlink 2728 (time 1786419691.60 total 20.00 last 134.90)
 - /unlink 4030 (time 1786419701.60 total 30.01 last 130.17)
total: 4500 /unlink in 33.50 seconds: 134.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 720 (time 1786419719.76 total 10.02 last 71.83)
 - open/close 1470 (time 1786419729.77 total 20.04 last 74.91)
 - open/close 2223 (time 1786419739.77 total 30.04 last 75.29)
 - open/close 2955 (time 1786419749.79 total 40.05 last 73.10)
 - open/close 3609 (time 1786419759.83 total 50.09 last 65.13)
 - open/close 4324 (time 1786419769.84 total 60.10 last 71.44)
total: 4500 open/close in 62.40 seconds: 72.11 ops/second
 - /unlink 1362 (time 1786419786.28 total 10.00 last 136.17)
 - /unlink 2616 (time 1786419796.30 total 20.01 last 125.25)
 - /unlink 3784 (time 1786419806.31 total 30.03 last 116.68)
total: 4500 /unlink in 35.87 seconds: 125.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 594 (time 1786419826.53 total 10.00 last 59.40)
 - open/close 1240 (time 1786419836.54 total 20.01 last 64.53)
 - open/close 1956 (time 1786419846.56 total 30.03 last 71.45)
 - open/close 2681 (time 1786419856.57 total 40.04 last 72.47)
 - open/close 3447 (time 1786419866.57 total 50.04 last 76.58)
 - open/close 4164 (time 1786419876.58 total 60.05 last 71.62)
total: 4500 open/close in 64.99 seconds: 69.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1350 (time 1786419896.52 total 10.00 last 134.98)
 - /unlink 2640 (time 1786419906.52 total 20.00 last 128.95)
 - /unlink 3951 (time 1786419916.52 total 30.01 last 131.07)
total: 4500 /unlink in 33.51 seconds: 134.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 734 (time 1786419935.20 total 10.01 last 73.36)
 - open/close 1546 (time 1786419945.21 total 20.01 last 81.17)
 - open/close 2357 (time 1786419955.22 total 30.02 last 81.04)
 - open/close 3162 (time 1786419965.22 total 40.03 last 80.44)
 - open/close 3910 (time 1786419975.23 total 50.03 last 74.75)
total: 4500 open/close in 56.83 seconds: 79.19 ops/second
 - /unlink 1378 (time 1786419996.02 total 10.01 last 137.65)
 - /unlink 2579 (time 1786420006.02 total 20.01 last 120.06)
 - /unlink 3824 (time 1786420016.03 total 30.02 last 124.42)
total: 4500 /unlink in 34.83 seconds: 129.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 796 (time 1786420035.27 total 10.00 last 79.57)
 - open/close 1499 (time 1786420045.27 total 20.01 last 70.29)
 - open/close 2276 (time 1786420055.27 total 30.01 last 77.70)
 - open/close 3014 (time 1786420065.28 total 40.02 last 73.73)
 - open/close 3738 (time 1786420075.29 total 50.03 last 72.31)
 - open/close 4443 (time 1786420085.31 total 60.05 last 70.35)
total: 4500 open/close in 60.78 seconds: 74.04 ops/second
 - /unlink 1286 (time 1786420101.64 total 10.00 last 128.59)
 - /unlink 2577 (time 1786420111.64 total 20.01 last 129.04)
 - /unlink 3864 (time 1786420121.65 total 30.01 last 128.64)
total: 4500 /unlink in 34.69 seconds: 129.71 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95010 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user 'cl1': success
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user 'cl1': success

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg317-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg317-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server
stop mds service on oleg317-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server
stop mds service on oleg317-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server

unloading modules via unload_modules_local on: 'oleg317-server'
oleg317-server: oleg317-server.virtnet: executing unload_modules_local
oleg317-server: 
modules unloaded.