== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:17:57 (1785237477)
Stopping clients: oleg253-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg253-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg253-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg253-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg253-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg253-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg253-server
oleg253-server: oleg253-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg253-server'
oleg253-server: oleg253-server.virtnet: executing load_modules_local
oleg253-server: Loading modules from /home/green/git/lustre-release/lustre
oleg253-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg253-server: detected 4 online CPUs by sysfs
oleg253-server: MODOPTS_LIBCFS=
oleg253-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg253-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all
pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg253-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all
pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg253-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all
pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg253-client.virtnet:  -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 740 (time 1785237682.45 total 10.01 last 73.91)
 - open/close 1441 (time 1785237692.46 total 20.02 last 70.04)
 - open/close 1977 (time 1785237702.47 total 30.03 last 53.54)
 - open/close 2661 (time 1785237712.48 total 40.04 last 68.38)
 - open/close 3310 (time 1785237722.48 total 50.04 last 64.86)
 - open/close 4042 (time 1785237732.49 total 60.05 last 73.15)
total: 4500 open/close in 66.61 seconds: 67.56 ops/second
 - /unlink 1384 (time 1785237753.88 total 10.01 last 138.30)
 - /unlink 2826 (time 1785237763.88 total 20.01 last 144.14)
 - /unlink 4061 (time 1785237773.88 total 30.01 last 123.47)
total: 4500 /unlink in 34.18 seconds: 131.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 552 (time 1785237794.24 total 10.03 last 55.05)
 - open/close 1283 (time 1785237804.24 total 20.03 last 73.09)
 - open/close 1972 (time 1785237814.25 total 30.04 last 68.86)
 - open/close 2701 (time 1785237824.25 total 40.04 last 72.84)
 - open/close 3389 (time 1785237834.26 total 50.05 last 68.76)
 - open/close 4173 (time 1785237844.27 total 60.06 last 78.35)
total: 4500 open/close in 64.81 seconds: 69.43 ops/second
 - /unlink 1321 (time 1785237864.21 total 10.00 last 132.04)
 - /unlink 2670 (time 1785237874.21 total 20.01 last 134.88)
 - /unlink 4053 (time 1785237884.22 total 30.01 last 138.20)
total: 4500 /unlink in 33.34 seconds: 134.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 616 (time 1785237902.89 total 10.00 last 61.60)
 - open/close 1326 (time 1785237912.93 total 20.04 last 70.72)
 - open/close 2143 (time 1785237922.94 total 30.05 last 81.62)
 - open/close 2824 (time 1785237932.96 total 40.07 last 67.98)
 - open/close 3481 (time 1785237942.97 total 50.08 last 65.64)
 - open/close 4112 (time 1785237952.97 total 60.08 last 63.07)
total: 4500 open/close in 66.20 seconds: 67.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1171 (time 1785237973.45 total 10.01 last 117.02)
 - /unlink 2309 (time 1785237983.45 total 20.01 last 113.76)
 - /unlink 3698 (time 1785237993.45 total 30.01 last 138.89)
total: 4500 /unlink in 35.68 seconds: 126.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 752 (time 1785238013.62 total 10.01 last 75.12)
 - open/close 1518 (time 1785238023.63 total 20.02 last 76.56)
 - open/close 2296 (time 1785238033.63 total 30.02 last 77.79)
 - open/close 3049 (time 1785238043.63 total 40.02 last 75.26)
 - open/close 3790 (time 1785238053.64 total 50.03 last 74.02)
total: 4500 open/close in 59.74 seconds: 75.33 ops/second
 - /unlink 1430 (time 1785238077.54 total 10.01 last 142.88)
 - /unlink 2734 (time 1785238087.55 total 20.01 last 130.36)
 - /unlink 4138 (time 1785238097.55 total 30.01 last 140.37)
total: 4500 /unlink in 32.28 seconds: 139.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 789 (time 1785238114.47 total 10.01 last 78.83)
 - open/close 1641 (time 1785238124.48 total 20.02 last 85.07)
 - open/close 2392 (time 1785238134.49 total 30.04 last 75.01)
 - open/close 3069 (time 1785238144.50 total 40.04 last 67.67)
 - open/close 3724 (time 1785238154.51 total 50.05 last 65.45)
 - open/close 4315 (time 1785238164.52 total 60.06 last 59.03)
total: 4500 open/close in 62.72 seconds: 71.74 ops/second
 - /unlink 1246 (time 1785238182.92 total 10.01 last 124.48)
 - /unlink 2538 (time 1785238192.92 total 20.01 last 129.20)
 - /unlink 3930 (time 1785238202.93 total 30.02 last 139.08)
total: 4500 /unlink in 34.43 seconds: 130.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 790 (time 1785238221.72 total 10.01 last 78.94)
 - open/close 1579 (time 1785238231.72 total 20.01 last 78.88)
 - open/close 2320 (time 1785238241.73 total 30.01 last 74.06)
 - open/close 3100 (time 1785238251.73 total 40.02 last 77.96)
 - open/close 3709 (time 1785238261.74 total 50.03 last 60.86)
 - open/close 4280 (time 1785238271.75 total 60.04 last 57.05)
total: 4500 open/close in 63.67 seconds: 70.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1495 (time 1785238290.72 total 10.02 last 149.14)
 - /unlink 2747 (time 1785238300.72 total 20.03 last 125.13)
 - /unlink 4062 (time 1785238310.72 total 30.03 last 131.49)
total: 4500 /unlink in 33.79 seconds: 133.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 646 (time 1785238329.81 total 10.01 last 64.54)
 - open/close 1309 (time 1785238339.82 total 20.02 last 66.25)
 - open/close 1890 (time 1785238349.83 total 30.02 last 58.07)
 - open/close 2539 (time 1785238359.85 total 40.05 last 64.71)
 - open/close 3263 (time 1785238369.86 total 50.05 last 72.39)
 - open/close 3914 (time 1785238379.86 total 60.05 last 65.08)
total: 4500 open/close in 68.51 seconds: 65.69 ops/second
 - /unlink 1439 (time 1785238402.96 total 10.00 last 143.85)
 - /unlink 2622 (time 1785238412.96 total 20.00 last 118.30)
 - /unlink 3989 (time 1785238422.96 total 30.00 last 136.69)
total: 4500 /unlink in 33.97 seconds: 132.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 905 (time 1785238441.59 total 10.00 last 90.47)
 - open/close 2191 (time 1785238451.60 total 20.01 last 128.55)
 - open/close 4156 (time 1785238461.60 total 30.01 last 196.48)
total: 4500 open/close in 31.25 seconds: 144.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 4418 (time 1785238474.33 total 10.01 last 441.35)
total: 4500 /unlink in 11.14 seconds: 403.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 669 (time 1785238491.16 total 10.03 last 66.71)
 - open/close 1360 (time 1785238501.17 total 20.04 last 69.04)
 - open/close 2086 (time 1785238511.20 total 30.07 last 72.37)
 - open/close 2731 (time 1785238521.20 total 40.07 last 64.48)
 - open/close 3379 (time 1785238531.20 total 50.07 last 64.78)
 - open/close 4014 (time 1785238541.21 total 60.08 last 63.47)
total: 4500 open/close in 66.68 seconds: 67.49 ops/second
 - /unlink 1338 (time 1785238563.98 total 10.02 last 133.56)
 - /unlink 2624 (time 1785238573.99 total 20.02 last 128.59)
 - /unlink 3965 (time 1785238583.99 total 30.03 last 134.00)
total: 4500 /unlink in 34.40 seconds: 130.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 676 (time 1785238604.62 total 10.02 last 67.46)
 - open/close 1251 (time 1785238614.62 total 20.02 last 57.49)
 - open/close 1853 (time 1785238624.62 total 30.03 last 60.17)
 - open/close 2594 (time 1785238634.62 total 40.03 last 74.10)
 - open/close 3279 (time 1785238644.63 total 50.04 last 68.43)
 - open/close 3993 (time 1785238654.65 total 60.05 last 71.30)
total: 4500 open/close in 67.19 seconds: 66.98 ops/second
 - /unlink 1487 (time 1785238676.11 total 10.00 last 148.70)
 - /unlink 2893 (time 1785238686.11 total 20.00 last 140.56)
 - /unlink 4186 (time 1785238696.11 total 30.00 last 129.29)
total: 4500 /unlink in 32.15 seconds: 139.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 751 (time 1785238712.95 total 10.00 last 75.08)
 - open/close 1490 (time 1785238722.96 total 20.01 last 73.82)
 - open/close 2159 (time 1785238732.98 total 30.03 last 66.78)
 - open/close 2917 (time 1785238742.98 total 40.03 last 75.78)
 - open/close 3627 (time 1785238752.99 total 50.04 last 70.93)
 - open/close 4316 (time 1785238763.00 total 60.05 last 68.84)
total: 4500 open/close in 62.75 seconds: 71.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1480 (time 1785238780.18 total 10.00 last 147.97)
 - /unlink 2780 (time 1785238790.18 total 20.00 last 129.97)
 - /unlink 4133 (time 1785238800.19 total 30.01 last 135.19)
total: 4500 /unlink in 32.69 seconds: 137.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 671 (time 1785238817.57 total 10.02 last 66.99)
 - open/close 1391 (time 1785238827.57 total 20.02 last 71.97)
 - open/close 2085 (time 1785238837.58 total 30.03 last 69.37)
 - open/close 2766 (time 1785238847.58 total 40.03 last 68.05)
 - open/close 3498 (time 1785238857.59 total 50.04 last 73.18)
 - open/close 4243 (time 1785238867.59 total 60.04 last 74.49)
total: 4500 open/close in 63.52 seconds: 70.84 ops/second
 - /unlink 1571 (time 1785238885.85 total 10.00 last 157.07)
 - /unlink 3021 (time 1785238895.85 total 20.01 last 144.94)
 - /unlink 4462 (time 1785238905.85 total 30.01 last 144.07)
total: 4500 /unlink in 30.20 seconds: 149.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 763 (time 1785238920.13 total 10.00 last 76.28)
 - open/close 1558 (time 1785238930.14 total 20.01 last 79.42)
 - open/close 2254 (time 1785238940.14 total 30.02 last 69.57)
 - open/close 2953 (time 1785238950.15 total 40.03 last 69.85)
 - open/close 3585 (time 1785238960.15 total 50.03 last 63.19)
 - open/close 4318 (time 1785238970.16 total 60.04 last 73.24)
total: 4500 open/close in 62.41 seconds: 72.10 ops/second
 - /unlink 1160 (time 1785238987.17 total 10.00 last 115.98)
 - /unlink 2424 (time 1785238997.17 total 20.01 last 126.35)
 - /unlink 3632 (time 1785239007.17 total 30.01 last 120.75)
total: 4500 /unlink in 37.87 seconds: 118.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 594 (time 1785239031.09 total 10.01 last 59.34)
 - open/close 1245 (time 1785239041.09 total 20.01 last 65.08)
 - open/close 1836 (time 1785239051.09 total 30.01 last 59.08)
 - open/close 2309 (time 1785239061.10 total 40.02 last 47.29)
 - open/close 2858 (time 1785239071.10 total 50.02 last 54.86)
 - open/close 3482 (time 1785239081.11 total 60.03 last 62.35)
 - open/close 4176 (time 1785239091.12 total 70.04 last 69.31)
total: 4500 open/close in 75.98 seconds: 59.22 ops/second
 - /unlink 1590 (time 1785239112.33 total 10.00 last 158.96)
 - /unlink 2812 (time 1785239122.34 total 20.01 last 122.11)
 - /unlink 3935 (time 1785239132.34 total 30.01 last 112.30)
total: 4500 /unlink in 34.97 seconds: 128.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 538 (time 1785239154.19 total 10.01 last 53.76)
 - open/close 1176 (time 1785239164.19 total 20.01 last 63.78)
 - open/close 1829 (time 1785239174.20 total 30.02 last 65.26)
 - open/close 2370 (time 1785239184.21 total 40.02 last 54.06)
 - open/close 2826 (time 1785239194.22 total 50.03 last 45.56)
 - open/close 3524 (time 1785239204.22 total 60.04 last 69.76)
 - open/close 4187 (time 1785239214.23 total 70.05 last 66.26)
total: 4500 open/close in 74.14 seconds: 60.69 ops/second
 - /unlink 1294 (time 1785239233.03 total 10.00 last 129.37)
 - /unlink 2455 (time 1785239243.04 total 20.01 last 116.04)
 - /unlink 3794 (time 1785239253.04 total 30.01 last 133.90)
total: 4500 /unlink in 35.67 seconds: 126.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 645 (time 1785239273.61 total 10.02 last 64.39)
 - open/close 1216 (time 1785239283.62 total 20.03 last 57.03)
 - open/close 1908 (time 1785239293.62 total 30.03 last 69.19)
 - open/close 2566 (time 1785239303.63 total 40.04 last 65.71)
 - open/close 3236 (time 1785239313.63 total 50.05 last 66.99)
 - open/close 3971 (time 1785239323.64 total 60.05 last 73.43)
total: 4500 open/close in 66.47 seconds: 67.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1430 (time 1785239344.70 total 10.00 last 142.93)
 - /unlink 2666 (time 1785239354.70 total 20.01 last 123.57)
 - /unlink 3892 (time 1785239364.71 total 30.01 last 122.50)
total: 4500 /unlink in 35.82 seconds: 125.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 531 (time 1785239385.99 total 10.03 last 52.95)
 - open/close 1055 (time 1785239396.00 total 20.03 last 52.37)
 - open/close 1761 (time 1785239406.00 total 30.04 last 70.55)
 - open/close 2424 (time 1785239416.01 total 40.05 last 66.26)
 - open/close 3030 (time 1785239426.02 total 50.05 last 60.56)
 - open/close 3632 (time 1785239436.02 total 60.06 last 60.19)
 - open/close 4287 (time 1785239446.02 total 70.06 last 65.47)
total: 4500 open/close in 73.26 seconds: 61.43 ops/second
 - /unlink 1084 (time 1785239464.25 total 10.01 last 108.32)
 - /unlink 2045 (time 1785239474.25 total 20.01 last 96.10)
 - /unlink 3037 (time 1785239484.25 total 30.01 last 99.19)
 - /unlink 4215 (time 1785239494.25 total 40.01 last 117.79)
total: 4500 /unlink in 43.50 seconds: 103.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 532 (time 1785239515.70 total 10.01 last 53.16)
 - open/close 1215 (time 1785239525.71 total 20.01 last 68.27)
 - open/close 1836 (time 1785239535.71 total 30.01 last 62.08)
 - open/close 2413 (time 1785239545.71 total 40.02 last 57.69)
 - open/close 2853 (time 1785239555.75 total 50.06 last 43.83)
 - open/close 3476 (time 1785239565.75 total 60.06 last 62.28)
 - open/close 4050 (time 1785239575.77 total 70.07 last 57.33)
total: 4500 open/close in 76.44 seconds: 58.87 ops/second
 - /unlink 1074 (time 1785239596.88 total 10.01 last 107.32)
 - /unlink 1820 (time 1785239606.90 total 20.02 last 74.48)
 - /unlink 2821 (time 1785239616.90 total 30.03 last 100.04)
 - /unlink 3675 (time 1785239626.91 total 40.03 last 85.38)
total: 4500 /unlink in 46.64 seconds: 96.48 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95271 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg253-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg253-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg253-server
stop mds service on oleg253-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg253-server
stop mds service on oleg253-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg253-server

unloading modules via unload_modules_local on: 'oleg253-server'
oleg253-server: oleg253-server.virtnet: executing unload_modules_local
oleg253-server: 
modules unloaded.