== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:44:30 (1782261870)
Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg225-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg225-server
oleg225-server: oleg225-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing load_modules_local
oleg225-server: Loading modules from /home/green/git/lustre-release/lustre
oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg225-server: detected 4 online CPUs by sysfs
oleg225-server: MODOPTS_LIBCFS=
oleg225-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg225-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg225-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg225-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg225-client.virtnet:  -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 739 (time 1782262087.94 total 10.00 last 73.86)
 - open/close 1457 (time 1782262097.94 total 20.01 last 71.78)
 - open/close 2198 (time 1782262107.94 total 30.01 last 74.09)
 - open/close 2857 (time 1782262117.94 total 40.01 last 65.90)
 - open/close 3447 (time 1782262127.95 total 50.02 last 58.96)
 - open/close 4132 (time 1782262137.95 total 60.02 last 68.49)
total: 4500 open/close in 65.68 seconds: 68.51 ops/second
 - /unlink 1175 (time 1782262158.89 total 10.00 last 117.50)
 - /unlink 2397 (time 1782262168.90 total 20.00 last 122.18)
 - /unlink 3685 (time 1782262178.90 total 30.01 last 128.74)
total: 4500 /unlink in 36.58 seconds: 123.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 735 (time 1782262199.59 total 10.00 last 73.46)
 - open/close 1588 (time 1782262209.59 total 20.01 last 85.27)
 - open/close 2334 (time 1782262219.60 total 30.02 last 74.50)
 - open/close 2996 (time 1782262229.61 total 40.02 last 66.18)
 - open/close 3596 (time 1782262239.63 total 50.05 last 59.84)
 - open/close 4231 (time 1782262249.64 total 60.06 last 63.46)
total: 4500 open/close in 63.95 seconds: 70.37 ops/second
 - /unlink 1424 (time 1782262268.17 total 10.01 last 142.29)
 - /unlink 2754 (time 1782262278.19 total 20.03 last 132.76)
 - /unlink 3915 (time 1782262288.20 total 30.03 last 116.07)
total: 4500 /unlink in 35.47 seconds: 126.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 739 (time 1782262308.00 total 10.02 last 73.75)
 - open/close 1502 (time 1782262318.02 total 20.04 last 76.14)
 - open/close 2344 (time 1782262328.03 total 30.06 last 84.08)
 - open/close 3021 (time 1782262338.04 total 40.06 last 67.66)
 - open/close 3678 (time 1782262348.05 total 50.07 last 65.63)
 - open/close 4315 (time 1782262358.06 total 60.08 last 63.64)
total: 4500 open/close in 62.39 seconds: 72.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1428 (time 1782262375.51 total 10.01 last 142.73)
 - /unlink 2765 (time 1782262385.52 total 20.01 last 133.61)
 - /unlink 4241 (time 1782262395.52 total 30.01 last 147.59)
total: 4500 /unlink in 32.21 seconds: 139.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 658 (time 1782262412.72 total 10.02 last 65.67)
 - open/close 1361 (time 1782262422.72 total 20.02 last 70.29)
 - open/close 2073 (time 1782262432.73 total 30.03 last 71.14)
 - open/close 2737 (time 1782262442.73 total 40.03 last 66.37)
 - open/close 3458 (time 1782262452.74 total 50.04 last 72.08)
 - open/close 4166 (time 1782262462.75 total 60.05 last 70.70)
total: 4500 open/close in 65.01 seconds: 69.22 ops/second
 - /unlink 1242 (time 1782262482.49 total 10.00 last 124.20)
 - /unlink 2648 (time 1782262492.50 total 20.00 last 140.54)
 - /unlink 3911 (time 1782262502.50 total 30.01 last 126.23)
total: 4500 /unlink in 34.70 seconds: 129.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 817 (time 1782262521.77 total 10.00 last 81.67)
 - open/close 1439 (time 1782262531.78 total 20.02 last 62.13)
 - open/close 2077 (time 1782262541.78 total 30.02 last 63.77)
 - open/close 2753 (time 1782262551.79 total 40.03 last 67.55)
 - open/close 3550 (time 1782262561.83 total 50.07 last 79.40)
 - open/close 4345 (time 1782262571.84 total 60.08 last 79.41)
total: 4500 open/close in 61.96 seconds: 72.62 ops/second
 - /unlink 1399 (time 1782262588.32 total 10.00 last 139.85)
 - /unlink 2576 (time 1782262598.34 total 20.02 last 117.47)
 - /unlink 3779 (time 1782262608.34 total 30.03 last 120.24)
total: 4500 /unlink in 36.08 seconds: 124.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 557 (time 1782262630.21 total 10.03 last 55.52)
 - open/close 1295 (time 1782262640.22 total 20.04 last 73.78)
 - open/close 2049 (time 1782262650.23 total 30.05 last 75.27)
 - open/close 2761 (time 1782262660.24 total 40.06 last 71.15)
 - open/close 3435 (time 1782262670.25 total 50.07 last 67.35)
 - open/close 4057 (time 1782262680.28 total 60.10 last 61.99)
total: 4500 open/close in 67.71 seconds: 66.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 975 (time 1782262703.05 total 10.00 last 97.46)
 - /unlink 2045 (time 1782262713.05 total 20.00 last 106.99)
 - /unlink 3263 (time 1782262723.06 total 30.01 last 121.76)
total: 4500 /unlink in 39.03 seconds: 115.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 684 (time 1782262747.69 total 10.00 last 68.38)
 - open/close 1384 (time 1782262757.70 total 20.01 last 69.93)
 - open/close 2015 (time 1782262767.71 total 30.02 last 63.05)
 - open/close 2676 (time 1782262777.72 total 40.03 last 66.03)
 - open/close 3377 (time 1782262787.73 total 50.05 last 70.00)
 - open/close 4177 (time 1782262797.74 total 60.05 last 79.95)
total: 4500 open/close in 63.80 seconds: 70.53 ops/second
 - /unlink 1190 (time 1782262817.03 total 10.00 last 118.98)
 - /unlink 2365 (time 1782262827.04 total 20.01 last 117.44)
 - /unlink 3611 (time 1782262837.04 total 30.01 last 124.58)
total: 4500 /unlink in 37.38 seconds: 120.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 727 (time 1782262859.63 total 10.00 last 72.68)
 - open/close 1420 (time 1782262869.66 total 20.03 last 69.14)
 - open/close 2227 (time 1782262879.66 total 30.03 last 80.67)
 - open/close 2984 (time 1782262889.68 total 40.05 last 75.58)
 - open/close 3743 (time 1782262899.68 total 50.05 last 75.85)
total: 4500 open/close in 59.99 seconds: 75.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1380 (time 1782262924.59 total 10.00 last 137.95)
 - /unlink 2614 (time 1782262934.60 total 20.01 last 123.36)
 - /unlink 3824 (time 1782262944.60 total 30.01 last 120.97)
total: 4500 /unlink in 36.39 seconds: 123.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 656 (time 1782262965.77 total 10.00 last 65.60)
 - open/close 1307 (time 1782262975.78 total 20.00 last 65.07)
 - open/close 2072 (time 1782262985.79 total 30.01 last 76.43)
 - open/close 2941 (time 1782262995.79 total 40.02 last 86.83)
 - open/close 3632 (time 1782263005.81 total 50.03 last 69.01)
 - open/close 4461 (time 1782263015.83 total 60.05 last 82.75)
total: 4500 open/close in 60.84 seconds: 73.96 ops/second
 - /unlink 1145 (time 1782263031.62 total 10.00 last 114.48)
 - /unlink 2322 (time 1782263041.62 total 20.01 last 117.65)
 - /unlink 3535 (time 1782263051.62 total 30.01 last 121.29)
total: 4500 /unlink in 38.95 seconds: 115.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 713 (time 1782263075.41 total 10.01 last 71.22)
 - open/close 1500 (time 1782263085.42 total 20.02 last 78.63)
 - open/close 2249 (time 1782263095.43 total 30.03 last 74.86)
 - open/close 2884 (time 1782263105.43 total 40.03 last 63.48)
 - open/close 3519 (time 1782263115.44 total 50.04 last 63.43)
 - open/close 4172 (time 1782263125.47 total 60.07 last 65.13)
total: 4500 open/close in 65.20 seconds: 69.02 ops/second
 - /unlink 1163 (time 1782263145.81 total 10.00 last 116.30)
 - /unlink 2490 (time 1782263155.81 total 20.00 last 132.69)
 - /unlink 3708 (time 1782263165.82 total 30.00 last 121.78)
total: 4500 /unlink in 36.81 seconds: 122.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 727 (time 1782263187.45 total 10.03 last 72.50)
 - open/close 1483 (time 1782263197.47 total 20.05 last 75.44)
 - open/close 2206 (time 1782263207.47 total 30.05 last 72.27)
 - open/close 2902 (time 1782263217.48 total 40.06 last 69.55)
 - open/close 3551 (time 1782263227.48 total 50.06 last 64.86)
 - open/close 4131 (time 1782263237.50 total 60.08 last 57.90)
total: 4500 open/close in 66.77 seconds: 67.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1250 (time 1782263259.37 total 10.00 last 124.96)
 - /unlink 2375 (time 1782263269.38 total 20.01 last 112.46)
 - /unlink 3357 (time 1782263279.40 total 30.03 last 97.94)
 - /unlink 4478 (time 1782263289.41 total 40.04 last 112.08)
total: 4500 /unlink in 40.26 seconds: 111.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 813 (time 1782263305.39 total 10.02 last 81.17)
 - open/close 1554 (time 1782263315.40 total 20.03 last 74.01)
 - open/close 2212 (time 1782263325.41 total 30.04 last 65.75)
 - open/close 2822 (time 1782263335.42 total 40.04 last 60.95)
 - open/close 3402 (time 1782263345.42 total 50.05 last 57.97)
 - open/close 3814 (time 1782263355.45 total 60.08 last 41.07)
 - open/close 4370 (time 1782263365.47 total 70.10 last 55.53)
total: 4500 open/close in 71.71 seconds: 62.76 ops/second
 - /unlink 1372 (time 1782263382.96 total 10.02 last 136.96)
 - /unlink 2770 (time 1782263392.97 total 20.03 last 139.62)
 - /unlink 4095 (time 1782263402.98 total 30.04 last 132.43)
total: 4500 /unlink in 34.05 seconds: 132.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 608 (time 1782263422.81 total 10.01 last 60.73)
 - open/close 1207 (time 1782263432.82 total 20.03 last 59.81)
 - open/close 1818 (time 1782263442.83 total 30.03 last 61.05)
 - open/close 2655 (time 1782263452.84 total 40.04 last 83.62)
 - open/close 3392 (time 1782263462.85 total 50.06 last 73.62)
 - open/close 4114 (time 1782263472.86 total 60.06 last 72.15)
total: 4500 open/close in 66.38 seconds: 67.79 ops/second
 - /unlink 1150 (time 1782263494.15 total 10.01 last 114.94)
 - /unlink 2447 (time 1782263504.15 total 20.01 last 129.67)
 - /unlink 3749 (time 1782263514.16 total 30.01 last 130.14)
total: 4500 /unlink in 36.49 seconds: 123.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 652 (time 1782263536.47 total 10.01 last 65.17)
 - open/close 1335 (time 1782263546.49 total 20.02 last 68.17)
 - open/close 1915 (time 1782263556.50 total 30.03 last 57.98)
 - open/close 2590 (time 1782263566.50 total 40.03 last 67.46)
 - open/close 3117 (time 1782263576.52 total 50.05 last 52.63)
 - open/close 3798 (time 1782263586.52 total 60.05 last 68.06)
 - open/close 4466 (time 1782263596.53 total 70.06 last 66.76)
total: 4500 open/close in 70.49 seconds: 63.84 ops/second
 - /unlink 1263 (time 1782263613.26 total 10.00 last 126.29)
 - /unlink 2246 (time 1782263623.26 total 20.00 last 98.27)
 - /unlink 3049 (time 1782263633.27 total 30.01 last 80.22)
 - /unlink 3828 (time 1782263643.28 total 40.02 last 77.85)
total: 4500 /unlink in 47.75 seconds: 94.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 693 (time 1782263667.25 total 10.01 last 69.26)
 - open/close 1371 (time 1782263677.26 total 20.02 last 67.72)
 - open/close 2019 (time 1782263687.28 total 30.04 last 64.66)
 - open/close 2761 (time 1782263697.28 total 40.04 last 74.17)
 - open/close 3307 (time 1782263707.28 total 50.04 last 54.60)
 - open/close 4029 (time 1782263717.30 total 60.06 last 72.11)
total: 4500 open/close in 67.98 seconds: 66.20 ops/second
 - /unlink 1236 (time 1782263741.80 total 10.00 last 123.56)
 - /unlink 2551 (time 1782263751.81 total 20.02 last 131.29)
 - /unlink 3794 (time 1782263761.82 total 30.02 last 124.25)
total: 4500 /unlink in 34.73 seconds: 129.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 807 (time 1782263780.89 total 10.00 last 80.67)
 - open/close 1497 (time 1782263790.91 total 20.02 last 68.89)
 - open/close 2297 (time 1782263800.92 total 30.03 last 79.88)
 - open/close 3055 (time 1782263810.93 total 40.05 last 75.72)
 - open/close 3711 (time 1782263820.94 total 50.05 last 65.54)
 - open/close 4427 (time 1782263830.95 total 60.06 last 71.58)
total: 4500 open/close in 60.64 seconds: 74.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1096 (time 1782263846.43 total 10.00 last 109.55)
 - /unlink 2195 (time 1782263856.43 total 20.00 last 109.90)
 - /unlink 3396 (time 1782263866.46 total 30.03 last 119.79)
total: 4500 /unlink in 39.01 seconds: 115.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 603 (time 1782263890.71 total 10.00 last 60.27)
 - open/close 1141 (time 1782263900.72 total 20.02 last 53.73)
 - open/close 1670 (time 1782263910.73 total 30.02 last 52.88)
 - open/close 2422 (time 1782263920.74 total 40.03 last 75.12)
 - open/close 3214 (time 1782263930.74 total 50.04 last 79.16)
 - open/close 4003 (time 1782263940.75 total 60.04 last 78.87)
total: 4500 open/close in 66.52 seconds: 67.65 ops/second
 - /unlink 1300 (time 1782263962.53 total 10.00 last 129.98)
 - /unlink 2455 (time 1782263972.54 total 20.01 last 115.37)
 - /unlink 3724 (time 1782263982.55 total 30.02 last 126.80)
total: 4500 /unlink in 35.67 seconds: 126.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 796 (time 1782264003.47 total 10.00 last 79.57)
 - open/close 1528 (time 1782264013.47 total 20.00 last 73.20)
 - open/close 2143 (time 1782264023.48 total 30.01 last 61.45)
 - open/close 2834 (time 1782264033.50 total 40.03 last 69.00)
 - open/close 3484 (time 1782264043.50 total 50.03 last 65.00)
 - open/close 3965 (time 1782264053.52 total 60.04 last 48.02)
 - open/close 4364 (time 1782264063.52 total 70.05 last 39.87)
total: 4500 open/close in 72.58 seconds: 62.00 ops/second
 - /unlink 1126 (time 1782264081.20 total 10.00 last 112.57)
 - /unlink 2184 (time 1782264091.21 total 20.01 last 105.74)
 - /unlink 3267 (time 1782264101.21 total 30.01 last 108.29)
total: 4500 /unlink in 38.76 seconds: 116.11 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95029 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg225-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg225-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server
stop mds service on oleg225-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg225-server

unloading modules via unload_modules_local on: 'oleg225-server'
oleg225-server: oleg225-server.virtnet: executing unload_modules_local
oleg225-server: 
modules unloaded.