== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:16:07 (1770448567)
Stopping clients: oleg146-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg146-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg146-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg146-server
oleg146-server: oleg146-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg146-server'
oleg146-server: oleg146-server.virtnet: executing load_modules_local
oleg146-server: Loading modules from /home/green/git/lustre-release/lustre
oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg146-server: detected 4 online CPUs by sysfs
oleg146-server: MODOPTS_LIBCFS=
oleg146-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg146-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all
pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg146-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all
pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg146-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all
pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg146-client.virtnet:  -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 710 (time 1770448792.24 total 10.01 last 70.94)
 - open/close 1347 (time 1770448802.24 total 20.01 last 63.66)
 - open/close 2047 (time 1770448812.28 total 30.05 last 69.74)
 - open/close 2771 (time 1770448822.29 total 40.06 last 72.34)
 - open/close 3409 (time 1770448832.30 total 50.07 last 63.73)
 - open/close 4048 (time 1770448842.32 total 60.09 last 63.79)
total: 4500 open/close in 66.98 seconds: 67.18 ops/second
 - /unlink 1221 (time 1770448864.39 total 10.00 last 122.07)
 - /unlink 2283 (time 1770448874.40 total 20.01 last 106.07)
 - /unlink 3504 (time 1770448884.41 total 30.02 last 122.02)
total: 4500 /unlink in 39.29 seconds: 114.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 701 (time 1770448910.28 total 10.00 last 70.08)
 - open/close 1380 (time 1770448920.29 total 20.02 last 67.81)
 - open/close 2046 (time 1770448930.29 total 30.02 last 66.59)
 - open/close 2635 (time 1770448940.30 total 40.02 last 58.86)
 - open/close 3162 (time 1770448950.31 total 50.03 last 52.65)
 - open/close 3771 (time 1770448960.31 total 60.04 last 60.86)
 - open/close 4386 (time 1770448970.33 total 70.06 last 61.39)
total: 4500 open/close in 71.85 seconds: 62.63 ops/second
 - /unlink 1127 (time 1770448987.71 total 10.00 last 112.70)
 - /unlink 2233 (time 1770448997.71 total 20.00 last 110.56)
 - /unlink 3324 (time 1770449007.72 total 30.01 last 109.04)
total: 4500 /unlink in 40.01 seconds: 112.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 580 (time 1770449033.94 total 10.01 last 57.96)
 - open/close 1194 (time 1770449043.94 total 20.01 last 61.39)
 - open/close 1809 (time 1770449053.95 total 30.02 last 61.43)
 - open/close 2371 (time 1770449063.95 total 40.03 last 56.16)
 - open/close 3011 (time 1770449073.96 total 50.03 last 63.98)
 - open/close 3604 (time 1770449083.96 total 60.04 last 59.27)
 - open/close 4201 (time 1770449093.98 total 70.05 last 59.62)
total: 4500 open/close in 75.45 seconds: 59.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1003 (time 1770449115.51 total 10.00 last 100.27)
 - /unlink 1846 (time 1770449125.53 total 20.03 last 84.10)
 - /unlink 2937 (time 1770449135.54 total 30.03 last 109.09)
 - /unlink 4143 (time 1770449145.55 total 40.04 last 120.48)
total: 4500 /unlink in 43.66 seconds: 103.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 603 (time 1770449165.26 total 10.02 last 60.20)
 - open/close 1197 (time 1770449175.26 total 20.02 last 59.38)
 - open/close 1849 (time 1770449185.26 total 30.02 last 65.20)
 - open/close 2535 (time 1770449195.27 total 40.03 last 68.54)
 - open/close 3142 (time 1770449205.28 total 50.04 last 60.62)
 - open/close 3651 (time 1770449215.29 total 60.05 last 50.86)
 - open/close 4249 (time 1770449225.29 total 70.06 last 59.77)
total: 4500 open/close in 73.74 seconds: 61.03 ops/second
 - /unlink 1240 (time 1770449243.53 total 10.01 last 123.93)
 - /unlink 2419 (time 1770449253.54 total 20.01 last 117.84)
 - /unlink 3599 (time 1770449263.55 total 30.02 last 117.86)
total: 4500 /unlink in 38.25 seconds: 117.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 560 (time 1770449287.36 total 10.01 last 55.94)
 - open/close 1168 (time 1770449297.38 total 20.03 last 60.70)
 - open/close 1847 (time 1770449307.39 total 30.04 last 67.83)
 - open/close 2551 (time 1770449317.40 total 40.05 last 70.33)
 - open/close 3241 (time 1770449327.40 total 50.05 last 68.99)
 - open/close 3920 (time 1770449337.40 total 60.05 last 67.89)
total: 4500 open/close in 69.33 seconds: 64.90 ops/second
 - /unlink 1304 (time 1770449361.22 total 10.00 last 130.38)
 - /unlink 2639 (time 1770449371.23 total 20.01 last 133.43)
 - /unlink 3779 (time 1770449381.23 total 30.01 last 113.92)
total: 4500 /unlink in 37.71 seconds: 119.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 647 (time 1770449403.85 total 10.02 last 64.59)
 - open/close 1327 (time 1770449413.88 total 20.05 last 67.79)
 - open/close 1997 (time 1770449423.90 total 30.07 last 66.87)
 - open/close 2611 (time 1770449433.91 total 40.08 last 61.34)
 - open/close 3283 (time 1770449443.94 total 50.11 last 66.98)
 - open/close 3900 (time 1770449453.95 total 60.12 last 61.63)
total: 4500 open/close in 69.73 seconds: 64.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1266 (time 1770449479.46 total 10.00 last 126.58)
 - /unlink 2592 (time 1770449489.47 total 20.01 last 132.50)
 - /unlink 3741 (time 1770449499.48 total 30.01 last 114.84)
total: 4500 /unlink in 35.85 seconds: 125.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 589 (time 1770449520.09 total 10.01 last 58.84)
 - open/close 1189 (time 1770449530.10 total 20.02 last 59.91)
 - open/close 1906 (time 1770449540.11 total 30.03 last 71.63)
 - open/close 2588 (time 1770449550.11 total 40.04 last 68.18)
 - open/close 3322 (time 1770449560.12 total 50.05 last 73.34)
 - open/close 3876 (time 1770449570.13 total 60.06 last 55.33)
total: 4500 open/close in 69.77 seconds: 64.50 ops/second
 - /unlink 1230 (time 1770449595.03 total 10.00 last 122.99)
 - /unlink 2430 (time 1770449605.04 total 20.00 last 119.98)
 - /unlink 3719 (time 1770449615.04 total 30.00 last 128.89)
total: 4500 /unlink in 36.07 seconds: 124.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 630 (time 1770449635.89 total 10.02 last 62.88)
 - open/close 1298 (time 1770449645.90 total 20.03 last 66.70)
 - open/close 2008 (time 1770449655.91 total 30.04 last 70.94)
 - open/close 2704 (time 1770449665.92 total 40.05 last 69.57)
 - open/close 3231 (time 1770449675.93 total 50.06 last 52.64)
 - open/close 3830 (time 1770449685.95 total 60.08 last 59.78)
 - open/close 4484 (time 1770449695.95 total 70.08 last 65.36)
total: 4500 open/close in 70.33 seconds: 63.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1448 (time 1770449710.33 total 10.00 last 144.79)
 - /unlink 2737 (time 1770449720.33 total 20.01 last 128.82)
 - /unlink 4022 (time 1770449730.34 total 30.01 last 128.44)
total: 4500 /unlink in 33.79 seconds: 133.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 700 (time 1770449748.75 total 10.01 last 69.95)
 - open/close 1404 (time 1770449758.76 total 20.01 last 70.35)
 - open/close 2204 (time 1770449768.76 total 30.02 last 79.98)
 - open/close 3008 (time 1770449778.77 total 40.02 last 80.34)
 - open/close 3791 (time 1770449788.77 total 50.02 last 78.30)
total: 4500 open/close in 59.26 seconds: 75.94 ops/second
 - /unlink 1391 (time 1770449811.76 total 10.00 last 139.04)
 - /unlink 2634 (time 1770449821.76 total 20.01 last 124.29)
 - /unlink 4275 (time 1770449831.76 total 30.01 last 164.02)
total: 4500 /unlink in 31.54 seconds: 142.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 670 (time 1770449847.32 total 10.01 last 66.96)
 - open/close 1450 (time 1770449857.32 total 20.01 last 77.99)
 - open/close 2257 (time 1770449867.33 total 30.01 last 80.69)
 - open/close 3020 (time 1770449877.33 total 40.01 last 76.28)
 - open/close 3771 (time 1770449887.34 total 50.02 last 75.00)
 - open/close 4487 (time 1770449897.34 total 60.03 last 71.58)
total: 4500 open/close in 60.22 seconds: 74.73 ops/second
 - /unlink 1434 (time 1770449911.77 total 10.01 last 143.26)
 - /unlink 2795 (time 1770449921.78 total 20.02 last 135.99)
 - /unlink 4163 (time 1770449931.79 total 30.03 last 136.68)
total: 4500 /unlink in 32.61 seconds: 137.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 777 (time 1770449947.77 total 10.01 last 77.65)
 - open/close 1498 (time 1770449957.77 total 20.01 last 72.08)
 - open/close 2241 (time 1770449967.78 total 30.02 last 74.25)
 - open/close 2909 (time 1770449977.78 total 40.02 last 66.78)
 - open/close 3721 (time 1770449987.80 total 50.03 last 81.09)
 - open/close 4434 (time 1770449997.80 total 60.04 last 71.27)
total: 4500 open/close in 60.77 seconds: 74.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1478 (time 1770450012.13 total 10.01 last 147.70)
 - /unlink 2896 (time 1770450022.13 total 20.01 last 141.72)
 - /unlink 4142 (time 1770450032.14 total 30.02 last 124.52)
total: 4500 /unlink in 32.67 seconds: 137.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 703 (time 1770450048.76 total 10.01 last 70.21)
 - open/close 1474 (time 1770450058.77 total 20.02 last 77.07)
 - open/close 2274 (time 1770450068.77 total 30.02 last 79.96)
 - open/close 3031 (time 1770450078.77 total 40.03 last 75.68)
 - open/close 3709 (time 1770450088.78 total 50.03 last 67.78)
 - open/close 4394 (time 1770450098.78 total 60.03 last 68.48)
total: 4500 open/close in 61.51 seconds: 73.16 ops/second
 - /unlink 1283 (time 1770450115.00 total 10.01 last 128.23)
 - /unlink 2686 (time 1770450125.00 total 20.01 last 140.23)
 - /unlink 4113 (time 1770450135.00 total 30.01 last 142.68)
total: 4500 /unlink in 33.18 seconds: 135.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 794 (time 1770450152.17 total 10.02 last 79.26)
 - open/close 1473 (time 1770450162.17 total 20.02 last 67.87)
 - open/close 2238 (time 1770450172.18 total 30.03 last 76.47)
 - open/close 3111 (time 1770450182.18 total 40.03 last 87.27)
 - open/close 3877 (time 1770450192.18 total 50.03 last 76.59)
total: 4500 open/close in 59.21 seconds: 76.00 ops/second
 - /unlink 1353 (time 1770450215.10 total 10.00 last 135.27)
 - /unlink 2800 (time 1770450225.11 total 20.01 last 144.59)
 - /unlink 4201 (time 1770450235.11 total 30.01 last 140.06)
total: 4500 /unlink in 31.71 seconds: 141.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 678 (time 1770450251.37 total 10.01 last 67.72)
 - open/close 1407 (time 1770450261.37 total 20.02 last 72.88)
 - open/close 2078 (time 1770450271.38 total 30.03 last 67.00)
 - open/close 2784 (time 1770450281.39 total 40.04 last 70.53)
 - open/close 3559 (time 1770450291.39 total 50.04 last 77.50)
 - open/close 4352 (time 1770450301.40 total 60.04 last 79.27)
total: 4500 open/close in 62.02 seconds: 72.56 ops/second
 - /unlink 1348 (time 1770450317.85 total 10.00 last 134.77)
 - /unlink 2641 (time 1770450327.86 total 20.01 last 129.17)
 - /unlink 4089 (time 1770450337.86 total 30.01 last 144.76)
total: 4500 /unlink in 32.98 seconds: 136.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 712 (time 1770450354.91 total 10.00 last 71.19)
 - open/close 1464 (time 1770450364.93 total 20.01 last 75.10)
 - open/close 2139 (time 1770450374.95 total 30.03 last 67.38)
 - open/close 2876 (time 1770450384.95 total 40.04 last 73.64)
 - open/close 3592 (time 1770450394.96 total 50.05 last 71.52)
 - open/close 4345 (time 1770450404.98 total 60.07 last 75.17)
total: 4500 open/close in 62.18 seconds: 72.37 ops/second
 - /unlink 1522 (time 1770450421.09 total 10.00 last 152.15)
 - /unlink 2957 (time 1770450431.09 total 20.00 last 143.49)
 - /unlink 4284 (time 1770450441.10 total 30.01 last 132.57)
total: 4500 /unlink in 31.73 seconds: 141.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 714 (time 1770450456.65 total 10.01 last 71.32)
 - open/close 1418 (time 1770450466.65 total 20.01 last 70.39)
 - open/close 2197 (time 1770450476.66 total 30.02 last 77.83)
 - open/close 2868 (time 1770450486.66 total 40.02 last 67.09)
 - open/close 3635 (time 1770450496.66 total 50.03 last 76.67)
 - open/close 4379 (time 1770450506.66 total 60.03 last 74.38)
total: 4500 open/close in 61.79 seconds: 72.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1481 (time 1770450522.06 total 10.00 last 148.08)
 - /unlink 2885 (time 1770450532.06 total 20.01 last 140.30)
 - /unlink 4194 (time 1770450542.07 total 30.01 last 130.87)
total: 4500 /unlink in 32.47 seconds: 138.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 744 (time 1770450558.97 total 10.00 last 74.37)
 - open/close 1474 (time 1770450569.00 total 20.04 last 72.77)
 - open/close 2254 (time 1770450579.01 total 30.04 last 77.93)
 - open/close 2983 (time 1770450589.01 total 40.05 last 72.87)
 - open/close 3677 (time 1770450599.02 total 50.05 last 69.36)
 - open/close 4369 (time 1770450609.02 total 60.06 last 69.18)
total: 4500 open/close in 61.88 seconds: 72.72 ops/second
 - /unlink 1338 (time 1770450625.25 total 10.01 last 133.73)
 - /unlink 2705 (time 1770450635.26 total 20.01 last 136.64)
 - /unlink 4029 (time 1770450645.26 total 30.01 last 132.37)
total: 4500 /unlink in 33.72 seconds: 133.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 695 (time 1770450663.46 total 10.00 last 69.47)
 - open/close 1352 (time 1770450673.47 total 20.02 last 65.61)
 - open/close 2033 (time 1770450683.49 total 30.03 last 68.00)
 - open/close 2780 (time 1770450693.51 total 40.06 last 74.52)
 - open/close 3502 (time 1770450703.51 total 50.06 last 72.16)
 - open/close 4198 (time 1770450713.53 total 60.08 last 69.50)
total: 4500 open/close in 64.20 seconds: 70.10 ops/second
 - /unlink 1312 (time 1770450731.89 total 10.01 last 131.11)
 - /unlink 2650 (time 1770450741.89 total 20.01 last 133.77)
 - /unlink 4252 (time 1770450751.90 total 30.02 last 160.02)
total: 4500 /unlink in 31.50 seconds: 142.85 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11101: 93259 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg146-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg146-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server
stop mds service on oleg146-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server
stop mds service on oleg146-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg146-server'
oleg146-server: oleg146-server.virtnet: executing unload_modules_local
oleg146-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg146-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.