== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:31:21 (1763346681)
Stopping clients: oleg234-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg234-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg234-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg234-server
oleg234-server: oleg234-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg234-server'
oleg234-server: oleg234-server.virtnet: executing load_modules_local
oleg234-server: Loading modules from /home/green/git/lustre-release/lustre
oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy
oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy
oleg234-server: detected 4 online CPUs by sysfs
oleg234-server: MODOPTS_LIBCFS=
oleg234-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg234-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all
pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg234-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all
pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg234-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all
pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg234-client.virtnet:  -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 602 (time 1763346858.51 total 10.01 last 60.14)
 - open/close 1279 (time 1763346868.52 total 20.02 last 67.63)
 - open/close 1864 (time 1763346878.53 total 30.02 last 58.49)
 - open/close 2462 (time 1763346888.53 total 40.03 last 59.76)
 - open/close 3122 (time 1763346898.54 total 50.04 last 65.93)
 - open/close 3821 (time 1763346908.55 total 60.05 last 69.85)
 - open/close 4444 (time 1763346918.56 total 70.05 last 62.27)
total: 4500 open/close in 71.13 seconds: 63.26 ops/second
 - /unlink 1213 (time 1763346934.50 total 10.00 last 121.28)
 - /unlink 2413 (time 1763346944.50 total 20.00 last 119.98)
 - /unlink 3705 (time 1763346954.51 total 30.01 last 129.11)
total: 4500 /unlink in 36.05 seconds: 124.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 703 (time 1763346974.92 total 10.00 last 70.27)
 - open/close 1437 (time 1763346984.93 total 20.01 last 73.33)
 - open/close 2135 (time 1763346994.94 total 30.02 last 69.76)
 - open/close 2762 (time 1763347004.94 total 40.02 last 62.67)
 - open/close 3379 (time 1763347014.96 total 50.04 last 61.59)
 - open/close 3950 (time 1763347024.96 total 60.05 last 57.07)
total: 4500 open/close in 69.45 seconds: 64.80 ops/second
 - /unlink 1251 (time 1763347049.08 total 10.01 last 124.95)
 - /unlink 2502 (time 1763347059.09 total 20.02 last 124.96)
 - /unlink 3747 (time 1763347069.09 total 30.03 last 124.41)
total: 4500 /unlink in 36.54 seconds: 123.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 621 (time 1763347089.79 total 10.03 last 61.93)
 - open/close 1252 (time 1763347099.80 total 20.04 last 63.00)
 - open/close 1905 (time 1763347109.83 total 30.07 last 65.11)
 - open/close 2474 (time 1763347119.85 total 40.09 last 56.79)
 - open/close 3025 (time 1763347129.87 total 50.11 last 54.99)
 - open/close 3678 (time 1763347139.88 total 60.12 last 65.22)
 - open/close 4245 (time 1763347149.88 total 70.12 last 56.69)
total: 4500 open/close in 74.19 seconds: 60.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1221 (time 1763347168.70 total 10.01 last 121.99)
 - /unlink 2425 (time 1763347178.71 total 20.02 last 120.29)
 - /unlink 3676 (time 1763347188.71 total 30.02 last 125.03)
total: 4500 /unlink in 36.68 seconds: 122.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 704 (time 1763347209.48 total 10.03 last 70.22)
 - open/close 1387 (time 1763347219.48 total 20.03 last 68.27)
 - open/close 2014 (time 1763347229.49 total 30.03 last 62.69)
 - open/close 2677 (time 1763347239.49 total 40.03 last 66.29)
 - open/close 3341 (time 1763347249.49 total 50.04 last 66.38)
 - open/close 4031 (time 1763347259.50 total 60.05 last 68.91)
total: 4500 open/close in 66.98 seconds: 67.19 ops/second
 - /unlink 1255 (time 1763347281.43 total 10.00 last 125.50)
 - /unlink 2274 (time 1763347291.43 total 20.01 last 101.83)
 - /unlink 3373 (time 1763347301.44 total 30.01 last 109.84)
 - /unlink 4499 (time 1763347311.44 total 40.01 last 112.59)
total: 4500 /unlink in 40.01 seconds: 112.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 552 (time 1763347326.91 total 10.01 last 55.12)
 - open/close 1159 (time 1763347336.91 total 20.02 last 60.69)
 - open/close 1748 (time 1763347346.91 total 30.02 last 58.88)
 - open/close 2309 (time 1763347356.92 total 40.03 last 56.05)
 - open/close 2977 (time 1763347366.93 total 50.04 last 66.74)
 - open/close 3660 (time 1763347376.94 total 60.05 last 68.24)
 - open/close 4315 (time 1763347386.96 total 70.07 last 65.37)
total: 4500 open/close in 72.94 seconds: 61.70 ops/second
 - /unlink 1254 (time 1763347404.10 total 10.00 last 125.36)
 - /unlink 2590 (time 1763347414.10 total 20.01 last 133.50)
 - /unlink 3708 (time 1763347424.11 total 30.02 last 111.72)
total: 4500 /unlink in 36.74 seconds: 122.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 651 (time 1763347445.41 total 10.01 last 65.04)
 - open/close 1317 (time 1763347455.43 total 20.03 last 66.47)
 - open/close 2002 (time 1763347465.44 total 30.04 last 68.45)
 - open/close 2627 (time 1763347475.44 total 40.04 last 62.49)
 - open/close 3300 (time 1763347485.46 total 50.05 last 67.20)
 - open/close 4013 (time 1763347495.48 total 60.08 last 71.13)
total: 4500 open/close in 67.55 seconds: 66.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1235 (time 1763347517.75 total 10.01 last 123.40)
 - /unlink 2490 (time 1763347527.75 total 20.01 last 125.48)
 - /unlink 3769 (time 1763347537.76 total 30.02 last 127.83)
total: 4500 /unlink in 35.81 seconds: 125.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 726 (time 1763347557.90 total 10.00 last 72.59)
 - open/close 1420 (time 1763347567.92 total 20.02 last 69.28)
 - open/close 2100 (time 1763347577.93 total 30.03 last 67.94)
 - open/close 2815 (time 1763347587.93 total 40.03 last 71.50)
 - open/close 3509 (time 1763347597.94 total 50.04 last 69.32)
 - open/close 4170 (time 1763347607.94 total 60.04 last 66.10)
total: 4500 open/close in 64.66 seconds: 69.59 ops/second
 - /unlink 1351 (time 1763347627.14 total 10.01 last 135.02)
 - /unlink 2498 (time 1763347637.14 total 20.01 last 114.68)
 - /unlink 3598 (time 1763347647.14 total 30.01 last 109.93)
total: 4500 /unlink in 37.39 seconds: 120.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 699 (time 1763347668.86 total 10.01 last 69.83)
 - open/close 1369 (time 1763347678.87 total 20.02 last 66.96)
 - open/close 2143 (time 1763347688.87 total 30.02 last 77.36)
 - open/close 2861 (time 1763347698.88 total 40.03 last 71.72)
 - open/close 3531 (time 1763347708.89 total 50.03 last 66.99)
 - open/close 4132 (time 1763347718.89 total 60.04 last 60.06)
total: 4500 open/close in 65.47 seconds: 68.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1291 (time 1763347738.54 total 10.00 last 129.07)
 - /unlink 2525 (time 1763347748.55 total 20.01 last 123.33)
 - /unlink 3758 (time 1763347758.55 total 30.01 last 123.29)
total: 4500 /unlink in 36.04 seconds: 124.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 721 (time 1763347777.91 total 10.01 last 72.06)
 - open/close 1446 (time 1763347787.91 total 20.01 last 72.47)
 - open/close 2127 (time 1763347797.91 total 30.01 last 68.10)
 - open/close 2818 (time 1763347807.93 total 40.03 last 68.97)
 - open/close 3542 (time 1763347817.94 total 50.04 last 72.31)
 - open/close 4117 (time 1763347827.97 total 60.06 last 57.37)
total: 4500 open/close in 66.70 seconds: 67.46 ops/second
 - /unlink 1126 (time 1763347849.62 total 10.00 last 112.58)
 - /unlink 2378 (time 1763347859.63 total 20.00 last 125.16)
 - /unlink 3704 (time 1763347869.63 total 30.01 last 132.56)
total: 4500 /unlink in 36.56 seconds: 123.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 671 (time 1763347890.50 total 10.01 last 67.01)
 - open/close 1352 (time 1763347900.51 total 20.02 last 68.02)
 - open/close 2059 (time 1763347910.52 total 30.04 last 70.61)
 - open/close 2748 (time 1763347920.55 total 40.07 last 68.68)
 - open/close 3366 (time 1763347930.56 total 50.08 last 61.77)
 - open/close 4017 (time 1763347940.57 total 60.08 last 65.05)
total: 4500 open/close in 67.48 seconds: 66.69 ops/second
 - /unlink 1418 (time 1763347962.33 total 10.00 last 141.77)
 - /unlink 2762 (time 1763347972.34 total 20.01 last 134.24)
 - /unlink 4132 (time 1763347982.34 total 30.01 last 136.99)
total: 4500 /unlink in 32.28 seconds: 139.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 788 (time 1763347997.94 total 10.00 last 78.79)
 - open/close 1492 (time 1763348007.95 total 20.01 last 70.34)
 - open/close 2199 (time 1763348017.97 total 30.03 last 70.54)
 - open/close 2871 (time 1763348027.97 total 40.03 last 67.19)
 - open/close 3605 (time 1763348037.98 total 50.04 last 73.35)
 - open/close 4342 (time 1763348047.99 total 60.05 last 73.63)
total: 4500 open/close in 62.18 seconds: 72.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1357 (time 1763348063.81 total 10.01 last 135.61)
 - /unlink 2558 (time 1763348073.81 total 20.01 last 120.10)
 - /unlink 3889 (time 1763348083.82 total 30.01 last 133.08)
total: 4500 /unlink in 34.70 seconds: 129.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 757 (time 1763348102.11 total 10.01 last 75.65)
 - open/close 1473 (time 1763348112.12 total 20.02 last 71.54)
 - open/close 2250 (time 1763348122.13 total 30.03 last 77.60)
 - open/close 3010 (time 1763348132.15 total 40.05 last 75.85)
 - open/close 3694 (time 1763348142.17 total 50.06 last 68.30)
 - open/close 4383 (time 1763348152.17 total 60.06 last 68.89)
total: 4500 open/close in 61.85 seconds: 72.75 ops/second
 - /unlink 1367 (time 1763348167.98 total 10.00 last 136.66)
 - /unlink 2678 (time 1763348177.99 total 20.00 last 131.08)
 - /unlink 4031 (time 1763348187.99 total 30.01 last 135.24)
total: 4500 /unlink in 33.71 seconds: 133.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 677 (time 1763348205.71 total 10.01 last 67.65)
 - open/close 1360 (time 1763348215.72 total 20.02 last 68.24)
 - open/close 2007 (time 1763348225.72 total 30.02 last 64.69)
 - open/close 2646 (time 1763348235.72 total 40.02 last 63.88)
 - open/close 3386 (time 1763348245.74 total 50.03 last 73.90)
 - open/close 4083 (time 1763348255.74 total 60.04 last 69.67)
total: 4500 open/close in 66.29 seconds: 67.89 ops/second
 - /unlink 1317 (time 1763348276.16 total 10.00 last 131.68)
 - /unlink 2638 (time 1763348286.18 total 20.02 last 131.90)
 - /unlink 3928 (time 1763348296.18 total 30.02 last 128.94)
total: 4500 /unlink in 34.34 seconds: 131.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 692 (time 1763348315.04 total 10.01 last 69.14)
 - open/close 1369 (time 1763348325.05 total 20.02 last 67.64)
 - open/close 2125 (time 1763348335.05 total 30.02 last 75.60)
 - open/close 2892 (time 1763348345.07 total 40.03 last 76.60)
 - open/close 3583 (time 1763348355.07 total 50.04 last 69.07)
 - open/close 4285 (time 1763348365.09 total 60.05 last 70.10)
total: 4500 open/close in 63.20 seconds: 71.20 ops/second
 - /unlink 1368 (time 1763348382.52 total 10.00 last 136.79)
 - /unlink 2636 (time 1763348392.52 total 20.01 last 126.73)
 - /unlink 3893 (time 1763348402.53 total 30.01 last 125.66)
total: 4500 /unlink in 34.65 seconds: 129.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 769 (time 1763348421.41 total 10.01 last 76.80)
 - open/close 1507 (time 1763348431.42 total 20.03 last 73.68)
 - open/close 2194 (time 1763348441.43 total 30.03 last 68.68)
 - open/close 2864 (time 1763348451.43 total 40.04 last 66.97)
 - open/close 3563 (time 1763348461.44 total 50.04 last 69.84)
 - open/close 4277 (time 1763348471.45 total 60.06 last 71.31)
total: 4500 open/close in 63.17 seconds: 71.24 ops/second
 - /unlink 1227 (time 1763348488.16 total 10.00 last 122.66)
 - /unlink 2556 (time 1763348498.17 total 20.01 last 132.86)
 - /unlink 3851 (time 1763348508.18 total 30.02 last 129.35)
total: 4500 /unlink in 34.63 seconds: 129.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 819 (time 1763348526.99 total 10.00 last 81.88)
 - open/close 1650 (time 1763348537.01 total 20.01 last 83.00)
 - open/close 2415 (time 1763348547.01 total 30.02 last 76.46)
 - open/close 3154 (time 1763348557.02 total 40.02 last 73.87)
 - open/close 3851 (time 1763348567.02 total 50.03 last 69.67)
total: 4500 open/close in 59.36 seconds: 75.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1334 (time 1763348590.54 total 10.00 last 133.40)
 - /unlink 2518 (time 1763348600.54 total 20.00 last 118.36)
 - /unlink 3841 (time 1763348610.55 total 30.01 last 132.23)
total: 4500 /unlink in 34.97 seconds: 128.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 712 (time 1763348634.21 total 10.01 last 71.16)
 - open/close 1413 (time 1763348644.23 total 20.02 last 70.03)
 - open/close 2136 (time 1763348654.23 total 30.02 last 72.26)
 - open/close 2793 (time 1763348664.24 total 40.04 last 65.60)
 - open/close 3497 (time 1763348674.25 total 50.05 last 70.33)
 - open/close 4219 (time 1763348684.26 total 60.05 last 72.20)
total: 4500 open/close in 63.69 seconds: 70.66 ops/second
 - /unlink 1482 (time 1763348701.32 total 10.01 last 148.08)
 - /unlink 3016 (time 1763348711.32 total 20.01 last 153.36)
 - /unlink 4294 (time 1763348721.33 total 30.01 last 127.78)
total: 4500 /unlink in 31.74 seconds: 141.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 725 (time 1763348737.31 total 10.01 last 72.44)
 - open/close 1492 (time 1763348747.31 total 20.01 last 76.66)
 - open/close 2253 (time 1763348757.33 total 30.03 last 75.98)
 - open/close 3026 (time 1763348767.34 total 40.04 last 77.25)
 - open/close 3790 (time 1763348777.35 total 50.05 last 76.31)
total: 4500 open/close in 59.10 seconds: 76.15 ops/second
 - /unlink 1390 (time 1763348799.37 total 10.00 last 139.00)
 - /unlink 2550 (time 1763348809.38 total 20.01 last 115.90)
 - /unlink 3861 (time 1763348819.39 total 30.02 last 131.00)
total: 4500 /unlink in 34.53 seconds: 130.30 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 94019 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg234-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg234-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server
stop mds service on oleg234-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server
stop mds service on oleg234-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg234-server'
oleg234-server: oleg234-server.virtnet: executing unload_modules_local
oleg234-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg234-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.