== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:16:50 (1762492610)
Stopping clients: oleg458-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg458-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg458-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg458-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg458-server
oleg458-server: oleg458-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg458-server'
oleg458-server: oleg458-server.virtnet: executing load_modules_local
oleg458-server: Loading modules from /home/green/git/lustre-release/lustre
oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy
oleg458-server: detected 4 online CPUs by sysfs
oleg458-server: MODOPTS_LIBCFS=
oleg458-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg458-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all
pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg458-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all
pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg458-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all
pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg458-client.virtnet:  -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 777 (time 1762492782.05 total 10.01 last 77.59)
 - open/close 1631 (time 1762492792.05 total 20.02 last 85.35)
 - open/close 2465 (time 1762492802.06 total 30.03 last 83.34)
 - open/close 3253 (time 1762492812.06 total 40.03 last 78.77)
 - open/close 4077 (time 1762492822.07 total 50.04 last 82.35)
total: 4500 open/close in 55.13 seconds: 81.63 ops/second
 - /unlink 1460 (time 1762492840.61 total 10.00 last 145.98)
 - /unlink 2975 (time 1762492850.62 total 20.01 last 151.36)
 - /unlink 4396 (time 1762492860.62 total 30.01 last 142.06)
total: 4500 /unlink in 30.92 seconds: 145.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 843 (time 1762492875.71 total 10.00 last 84.27)
 - open/close 1637 (time 1762492885.71 total 20.00 last 79.40)
 - open/close 2447 (time 1762492895.72 total 30.01 last 80.98)
 - open/close 3349 (time 1762492905.72 total 40.01 last 90.13)
 - open/close 4191 (time 1762492915.73 total 50.02 last 84.13)
total: 4500 open/close in 53.69 seconds: 83.82 ops/second
 - /unlink 1474 (time 1762492932.72 total 10.00 last 147.35)
 - /unlink 3036 (time 1762492942.72 total 20.01 last 156.16)
 - /unlink 4487 (time 1762492952.72 total 30.01 last 145.09)
total: 4500 /unlink in 30.22 seconds: 148.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 728 (time 1762492967.37 total 10.01 last 72.72)
 - open/close 1477 (time 1762492977.38 total 20.01 last 74.89)
 - open/close 2348 (time 1762492987.39 total 30.03 last 86.95)
 - open/close 3222 (time 1762492997.41 total 40.05 last 87.24)
 - open/close 4036 (time 1762493007.41 total 50.05 last 81.39)
total: 4500 open/close in 55.03 seconds: 81.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1516 (time 1762493025.26 total 10.00 last 151.55)
 - /unlink 3004 (time 1762493035.26 total 20.00 last 148.79)
total: 4500 /unlink in 29.51 seconds: 152.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 970 (time 1762493058.69 total 10.00 last 96.97)
 - open/close 1699 (time 1762493068.69 total 20.01 last 72.86)
 - open/close 2343 (time 1762493078.70 total 30.02 last 64.32)
 - open/close 3187 (time 1762493088.71 total 40.02 last 84.39)
 - open/close 3980 (time 1762493098.72 total 50.03 last 79.23)
total: 4500 open/close in 56.15 seconds: 80.14 ops/second
 - /unlink 1602 (time 1762493118.48 total 10.00 last 160.17)
 - /unlink 3189 (time 1762493128.49 total 20.00 last 158.68)
total: 4500 /unlink in 29.19 seconds: 154.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 882 (time 1762493151.78 total 10.01 last 88.08)
 - open/close 1745 (time 1762493161.79 total 20.02 last 86.26)
 - open/close 2594 (time 1762493171.79 total 30.03 last 84.83)
 - open/close 3410 (time 1762493181.80 total 40.03 last 81.57)
 - open/close 4179 (time 1762493191.82 total 50.06 last 76.71)
total: 4500 open/close in 54.47 seconds: 82.61 ops/second
 - /unlink 1509 (time 1762493210.25 total 10.01 last 150.72)
 - /unlink 2905 (time 1762493220.26 total 20.01 last 139.56)
 - /unlink 4455 (time 1762493230.29 total 30.05 last 154.50)
total: 4500 /unlink in 30.52 seconds: 147.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 902 (time 1762493244.71 total 10.01 last 90.14)
 - open/close 1702 (time 1762493254.74 total 20.03 last 79.80)
 - open/close 2481 (time 1762493264.75 total 30.04 last 77.80)
 - open/close 3183 (time 1762493274.75 total 40.05 last 70.17)
 - open/close 3958 (time 1762493284.77 total 50.07 last 77.37)
total: 4500 open/close in 56.12 seconds: 80.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1526 (time 1762493303.42 total 10.01 last 152.50)
 - /unlink 2919 (time 1762493313.43 total 20.01 last 139.24)
 - /unlink 4461 (time 1762493323.44 total 30.02 last 154.04)
total: 4500 /unlink in 30.26 seconds: 148.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 918 (time 1762493336.94 total 10.01 last 91.69)
 - open/close 1754 (time 1762493346.95 total 20.02 last 83.55)
 - open/close 2620 (time 1762493356.95 total 30.02 last 86.59)
 - open/close 3308 (time 1762493366.96 total 40.03 last 68.75)
 - open/close 3958 (time 1762493376.97 total 50.04 last 64.95)
total: 4500 open/close in 57.74 seconds: 77.94 ops/second
 - /unlink 1452 (time 1762493398.69 total 10.00 last 145.15)
 - /unlink 2984 (time 1762493408.70 total 20.00 last 153.18)
total: 4500 /unlink in 29.74 seconds: 151.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 814 (time 1762493432.07 total 10.02 last 81.28)
 - open/close 1503 (time 1762493442.08 total 20.02 last 68.84)
 - open/close 2309 (time 1762493452.08 total 30.03 last 80.56)
 - open/close 3101 (time 1762493462.09 total 40.04 last 79.13)
 - open/close 3973 (time 1762493472.09 total 50.04 last 87.20)
total: 4500 open/close in 56.09 seconds: 80.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1429 (time 1762493492.01 total 10.00 last 142.86)
 - /unlink 2952 (time 1762493502.02 total 20.01 last 152.17)
 - /unlink 4376 (time 1762493512.02 total 30.02 last 142.30)
total: 4500 /unlink in 30.96 seconds: 145.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 755 (time 1762493528.00 total 10.01 last 75.45)
 - open/close 1614 (time 1762493538.02 total 20.02 last 85.76)
 - open/close 2404 (time 1762493548.02 total 30.02 last 78.99)
 - open/close 3209 (time 1762493558.02 total 40.03 last 80.46)
 - open/close 4014 (time 1762493568.03 total 50.04 last 80.42)
total: 4500 open/close in 56.38 seconds: 79.81 ops/second
 - /unlink 1548 (time 1762493589.21 total 10.00 last 154.78)
 - /unlink 2960 (time 1762493599.21 total 20.00 last 141.20)
 - /unlink 4331 (time 1762493609.21 total 30.01 last 137.00)
total: 4500 /unlink in 31.78 seconds: 141.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 786 (time 1762493625.08 total 10.01 last 78.55)
 - open/close 1595 (time 1762493635.09 total 20.01 last 80.85)
 - open/close 2383 (time 1762493645.09 total 30.01 last 78.80)
 - open/close 3257 (time 1762493655.09 total 40.01 last 87.39)
 - open/close 4125 (time 1762493665.10 total 50.03 last 86.70)
total: 4500 open/close in 54.80 seconds: 82.12 ops/second
 - /unlink 1427 (time 1762493683.91 total 10.01 last 142.62)
 - /unlink 3067 (time 1762493693.92 total 20.01 last 163.88)
total: 4500 /unlink in 27.47 seconds: 163.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 947 (time 1762493714.11 total 10.00 last 94.67)
 - open/close 1846 (time 1762493724.12 total 20.01 last 89.81)
 - open/close 2755 (time 1762493734.13 total 30.02 last 90.82)
 - open/close 3562 (time 1762493744.17 total 40.06 last 80.42)
 - open/close 4249 (time 1762493754.17 total 50.06 last 68.66)
total: 4500 open/close in 53.00 seconds: 84.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1459 (time 1762493770.27 total 10.01 last 145.81)
 - /unlink 3129 (time 1762493780.27 total 20.01 last 166.99)
 - /unlink 4291 (time 1762493790.27 total 30.01 last 116.17)
total: 4500 /unlink in 31.48 seconds: 142.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 907 (time 1762493805.61 total 10.00 last 90.70)
 - open/close 1687 (time 1762493815.62 total 20.01 last 77.95)
 - open/close 2552 (time 1762493825.63 total 30.01 last 86.44)
 - open/close 3403 (time 1762493835.66 total 40.05 last 84.79)
 - open/close 4219 (time 1762493845.67 total 50.06 last 81.51)
total: 4500 open/close in 53.07 seconds: 84.79 ops/second
 - /unlink 1424 (time 1762493862.57 total 10.00 last 142.40)
 - /unlink 2858 (time 1762493872.58 total 20.01 last 143.28)
 - /unlink 4279 (time 1762493882.58 total 30.01 last 142.07)
total: 4500 /unlink in 31.68 seconds: 142.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 743 (time 1762493898.12 total 10.01 last 74.26)
 - open/close 1426 (time 1762493908.12 total 20.01 last 68.29)
 - open/close 2239 (time 1762493918.14 total 30.03 last 81.13)
 - open/close 3024 (time 1762493928.15 total 40.04 last 78.43)
 - open/close 3895 (time 1762493938.16 total 50.04 last 87.05)
total: 4500 open/close in 57.99 seconds: 77.60 ops/second
 - /unlink 1332 (time 1762493959.66 total 10.00 last 133.17)
 - /unlink 2816 (time 1762493969.66 total 20.00 last 148.39)
 - /unlink 4127 (time 1762493979.66 total 30.01 last 131.04)
total: 4500 /unlink in 33.13 seconds: 135.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 781 (time 1762493997.53 total 10.01 last 78.04)
 - open/close 1651 (time 1762494007.53 total 20.01 last 86.95)
 - open/close 2496 (time 1762494017.54 total 30.02 last 84.45)
 - open/close 3364 (time 1762494027.55 total 40.03 last 86.70)
 - open/close 4300 (time 1762494037.56 total 50.04 last 93.50)
total: 4500 open/close in 52.40 seconds: 85.87 ops/second
 - /unlink 1559 (time 1762494052.99 total 10.00 last 155.86)
 - /unlink 2893 (time 1762494062.99 total 20.01 last 133.35)
 - /unlink 4387 (time 1762494072.99 total 30.01 last 149.37)
total: 4500 /unlink in 30.92 seconds: 145.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 899 (time 1762494087.44 total 10.00 last 89.87)
 - open/close 1755 (time 1762494097.44 total 20.00 last 85.59)
 - open/close 2536 (time 1762494107.44 total 30.01 last 78.08)
 - open/close 3423 (time 1762494117.45 total 40.01 last 88.65)
 - open/close 4346 (time 1762494127.45 total 50.02 last 92.25)
total: 4500 open/close in 51.76 seconds: 86.94 ops/second
 - /unlink 1445 (time 1762494142.14 total 10.00 last 144.48)
 - /unlink 2981 (time 1762494152.15 total 20.01 last 153.50)
 - /unlink 4313 (time 1762494162.15 total 30.01 last 133.18)
total: 4500 /unlink in 31.38 seconds: 143.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 844 (time 1762494177.32 total 10.01 last 84.35)
 - open/close 1664 (time 1762494187.34 total 20.02 last 81.84)
 - open/close 2448 (time 1762494197.34 total 30.03 last 78.39)
 - open/close 3235 (time 1762494207.35 total 40.04 last 78.62)
 - open/close 4001 (time 1762494217.35 total 50.04 last 76.58)
total: 4500 open/close in 56.31 seconds: 79.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1077 (time 1762494237.81 total 10.01 last 107.65)
 - /unlink 2086 (time 1762494247.83 total 20.03 last 100.64)
 - /unlink 3556 (time 1762494257.84 total 30.03 last 146.96)
total: 4500 /unlink in 36.84 seconds: 122.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 765 (time 1762494278.39 total 10.01 last 76.41)
 - open/close 1529 (time 1762494288.39 total 20.01 last 76.39)
 - open/close 2385 (time 1762494298.40 total 30.03 last 85.49)
 - open/close 3194 (time 1762494308.41 total 40.03 last 80.88)
 - open/close 3989 (time 1762494318.41 total 50.03 last 79.45)
total: 4500 open/close in 56.67 seconds: 79.41 ops/second
 - /unlink 1458 (time 1762494338.66 total 10.01 last 145.72)
 - /unlink 2858 (time 1762494348.67 total 20.01 last 139.91)
 - /unlink 4301 (time 1762494358.67 total 30.02 last 144.23)
total: 4500 /unlink in 31.55 seconds: 142.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 753 (time 1762494374.23 total 10.00 last 75.27)
 - open/close 1575 (time 1762494384.24 total 20.01 last 82.12)
 - open/close 2361 (time 1762494394.25 total 30.03 last 78.50)
 - open/close 3256 (time 1762494404.26 total 40.03 last 89.46)
 - open/close 3954 (time 1762494414.26 total 50.03 last 69.77)
total: 4500 open/close in 58.20 seconds: 77.32 ops/second
 - /unlink 1476 (time 1762494436.85 total 10.00 last 147.57)
 - /unlink 2872 (time 1762494446.85 total 20.00 last 139.58)
 - /unlink 4356 (time 1762494456.85 total 30.01 last 148.37)
total: 4500 /unlink in 31.07 seconds: 144.85 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11154: 93962 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg458-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg458-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server
stop mds service on oleg458-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server
stop mds service on oleg458-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg458-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg458-server'
oleg458-server: oleg458-server.virtnet: executing unload_modules_local
oleg458-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg458-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.