== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:47:43 (1763405263)
Stopping clients: oleg135-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg135-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg135-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg135-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg135-server
oleg135-server: oleg135-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg135-server'
oleg135-server: oleg135-server.virtnet: executing load_modules_local
oleg135-server: Loading modules from /home/green/git/lustre-release/lustre
oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy
oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy
oleg135-server: detected 4 online CPUs by sysfs
oleg135-server: MODOPTS_LIBCFS=
oleg135-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg135-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg135-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg135-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg135-client.virtnet:  -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 633 (time 1763405430.01 total 10.00 last 63.28)
 - open/close 1326 (time 1763405440.02 total 20.02 last 69.21)
 - open/close 1977 (time 1763405450.04 total 30.03 last 65.02)
 - open/close 2589 (time 1763405460.04 total 40.03 last 61.16)
 - open/close 3239 (time 1763405470.05 total 50.04 last 64.97)
 - open/close 3898 (time 1763405480.05 total 60.04 last 65.86)
total: 4500 open/close in 68.88 seconds: 65.33 ops/second
 - /unlink 1167 (time 1763405503.45 total 10.01 last 116.59)
 - /unlink 2348 (time 1763405513.45 total 20.01 last 118.04)
 - /unlink 3616 (time 1763405523.45 total 30.02 last 126.78)
total: 4500 /unlink in 37.68 seconds: 119.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 723 (time 1763405545.84 total 10.03 last 72.11)
 - open/close 1388 (time 1763405555.85 total 20.04 last 66.43)
 - open/close 2056 (time 1763405565.86 total 30.05 last 66.68)
 - open/close 2713 (time 1763405575.88 total 40.07 last 65.62)
 - open/close 3405 (time 1763405585.89 total 50.08 last 69.13)
 - open/close 4075 (time 1763405595.89 total 60.08 last 66.99)
total: 4500 open/close in 66.24 seconds: 67.93 ops/second
 - /unlink 1199 (time 1763405616.59 total 10.00 last 119.87)
 - /unlink 2332 (time 1763405626.59 total 20.00 last 113.29)
 - /unlink 3599 (time 1763405636.60 total 30.01 last 126.63)
total: 4500 /unlink in 37.79 seconds: 119.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 755 (time 1763405658.34 total 10.01 last 75.44)
 - open/close 1496 (time 1763405668.34 total 20.01 last 74.09)
 - open/close 2169 (time 1763405678.35 total 30.02 last 67.24)
 - open/close 2875 (time 1763405688.36 total 40.02 last 70.55)
 - open/close 3597 (time 1763405698.37 total 50.03 last 72.13)
 - open/close 4304 (time 1763405708.38 total 60.05 last 70.62)
total: 4500 open/close in 63.13 seconds: 71.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1222 (time 1763405725.59 total 10.00 last 122.17)
 - /unlink 2353 (time 1763405735.60 total 20.01 last 113.07)
 - /unlink 3589 (time 1763405745.60 total 30.01 last 123.51)
total: 4500 /unlink in 37.35 seconds: 120.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 660 (time 1763405766.84 total 10.00 last 66.00)
 - open/close 1362 (time 1763405776.85 total 20.01 last 70.10)
 - open/close 2049 (time 1763405786.86 total 30.02 last 68.68)
 - open/close 2762 (time 1763405796.86 total 40.02 last 71.28)
 - open/close 3465 (time 1763405806.87 total 50.03 last 70.27)
 - open/close 4211 (time 1763405816.87 total 60.03 last 74.57)
total: 4500 open/close in 64.41 seconds: 69.87 ops/second
 - /unlink 1335 (time 1763405835.75 total 10.00 last 133.48)
 - /unlink 2620 (time 1763405845.75 total 20.00 last 128.48)
 - /unlink 3828 (time 1763405855.75 total 30.01 last 120.77)
total: 4500 /unlink in 34.89 seconds: 128.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 599 (time 1763405876.10 total 10.00 last 59.88)
 - open/close 1170 (time 1763405886.11 total 20.01 last 57.05)
 - open/close 1886 (time 1763405896.14 total 30.04 last 71.39)
 - open/close 2549 (time 1763405906.14 total 40.05 last 66.27)
 - open/close 3172 (time 1763405916.15 total 50.05 last 62.25)
 - open/close 3757 (time 1763405926.16 total 60.06 last 58.45)
 - open/close 4289 (time 1763405936.16 total 70.07 last 53.19)
total: 4500 open/close in 73.41 seconds: 61.30 ops/second
 - /unlink 1181 (time 1763405954.62 total 10.00 last 118.10)
 - /unlink 2228 (time 1763405964.63 total 20.01 last 104.63)
 - /unlink 3425 (time 1763405974.63 total 30.01 last 119.68)
total: 4500 /unlink in 39.74 seconds: 113.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 622 (time 1763405999.56 total 10.00 last 62.20)
 - open/close 1300 (time 1763406009.57 total 20.02 last 67.68)
 - open/close 1996 (time 1763406019.58 total 30.03 last 69.54)
 - open/close 2660 (time 1763406029.59 total 40.04 last 66.34)
 - open/close 3324 (time 1763406039.61 total 50.06 last 66.26)
 - open/close 3940 (time 1763406049.62 total 60.07 last 61.55)
total: 4500 open/close in 68.33 seconds: 65.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1156 (time 1763406072.68 total 10.00 last 115.59)
 - /unlink 2252 (time 1763406082.68 total 20.00 last 109.60)
 - /unlink 3621 (time 1763406092.68 total 30.00 last 136.87)
total: 4500 /unlink in 37.48 seconds: 120.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 652 (time 1763406115.81 total 10.01 last 65.16)
 - open/close 1306 (time 1763406125.82 total 20.01 last 65.35)
 - open/close 1997 (time 1763406135.82 total 30.02 last 69.07)
 - open/close 2635 (time 1763406145.83 total 40.03 last 63.75)
 - open/close 3345 (time 1763406155.84 total 50.04 last 70.92)
 - open/close 4028 (time 1763406165.85 total 60.05 last 68.25)
total: 4500 open/close in 66.43 seconds: 67.74 ops/second
 - /unlink 1265 (time 1763406186.89 total 10.01 last 126.41)
 - /unlink 2540 (time 1763406196.89 total 20.01 last 127.45)
 - /unlink 3741 (time 1763406206.90 total 30.02 last 119.98)
total: 4500 /unlink in 36.39 seconds: 123.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 666 (time 1763406228.20 total 10.00 last 66.59)
 - open/close 1357 (time 1763406238.22 total 20.02 last 69.00)
 - open/close 2038 (time 1763406248.22 total 30.02 last 68.05)
 - open/close 2732 (time 1763406258.22 total 40.02 last 69.40)
 - open/close 3408 (time 1763406268.23 total 50.03 last 67.57)
 - open/close 4091 (time 1763406278.23 total 60.03 last 68.28)
total: 4500 open/close in 66.53 seconds: 67.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1084 (time 1763406300.71 total 10.01 last 108.28)
 - /unlink 2365 (time 1763406310.72 total 20.02 last 128.04)
 - /unlink 3675 (time 1763406320.72 total 30.02 last 130.91)
total: 4500 /unlink in 36.82 seconds: 122.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 711 (time 1763406342.57 total 10.02 last 70.97)
 - open/close 1405 (time 1763406352.58 total 20.03 last 69.34)
 - open/close 2075 (time 1763406362.59 total 30.04 last 66.93)
 - open/close 2714 (time 1763406372.60 total 40.04 last 63.85)
 - open/close 3402 (time 1763406382.61 total 50.05 last 68.73)
 - open/close 4097 (time 1763406392.63 total 60.07 last 69.37)
total: 4500 open/close in 65.54 seconds: 68.66 ops/second
 - /unlink 1146 (time 1763406412.39 total 10.00 last 114.58)
 - /unlink 2354 (time 1763406422.39 total 20.00 last 120.78)
 - /unlink 3570 (time 1763406432.40 total 30.01 last 121.50)
total: 4500 /unlink in 37.58 seconds: 119.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 724 (time 1763406454.10 total 10.01 last 72.32)
 - open/close 1339 (time 1763406464.11 total 20.01 last 61.49)
 - open/close 2043 (time 1763406474.11 total 30.02 last 70.35)
 - open/close 2688 (time 1763406484.13 total 40.04 last 64.41)
 - open/close 3315 (time 1763406494.13 total 50.04 last 62.67)
 - open/close 3908 (time 1763406504.13 total 60.04 last 59.28)
total: 4500 open/close in 68.18 seconds: 66.00 ops/second
 - /unlink 1225 (time 1763406526.89 total 10.02 last 122.25)
 - /unlink 2272 (time 1763406536.89 total 20.02 last 104.66)
 - /unlink 3483 (time 1763406546.90 total 30.03 last 121.05)
total: 4500 /unlink in 38.72 seconds: 116.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 653 (time 1763406569.43 total 10.01 last 65.24)
 - open/close 1346 (time 1763406579.43 total 20.01 last 69.28)
 - open/close 2002 (time 1763406589.44 total 30.02 last 65.54)
 - open/close 2604 (time 1763406599.46 total 40.04 last 60.10)
 - open/close 3178 (time 1763406609.46 total 50.04 last 57.39)
 - open/close 3752 (time 1763406619.50 total 60.08 last 57.15)
 - open/close 4292 (time 1763406629.52 total 70.09 last 53.94)
total: 4500 open/close in 73.62 seconds: 61.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1136 (time 1763406647.50 total 10.00 last 113.57)
 - /unlink 2266 (time 1763406657.52 total 20.02 last 112.83)
 - /unlink 3401 (time 1763406667.53 total 30.03 last 113.41)
total: 4500 /unlink in 39.88 seconds: 112.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 621 (time 1763406691.95 total 10.01 last 62.06)
 - open/close 1296 (time 1763406701.96 total 20.02 last 67.43)
 - open/close 1906 (time 1763406711.97 total 30.03 last 60.94)
 - open/close 2428 (time 1763406721.99 total 40.04 last 52.12)
 - open/close 3002 (time 1763406732.00 total 50.05 last 57.35)
 - open/close 3587 (time 1763406742.01 total 60.07 last 58.41)
 - open/close 4288 (time 1763406752.02 total 70.07 last 70.09)
total: 4500 open/close in 72.94 seconds: 61.70 ops/second
 - /unlink 1266 (time 1763406769.30 total 10.01 last 126.51)
 - /unlink 2381 (time 1763406779.31 total 20.01 last 111.47)
 - /unlink 3426 (time 1763406789.31 total 30.01 last 104.49)
total: 4500 /unlink in 38.99 seconds: 115.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 585 (time 1763406813.54 total 10.02 last 58.38)
 - open/close 1208 (time 1763406823.55 total 20.04 last 62.21)
 - open/close 1808 (time 1763406833.55 total 30.04 last 59.97)
 - open/close 2478 (time 1763406843.57 total 40.05 last 66.93)
 - open/close 3123 (time 1763406853.57 total 50.06 last 64.45)
 - open/close 3729 (time 1763406863.58 total 60.07 last 60.55)
 - open/close 4344 (time 1763406873.58 total 70.07 last 61.50)
total: 4500 open/close in 72.75 seconds: 61.86 ops/second
 - /unlink 1196 (time 1763406891.24 total 10.01 last 119.51)
 - /unlink 2353 (time 1763406901.26 total 20.02 last 115.52)
 - /unlink 3570 (time 1763406911.26 total 30.03 last 121.64)
total: 4500 /unlink in 38.09 seconds: 118.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 543 (time 1763406935.77 total 10.01 last 54.24)
 - open/close 1196 (time 1763406945.78 total 20.01 last 65.29)
 - open/close 1858 (time 1763406955.80 total 30.03 last 66.07)
 - open/close 2376 (time 1763406965.80 total 40.04 last 51.79)
 - open/close 2995 (time 1763406975.81 total 50.05 last 61.82)
 - open/close 3511 (time 1763406985.81 total 60.05 last 51.58)
 - open/close 4119 (time 1763406995.82 total 70.06 last 60.74)
total: 4500 open/close in 76.02 seconds: 59.19 ops/second
 - /unlink 1272 (time 1763407017.03 total 10.00 last 127.19)
 - /unlink 2348 (time 1763407027.03 total 20.00 last 107.58)
 - /unlink 3378 (time 1763407037.04 total 30.01 last 102.94)
 - /unlink 4457 (time 1763407047.04 total 40.01 last 107.88)
total: 4500 /unlink in 40.41 seconds: 111.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 701 (time 1763407062.30 total 10.01 last 70.05)
 - open/close 1317 (time 1763407072.32 total 20.03 last 61.48)
 - open/close 1877 (time 1763407082.33 total 30.04 last 55.95)
 - open/close 2478 (time 1763407092.33 total 40.04 last 60.09)
 - open/close 3092 (time 1763407102.34 total 50.05 last 61.36)
 - open/close 3701 (time 1763407112.35 total 60.05 last 60.87)
 - open/close 4330 (time 1763407122.35 total 70.05 last 62.90)
total: 4500 open/close in 72.82 seconds: 61.80 ops/second
 - /unlink 1273 (time 1763407139.66 total 10.02 last 127.06)
 - /unlink 2384 (time 1763407149.67 total 20.03 last 111.01)
 - /unlink 3398 (time 1763407159.67 total 30.03 last 101.36)
 - /unlink 4485 (time 1763407169.68 total 40.04 last 108.64)
total: 4500 /unlink in 40.15 seconds: 112.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 666 (time 1763407184.30 total 10.01 last 66.52)
 - open/close 1302 (time 1763407194.33 total 20.04 last 63.41)
 - open/close 1869 (time 1763407204.34 total 30.04 last 56.69)
 - open/close 2394 (time 1763407214.34 total 40.05 last 52.45)
 - open/close 3049 (time 1763407224.35 total 50.05 last 65.49)
 - open/close 3688 (time 1763407234.35 total 60.06 last 63.90)
 - open/close 4369 (time 1763407244.35 total 70.06 last 68.08)
total: 4500 open/close in 72.03 seconds: 62.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1157 (time 1763407261.11 total 10.00 last 115.65)
 - /unlink 2257 (time 1763407271.11 total 20.01 last 109.98)
 - /unlink 3254 (time 1763407281.12 total 30.01 last 99.67)
 - /unlink 4340 (time 1763407291.12 total 40.01 last 108.56)
total: 4500 /unlink in 41.38 seconds: 108.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 628 (time 1763407306.80 total 10.01 last 62.71)
 - open/close 1183 (time 1763407316.80 total 20.02 last 55.49)
 - open/close 1773 (time 1763407326.81 total 30.03 last 58.92)
 - open/close 2427 (time 1763407336.81 total 40.03 last 65.40)
 - open/close 3040 (time 1763407346.82 total 50.04 last 61.25)
 - open/close 3608 (time 1763407356.82 total 60.04 last 56.79)
 - open/close 4274 (time 1763407366.83 total 70.05 last 66.57)
total: 4500 open/close in 73.43 seconds: 61.28 ops/second
 - /unlink 962 (time 1763407384.99 total 10.01 last 96.15)
 - /unlink 1725 (time 1763407394.99 total 20.01 last 76.28)
 - /unlink 2589 (time 1763407405.01 total 30.02 last 86.26)
 - /unlink 3447 (time 1763407415.02 total 40.03 last 85.74)
 - /unlink 4492 (time 1763407425.03 total 50.05 last 104.32)
total: 4500 /unlink in 50.11 seconds: 89.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 502 (time 1763407441.44 total 10.03 last 50.04)
 - open/close 1123 (time 1763407451.45 total 20.03 last 62.09)
 - open/close 1680 (time 1763407461.45 total 30.04 last 55.69)
 - open/close 2198 (time 1763407471.45 total 40.04 last 51.77)
 - open/close 2814 (time 1763407481.46 total 50.05 last 61.54)
 - open/close 3430 (time 1763407491.49 total 60.07 last 61.46)
 - open/close 4125 (time 1763407501.49 total 70.07 last 69.50)
total: 4500 open/close in 75.69 seconds: 59.45 ops/second
 - /unlink 1280 (time 1763407521.44 total 10.00 last 127.96)
 - /unlink 2446 (time 1763407531.45 total 20.01 last 116.52)
 - /unlink 3720 (time 1763407541.45 total 30.01 last 127.37)
total: 4500 /unlink in 36.56 seconds: 123.10 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 94089 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg135-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg135-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server
stop mds service on oleg135-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server
stop mds service on oleg135-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg135-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg135-server'
oleg135-server: oleg135-server.virtnet: executing unload_modules_local
oleg135-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg135-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.