== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:12:40 (1763341960)
Stopping clients: oleg117-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg117-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg117-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg117-server
oleg117-server: oleg117-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg117-server'
oleg117-server: oleg117-server.virtnet: executing load_modules_local
oleg117-server: Loading modules from /home/green/git/lustre-release/lustre
oleg117-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy
oleg117-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy
oleg117-server: detected 4 online CPUs by sysfs
oleg117-server: MODOPTS_LIBCFS=
oleg117-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg117-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all
pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg117-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all
pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg117-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all
pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg117-client.virtnet:  -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_loc=0x1312
fail_val=5
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 790 (time 1763342119.28 total 10.00 last 78.97)
 - open/close 1614 (time 1763342129.29 total 20.01 last 82.37)
 - open/close 2368 (time 1763342139.29 total 30.01 last 75.39)
 - open/close 3126 (time 1763342149.30 total 40.02 last 75.69)
 - open/close 3885 (time 1763342159.31 total 50.03 last 75.86)
total: 4500 open/close in 58.12 seconds: 77.42 ops/second
 - /unlink 1305 (time 1763342180.39 total 10.00 last 130.49)
 - /unlink 2583 (time 1763342190.40 total 20.00 last 127.77)
 - /unlink 3862 (time 1763342200.40 total 30.01 last 127.80)
total: 4500 /unlink in 34.68 seconds: 129.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 792 (time 1763342218.71 total 10.00 last 79.19)
 - open/close 1519 (time 1763342228.72 total 20.02 last 72.60)
 - open/close 2237 (time 1763342238.73 total 30.02 last 71.75)
 - open/close 2988 (time 1763342248.73 total 40.02 last 75.09)
 - open/close 3747 (time 1763342258.74 total 50.04 last 75.81)
total: 4500 open/close in 59.96 seconds: 75.05 ops/second
 - /unlink 1246 (time 1763342282.63 total 10.00 last 124.56)
 - /unlink 2547 (time 1763342292.63 total 20.01 last 130.03)
 - /unlink 3843 (time 1763342302.63 total 30.01 last 129.59)
total: 4500 /unlink in 35.79 seconds: 125.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 654 (time 1763342323.21 total 10.01 last 65.35)
 - open/close 1330 (time 1763342333.23 total 20.03 last 67.48)
 - open/close 2066 (time 1763342343.24 total 30.03 last 73.53)
 - open/close 2760 (time 1763342353.24 total 40.04 last 69.37)
 - open/close 3435 (time 1763342363.24 total 50.04 last 67.49)
 - open/close 4158 (time 1763342373.25 total 60.05 last 72.24)
total: 4500 open/close in 64.99 seconds: 69.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1294 (time 1763342392.11 total 10.00 last 129.39)
 - /unlink 2538 (time 1763342402.11 total 20.00 last 124.36)
 - /unlink 3716 (time 1763342412.11 total 30.01 last 117.75)
total: 4500 /unlink in 36.23 seconds: 124.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 726 (time 1763342433.31 total 10.01 last 72.54)
 - open/close 1433 (time 1763342443.32 total 20.02 last 70.65)
 - open/close 2134 (time 1763342453.32 total 30.02 last 70.06)
 - open/close 2845 (time 1763342463.32 total 40.02 last 71.08)
 - open/close 3492 (time 1763342473.34 total 50.04 last 64.62)
 - open/close 4179 (time 1763342483.34 total 60.04 last 68.69)
total: 4500 open/close in 64.28 seconds: 70.01 ops/second
 - /unlink 1350 (time 1763342501.32 total 10.00 last 134.96)
 - /unlink 2525 (time 1763342511.33 total 20.00 last 117.48)
 - /unlink 3749 (time 1763342521.33 total 30.01 last 122.39)
total: 4500 /unlink in 36.13 seconds: 124.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 721 (time 1763342540.65 total 10.01 last 72.06)
 - open/close 1423 (time 1763342550.65 total 20.01 last 70.19)
 - open/close 2154 (time 1763342560.65 total 30.01 last 73.08)
 - open/close 2906 (time 1763342570.67 total 40.03 last 75.05)
 - open/close 3641 (time 1763342580.69 total 50.05 last 73.37)
 - open/close 4363 (time 1763342590.71 total 60.07 last 72.06)
total: 4500 open/close in 61.91 seconds: 72.69 ops/second
 - /unlink 1276 (time 1763342606.18 total 10.00 last 127.58)
 - /unlink 2604 (time 1763342616.19 total 20.01 last 132.69)
 - /unlink 3913 (time 1763342626.19 total 30.01 last 130.90)
total: 4500 /unlink in 34.52 seconds: 130.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 700 (time 1763342644.33 total 10.01 last 69.93)
 - open/close 1399 (time 1763342654.33 total 20.02 last 69.86)
 - open/close 2112 (time 1763342664.34 total 30.02 last 71.29)
 - open/close 2844 (time 1763342674.35 total 40.03 last 73.13)
 - open/close 3598 (time 1763342684.35 total 50.03 last 75.37)
 - open/close 4354 (time 1763342694.36 total 60.05 last 75.49)
total: 4500 open/close in 62.07 seconds: 72.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1402 (time 1763342709.89 total 10.00 last 140.17)
 - /unlink 2771 (time 1763342719.89 total 20.00 last 136.89)
 - /unlink 4115 (time 1763342729.89 total 30.00 last 134.39)
total: 4500 /unlink in 32.82 seconds: 137.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 753 (time 1763342746.27 total 10.00 last 75.30)
 - open/close 1478 (time 1763342756.27 total 20.00 last 72.47)
 - open/close 2231 (time 1763342766.29 total 30.02 last 75.20)
 - open/close 2943 (time 1763342776.30 total 40.03 last 71.12)
 - open/close 3669 (time 1763342786.31 total 50.04 last 72.53)
 - open/close 4383 (time 1763342796.31 total 60.04 last 71.40)
total: 4500 open/close in 61.80 seconds: 72.82 ops/second
 - /unlink 1271 (time 1763342813.17 total 10.01 last 127.00)
 - /unlink 2534 (time 1763342823.17 total 20.01 last 126.26)
 - /unlink 3773 (time 1763342833.18 total 30.02 last 123.84)
total: 4500 /unlink in 35.90 seconds: 125.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 667 (time 1763342852.68 total 10.02 last 66.55)
 - open/close 1355 (time 1763342862.69 total 20.03 last 68.75)
 - open/close 2053 (time 1763342872.71 total 30.05 last 69.65)
 - open/close 2702 (time 1763342882.71 total 40.05 last 64.90)
 - open/close 3380 (time 1763342892.71 total 50.05 last 67.79)
 - open/close 4021 (time 1763342902.73 total 60.07 last 63.99)
total: 4500 open/close in 66.96 seconds: 67.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1361 (time 1763342923.59 total 10.00 last 136.09)
 - /unlink 2615 (time 1763342933.59 total 20.00 last 125.37)
 - /unlink 3828 (time 1763342943.59 total 30.00 last 121.29)
total: 4500 /unlink in 35.30 seconds: 127.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 675 (time 1763342962.71 total 10.00 last 67.47)
 - open/close 1272 (time 1763342972.73 total 20.02 last 59.62)
 - open/close 1921 (time 1763342982.74 total 30.03 last 64.81)
 - open/close 2588 (time 1763342992.76 total 40.05 last 66.59)
 - open/close 3302 (time 1763343002.76 total 50.05 last 71.37)
 - open/close 4037 (time 1763343012.77 total 60.06 last 73.45)
total: 4500 open/close in 66.59 seconds: 67.58 ops/second
 - /unlink 1306 (time 1763343032.46 total 10.01 last 130.40)
 - /unlink 2686 (time 1763343042.48 total 20.03 last 137.78)
 - /unlink 4116 (time 1763343052.48 total 30.03 last 142.98)
total: 4500 /unlink in 32.99 seconds: 136.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 720 (time 1763343069.37 total 10.01 last 71.93)
 - open/close 1441 (time 1763343079.38 total 20.01 last 72.08)
 - open/close 2246 (time 1763343089.38 total 30.01 last 80.50)
 - open/close 3002 (time 1763343099.39 total 40.02 last 75.52)
 - open/close 3739 (time 1763343109.40 total 50.04 last 73.60)
total: 4500 open/close in 59.70 seconds: 75.37 ops/second
 - /unlink 1457 (time 1763343131.62 total 10.00 last 145.69)
 - /unlink 2838 (time 1763343141.63 total 20.02 last 137.88)
 - /unlink 4199 (time 1763343151.64 total 30.02 last 136.04)
total: 4500 /unlink in 32.10 seconds: 140.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 761 (time 1763343167.00 total 10.02 last 75.98)
 - open/close 1385 (time 1763343177.02 total 20.04 last 62.26)
 - open/close 2118 (time 1763343187.04 total 30.05 last 73.21)
 - open/close 2797 (time 1763343197.04 total 40.05 last 67.89)
 - open/close 3485 (time 1763343207.04 total 50.06 last 68.79)
 - open/close 4241 (time 1763343217.04 total 60.06 last 75.59)
total: 4500 open/close in 63.64 seconds: 70.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1313 (time 1763343234.19 total 10.00 last 131.28)
 - /unlink 2574 (time 1763343244.19 total 20.00 last 126.10)
 - /unlink 3994 (time 1763343254.20 total 30.01 last 141.93)
total: 4500 /unlink in 33.96 seconds: 132.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 751 (time 1763343271.85 total 10.00 last 75.07)
 - open/close 1486 (time 1763343281.85 total 20.01 last 73.48)
 - open/close 2210 (time 1763343291.86 total 30.02 last 72.31)
 - open/close 2991 (time 1763343301.87 total 40.03 last 78.03)
 - open/close 3734 (time 1763343311.88 total 50.03 last 74.29)
 - open/close 4444 (time 1763343321.88 total 60.04 last 70.95)
total: 4500 open/close in 60.73 seconds: 74.10 ops/second
 - /unlink 1351 (time 1763343336.12 total 10.00 last 135.04)
 - /unlink 2656 (time 1763343346.12 total 20.01 last 130.47)
 - /unlink 3983 (time 1763343356.12 total 30.01 last 132.64)
total: 4500 /unlink in 33.72 seconds: 133.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 757 (time 1763343373.38 total 10.00 last 75.66)
 - open/close 1489 (time 1763343383.39 total 20.01 last 73.15)
 - open/close 2197 (time 1763343393.39 total 30.02 last 70.77)
 - open/close 2940 (time 1763343403.40 total 40.02 last 74.24)
 - open/close 3712 (time 1763343413.40 total 50.03 last 77.16)
 - open/close 4423 (time 1763343423.42 total 60.04 last 71.02)
total: 4500 open/close in 61.13 seconds: 73.61 ops/second
 - /unlink 1425 (time 1763343437.85 total 10.00 last 142.46)
 - /unlink 2679 (time 1763343447.85 total 20.01 last 125.36)
 - /unlink 3973 (time 1763343457.86 total 30.01 last 129.33)
total: 4500 /unlink in 34.03 seconds: 132.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 769 (time 1763343475.83 total 10.02 last 76.77)
 - open/close 1465 (time 1763343485.83 total 20.02 last 69.59)
 - open/close 2184 (time 1763343495.85 total 30.03 last 71.81)
 - open/close 2891 (time 1763343505.87 total 40.05 last 70.56)
 - open/close 3572 (time 1763343515.87 total 50.06 last 68.07)
 - open/close 4353 (time 1763343525.88 total 60.07 last 78.02)
total: 4500 open/close in 62.17 seconds: 72.38 ops/second
 - /unlink 1487 (time 1763343541.48 total 10.00 last 148.67)
 - /unlink 2997 (time 1763343551.50 total 20.01 last 150.83)
 - /unlink 4284 (time 1763343561.50 total 30.02 last 128.59)
total: 4500 /unlink in 31.95 seconds: 140.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 726 (time 1763343577.12 total 10.02 last 72.47)
 - open/close 1466 (time 1763343587.12 total 20.02 last 73.96)
 - open/close 2188 (time 1763343597.13 total 30.03 last 72.16)
 - open/close 2926 (time 1763343607.13 total 40.03 last 73.78)
 - open/close 3679 (time 1763343617.13 total 50.03 last 75.29)
 - open/close 4422 (time 1763343627.14 total 60.04 last 74.24)
total: 4500 open/close in 61.19 seconds: 73.54 ops/second
 - /unlink 1520 (time 1763343642.71 total 10.01 last 151.92)
 - /unlink 2899 (time 1763343652.71 total 20.01 last 137.88)
 - /unlink 4135 (time 1763343662.72 total 30.01 last 123.57)
total: 4500 /unlink in 32.90 seconds: 136.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 656 (time 1763343680.35 total 10.01 last 65.54)
 - open/close 1296 (time 1763343690.36 total 20.01 last 63.98)
 - open/close 1979 (time 1763343700.36 total 30.01 last 68.29)
 - open/close 2717 (time 1763343710.36 total 40.02 last 73.78)
 - open/close 3512 (time 1763343720.36 total 50.02 last 79.48)
 - open/close 4269 (time 1763343730.37 total 60.02 last 75.67)
total: 4500 open/close in 63.39 seconds: 70.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1290 (time 1763343747.61 total 10.00 last 128.96)
 - /unlink 2601 (time 1763343757.61 total 20.00 last 131.10)
 - /unlink 4020 (time 1763343767.61 total 30.00 last 141.88)
total: 4500 /unlink in 33.58 seconds: 133.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 794 (time 1763343784.97 total 10.00 last 79.38)
 - open/close 1518 (time 1763343794.99 total 20.03 last 72.23)
 - open/close 2223 (time 1763343805.00 total 30.04 last 70.43)
 - open/close 2840 (time 1763343815.01 total 40.04 last 61.67)
 - open/close 3568 (time 1763343825.02 total 50.05 last 72.74)
 - open/close 4240 (time 1763343835.02 total 60.06 last 67.16)
total: 4500 open/close in 63.22 seconds: 71.18 ops/second
 - /unlink 1262 (time 1763343851.52 total 10.00 last 126.16)
 - /unlink 2498 (time 1763343861.53 total 20.01 last 123.50)
 - /unlink 3759 (time 1763343871.53 total 30.01 last 126.09)
total: 4500 /unlink in 35.94 seconds: 125.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 693 (time 1763343890.60 total 10.00 last 69.30)
 - open/close 1397 (time 1763343900.60 total 20.00 last 70.38)
 - open/close 2119 (time 1763343910.60 total 30.00 last 72.19)
 - open/close 2845 (time 1763343920.61 total 40.01 last 72.56)
 - open/close 3539 (time 1763343930.61 total 50.02 last 69.36)
 - open/close 4237 (time 1763343940.62 total 60.03 last 69.72)
total: 4500 open/close in 64.06 seconds: 70.25 ops/second
 - /unlink 1218 (time 1763343960.08 total 10.00 last 121.75)
 - /unlink 2511 (time 1763343970.08 total 20.01 last 129.25)
 - /unlink 3839 (time 1763343980.09 total 30.02 last 132.69)
total: 4500 /unlink in 35.16 seconds: 127.97 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 93983 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg117-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg117-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server
stop mds service on oleg117-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server
stop mds service on oleg117-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg117-server'
oleg117-server: oleg117-server.virtnet: executing unload_modules_local
oleg117-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg117-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.