== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:58:29 (1786993109)
Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg241-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server
oleg241-server: oleg241-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing load_modules_local
oleg241-server: Loading modules from /home/green/git/lustre-release/lustre
oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg241-server: detected 4 online CPUs by sysfs
oleg241-server: MODOPTS_LIBCFS=
oleg241-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg241-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg241-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg241-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg241-client.virtnet:  -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 763 (time 1786993349.19 total 10.03 last 76.06)
 - open/close 1490 (time 1786993359.20 total 20.04 last 72.62)
 - open/close 2084 (time 1786993369.21 total 30.05 last 59.34)
 - open/close 2697 (time 1786993379.21 total 40.06 last 61.26)
 - open/close 3318 (time 1786993389.22 total 50.07 last 62.05)
 - open/close 3954 (time 1786993399.22 total 60.07 last 63.59)
total: 4500 open/close in 68.30 seconds: 65.89 ops/second
 - /unlink 1299 (time 1786993422.43 total 10.00 last 129.89)
 - /unlink 2584 (time 1786993432.44 total 20.00 last 128.45)
 - /unlink 3928 (time 1786993442.44 total 30.01 last 134.36)
total: 4500 /unlink in 34.59 seconds: 130.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 741 (time 1786993462.44 total 10.01 last 74.00)
 - open/close 1249 (time 1786993472.45 total 20.03 last 50.74)
 - open/close 1867 (time 1786993482.46 total 30.04 last 61.73)
 - open/close 2487 (time 1786993492.46 total 40.04 last 61.97)
 - open/close 3202 (time 1786993502.47 total 50.05 last 71.44)
 - open/close 3919 (time 1786993512.48 total 60.06 last 71.63)
total: 4500 open/close in 68.54 seconds: 65.66 ops/second
 - /unlink 1313 (time 1786993535.66 total 10.01 last 131.18)
 - /unlink 2474 (time 1786993545.66 total 20.02 last 116.03)
 - /unlink 3656 (time 1786993555.67 total 30.02 last 118.15)
total: 4500 /unlink in 37.06 seconds: 121.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 584 (time 1786993579.04 total 10.02 last 58.31)
 - open/close 1131 (time 1786993589.04 total 20.02 last 54.69)
 - open/close 1803 (time 1786993599.05 total 30.02 last 67.16)
 - open/close 2626 (time 1786993609.06 total 40.04 last 82.20)
 - open/close 3281 (time 1786993619.08 total 50.06 last 65.34)
 - open/close 3949 (time 1786993629.10 total 60.08 last 66.69)
total: 4500 open/close in 66.68 seconds: 67.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1413 (time 1786993650.71 total 10.00 last 141.28)
 - /unlink 2601 (time 1786993660.71 total 20.00 last 118.79)
 - /unlink 3716 (time 1786993670.72 total 30.01 last 111.39)
total: 4500 /unlink in 37.07 seconds: 121.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 568 (time 1786993694.70 total 10.00 last 56.79)
 - open/close 1217 (time 1786993704.71 total 20.01 last 64.87)
 - open/close 1940 (time 1786993714.71 total 30.01 last 72.26)
 - open/close 2546 (time 1786993724.72 total 40.02 last 60.57)
 - open/close 3336 (time 1786993734.72 total 50.02 last 78.99)
 - open/close 4063 (time 1786993744.72 total 60.02 last 72.67)
total: 4500 open/close in 65.10 seconds: 69.12 ops/second
 - /unlink 1291 (time 1786993764.55 total 10.01 last 128.99)
 - /unlink 2630 (time 1786993774.56 total 20.01 last 133.82)
 - /unlink 3963 (time 1786993784.56 total 30.02 last 133.23)
total: 4500 /unlink in 34.45 seconds: 130.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 704 (time 1786993804.25 total 10.01 last 70.36)
 - open/close 1380 (time 1786993814.26 total 20.01 last 67.56)
 - open/close 2091 (time 1786993824.27 total 30.02 last 71.04)
 - open/close 2817 (time 1786993834.27 total 40.02 last 72.59)
 - open/close 3508 (time 1786993844.27 total 50.03 last 69.07)
 - open/close 4208 (time 1786993854.29 total 60.04 last 69.91)
total: 4500 open/close in 65.35 seconds: 68.87 ops/second
 - /unlink 1348 (time 1786993874.89 total 10.00 last 134.80)
 - /unlink 2632 (time 1786993884.89 total 20.00 last 128.40)
 - /unlink 3897 (time 1786993894.89 total 30.00 last 126.47)
total: 4500 /unlink in 34.20 seconds: 131.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 707 (time 1786993914.28 total 10.01 last 70.64)
 - open/close 1381 (time 1786993924.30 total 20.02 last 67.31)
 - open/close 1990 (time 1786993934.31 total 30.03 last 60.84)
 - open/close 2674 (time 1786993944.32 total 40.05 last 68.30)
 - open/close 3320 (time 1786993954.35 total 50.07 last 64.43)
 - open/close 3930 (time 1786993964.35 total 60.08 last 60.98)
total: 4500 open/close in 68.66 seconds: 65.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1169 (time 1786993988.46 total 10.00 last 116.88)
 - /unlink 2548 (time 1786993998.47 total 20.01 last 137.76)
 - /unlink 3846 (time 1786994008.49 total 30.03 last 129.59)
total: 4500 /unlink in 36.53 seconds: 123.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 579 (time 1786994031.23 total 10.00 last 57.89)
 - open/close 1229 (time 1786994041.24 total 20.01 last 64.95)
 - open/close 1927 (time 1786994051.24 total 30.02 last 69.76)
 - open/close 2536 (time 1786994061.25 total 40.02 last 60.87)
 - open/close 3231 (time 1786994071.25 total 50.02 last 69.48)
 - open/close 4000 (time 1786994081.25 total 60.02 last 76.89)
total: 4500 open/close in 67.25 seconds: 66.91 ops/second
 - /unlink 1387 (time 1786994103.17 total 10.00 last 138.65)
 - /unlink 2615 (time 1786994113.18 total 20.01 last 122.69)
 - /unlink 3946 (time 1786994123.18 total 30.02 last 133.06)
total: 4500 /unlink in 34.14 seconds: 131.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 741 (time 1786994142.16 total 10.01 last 74.05)
 - open/close 1602 (time 1786994152.16 total 20.01 last 86.06)
 - open/close 2408 (time 1786994162.17 total 30.01 last 80.58)
 - open/close 3015 (time 1786994172.18 total 40.03 last 60.61)
 - open/close 3623 (time 1786994182.19 total 50.03 last 60.77)
 - open/close 4242 (time 1786994192.19 total 60.03 last 61.90)
total: 4500 open/close in 64.06 seconds: 70.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1101 (time 1786994211.61 total 10.01 last 110.00)
 - /unlink 2156 (time 1786994221.62 total 20.02 last 105.39)
 - /unlink 3576 (time 1786994231.64 total 30.04 last 141.74)
total: 4500 /unlink in 38.22 seconds: 117.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 658 (time 1786994254.96 total 10.00 last 65.79)
 - open/close 1471 (time 1786994264.97 total 20.01 last 81.26)
 - open/close 2276 (time 1786994274.97 total 30.01 last 80.50)
 - open/close 2907 (time 1786994284.98 total 40.02 last 62.99)
 - open/close 3455 (time 1786994294.99 total 50.03 last 54.76)
 - open/close 4117 (time 1786994305.00 total 60.04 last 66.17)
total: 4500 open/close in 65.61 seconds: 68.59 ops/second
 - /unlink 1281 (time 1786994325.84 total 10.00 last 128.09)
 - /unlink 2488 (time 1786994335.85 total 20.01 last 120.62)
 - /unlink 3908 (time 1786994345.85 total 30.01 last 141.97)
total: 4500 /unlink in 34.64 seconds: 129.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 684 (time 1786994365.76 total 10.00 last 68.37)
 - open/close 1384 (time 1786994375.77 total 20.01 last 69.93)
 - open/close 2116 (time 1786994385.79 total 30.03 last 73.06)
 - open/close 2789 (time 1786994395.80 total 40.04 last 67.24)
 - open/close 3546 (time 1786994405.80 total 50.05 last 75.68)
 - open/close 4131 (time 1786994415.83 total 60.08 last 58.29)
total: 4500 open/close in 66.11 seconds: 68.06 ops/second
 - /unlink 1520 (time 1786994437.20 total 10.00 last 151.97)
 - /unlink 2666 (time 1786994447.20 total 20.01 last 114.56)
 - /unlink 3925 (time 1786994457.21 total 30.02 last 125.70)
total: 4500 /unlink in 34.51 seconds: 130.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 643 (time 1786994476.95 total 10.01 last 64.27)
 - open/close 1292 (time 1786994486.96 total 20.02 last 64.80)
 - open/close 1833 (time 1786994496.98 total 30.04 last 53.99)
 - open/close 2394 (time 1786994506.99 total 40.05 last 56.07)
 - open/close 3050 (time 1786994517.01 total 50.07 last 65.46)
 - open/close 3752 (time 1786994527.01 total 60.07 last 70.17)
 - open/close 4411 (time 1786994537.03 total 70.09 last 65.78)
total: 4500 open/close in 71.58 seconds: 62.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1562 (time 1786994553.03 total 10.01 last 156.11)
 - /unlink 2872 (time 1786994563.04 total 20.01 last 130.98)
 - /unlink 4400 (time 1786994573.04 total 30.01 last 152.74)
total: 4500 /unlink in 30.96 seconds: 145.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 688 (time 1786994590.46 total 10.00 last 68.77)
 - open/close 1509 (time 1786994600.47 total 20.01 last 82.04)
 - open/close 2255 (time 1786994610.49 total 30.03 last 74.47)
 - open/close 3016 (time 1786994620.50 total 40.05 last 75.97)
 - open/close 3721 (time 1786994630.53 total 50.08 last 70.29)
total: 4500 open/close in 59.87 seconds: 75.17 ops/second
 - /unlink 1252 (time 1786994655.00 total 10.00 last 125.20)
 - /unlink 2860 (time 1786994665.00 total 20.00 last 160.76)
 - /unlink 4224 (time 1786994675.01 total 30.01 last 136.30)
total: 4500 /unlink in 32.21 seconds: 139.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 672 (time 1786994694.38 total 10.01 last 67.16)
 - open/close 1430 (time 1786994704.39 total 20.01 last 75.76)
 - open/close 2211 (time 1786994714.40 total 30.02 last 78.01)
 - open/close 2973 (time 1786994724.41 total 40.03 last 76.15)
 - open/close 3717 (time 1786994734.44 total 50.06 last 74.15)
 - open/close 4474 (time 1786994744.45 total 60.07 last 75.65)
total: 4500 open/close in 60.33 seconds: 74.59 ops/second
 - /unlink 1394 (time 1786994759.00 total 10.01 last 139.19)
 - /unlink 2602 (time 1786994769.00 total 20.02 last 120.75)
 - /unlink 4023 (time 1786994779.01 total 30.02 last 142.08)
total: 4500 /unlink in 33.81 seconds: 133.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 794 (time 1786994798.84 total 10.01 last 79.31)
 - open/close 1507 (time 1786994808.85 total 20.02 last 71.22)
 - open/close 2253 (time 1786994818.86 total 30.03 last 74.51)
 - open/close 2934 (time 1786994828.88 total 40.05 last 68.02)
 - open/close 3423 (time 1786994838.90 total 50.07 last 48.79)
 - open/close 4008 (time 1786994848.92 total 60.09 last 58.34)
 - open/close 4443 (time 1786994858.96 total 70.13 last 43.35)
total: 4500 open/close in 71.40 seconds: 63.03 ops/second
 - /unlink 1171 (time 1786994876.50 total 10.03 last 116.77)
 - /unlink 2269 (time 1786994886.52 total 20.05 last 109.58)
 - /unlink 3417 (time 1786994896.53 total 30.05 last 114.75)
total: 4500 /unlink in 39.30 seconds: 114.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 657 (time 1786994922.00 total 10.02 last 65.59)
 - open/close 1373 (time 1786994932.00 total 20.02 last 71.57)
 - open/close 2094 (time 1786994942.01 total 30.03 last 72.01)
 - open/close 2821 (time 1786994952.02 total 40.04 last 72.67)
 - open/close 3496 (time 1786994962.03 total 50.05 last 67.39)
 - open/close 4121 (time 1786994972.04 total 60.06 last 62.49)
total: 4500 open/close in 67.93 seconds: 66.25 ops/second
 - /unlink 914 (time 1786994996.59 total 10.01 last 91.34)
 - /unlink 1992 (time 1786995006.60 total 20.01 last 107.78)
 - /unlink 3070 (time 1786995016.60 total 30.02 last 107.72)
 - /unlink 4377 (time 1786995026.61 total 40.02 last 130.64)
total: 4500 /unlink in 40.97 seconds: 109.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 713 (time 1786995042.66 total 10.01 last 71.25)
 - open/close 1448 (time 1786995052.66 total 20.01 last 73.47)
 - open/close 2214 (time 1786995062.67 total 30.02 last 76.50)
 - open/close 2876 (time 1786995072.68 total 40.03 last 66.14)
 - open/close 3550 (time 1786995082.70 total 50.05 last 67.31)
 - open/close 4276 (time 1786995092.71 total 60.06 last 72.53)
total: 4500 open/close in 62.98 seconds: 71.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1386 (time 1786995111.33 total 10.00 last 138.58)
 - /unlink 2644 (time 1786995121.33 total 20.00 last 125.76)
 - /unlink 3718 (time 1786995131.33 total 30.01 last 107.39)
total: 4500 /unlink in 36.34 seconds: 123.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 731 (time 1786995152.96 total 10.02 last 72.98)
 - open/close 1431 (time 1786995162.97 total 20.03 last 69.93)
 - open/close 2154 (time 1786995172.98 total 30.03 last 72.25)
 - open/close 2804 (time 1786995182.98 total 40.04 last 64.94)
 - open/close 3408 (time 1786995193.01 total 50.07 last 60.26)
 - open/close 4009 (time 1786995203.01 total 60.07 last 60.08)
total: 4500 open/close in 67.36 seconds: 66.80 ops/second
 - /unlink 1159 (time 1786995225.26 total 10.00 last 115.88)
 - /unlink 2495 (time 1786995235.26 total 20.00 last 133.58)
 - /unlink 3754 (time 1786995245.27 total 30.00 last 125.88)
total: 4500 /unlink in 36.46 seconds: 123.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 647 (time 1786995267.39 total 10.00 last 64.69)
 - open/close 1261 (time 1786995277.45 total 20.05 last 61.08)
 - open/close 1906 (time 1786995287.46 total 30.07 last 64.41)
 - open/close 2475 (time 1786995297.47 total 40.07 last 56.86)
 - open/close 3022 (time 1786995307.48 total 50.09 last 54.61)
 - open/close 3550 (time 1786995317.50 total 60.11 last 52.69)
 - open/close 4090 (time 1786995327.51 total 70.12 last 53.95)
total: 4500 open/close in 76.93 seconds: 58.49 ops/second
 - /unlink 1060 (time 1786995350.43 total 10.01 last 105.89)
 - /unlink 2082 (time 1786995360.43 total 20.02 last 102.14)
 - /unlink 3121 (time 1786995370.45 total 30.04 last 103.70)
 - /unlink 4040 (time 1786995380.45 total 40.04 last 91.86)
total: 4500 /unlink in 44.55 seconds: 101.01 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95435 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg241-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg241-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server
stop mds service on oleg241-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg241-server

unloading modules via unload_modules_local on: 'oleg241-server'
oleg241-server: oleg241-server.virtnet: executing unload_modules_local
oleg241-server: 
modules unloaded.