== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:02:12 (1778655732)
Stopping clients: oleg211-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg211-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg211-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg211-server
oleg211-server: oleg211-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing load_modules_local
oleg211-server: Loading modules from /home/green/git/lustre-release/lustre
oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg211-server: detected 4 online CPUs by sysfs
oleg211-server: MODOPTS_LIBCFS=
oleg211-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg211-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg211-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg211-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all
pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg211-client.virtnet:  -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 1037 (time 1778655942.75 total 10.00 last 103.69)
 - open/close 1955 (time 1778655952.75 total 20.00 last 91.79)
 - open/close 2826 (time 1778655962.76 total 30.00 last 87.08)
 - open/close 3679 (time 1778655972.76 total 40.01 last 85.24)
total: 4500 open/close in 48.96 seconds: 91.91 ops/second
 - /unlink 1391 (time 1778655996.24 total 10.01 last 138.91)
 - /unlink 2826 (time 1778656006.24 total 20.01 last 143.48)
 - /unlink 4254 (time 1778656016.24 total 30.02 last 142.73)
total: 4500 /unlink in 31.79 seconds: 141.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 701 (time 1778656032.19 total 10.00 last 70.10)
 - open/close 1442 (time 1778656042.19 total 20.00 last 74.09)
 - open/close 2259 (time 1778656052.20 total 30.01 last 81.63)
 - open/close 2981 (time 1778656062.21 total 40.01 last 72.16)
 - open/close 3598 (time 1778656072.22 total 50.03 last 61.63)
 - open/close 4179 (time 1778656082.23 total 60.03 last 58.05)
total: 4500 open/close in 63.58 seconds: 70.78 ops/second
 - /unlink 1579 (time 1778656100.70 total 10.00 last 157.89)
 - /unlink 2807 (time 1778656110.70 total 20.01 last 122.74)
 - /unlink 4095 (time 1778656120.70 total 30.01 last 128.80)
total: 4500 /unlink in 33.22 seconds: 135.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 635 (time 1778656140.16 total 10.04 last 63.22)
 - open/close 1247 (time 1778656150.17 total 20.06 last 61.13)
 - open/close 1845 (time 1778656160.19 total 30.07 last 59.72)
 - open/close 2390 (time 1778656170.21 total 40.09 last 54.39)
 - open/close 3106 (time 1778656180.21 total 50.09 last 71.57)
 - open/close 3721 (time 1778656190.23 total 60.12 last 61.35)
 - open/close 4251 (time 1778656200.25 total 70.13 last 52.91)
total: 4500 open/close in 74.85 seconds: 60.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1265 (time 1778656221.13 total 10.01 last 126.42)
 - /unlink 2615 (time 1778656231.14 total 20.01 last 134.98)
 - /unlink 4109 (time 1778656241.14 total 30.01 last 149.37)
total: 4500 /unlink in 32.33 seconds: 139.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 813 (time 1778656258.32 total 10.01 last 81.24)
 - open/close 1527 (time 1778656268.34 total 20.04 last 71.19)
 - open/close 2307 (time 1778656278.35 total 30.04 last 77.98)
 - open/close 3091 (time 1778656288.35 total 40.04 last 78.38)
 - open/close 3754 (time 1778656298.36 total 50.05 last 66.25)
 - open/close 4385 (time 1778656308.36 total 60.06 last 63.06)
total: 4500 open/close in 61.96 seconds: 72.63 ops/second
 - /unlink 1230 (time 1778656327.27 total 10.01 last 122.93)
 - /unlink 2692 (time 1778656337.28 total 20.01 last 146.10)
 - /unlink 4031 (time 1778656347.28 total 30.01 last 133.90)
total: 4500 /unlink in 33.15 seconds: 135.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 820 (time 1778656364.55 total 10.00 last 81.97)
 - open/close 1487 (time 1778656374.56 total 20.02 last 66.62)
 - open/close 2289 (time 1778656384.57 total 30.03 last 80.11)
 - open/close 3135 (time 1778656394.59 total 40.04 last 84.49)
 - open/close 3944 (time 1778656404.59 total 50.05 last 80.85)
total: 4500 open/close in 57.46 seconds: 78.31 ops/second
 - /unlink 1207 (time 1778656428.26 total 10.00 last 120.68)
 - /unlink 2318 (time 1778656438.26 total 20.00 last 111.08)
 - /unlink 3842 (time 1778656448.26 total 30.01 last 152.37)
total: 4500 /unlink in 34.73 seconds: 129.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 765 (time 1778656468.90 total 10.00 last 76.48)
 - open/close 1457 (time 1778656478.90 total 20.01 last 69.18)
 - open/close 2112 (time 1778656488.92 total 30.03 last 65.35)
 - open/close 2790 (time 1778656498.93 total 40.04 last 67.74)
 - open/close 3470 (time 1778656508.93 total 50.04 last 67.99)
 - open/close 4087 (time 1778656518.96 total 60.06 last 61.55)
total: 4500 open/close in 65.58 seconds: 68.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1027 (time 1778656538.33 total 10.01 last 102.64)
 - /unlink 1863 (time 1778656548.36 total 20.04 last 83.32)
 - /unlink 3072 (time 1778656558.37 total 30.05 last 120.82)
 - /unlink 4339 (time 1778656568.38 total 40.06 last 126.54)
total: 4500 /unlink in 40.83 seconds: 110.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 751 (time 1778656584.42 total 10.01 last 74.99)
 - open/close 1436 (time 1778656594.42 total 20.02 last 68.47)
 - open/close 1952 (time 1778656604.42 total 30.02 last 51.59)
 - open/close 2485 (time 1778656614.43 total 40.03 last 53.27)
 - open/close 3214 (time 1778656624.44 total 50.03 last 72.85)
 - open/close 3879 (time 1778656634.47 total 60.06 last 66.30)
total: 4500 open/close in 69.63 seconds: 64.63 ops/second
 - /unlink 1424 (time 1778656659.17 total 10.01 last 142.27)
 - /unlink 2566 (time 1778656669.18 total 20.02 last 114.08)
 - /unlink 3768 (time 1778656679.18 total 30.02 last 120.15)
total: 4500 /unlink in 36.28 seconds: 124.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 671 (time 1778656701.28 total 10.05 last 66.77)
 - open/close 1389 (time 1778656711.29 total 20.06 last 71.70)
 - open/close 2171 (time 1778656721.30 total 30.07 last 78.16)
 - open/close 3056 (time 1778656731.30 total 40.08 last 88.43)
 - open/close 3689 (time 1778656741.31 total 50.08 last 63.27)
 - open/close 4448 (time 1778656751.32 total 60.09 last 75.82)
total: 4500 open/close in 61.30 seconds: 73.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1481 (time 1778656768.18 total 10.00 last 148.09)
 - /unlink 2896 (time 1778656778.18 total 20.00 last 141.47)
 - /unlink 4064 (time 1778656788.18 total 30.00 last 116.78)
total: 4500 /unlink in 34.86 seconds: 129.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 629 (time 1778656810.01 total 10.01 last 62.86)
 - open/close 1137 (time 1778656820.03 total 20.03 last 50.68)
 - open/close 1881 (time 1778656830.03 total 30.03 last 74.39)
 - open/close 2412 (time 1778656840.04 total 40.04 last 53.05)
 - open/close 2891 (time 1778656850.04 total 50.04 last 47.89)
 - open/close 3322 (time 1778656860.04 total 60.04 last 43.10)
 - open/close 4014 (time 1778656870.06 total 70.06 last 69.11)
total: 4500 open/close in 78.19 seconds: 57.55 ops/second
 - /unlink 1483 (time 1778656893.92 total 10.00 last 148.25)
 - /unlink 2693 (time 1778656903.93 total 20.01 last 120.98)
 - /unlink 3918 (time 1778656913.94 total 30.02 last 122.34)
total: 4500 /unlink in 34.35 seconds: 130.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 737 (time 1778656933.87 total 10.01 last 73.66)
 - open/close 1362 (time 1778656943.87 total 20.01 last 62.49)
 - open/close 2007 (time 1778656953.88 total 30.02 last 64.41)
 - open/close 2517 (time 1778656963.91 total 40.05 last 50.88)
 - open/close 2945 (time 1778656973.93 total 50.06 last 42.72)
 - open/close 3328 (time 1778656983.93 total 60.07 last 38.27)
 - open/close 3840 (time 1778656993.95 total 70.09 last 51.11)
 - open/close 4493 (time 1778657003.96 total 80.09 last 65.27)
total: 4500 open/close in 80.17 seconds: 56.13 ops/second
 - /unlink 1444 (time 1778657018.81 total 10.00 last 144.34)
 - /unlink 3052 (time 1778657028.82 total 20.01 last 160.66)
 - /unlink 4219 (time 1778657038.83 total 30.02 last 116.64)
total: 4500 /unlink in 32.91 seconds: 136.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 440 (time 1778657060.56 total 10.00 last 43.98)
 - open/close 1113 (time 1778657070.56 total 20.01 last 67.29)
 - open/close 1816 (time 1778657080.57 total 30.02 last 70.23)
 - open/close 2525 (time 1778657090.57 total 40.02 last 70.89)
 - open/close 3209 (time 1778657100.60 total 50.05 last 68.22)
 - open/close 3974 (time 1778657110.60 total 60.05 last 76.49)
total: 4500 open/close in 66.21 seconds: 67.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1534 (time 1778657131.42 total 10.00 last 153.38)
 - /unlink 2869 (time 1778657141.42 total 20.00 last 133.47)
 - /unlink 4232 (time 1778657151.43 total 30.01 last 136.22)
total: 4500 /unlink in 32.99 seconds: 136.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 629 (time 1778657170.40 total 10.01 last 62.87)
 - open/close 1174 (time 1778657180.40 total 20.01 last 54.48)
 - open/close 1897 (time 1778657190.41 total 30.02 last 72.22)
 - open/close 2684 (time 1778657200.42 total 40.03 last 78.64)
 - open/close 3474 (time 1778657210.43 total 50.03 last 78.96)
 - open/close 4275 (time 1778657220.43 total 60.04 last 80.05)
total: 4500 open/close in 63.16 seconds: 71.25 ops/second
 - /unlink 1630 (time 1778657238.66 total 10.00 last 162.99)
 - /unlink 2957 (time 1778657248.66 total 20.00 last 132.69)
 - /unlink 4267 (time 1778657258.66 total 30.00 last 131.00)
total: 4500 /unlink in 31.55 seconds: 142.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 484 (time 1778657275.52 total 10.02 last 48.32)
 - open/close 938 (time 1778657285.53 total 20.02 last 45.36)
 - open/close 1540 (time 1778657295.55 total 30.04 last 60.11)
 - open/close 2051 (time 1778657305.56 total 40.05 last 51.03)
 - open/close 2731 (time 1778657315.56 total 50.06 last 67.98)
 - open/close 3388 (time 1778657325.57 total 60.06 last 65.68)
 - open/close 4100 (time 1778657335.57 total 70.06 last 71.17)
total: 4500 open/close in 76.73 seconds: 58.65 ops/second
 - /unlink 834 (time 1778657358.36 total 10.01 last 83.35)
 - /unlink 1599 (time 1778657368.36 total 20.01 last 76.48)
 - /unlink 2782 (time 1778657378.37 total 30.01 last 118.29)
 - /unlink 3944 (time 1778657388.37 total 40.01 last 116.20)
total: 4500 /unlink in 45.38 seconds: 99.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 811 (time 1778657408.93 total 10.00 last 81.07)
 - open/close 1552 (time 1778657418.97 total 20.05 last 73.79)
 - open/close 2137 (time 1778657428.98 total 30.05 last 58.46)
 - open/close 2869 (time 1778657438.99 total 40.06 last 73.13)
 - open/close 3571 (time 1778657448.99 total 50.07 last 70.17)
 - open/close 4169 (time 1778657458.99 total 60.07 last 59.79)
total: 4500 open/close in 66.18 seconds: 67.99 ops/second
 - /unlink 1333 (time 1778657480.72 total 10.01 last 133.20)
 - /unlink 2426 (time 1778657490.72 total 20.01 last 109.23)
 - /unlink 3586 (time 1778657500.72 total 30.02 last 115.98)
total: 4500 /unlink in 39.26 seconds: 114.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 702 (time 1778657526.75 total 10.00 last 70.17)
 - open/close 1405 (time 1778657536.81 total 20.06 last 69.92)
 - open/close 2082 (time 1778657546.82 total 30.07 last 67.63)
 - open/close 2751 (time 1778657556.82 total 40.07 last 66.86)
 - open/close 3434 (time 1778657566.83 total 50.08 last 68.27)
 - open/close 4033 (time 1778657576.83 total 60.08 last 59.88)
total: 4500 open/close in 67.79 seconds: 66.38 ops/second
 - /unlink 862 (time 1778657599.83 total 10.00 last 86.16)
 - /unlink 2220 (time 1778657609.85 total 20.02 last 135.64)
 - /unlink 3755 (time 1778657619.85 total 30.02 last 153.48)
total: 4500 /unlink in 36.35 seconds: 123.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 569 (time 1778657640.76 total 10.02 last 56.81)
 - open/close 1312 (time 1778657650.77 total 20.02 last 74.27)
 - open/close 1986 (time 1778657660.77 total 30.03 last 67.34)
 - open/close 2818 (time 1778657670.78 total 40.03 last 83.18)
 - open/close 3597 (time 1778657680.78 total 50.04 last 77.85)
 - open/close 4354 (time 1778657690.78 total 60.04 last 75.70)
total: 4500 open/close in 61.94 seconds: 72.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1390 (time 1778657708.17 total 10.00 last 138.95)
 - /unlink 2728 (time 1778657718.17 total 20.00 last 133.79)
 - /unlink 4080 (time 1778657728.17 total 30.01 last 135.13)
total: 4500 /unlink in 33.23 seconds: 135.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 788 (time 1778657747.08 total 10.02 last 78.68)
 - open/close 1430 (time 1778657757.10 total 20.03 last 64.08)
 - open/close 2105 (time 1778657767.10 total 30.04 last 67.48)
 - open/close 2832 (time 1778657777.11 total 40.04 last 72.66)
 - open/close 3508 (time 1778657787.12 total 50.05 last 67.55)
 - open/close 4326 (time 1778657797.12 total 60.05 last 81.79)
total: 4500 open/close in 62.80 seconds: 71.66 ops/second
 - /unlink 1304 (time 1778657815.42 total 10.00 last 130.38)
 - /unlink 2535 (time 1778657825.43 total 20.01 last 123.01)
 - /unlink 3742 (time 1778657835.43 total 30.01 last 120.64)
total: 4500 /unlink in 36.41 seconds: 123.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 763 (time 1778657857.27 total 10.00 last 76.29)
 - open/close 1485 (time 1778657867.27 total 20.01 last 72.16)
 - open/close 2342 (time 1778657877.27 total 30.01 last 85.70)
 - open/close 3019 (time 1778657887.30 total 40.03 last 67.53)
 - open/close 3723 (time 1778657897.30 total 50.03 last 70.39)
 - open/close 4355 (time 1778657907.30 total 60.04 last 63.18)
total: 4500 open/close in 62.70 seconds: 71.77 ops/second
 - /unlink 1456 (time 1778657925.16 total 10.00 last 145.54)
 - /unlink 2764 (time 1778657935.17 total 20.01 last 130.69)
 - /unlink 4226 (time 1778657945.17 total 30.02 last 146.16)
total: 4500 /unlink in 32.56 seconds: 138.19 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 93951 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg211-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg211-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server
stop mds service on oleg211-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg211-server

unloading modules via unload_modules_local on: 'oleg211-server'
oleg211-server: oleg211-server.virtnet: executing unload_modules_local
oleg211-server: 
modules unloaded.