== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:35:41 (1781588141)
Stopping clients: oleg649-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg649-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg649-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg649-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg649-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg649-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg649-server
oleg649-server: oleg649-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg649-server'
oleg649-server: oleg649-server.virtnet: executing load_modules_local
oleg649-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
oleg649-server: Loading modules from /home/green/git/lustre-release/lustre
oleg649-server: detected 4 online CPUs by sysfs
oleg649-server: MODOPTS_LIBCFS=
oleg649-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg649-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all
pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg649-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all
pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg649-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all
pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg649-client.virtnet:  -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 642 (time 1781588386.03 total 10.00 last 64.19)
 - open/close 1281 (time 1781588396.03 total 20.00 last 63.89)
 - open/close 1918 (time 1781588406.04 total 30.01 last 63.67)
 - open/close 2440 (time 1781588416.05 total 40.02 last 52.14)
 - open/close 3037 (time 1781588426.05 total 50.02 last 59.67)
 - open/close 3573 (time 1781588436.05 total 60.02 last 53.59)
 - open/close 4119 (time 1781588446.05 total 70.03 last 54.59)
total: 4500 open/close in 76.24 seconds: 59.02 ops/second
 - /unlink 1336 (time 1781588467.34 total 10.00 last 133.58)
 - /unlink 2309 (time 1781588477.35 total 20.00 last 97.27)
 - /unlink 3424 (time 1781588487.36 total 30.01 last 111.39)
 - /unlink 4447 (time 1781588497.37 total 40.03 last 102.15)
total: 4500 /unlink in 40.71 seconds: 110.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 557 (time 1781588513.43 total 10.00 last 55.68)
 - open/close 1216 (time 1781588523.43 total 20.01 last 65.89)
 - open/close 1791 (time 1781588533.43 total 30.01 last 57.49)
 - open/close 2437 (time 1781588543.44 total 40.02 last 64.55)
 - open/close 3002 (time 1781588553.45 total 50.03 last 56.43)
 - open/close 3609 (time 1781588563.46 total 60.04 last 60.65)
 - open/close 4069 (time 1781588573.47 total 70.05 last 45.94)
total: 4500 open/close in 78.91 seconds: 57.03 ops/second
 - /unlink 980 (time 1781588598.10 total 10.00 last 97.96)
 - /unlink 1928 (time 1781588608.11 total 20.01 last 94.75)
 - /unlink 3000 (time 1781588618.11 total 30.01 last 107.17)
 - /unlink 4144 (time 1781588628.11 total 40.02 last 114.35)
total: 4500 /unlink in 43.49 seconds: 103.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 564 (time 1781588647.54 total 10.01 last 56.35)
 - open/close 1105 (time 1781588657.54 total 20.02 last 54.05)
 - open/close 1674 (time 1781588667.55 total 30.02 last 56.88)
 - open/close 2246 (time 1781588677.56 total 40.03 last 57.17)
 - open/close 2784 (time 1781588687.58 total 50.05 last 53.68)
 - open/close 3441 (time 1781588697.60 total 60.08 last 65.54)
 - open/close 4064 (time 1781588707.61 total 70.08 last 62.27)
total: 4500 open/close in 77.29 seconds: 58.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1112 (time 1781588729.60 total 10.00 last 111.19)
 - /unlink 2274 (time 1781588739.60 total 20.00 last 116.16)
 - /unlink 3427 (time 1781588749.61 total 30.01 last 115.26)
total: 4500 /unlink in 39.37 seconds: 114.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 651 (time 1781588773.82 total 10.00 last 65.08)
 - open/close 1345 (time 1781588783.82 total 20.01 last 69.38)
 - open/close 1920 (time 1781588793.83 total 30.01 last 57.48)
 - open/close 2559 (time 1781588803.84 total 40.02 last 63.83)
 - open/close 3253 (time 1781588813.84 total 50.02 last 69.38)
 - open/close 3922 (time 1781588823.85 total 60.03 last 66.85)
total: 4500 open/close in 68.70 seconds: 65.50 ops/second
 - /unlink 1312 (time 1781588847.49 total 10.02 last 131.00)
 - /unlink 2493 (time 1781588857.49 total 20.02 last 118.08)
 - /unlink 3487 (time 1781588867.49 total 30.02 last 99.38)
total: 4500 /unlink in 38.97 seconds: 115.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 694 (time 1781588891.17 total 10.01 last 69.34)
 - open/close 1320 (time 1781588901.18 total 20.02 last 62.54)
 - open/close 2075 (time 1781588911.18 total 30.02 last 75.48)
 - open/close 2824 (time 1781588921.20 total 40.04 last 74.78)
 - open/close 3530 (time 1781588931.20 total 50.04 last 70.55)
 - open/close 4283 (time 1781588941.21 total 60.05 last 75.26)
total: 4500 open/close in 63.37 seconds: 71.02 ops/second
 - /unlink 1229 (time 1781588958.60 total 10.00 last 122.85)
 - /unlink 2337 (time 1781588968.61 total 20.01 last 110.70)
 - /unlink 3472 (time 1781588978.62 total 30.02 last 113.38)
total: 4500 /unlink in 39.77 seconds: 113.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 601 (time 1781589003.23 total 10.01 last 60.04)
 - open/close 1190 (time 1781589013.25 total 20.02 last 58.82)
 - open/close 1838 (time 1781589023.25 total 30.03 last 64.78)
 - open/close 2500 (time 1781589033.25 total 40.03 last 66.18)
 - open/close 3091 (time 1781589043.27 total 50.05 last 59.00)
 - open/close 3697 (time 1781589053.27 total 60.05 last 60.58)
 - open/close 4258 (time 1781589063.28 total 70.06 last 56.06)
total: 4500 open/close in 73.93 seconds: 60.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1076 (time 1781589082.85 total 10.00 last 107.57)
 - /unlink 2060 (time 1781589092.86 total 20.01 last 98.34)
 - /unlink 3075 (time 1781589102.86 total 30.02 last 101.43)
 - /unlink 3907 (time 1781589112.87 total 40.02 last 83.16)
total: 4500 /unlink in 45.68 seconds: 98.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 498 (time 1781589133.42 total 10.01 last 49.75)
 - open/close 1058 (time 1781589143.44 total 20.03 last 55.90)
 - open/close 1548 (time 1781589153.45 total 30.04 last 48.95)
 - open/close 2072 (time 1781589163.46 total 40.04 last 52.36)
 - open/close 2669 (time 1781589173.48 total 50.06 last 59.58)
 - open/close 3290 (time 1781589183.48 total 60.07 last 62.06)
 - open/close 3923 (time 1781589193.50 total 70.09 last 63.21)
total: 4500 open/close in 78.33 seconds: 57.45 ops/second
 - /unlink 1155 (time 1781589216.35 total 10.01 last 115.42)
 - /unlink 2262 (time 1781589226.35 total 20.02 last 110.60)
 - /unlink 3413 (time 1781589236.37 total 30.03 last 114.97)
 - /unlink 4478 (time 1781589246.38 total 40.04 last 106.39)
total: 4500 /unlink in 40.22 seconds: 111.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 671 (time 1781589261.17 total 10.02 last 66.98)
 - open/close 1299 (time 1781589271.18 total 20.02 last 62.75)
 - open/close 1919 (time 1781589281.19 total 30.03 last 61.94)
 - open/close 2584 (time 1781589291.21 total 40.05 last 66.37)
 - open/close 3253 (time 1781589301.21 total 50.05 last 66.90)
 - open/close 3876 (time 1781589311.22 total 60.06 last 62.24)
 - open/close 4470 (time 1781589321.22 total 70.07 last 59.38)
total: 4500 open/close in 70.50 seconds: 63.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1263 (time 1781589335.48 total 10.01 last 126.11)
 - /unlink 2372 (time 1781589345.48 total 20.02 last 110.88)
 - /unlink 3504 (time 1781589355.49 total 30.02 last 113.13)
total: 4500 /unlink in 38.12 seconds: 118.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 675 (time 1781589379.11 total 10.00 last 67.47)
 - open/close 1316 (time 1781589389.12 total 20.01 last 64.03)
 - open/close 1924 (time 1781589399.14 total 30.03 last 60.73)
 - open/close 2531 (time 1781589409.15 total 40.04 last 60.64)
 - open/close 3134 (time 1781589419.16 total 50.05 last 60.23)
 - open/close 3792 (time 1781589429.16 total 60.05 last 65.77)
 - open/close 4381 (time 1781589439.17 total 70.06 last 58.86)
total: 4500 open/close in 72.15 seconds: 62.37 ops/second
 - /unlink 1036 (time 1781589457.50 total 10.00 last 103.59)
 - /unlink 2214 (time 1781589467.50 total 20.00 last 117.75)
 - /unlink 3256 (time 1781589477.51 total 30.02 last 104.08)
 - /unlink 4465 (time 1781589487.52 total 40.02 last 120.84)
total: 4500 /unlink in 40.45 seconds: 111.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 605 (time 1781589503.01 total 10.00 last 60.49)
 - open/close 1268 (time 1781589513.02 total 20.01 last 66.21)
 - open/close 1899 (time 1781589523.02 total 30.02 last 63.08)
 - open/close 2480 (time 1781589533.03 total 40.02 last 58.08)
 - open/close 3077 (time 1781589543.03 total 50.03 last 59.66)
 - open/close 3680 (time 1781589553.05 total 60.04 last 60.24)
 - open/close 4275 (time 1781589563.05 total 70.05 last 59.45)
total: 4500 open/close in 73.33 seconds: 61.37 ops/second
 - /unlink 1217 (time 1781589580.86 total 10.00 last 121.66)
 - /unlink 2502 (time 1781589590.86 total 20.01 last 128.46)
 - /unlink 3545 (time 1781589600.87 total 30.01 last 104.28)
total: 4500 /unlink in 38.09 seconds: 118.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 655 (time 1781589625.14 total 10.00 last 65.49)
 - open/close 1262 (time 1781589635.16 total 20.02 last 60.58)
 - open/close 1849 (time 1781589645.16 total 30.02 last 58.68)
 - open/close 2475 (time 1781589655.17 total 40.04 last 62.52)
 - open/close 3042 (time 1781589665.19 total 50.05 last 56.63)
 - open/close 3707 (time 1781589675.19 total 60.05 last 66.49)
 - open/close 4372 (time 1781589685.20 total 70.07 last 66.39)
total: 4500 open/close in 73.40 seconds: 61.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1306 (time 1781589704.66 total 10.01 last 130.41)
 - /unlink 2405 (time 1781589714.66 total 20.02 last 109.85)
 - /unlink 3608 (time 1781589724.66 total 30.02 last 120.29)
total: 4500 /unlink in 37.38 seconds: 120.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 696 (time 1781589747.06 total 10.01 last 69.55)
 - open/close 1422 (time 1781589757.08 total 20.02 last 72.52)
 - open/close 2035 (time 1781589767.08 total 30.03 last 61.25)
 - open/close 2610 (time 1781589777.08 total 40.03 last 57.50)
 - open/close 3239 (time 1781589787.09 total 50.04 last 62.84)
 - open/close 3809 (time 1781589797.12 total 60.07 last 56.83)
 - open/close 4321 (time 1781589807.14 total 70.08 last 51.13)
total: 4500 open/close in 73.91 seconds: 60.89 ops/second
 - /unlink 1044 (time 1781589826.88 total 10.00 last 104.38)
 - /unlink 1998 (time 1781589836.89 total 20.01 last 95.29)
 - /unlink 3119 (time 1781589846.90 total 30.03 last 111.96)
 - /unlink 4303 (time 1781589856.90 total 40.03 last 118.38)
total: 4500 /unlink in 41.64 seconds: 108.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 661 (time 1781589873.75 total 10.01 last 66.03)
 - open/close 1318 (time 1781589883.77 total 20.03 last 65.60)
 - open/close 2067 (time 1781589893.77 total 30.03 last 74.89)
 - open/close 2708 (time 1781589903.77 total 40.03 last 64.06)
 - open/close 3279 (time 1781589913.78 total 50.04 last 57.05)
 - open/close 3920 (time 1781589923.80 total 60.06 last 64.02)
total: 4500 open/close in 69.18 seconds: 65.05 ops/second
 - /unlink 1068 (time 1781589948.50 total 10.00 last 106.79)
 - /unlink 2220 (time 1781589958.51 total 20.01 last 115.15)
 - /unlink 3317 (time 1781589968.51 total 30.01 last 109.65)
total: 4500 /unlink in 39.93 seconds: 112.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 630 (time 1781589993.57 total 10.02 last 62.89)
 - open/close 1230 (time 1781590003.58 total 20.02 last 59.98)
 - open/close 1899 (time 1781590013.59 total 30.03 last 66.83)
 - open/close 2407 (time 1781590023.60 total 40.05 last 50.71)
 - open/close 3054 (time 1781590033.62 total 50.06 last 64.61)
 - open/close 3599 (time 1781590043.63 total 60.07 last 54.46)
 - open/close 4108 (time 1781590053.63 total 70.08 last 50.86)
total: 4500 open/close in 76.41 seconds: 58.90 ops/second
 - /unlink 1209 (time 1781590075.11 total 10.01 last 120.80)
 - /unlink 2324 (time 1781590085.12 total 20.01 last 111.45)
 - /unlink 3519 (time 1781590095.12 total 30.02 last 119.46)
total: 4500 /unlink in 38.59 seconds: 116.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 622 (time 1781590118.55 total 10.01 last 62.14)
 - open/close 1224 (time 1781590128.55 total 20.01 last 60.19)
 - open/close 1885 (time 1781590138.55 total 30.01 last 66.09)
 - open/close 2586 (time 1781590148.56 total 40.02 last 70.03)
 - open/close 3211 (time 1781590158.59 total 50.04 last 62.37)
 - open/close 3907 (time 1781590168.62 total 60.08 last 69.38)
total: 4500 open/close in 69.99 seconds: 64.30 ops/second
 - /unlink 1076 (time 1781590192.70 total 10.01 last 107.54)
 - /unlink 2128 (time 1781590202.71 total 20.01 last 105.13)
 - /unlink 3263 (time 1781590212.71 total 30.01 last 113.49)
 - /unlink 4163 (time 1781590222.72 total 40.02 last 89.95)
total: 4500 /unlink in 43.34 seconds: 103.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 615 (time 1781590242.10 total 10.00 last 61.47)
 - open/close 1273 (time 1781590252.11 total 20.01 last 65.78)
 - open/close 1929 (time 1781590262.11 total 30.02 last 65.55)
 - open/close 2593 (time 1781590272.12 total 40.02 last 66.39)
 - open/close 3283 (time 1781590282.14 total 50.04 last 68.85)
 - open/close 3882 (time 1781590292.16 total 60.06 last 59.79)
total: 4500 open/close in 69.91 seconds: 64.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1123 (time 1781590316.95 total 10.01 last 112.16)
 - /unlink 2248 (time 1781590326.95 total 20.02 last 112.47)
 - /unlink 3415 (time 1781590336.96 total 30.02 last 116.63)
 - /unlink 4492 (time 1781590346.96 total 40.02 last 107.66)
total: 4500 /unlink in 40.07 seconds: 112.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 696 (time 1781590363.26 total 10.00 last 69.58)
 - open/close 1315 (time 1781590373.28 total 20.03 last 61.75)
 - open/close 1963 (time 1781590383.32 total 30.06 last 64.58)
 - open/close 2626 (time 1781590393.34 total 40.08 last 66.16)
 - open/close 3255 (time 1781590403.35 total 50.09 last 62.83)
 - open/close 3849 (time 1781590413.35 total 60.10 last 59.39)
 - open/close 4435 (time 1781590423.36 total 70.10 last 58.56)
total: 4500 open/close in 71.36 seconds: 63.06 ops/second
 - /unlink 1202 (time 1781590439.71 total 10.00 last 120.18)
 - /unlink 2105 (time 1781590449.73 total 20.02 last 90.17)
 - /unlink 3061 (time 1781590459.73 total 30.02 last 95.57)
 - /unlink 4157 (time 1781590469.73 total 40.02 last 109.55)
total: 4500 /unlink in 43.25 seconds: 104.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 604 (time 1781590488.08 total 10.00 last 60.38)
 - open/close 1210 (time 1781590498.11 total 20.02 last 60.47)
 - open/close 1681 (time 1781590508.11 total 30.03 last 47.06)
 - open/close 2344 (time 1781590518.12 total 40.03 last 66.28)
 - open/close 3030 (time 1781590528.13 total 50.05 last 68.53)
 - open/close 3686 (time 1781590538.13 total 60.05 last 65.58)
 - open/close 4375 (time 1781590548.13 total 70.05 last 68.86)
total: 4500 open/close in 72.56 seconds: 62.02 ops/second
 - /unlink 1310 (time 1781590565.73 total 10.01 last 130.84)
 - /unlink 2477 (time 1781590575.74 total 20.02 last 116.62)
 - /unlink 3534 (time 1781590585.74 total 30.02 last 105.70)
total: 4500 /unlink in 39.91 seconds: 112.75 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11225: 95147 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg649-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg649-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg649-server
stop mds service on oleg649-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg649-server
stop mds service on oleg649-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg649-server

unloading modules via unload_modules_local on: 'oleg649-server'
oleg649-server: oleg649-server.virtnet: executing unload_modules_local
oleg649-server: 
modules unloaded.