== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:11:13 (1788441073)
Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg424-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg424-server
oleg424-server: oleg424-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg424-server'
oleg424-server: oleg424-server.virtnet: executing load_modules_local
oleg424-server: kmemleak disabled
oleg424-server: Loading modules from /home/green/git/lustre-release/lustre
oleg424-server: detected 4 online CPUs by sysfs
oleg424-server: MODOPTS_LIBCFS=
oleg424-server: Force libcfs to create 2 CPU partitions
oleg424-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg424-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg424-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg424-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg424-client.virtnet:  -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 437 (time 1788441292.86 total 10.02 last 43.61)
 - open/close 961 (time 1788441302.87 total 20.03 last 52.35)
 - open/close 1571 (time 1788441312.89 total 30.05 last 60.84)
 - open/close 2179 (time 1788441322.90 total 40.06 last 60.75)
 - open/close 2793 (time 1788441332.90 total 50.06 last 61.40)
 - open/close 3391 (time 1788441342.90 total 60.07 last 59.78)
 - open/close 4013 (time 1788441352.91 total 70.07 last 62.17)
total: 4500 open/close in 78.76 seconds: 57.13 ops/second
 - /unlink 1240 (time 1788441377.33 total 10.01 last 123.87)
 - /unlink 2508 (time 1788441387.34 total 20.01 last 126.77)
 - /unlink 3704 (time 1788441397.34 total 30.01 last 119.58)
total: 4500 /unlink in 37.23 seconds: 120.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 552 (time 1788441420.18 total 10.00 last 55.19)
 - open/close 1172 (time 1788441430.18 total 20.01 last 61.95)
 - open/close 1798 (time 1788441440.19 total 30.01 last 62.59)
 - open/close 2439 (time 1788441450.19 total 40.01 last 64.09)
 - open/close 3118 (time 1788441460.19 total 50.01 last 67.88)
 - open/close 3779 (time 1788441470.20 total 60.02 last 66.05)
 - open/close 4431 (time 1788441480.21 total 70.04 last 65.10)
total: 4500 open/close in 71.05 seconds: 63.34 ops/second
 - /unlink 1089 (time 1788441496.41 total 10.01 last 108.85)
 - /unlink 2267 (time 1788441506.41 total 20.01 last 117.74)
 - /unlink 3476 (time 1788441516.42 total 30.02 last 120.81)
total: 4500 /unlink in 38.09 seconds: 118.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 648 (time 1788441540.95 total 10.01 last 64.72)
 - open/close 1266 (time 1788441550.96 total 20.03 last 61.72)
 - open/close 1891 (time 1788441560.97 total 30.04 last 62.41)
 - open/close 2526 (time 1788441570.98 total 40.04 last 63.48)
 - open/close 3169 (time 1788441580.98 total 50.05 last 64.25)
 - open/close 3728 (time 1788441590.99 total 60.05 last 55.87)
 - open/close 4272 (time 1788441600.99 total 70.06 last 54.38)
total: 4500 open/close in 74.10 seconds: 60.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1235 (time 1788441620.14 total 10.01 last 123.43)
 - /unlink 2425 (time 1788441630.14 total 20.01 last 118.93)
 - /unlink 3611 (time 1788441640.14 total 30.01 last 118.56)
total: 4500 /unlink in 37.19 seconds: 120.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 621 (time 1788441662.58 total 10.00 last 62.07)
 - open/close 1218 (time 1788441672.58 total 20.01 last 59.67)
 - open/close 1824 (time 1788441682.59 total 30.01 last 60.57)
 - open/close 2449 (time 1788441692.59 total 40.02 last 62.48)
 - open/close 3074 (time 1788441702.59 total 50.02 last 62.49)
 - open/close 3703 (time 1788441712.60 total 60.03 last 62.82)
 - open/close 4231 (time 1788441722.62 total 70.05 last 52.69)
total: 4500 open/close in 74.43 seconds: 60.46 ops/second
 - /unlink 1159 (time 1788441743.12 total 10.00 last 115.85)
 - /unlink 2250 (time 1788441753.12 total 20.01 last 109.07)
 - /unlink 3258 (time 1788441763.13 total 30.01 last 100.76)
 - /unlink 4324 (time 1788441773.13 total 40.01 last 106.56)
total: 4500 /unlink in 41.28 seconds: 109.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 577 (time 1788441789.51 total 10.00 last 57.69)
 - open/close 1244 (time 1788441799.52 total 20.02 last 66.61)
 - open/close 1897 (time 1788441809.52 total 30.02 last 65.26)
 - open/close 2538 (time 1788441819.53 total 40.03 last 64.08)
 - open/close 3172 (time 1788441829.55 total 50.05 last 63.24)
 - open/close 3859 (time 1788441839.57 total 60.07 last 68.59)
total: 4500 open/close in 68.87 seconds: 65.34 ops/second
 - /unlink 1218 (time 1788441862.96 total 10.00 last 121.80)
 - /unlink 2313 (time 1788441872.96 total 20.00 last 109.47)
 - /unlink 3561 (time 1788441882.98 total 30.02 last 124.60)
total: 4500 /unlink in 38.19 seconds: 117.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 643 (time 1788441906.36 total 10.01 last 64.25)
 - open/close 1302 (time 1788441916.37 total 20.02 last 65.80)
 - open/close 1943 (time 1788441926.38 total 30.02 last 64.09)
 - open/close 2587 (time 1788441936.39 total 40.04 last 64.30)
 - open/close 3237 (time 1788441946.40 total 50.05 last 64.91)
 - open/close 3907 (time 1788441956.42 total 60.07 last 66.90)
 - open/close 4472 (time 1788441966.43 total 70.08 last 56.46)
total: 4500 open/close in 70.55 seconds: 63.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1106 (time 1788441981.99 total 10.00 last 110.58)
 - /unlink 2200 (time 1788441991.99 total 20.01 last 109.33)
 - /unlink 3330 (time 1788442002.00 total 30.01 last 112.96)
total: 4500 /unlink in 39.77 seconds: 113.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 624 (time 1788442027.25 total 10.01 last 62.35)
 - open/close 1240 (time 1788442037.25 total 20.01 last 61.59)
 - open/close 1852 (time 1788442047.27 total 30.03 last 61.11)
 - open/close 2425 (time 1788442057.27 total 40.03 last 57.29)
 - open/close 2985 (time 1788442067.29 total 50.04 last 55.90)
 - open/close 3589 (time 1788442077.31 total 60.07 last 60.26)
 - open/close 4203 (time 1788442087.33 total 70.09 last 61.27)
total: 4500 open/close in 74.66 seconds: 60.27 ops/second
 - /unlink 1228 (time 1788442107.26 total 10.01 last 122.68)
 - /unlink 2409 (time 1788442117.26 total 20.01 last 118.07)
 - /unlink 3532 (time 1788442127.27 total 30.02 last 112.26)
total: 4500 /unlink in 37.83 seconds: 118.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 600 (time 1788442150.52 total 10.01 last 59.94)
 - open/close 1151 (time 1788442160.53 total 20.02 last 55.05)
 - open/close 1763 (time 1788442170.53 total 30.02 last 61.20)
 - open/close 2426 (time 1788442180.54 total 40.02 last 66.27)
 - open/close 2989 (time 1788442190.54 total 50.03 last 56.25)
 - open/close 3602 (time 1788442200.55 total 60.04 last 61.27)
 - open/close 4209 (time 1788442210.56 total 70.05 last 60.64)
total: 4500 open/close in 75.11 seconds: 59.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1133 (time 1788442231.63 total 10.00 last 113.25)
 - /unlink 2352 (time 1788442241.63 total 20.01 last 121.87)
 - /unlink 3622 (time 1788442251.64 total 30.01 last 126.89)
total: 4500 /unlink in 37.41 seconds: 120.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 662 (time 1788442273.85 total 10.01 last 66.17)
 - open/close 1271 (time 1788442283.85 total 20.01 last 60.89)
 - open/close 1991 (time 1788442293.87 total 30.03 last 71.84)
 - open/close 2599 (time 1788442303.88 total 40.03 last 60.77)
 - open/close 3206 (time 1788442313.88 total 50.04 last 60.67)
 - open/close 3890 (time 1788442323.90 total 60.06 last 68.27)
total: 4500 open/close in 70.05 seconds: 64.24 ops/second
 - /unlink 1153 (time 1788442349.46 total 10.00 last 115.28)
 - /unlink 2167 (time 1788442359.47 total 20.01 last 101.32)
 - /unlink 3068 (time 1788442369.47 total 30.01 last 90.07)
 - /unlink 4213 (time 1788442379.47 total 40.01 last 114.50)
total: 4500 /unlink in 42.63 seconds: 105.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 634 (time 1788442396.72 total 10.00 last 63.38)
 - open/close 1215 (time 1788442406.73 total 20.01 last 58.06)
 - open/close 1848 (time 1788442416.74 total 30.02 last 63.22)
 - open/close 2474 (time 1788442426.75 total 40.03 last 62.55)
 - open/close 3099 (time 1788442436.76 total 50.04 last 62.45)
 - open/close 3721 (time 1788442446.76 total 60.04 last 62.19)
 - open/close 4298 (time 1788442456.77 total 70.06 last 57.61)
total: 4500 open/close in 73.47 seconds: 61.25 ops/second
 - /unlink 1293 (time 1788442475.16 total 10.01 last 129.17)
 - /unlink 2392 (time 1788442485.17 total 20.01 last 109.85)
 - /unlink 3599 (time 1788442495.17 total 30.02 last 120.66)
total: 4500 /unlink in 37.95 seconds: 118.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 617 (time 1788442518.32 total 10.00 last 61.69)
 - open/close 1241 (time 1788442528.32 total 20.00 last 62.39)
 - open/close 1862 (time 1788442538.34 total 30.01 last 62.03)
 - open/close 2498 (time 1788442548.36 total 40.03 last 63.47)
 - open/close 3145 (time 1788442558.37 total 50.05 last 64.62)
 - open/close 3746 (time 1788442568.39 total 60.07 last 59.99)
 - open/close 4446 (time 1788442578.39 total 70.07 last 69.98)
total: 4500 open/close in 70.70 seconds: 63.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1229 (time 1788442593.72 total 10.02 last 122.71)
 - /unlink 2433 (time 1788442603.73 total 20.02 last 120.32)
 - /unlink 3640 (time 1788442613.73 total 30.03 last 120.59)
total: 4500 /unlink in 37.15 seconds: 121.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 623 (time 1788442635.69 total 10.01 last 62.24)
 - open/close 1265 (time 1788442645.69 total 20.02 last 64.16)
 - open/close 1864 (time 1788442655.71 total 30.03 last 59.81)
 - open/close 2532 (time 1788442665.71 total 40.04 last 66.77)
 - open/close 3185 (time 1788442675.73 total 50.05 last 65.20)
 - open/close 3819 (time 1788442685.74 total 60.06 last 63.36)
 - open/close 4471 (time 1788442695.75 total 70.07 last 65.13)
total: 4500 open/close in 70.65 seconds: 63.69 ops/second
 - /unlink 1233 (time 1788442712.31 total 10.00 last 123.27)
 - /unlink 2442 (time 1788442722.33 total 20.02 last 120.68)
 - /unlink 3639 (time 1788442732.33 total 30.02 last 119.69)
total: 4500 /unlink in 37.19 seconds: 121.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 529 (time 1788442754.87 total 10.00 last 52.89)
 - open/close 1153 (time 1788442764.88 total 20.01 last 62.34)
 - open/close 1722 (time 1788442774.88 total 30.02 last 56.87)
 - open/close 2294 (time 1788442784.89 total 40.03 last 57.14)
 - open/close 2891 (time 1788442794.89 total 50.03 last 59.69)
 - open/close 3494 (time 1788442804.90 total 60.04 last 60.25)
 - open/close 4117 (time 1788442814.91 total 70.05 last 62.23)
total: 4500 open/close in 76.10 seconds: 59.13 ops/second
 - /unlink 1147 (time 1788442836.08 total 10.01 last 114.61)
 - /unlink 2249 (time 1788442846.09 total 20.02 last 110.11)
 - /unlink 3450 (time 1788442856.09 total 30.02 last 120.08)
total: 4500 /unlink in 38.46 seconds: 116.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 620 (time 1788442880.19 total 10.01 last 61.94)
 - open/close 1267 (time 1788442890.22 total 20.04 last 64.52)
 - open/close 1896 (time 1788442900.24 total 30.05 last 62.82)
 - open/close 2444 (time 1788442910.27 total 40.09 last 54.59)
 - open/close 3051 (time 1788442920.28 total 50.09 last 60.67)
 - open/close 3630 (time 1788442930.28 total 60.09 last 57.90)
 - open/close 4144 (time 1788442940.33 total 70.14 last 51.15)
total: 4500 open/close in 75.69 seconds: 59.45 ops/second
 - /unlink 1240 (time 1788442961.77 total 10.01 last 123.90)
 - /unlink 2467 (time 1788442971.78 total 20.02 last 122.53)
 - /unlink 3709 (time 1788442981.79 total 30.03 last 124.10)
total: 4500 /unlink in 37.18 seconds: 121.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 624 (time 1788443004.54 total 10.00 last 62.39)
 - open/close 1177 (time 1788443014.55 total 20.01 last 55.25)
 - open/close 1786 (time 1788443024.56 total 30.01 last 60.87)
 - open/close 2497 (time 1788443034.56 total 40.02 last 71.07)
 - open/close 3103 (time 1788443044.58 total 50.04 last 60.50)
 - open/close 3683 (time 1788443054.59 total 60.04 last 57.96)
 - open/close 4312 (time 1788443064.59 total 70.05 last 62.89)
total: 4500 open/close in 72.71 seconds: 61.89 ops/second
 - /unlink 1306 (time 1788443082.41 total 10.00 last 130.60)
 - /unlink 2415 (time 1788443092.42 total 20.01 last 110.81)
 - /unlink 3465 (time 1788443102.43 total 30.02 last 104.92)
total: 4500 /unlink in 38.65 seconds: 116.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 500 (time 1788443127.58 total 10.01 last 49.97)
 - open/close 1074 (time 1788443137.58 total 20.01 last 57.36)
 - open/close 1732 (time 1788443147.58 total 30.01 last 65.80)
 - open/close 2315 (time 1788443157.59 total 40.02 last 58.26)
 - open/close 2926 (time 1788443167.59 total 50.02 last 61.10)
 - open/close 3456 (time 1788443177.60 total 60.03 last 52.94)
 - open/close 4121 (time 1788443187.61 total 70.05 last 66.41)
total: 4500 open/close in 76.70 seconds: 58.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1246 (time 1788443210.66 total 10.01 last 124.49)
 - /unlink 2410 (time 1788443220.67 total 20.01 last 116.35)
 - /unlink 3539 (time 1788443230.67 total 30.01 last 112.89)
total: 4500 /unlink in 37.61 seconds: 119.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 604 (time 1788443254.75 total 10.03 last 60.22)
 - open/close 1123 (time 1788443264.76 total 20.03 last 51.89)
 - open/close 1711 (time 1788443274.77 total 30.04 last 58.73)
 - open/close 2281 (time 1788443284.77 total 40.04 last 57.00)
 - open/close 2857 (time 1788443294.78 total 50.05 last 57.54)
 - open/close 3413 (time 1788443304.80 total 60.07 last 55.50)
 - open/close 3985 (time 1788443314.80 total 70.08 last 57.17)
total: 4500 open/close in 79.51 seconds: 56.60 ops/second
 - /unlink 1054 (time 1788443340.10 total 10.00 last 105.40)
 - /unlink 2307 (time 1788443350.11 total 20.01 last 125.13)
 - /unlink 3465 (time 1788443360.12 total 30.02 last 115.74)
total: 4500 /unlink in 38.83 seconds: 115.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 648 (time 1788443385.00 total 10.02 last 64.70)
 - open/close 1324 (time 1788443395.01 total 20.02 last 67.56)
 - open/close 1902 (time 1788443405.02 total 30.04 last 57.71)
 - open/close 2499 (time 1788443415.02 total 40.04 last 59.68)
 - open/close 3130 (time 1788443425.03 total 50.05 last 63.04)
 - open/close 3655 (time 1788443435.03 total 60.05 last 52.50)
 - open/close 4260 (time 1788443445.03 total 70.05 last 60.50)
total: 4500 open/close in 74.44 seconds: 60.45 ops/second
 - /unlink 1283 (time 1788443465.56 total 10.00 last 128.29)
 - /unlink 2489 (time 1788443475.56 total 20.00 last 120.60)
 - /unlink 3681 (time 1788443485.56 total 30.00 last 119.17)
total: 4500 /unlink in 37.68 seconds: 119.41 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95338 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg424-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg424-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server
stop mds service on oleg424-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server
stop mds service on oleg424-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server
unloading modules via unload_modules_local on: 'oleg424-server'
oleg424-server: oleg424-server.virtnet: executing unload_modules_local
oleg424-server: kmemleak disabled
modules unloaded.