== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:44:17 (1788623057)
Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg407-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg407-server
oleg407-server: oleg407-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg407-server'
oleg407-server: oleg407-server.virtnet: executing load_modules_local
oleg407-server: kmemleak disabled
oleg407-server: Loading modules from /home/green/git/lustre-release/lustre
oleg407-server: detected 4 online CPUs by sysfs
oleg407-server: MODOPTS_LIBCFS=
oleg407-server: Force libcfs to create 2 CPU partitions
oleg407-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg407-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all
pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg407-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all
pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg407-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all
pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg407-client.virtnet:  -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 718 (time 1788623272.24 total 10.01 last 71.72)
 - open/close 1406 (time 1788623282.25 total 20.02 last 68.74)
 - open/close 2057 (time 1788623292.26 total 30.03 last 65.05)
 - open/close 2702 (time 1788623302.26 total 40.03 last 64.49)
 - open/close 3410 (time 1788623312.26 total 50.03 last 70.79)
 - open/close 4111 (time 1788623322.27 total 60.04 last 70.03)
total: 4500 open/close in 65.79 seconds: 68.40 ops/second
 - /unlink 1299 (time 1788623343.19 total 10.01 last 129.83)
 - /unlink 2703 (time 1788623353.20 total 20.01 last 140.29)
 - /unlink 3992 (time 1788623363.21 total 30.02 last 128.82)
total: 4500 /unlink in 33.69 seconds: 133.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 671 (time 1788623382.07 total 10.00 last 67.08)
 - open/close 1192 (time 1788623392.08 total 20.01 last 52.08)
 - open/close 1788 (time 1788623402.09 total 30.02 last 59.52)
 - open/close 2316 (time 1788623412.10 total 40.03 last 52.75)
 - open/close 2802 (time 1788623422.10 total 50.03 last 48.60)
 - open/close 3335 (time 1788623432.11 total 60.04 last 53.24)
 - open/close 3972 (time 1788623442.12 total 70.05 last 63.66)
total: 4500 open/close in 77.54 seconds: 58.04 ops/second
 - /unlink 1298 (time 1788623464.91 total 10.01 last 129.69)
 - /unlink 2641 (time 1788623474.92 total 20.02 last 134.13)
 - /unlink 3867 (time 1788623484.92 total 30.02 last 122.58)
total: 4500 /unlink in 36.00 seconds: 125.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 633 (time 1788623506.99 total 10.00 last 63.27)
 - open/close 1250 (time 1788623517.01 total 20.02 last 61.61)
 - open/close 1852 (time 1788623527.02 total 30.03 last 60.14)
 - open/close 2484 (time 1788623537.03 total 40.04 last 63.14)
 - open/close 3099 (time 1788623547.03 total 50.04 last 61.49)
 - open/close 3695 (time 1788623557.04 total 60.05 last 59.54)
 - open/close 4412 (time 1788623567.04 total 70.05 last 71.68)
total: 4500 open/close in 71.24 seconds: 63.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1239 (time 1788623583.18 total 10.00 last 123.88)
 - /unlink 2500 (time 1788623593.19 total 20.01 last 126.02)
 - /unlink 3671 (time 1788623603.19 total 30.01 last 117.06)
total: 4500 /unlink in 37.01 seconds: 121.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 633 (time 1788623625.21 total 10.01 last 63.26)
 - open/close 1222 (time 1788623635.22 total 20.02 last 58.84)
 - open/close 1932 (time 1788623645.23 total 30.02 last 70.98)
 - open/close 2599 (time 1788623655.24 total 40.03 last 66.63)
 - open/close 3261 (time 1788623665.24 total 50.03 last 66.17)
 - open/close 3970 (time 1788623675.24 total 60.03 last 70.90)
total: 4500 open/close in 68.13 seconds: 66.05 ops/second
 - /unlink 1196 (time 1788623698.69 total 10.02 last 119.38)
 - /unlink 2376 (time 1788623708.70 total 20.03 last 117.90)
 - /unlink 3537 (time 1788623718.70 total 30.03 last 116.09)
total: 4500 /unlink in 37.02 seconds: 121.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 737 (time 1788623740.66 total 10.01 last 73.65)
 - open/close 1393 (time 1788623750.67 total 20.01 last 65.58)
 - open/close 2036 (time 1788623760.69 total 30.04 last 64.13)
 - open/close 2653 (time 1788623770.70 total 40.05 last 61.64)
 - open/close 3290 (time 1788623780.71 total 50.05 last 63.65)
 - open/close 3980 (time 1788623790.72 total 60.07 last 68.92)
total: 4500 open/close in 68.07 seconds: 66.11 ops/second
 - /unlink 851 (time 1788623813.49 total 10.00 last 85.08)
 - /unlink 1741 (time 1788623823.49 total 20.01 last 88.97)
 - /unlink 2941 (time 1788623833.49 total 30.01 last 119.98)
 - /unlink 4108 (time 1788623843.50 total 40.01 last 116.65)
total: 4500 /unlink in 42.90 seconds: 104.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 672 (time 1788623863.05 total 10.01 last 67.10)
 - open/close 1335 (time 1788623873.07 total 20.03 last 66.23)
 - open/close 1968 (time 1788623883.08 total 30.04 last 63.23)
 - open/close 2604 (time 1788623893.09 total 40.05 last 63.49)
 - open/close 3257 (time 1788623903.10 total 50.06 last 65.29)
 - open/close 3864 (time 1788623913.10 total 60.06 last 60.65)
total: 4500 open/close in 69.85 seconds: 64.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1081 (time 1788623938.55 total 10.00 last 108.10)
 - /unlink 2203 (time 1788623948.56 total 20.01 last 112.14)
 - /unlink 3198 (time 1788623958.56 total 30.01 last 99.50)
 - /unlink 4313 (time 1788623968.56 total 40.01 last 111.49)
total: 4500 /unlink in 42.07 seconds: 106.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 623 (time 1788623987.13 total 10.01 last 62.24)
 - open/close 1154 (time 1788623997.14 total 20.01 last 53.09)
 - open/close 1756 (time 1788624007.14 total 30.01 last 60.20)
 - open/close 2438 (time 1788624017.14 total 40.02 last 68.18)
 - open/close 3084 (time 1788624027.15 total 50.03 last 64.51)
 - open/close 3670 (time 1788624037.15 total 60.03 last 58.59)
 - open/close 4283 (time 1788624047.17 total 70.05 last 61.17)
total: 4500 open/close in 73.40 seconds: 61.31 ops/second
 - /unlink 1034 (time 1788624066.32 total 10.01 last 103.34)
 - /unlink 2333 (time 1788624076.32 total 20.01 last 129.84)
 - /unlink 3578 (time 1788624086.33 total 30.02 last 124.44)
total: 4500 /unlink in 38.59 seconds: 116.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 580 (time 1788624110.44 total 10.01 last 57.97)
 - open/close 1125 (time 1788624120.45 total 20.01 last 54.47)
 - open/close 1754 (time 1788624130.46 total 30.02 last 62.82)
 - open/close 2402 (time 1788624140.47 total 40.03 last 64.76)
 - open/close 3040 (time 1788624150.47 total 50.04 last 63.76)
 - open/close 3644 (time 1788624160.49 total 60.05 last 60.31)
 - open/close 4238 (time 1788624170.50 total 70.06 last 59.34)
total: 4500 open/close in 73.88 seconds: 60.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1123 (time 1788624191.74 total 10.01 last 112.15)
 - /unlink 2253 (time 1788624201.75 total 20.02 last 112.93)
 - /unlink 3390 (time 1788624211.75 total 30.02 last 113.66)
total: 4500 /unlink in 39.53 seconds: 113.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 614 (time 1788624237.35 total 10.01 last 61.34)
 - open/close 1205 (time 1788624247.36 total 20.01 last 59.07)
 - open/close 1777 (time 1788624257.38 total 30.04 last 57.07)
 - open/close 2335 (time 1788624267.40 total 40.06 last 55.66)
 - open/close 2892 (time 1788624277.41 total 50.07 last 55.67)
 - open/close 3470 (time 1788624287.42 total 60.07 last 57.77)
 - open/close 4089 (time 1788624297.42 total 70.07 last 61.88)
total: 4500 open/close in 77.34 seconds: 58.19 ops/second
 - /unlink 1091 (time 1788624319.94 total 10.00 last 109.07)
 - /unlink 2271 (time 1788624329.95 total 20.02 last 117.81)
 - /unlink 3219 (time 1788624339.96 total 30.03 last 94.73)
 - /unlink 4092 (time 1788624349.97 total 40.04 last 87.22)
total: 4500 /unlink in 44.31 seconds: 101.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 667 (time 1788624370.16 total 10.01 last 66.61)
 - open/close 1301 (time 1788624380.16 total 20.02 last 63.37)
 - open/close 2009 (time 1788624390.17 total 30.03 last 70.74)
 - open/close 2645 (time 1788624400.20 total 40.06 last 63.42)
 - open/close 3322 (time 1788624410.20 total 50.06 last 67.67)
 - open/close 3950 (time 1788624420.21 total 60.07 last 62.75)
total: 4500 open/close in 68.49 seconds: 65.70 ops/second
 - /unlink 1248 (time 1788624443.35 total 10.00 last 124.77)
 - /unlink 2290 (time 1788624453.36 total 20.01 last 104.09)
 - /unlink 3569 (time 1788624463.36 total 30.02 last 127.86)
total: 4500 /unlink in 37.66 seconds: 119.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 682 (time 1788624486.21 total 10.01 last 68.15)
 - open/close 1269 (time 1788624496.22 total 20.01 last 58.67)
 - open/close 1961 (time 1788624506.22 total 30.02 last 69.18)
 - open/close 2559 (time 1788624516.23 total 40.03 last 59.75)
 - open/close 3122 (time 1788624526.24 total 50.04 last 56.24)
 - open/close 3770 (time 1788624536.24 total 60.04 last 64.79)
 - open/close 4402 (time 1788624546.24 total 70.04 last 63.19)
total: 4500 open/close in 71.60 seconds: 62.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1264 (time 1788624562.57 total 10.02 last 126.10)
 - /unlink 2378 (time 1788624572.57 total 20.03 last 111.38)
 - /unlink 3583 (time 1788624582.57 total 30.03 last 120.48)
total: 4500 /unlink in 38.18 seconds: 117.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 630 (time 1788624605.92 total 10.01 last 62.92)
 - open/close 1280 (time 1788624615.92 total 20.02 last 64.97)
 - open/close 1969 (time 1788624625.94 total 30.03 last 68.79)
 - open/close 2540 (time 1788624635.95 total 40.05 last 57.03)
 - open/close 3174 (time 1788624645.96 total 50.05 last 63.37)
 - open/close 3758 (time 1788624655.96 total 60.06 last 58.36)
 - open/close 4350 (time 1788624665.97 total 70.06 last 59.18)
total: 4500 open/close in 72.55 seconds: 62.03 ops/second
 - /unlink 1192 (time 1788624683.47 total 10.01 last 119.14)
 - /unlink 2402 (time 1788624693.47 total 20.01 last 120.99)
 - /unlink 3444 (time 1788624703.49 total 30.02 last 104.06)
 - /unlink 4448 (time 1788624713.50 total 40.03 last 100.33)
total: 4500 /unlink in 40.81 seconds: 110.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 607 (time 1788624729.13 total 10.01 last 60.62)
 - open/close 1196 (time 1788624739.15 total 20.03 last 58.82)
 - open/close 1866 (time 1788624749.17 total 30.05 last 66.85)
 - open/close 2516 (time 1788624759.18 total 40.06 last 64.92)
 - open/close 3146 (time 1788624769.18 total 50.06 last 62.99)
 - open/close 3831 (time 1788624779.19 total 60.07 last 68.42)
total: 4500 open/close in 69.33 seconds: 64.91 ops/second
 - /unlink 1248 (time 1788624802.59 total 10.01 last 124.68)
 - /unlink 2478 (time 1788624812.59 total 20.01 last 122.93)
 - /unlink 3776 (time 1788624822.60 total 30.02 last 129.69)
total: 4500 /unlink in 35.47 seconds: 126.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 702 (time 1788624842.13 total 10.01 last 70.16)
 - open/close 1335 (time 1788624852.14 total 20.01 last 63.26)
 - open/close 2068 (time 1788624862.15 total 30.02 last 73.23)
 - open/close 2787 (time 1788624872.16 total 40.03 last 71.82)
 - open/close 3470 (time 1788624882.16 total 50.04 last 68.27)
 - open/close 4114 (time 1788624892.18 total 60.05 last 64.32)
total: 4500 open/close in 66.19 seconds: 67.98 ops/second
 - /unlink 1439 (time 1788624913.29 total 10.01 last 143.81)
 - /unlink 2796 (time 1788624923.29 total 20.01 last 135.70)
 - /unlink 4081 (time 1788624933.29 total 30.01 last 128.42)
total: 4500 /unlink in 33.47 seconds: 134.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 656 (time 1788624951.14 total 10.01 last 65.56)
 - open/close 1339 (time 1788624961.15 total 20.01 last 68.25)
 - open/close 1936 (time 1788624971.16 total 30.03 last 59.62)
 - open/close 2608 (time 1788624981.16 total 40.03 last 67.18)
 - open/close 3384 (time 1788624991.17 total 50.03 last 77.59)
 - open/close 4006 (time 1788625001.17 total 60.03 last 62.19)
total: 4500 open/close in 67.19 seconds: 66.98 ops/second
 - /unlink 1366 (time 1788625022.72 total 10.00 last 136.58)
 - /unlink 2447 (time 1788625032.72 total 20.01 last 108.04)
 - /unlink 3802 (time 1788625042.72 total 30.01 last 135.50)
total: 4500 /unlink in 35.83 seconds: 125.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 598 (time 1788625063.62 total 10.01 last 59.76)
 - open/close 1304 (time 1788625073.62 total 20.01 last 70.59)
 - open/close 2011 (time 1788625083.63 total 30.02 last 70.65)
 - open/close 2728 (time 1788625093.64 total 40.03 last 71.62)
 - open/close 3425 (time 1788625103.65 total 50.03 last 69.67)
 - open/close 4092 (time 1788625113.65 total 60.04 last 66.67)
total: 4500 open/close in 66.35 seconds: 67.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1205 (time 1788625134.65 total 10.00 last 120.49)
 - /unlink 2397 (time 1788625144.66 total 20.00 last 119.16)
 - /unlink 3684 (time 1788625154.66 total 30.01 last 128.62)
total: 4500 /unlink in 36.80 seconds: 122.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 706 (time 1788625176.19 total 10.01 last 70.56)
 - open/close 1381 (time 1788625186.19 total 20.01 last 67.48)
 - open/close 2106 (time 1788625196.20 total 30.01 last 72.47)
 - open/close 2802 (time 1788625206.20 total 40.01 last 69.60)
 - open/close 3509 (time 1788625216.20 total 50.02 last 70.65)
 - open/close 4193 (time 1788625226.22 total 60.03 last 68.30)
total: 4500 open/close in 64.08 seconds: 70.22 ops/second
 - /unlink 1455 (time 1788625244.23 total 10.00 last 145.44)
 - /unlink 2754 (time 1788625254.23 total 20.01 last 129.85)
 - /unlink 4082 (time 1788625264.24 total 30.01 last 132.72)
total: 4500 /unlink in 33.24 seconds: 135.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 681 (time 1788625282.20 total 10.00 last 68.07)
 - open/close 1280 (time 1788625292.21 total 20.02 last 59.83)
 - open/close 1940 (time 1788625302.21 total 30.02 last 65.97)
 - open/close 2544 (time 1788625312.22 total 40.03 last 60.35)
 - open/close 3198 (time 1788625322.22 total 50.03 last 65.39)
 - open/close 3836 (time 1788625332.23 total 60.04 last 63.74)
total: 4500 open/close in 70.00 seconds: 64.29 ops/second
 - /unlink 1361 (time 1788625357.11 total 10.00 last 136.07)
 - /unlink 2582 (time 1788625367.11 total 20.00 last 122.08)
 - /unlink 3873 (time 1788625377.12 total 30.01 last 129.07)
total: 4500 /unlink in 35.02 seconds: 128.49 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95299 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg407-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg407-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server
stop mds service on oleg407-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server
stop mds service on oleg407-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server
unloading modules via unload_modules_local on: 'oleg407-server'
oleg407-server: oleg407-server.virtnet: executing unload_modules_local
oleg407-server: kmemleak disabled
modules unloaded.