== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:29:48 (1788834588)
Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg402-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg402-server
oleg402-server: oleg402-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg402-server'
oleg402-server: oleg402-server.virtnet: executing load_modules_local
oleg402-server: kmemleak disabled
oleg402-server: Loading modules from /home/green/git/lustre-release/lustre
oleg402-server: detected 4 online CPUs by sysfs
oleg402-server: MODOPTS_LIBCFS=
oleg402-server: Force libcfs to create 2 CPU partitions
oleg402-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg402-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all
pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg402-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all
pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg402-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all
pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg402-client.virtnet:  -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 657 (time 1788834807.77 total 10.01 last 65.64)
 - open/close 1300 (time 1788834817.77 total 20.02 last 64.26)
 - open/close 1999 (time 1788834827.78 total 30.03 last 69.84)
 - open/close 2670 (time 1788834837.79 total 40.04 last 67.03)
 - open/close 3307 (time 1788834847.81 total 50.05 last 63.62)
 - open/close 3847 (time 1788834857.81 total 60.06 last 53.96)
 - open/close 4465 (time 1788834867.82 total 70.06 last 61.78)
total: 4500 open/close in 70.57 seconds: 63.76 ops/second
 - /unlink 1288 (time 1788834883.67 total 10.00 last 128.75)
 - /unlink 2531 (time 1788834893.67 total 20.01 last 124.28)
 - /unlink 3698 (time 1788834903.67 total 30.01 last 116.67)
total: 4500 /unlink in 36.65 seconds: 122.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 623 (time 1788834925.32 total 10.02 last 62.16)
 - open/close 1289 (time 1788834935.33 total 20.03 last 66.55)
 - open/close 1958 (time 1788834945.33 total 30.03 last 66.87)
 - open/close 2671 (time 1788834955.34 total 40.04 last 71.26)
 - open/close 3391 (time 1788834965.35 total 50.05 last 71.92)
 - open/close 4045 (time 1788834975.37 total 60.08 last 65.22)
total: 4500 open/close in 66.46 seconds: 67.71 ops/second
 - /unlink 1338 (time 1788834995.90 total 10.01 last 133.66)
 - /unlink 2678 (time 1788835005.90 total 20.01 last 133.99)
 - /unlink 4081 (time 1788835015.90 total 30.01 last 140.28)
total: 4500 /unlink in 33.32 seconds: 135.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 673 (time 1788835034.52 total 10.00 last 67.27)
 - open/close 1379 (time 1788835044.52 total 20.01 last 70.59)
 - open/close 2080 (time 1788835054.55 total 30.03 last 69.91)
 - open/close 2828 (time 1788835064.55 total 40.04 last 74.76)
 - open/close 3473 (time 1788835074.56 total 50.04 last 64.45)
 - open/close 4115 (time 1788835084.57 total 60.06 last 64.12)
total: 4500 open/close in 65.72 seconds: 68.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1314 (time 1788835104.54 total 10.01 last 131.32)
 - /unlink 2599 (time 1788835114.55 total 20.02 last 128.35)
 - /unlink 3836 (time 1788835124.55 total 30.02 last 123.66)
total: 4500 /unlink in 35.46 seconds: 126.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 687 (time 1788835143.92 total 10.02 last 68.58)
 - open/close 1368 (time 1788835153.92 total 20.02 last 68.09)
 - open/close 2066 (time 1788835163.95 total 30.05 last 69.61)
 - open/close 2768 (time 1788835173.95 total 40.05 last 70.18)
 - open/close 3434 (time 1788835183.96 total 50.05 last 66.57)
 - open/close 4108 (time 1788835193.96 total 60.06 last 67.37)
total: 4500 open/close in 65.73 seconds: 68.47 ops/second
 - /unlink 1387 (time 1788835214.07 total 10.00 last 138.64)
 - /unlink 2667 (time 1788835224.07 total 20.01 last 127.98)
 - /unlink 3957 (time 1788835234.08 total 30.01 last 128.95)
total: 4500 /unlink in 34.03 seconds: 132.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 732 (time 1788835252.12 total 10.00 last 73.19)
 - open/close 1414 (time 1788835262.12 total 20.00 last 68.20)
 - open/close 2091 (time 1788835272.13 total 30.01 last 67.63)
 - open/close 2671 (time 1788835282.14 total 40.02 last 57.94)
 - open/close 3248 (time 1788835292.15 total 50.03 last 57.64)
 - open/close 3832 (time 1788835302.16 total 60.05 last 58.30)
 - open/close 4393 (time 1788835312.18 total 70.06 last 56.03)
total: 4500 open/close in 71.53 seconds: 62.91 ops/second
 - /unlink 1301 (time 1788835327.96 total 10.00 last 130.05)
 - /unlink 2524 (time 1788835337.97 total 20.01 last 122.22)
 - /unlink 3793 (time 1788835347.97 total 30.02 last 126.83)
total: 4500 /unlink in 36.25 seconds: 124.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 684 (time 1788835368.77 total 10.02 last 68.28)
 - open/close 1315 (time 1788835378.78 total 20.02 last 63.06)
 - open/close 1991 (time 1788835388.78 total 30.03 last 67.55)
 - open/close 2601 (time 1788835398.79 total 40.03 last 60.98)
 - open/close 3261 (time 1788835408.79 total 50.04 last 65.95)
 - open/close 3895 (time 1788835418.80 total 60.05 last 63.36)
total: 4500 open/close in 69.05 seconds: 65.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1161 (time 1788835443.64 total 10.00 last 116.08)
 - /unlink 2399 (time 1788835453.64 total 20.00 last 123.77)
 - /unlink 3692 (time 1788835463.64 total 30.01 last 129.28)
total: 4500 /unlink in 36.68 seconds: 122.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 684 (time 1788835484.42 total 10.01 last 68.34)
 - open/close 1347 (time 1788835494.44 total 20.03 last 66.15)
 - open/close 2004 (time 1788835504.45 total 30.04 last 65.63)
 - open/close 2710 (time 1788835514.47 total 40.06 last 70.50)
 - open/close 3403 (time 1788835524.47 total 50.06 last 69.28)
 - open/close 4085 (time 1788835534.47 total 60.06 last 68.17)
total: 4500 open/close in 66.92 seconds: 67.25 ops/second
 - /unlink 1213 (time 1788835556.43 total 10.00 last 121.26)
 - /unlink 2417 (time 1788835566.44 total 20.01 last 120.30)
 - /unlink 3650 (time 1788835576.46 total 30.03 last 123.09)
total: 4500 /unlink in 37.06 seconds: 121.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 662 (time 1788835598.58 total 10.01 last 66.16)
 - open/close 1364 (time 1788835608.58 total 20.01 last 70.16)
 - open/close 2010 (time 1788835618.58 total 30.01 last 64.60)
 - open/close 2572 (time 1788835628.59 total 40.02 last 56.13)
 - open/close 3102 (time 1788835638.60 total 50.03 last 52.97)
 - open/close 3719 (time 1788835648.60 total 60.03 last 61.69)
 - open/close 4294 (time 1788835658.61 total 70.04 last 57.44)
total: 4500 open/close in 73.83 seconds: 60.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1222 (time 1788835677.34 total 10.00 last 122.19)
 - /unlink 2503 (time 1788835687.34 total 20.00 last 128.06)
 - /unlink 3680 (time 1788835697.35 total 30.01 last 117.67)
total: 4500 /unlink in 36.75 seconds: 122.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 677 (time 1788835719.10 total 10.00 last 67.68)
 - open/close 1324 (time 1788835729.11 total 20.01 last 64.68)
 - open/close 1955 (time 1788835739.13 total 30.02 last 62.98)
 - open/close 2631 (time 1788835749.13 total 40.03 last 67.58)
 - open/close 3345 (time 1788835759.13 total 50.03 last 71.38)
 - open/close 4008 (time 1788835769.14 total 60.04 last 66.23)
total: 4500 open/close in 66.82 seconds: 67.35 ops/second
 - /unlink 1325 (time 1788835790.72 total 10.00 last 132.50)
 - /unlink 2585 (time 1788835800.72 total 20.00 last 125.96)
 - /unlink 3784 (time 1788835810.72 total 30.00 last 119.90)
total: 4500 /unlink in 36.91 seconds: 121.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 602 (time 1788835832.97 total 10.03 last 60.03)
 - open/close 1213 (time 1788835842.98 total 20.03 last 61.06)
 - open/close 1863 (time 1788835853.00 total 30.06 last 64.84)
 - open/close 2528 (time 1788835863.01 total 40.07 last 66.45)
 - open/close 3235 (time 1788835873.01 total 50.07 last 70.69)
 - open/close 3895 (time 1788835883.02 total 60.07 last 65.96)
total: 4500 open/close in 68.80 seconds: 65.41 ops/second
 - /unlink 1280 (time 1788835906.17 total 10.00 last 127.94)
 - /unlink 2563 (time 1788835916.17 total 20.01 last 128.22)
 - /unlink 3845 (time 1788835926.18 total 30.01 last 128.15)
total: 4500 /unlink in 35.10 seconds: 128.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 680 (time 1788835945.73 total 10.00 last 67.99)
 - open/close 1371 (time 1788835955.74 total 20.01 last 69.05)
 - open/close 2069 (time 1788835965.74 total 30.01 last 69.80)
 - open/close 2784 (time 1788835975.74 total 40.01 last 71.47)
 - open/close 3481 (time 1788835985.76 total 50.03 last 69.59)
 - open/close 4182 (time 1788835995.76 total 60.04 last 70.05)
total: 4500 open/close in 64.48 seconds: 69.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1316 (time 1788836014.53 total 10.00 last 131.59)
 - /unlink 2688 (time 1788836024.54 total 20.00 last 137.14)
 - /unlink 4051 (time 1788836034.54 total 30.01 last 136.22)
total: 4500 /unlink in 32.91 seconds: 136.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 752 (time 1788836051.70 total 10.00 last 75.20)
 - open/close 1492 (time 1788836061.72 total 20.02 last 73.88)
 - open/close 2053 (time 1788836071.74 total 30.04 last 55.96)
 - open/close 2570 (time 1788836081.75 total 40.05 last 51.68)
 - open/close 3165 (time 1788836091.75 total 50.05 last 59.50)
 - open/close 3746 (time 1788836101.76 total 60.06 last 58.02)
 - open/close 4393 (time 1788836111.77 total 70.07 last 64.65)
total: 4500 open/close in 71.97 seconds: 62.53 ops/second
 - /unlink 1189 (time 1788836128.81 total 10.01 last 118.81)
 - /unlink 2198 (time 1788836138.81 total 20.01 last 100.88)
 - /unlink 3415 (time 1788836148.81 total 30.01 last 121.70)
total: 4500 /unlink in 39.14 seconds: 114.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 681 (time 1788836173.01 total 10.00 last 68.07)
 - open/close 1354 (time 1788836183.02 total 20.01 last 67.27)
 - open/close 1983 (time 1788836193.03 total 30.02 last 62.80)
 - open/close 2641 (time 1788836203.03 total 40.02 last 65.80)
 - open/close 3292 (time 1788836213.04 total 50.03 last 65.06)
 - open/close 3944 (time 1788836223.06 total 60.05 last 65.05)
total: 4500 open/close in 67.94 seconds: 66.24 ops/second
 - /unlink 1309 (time 1788836245.18 total 10.00 last 130.88)
 - /unlink 2599 (time 1788836255.18 total 20.00 last 128.99)
 - /unlink 3797 (time 1788836265.18 total 30.01 last 119.72)
total: 4500 /unlink in 37.05 seconds: 121.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 754 (time 1788836287.75 total 10.00 last 75.39)
 - open/close 1480 (time 1788836297.76 total 20.01 last 72.54)
 - open/close 2202 (time 1788836307.76 total 30.01 last 72.19)
 - open/close 2885 (time 1788836317.77 total 40.02 last 68.25)
 - open/close 3546 (time 1788836327.77 total 50.02 last 66.07)
 - open/close 4295 (time 1788836337.78 total 60.03 last 74.87)
total: 4500 open/close in 63.19 seconds: 71.21 ops/second
 - /unlink 1278 (time 1788836355.67 total 10.00 last 127.76)
 - /unlink 2495 (time 1788836365.67 total 20.00 last 121.70)
 - /unlink 3733 (time 1788836375.67 total 30.01 last 123.77)
total: 4500 /unlink in 35.96 seconds: 125.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 578 (time 1788836396.53 total 10.02 last 57.70)
 - open/close 1218 (time 1788836406.54 total 20.03 last 63.94)
 - open/close 1844 (time 1788836416.54 total 30.03 last 62.58)
 - open/close 2453 (time 1788836426.55 total 40.03 last 60.88)
 - open/close 3103 (time 1788836436.55 total 50.04 last 64.98)
 - open/close 3736 (time 1788836446.56 total 60.04 last 63.25)
 - open/close 4366 (time 1788836456.56 total 70.05 last 62.97)
total: 4500 open/close in 72.23 seconds: 62.30 ops/second
 - /unlink 1172 (time 1788836475.26 total 10.00 last 117.15)
 - /unlink 2359 (time 1788836485.26 total 20.01 last 118.67)
 - /unlink 3498 (time 1788836495.27 total 30.01 last 113.84)
total: 4500 /unlink in 38.33 seconds: 117.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 665 (time 1788836519.66 total 10.00 last 66.50)
 - open/close 1318 (time 1788836529.66 total 20.00 last 65.29)
 - open/close 1987 (time 1788836539.67 total 30.01 last 66.85)
 - open/close 2650 (time 1788836549.67 total 40.01 last 66.29)
 - open/close 3314 (time 1788836559.69 total 50.03 last 66.27)
 - open/close 3986 (time 1788836569.69 total 60.03 last 67.18)
total: 4500 open/close in 68.05 seconds: 66.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1185 (time 1788836592.38 total 10.00 last 118.45)
 - /unlink 2358 (time 1788836602.40 total 20.02 last 117.13)
 - /unlink 3473 (time 1788836612.40 total 30.02 last 111.49)
total: 4500 /unlink in 39.46 seconds: 114.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 569 (time 1788836638.01 total 10.00 last 56.89)
 - open/close 1164 (time 1788836648.02 total 20.01 last 59.43)
 - open/close 1761 (time 1788836658.03 total 30.02 last 59.68)
 - open/close 2426 (time 1788836668.04 total 40.03 last 66.39)
 - open/close 3113 (time 1788836678.05 total 50.04 last 68.65)
 - open/close 3803 (time 1788836688.07 total 60.06 last 68.90)
 - open/close 4470 (time 1788836698.08 total 70.08 last 66.57)
total: 4500 open/close in 70.68 seconds: 63.67 ops/second
 - /unlink 1270 (time 1788836713.51 total 10.00 last 126.99)
 - /unlink 2388 (time 1788836723.51 total 20.00 last 111.78)
 - /unlink 3336 (time 1788836733.51 total 30.01 last 94.77)
 - /unlink 4082 (time 1788836743.52 total 40.01 last 74.55)
total: 4500 /unlink in 46.08 seconds: 97.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 544 (time 1788836767.21 total 10.02 last 54.29)
 - open/close 1148 (time 1788836777.23 total 20.04 last 60.31)
 - open/close 1774 (time 1788836787.24 total 30.05 last 62.54)
 - open/close 2332 (time 1788836797.25 total 40.06 last 55.71)
 - open/close 2893 (time 1788836807.26 total 50.07 last 56.08)
 - open/close 3492 (time 1788836817.26 total 60.07 last 59.88)
 - open/close 4114 (time 1788836827.27 total 70.08 last 62.12)
total: 4500 open/close in 75.96 seconds: 59.24 ops/second
 - /unlink 1265 (time 1788836848.00 total 10.00 last 126.46)
 - /unlink 2372 (time 1788836858.01 total 20.01 last 110.57)
 - /unlink 3045 (time 1788836868.01 total 30.02 last 67.30)
 - /unlink 3711 (time 1788836878.01 total 40.02 last 66.57)
total: 4500 /unlink in 49.00 seconds: 91.83 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95311 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg402-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg402-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server
stop mds service on oleg402-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server
stop mds service on oleg402-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server
unloading modules via unload_modules_local on: 'oleg402-server'
oleg402-server: oleg402-server.virtnet: executing unload_modules_local
oleg402-server: kmemleak disabled
modules unloaded.