== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:17:35 (1789067855)
Stopping clients: oleg446-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg446-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg446-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg446-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg446-server
oleg446-server: oleg446-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg446-server'
oleg446-server: oleg446-server.virtnet: executing load_modules_local
oleg446-server: kmemleak disabled
oleg446-server: Loading modules from /home/green/git/lustre-release/lustre
oleg446-server: detected 4 online CPUs by sysfs
oleg446-server: MODOPTS_LIBCFS=
oleg446-server: Force libcfs to create 2 CPU partitions
oleg446-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg446-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg446-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg446-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg446-client.virtnet:  -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 599 (time 1789068088.78 total 10.00 last 59.88)
 - open/close 1262 (time 1789068098.79 total 20.01 last 66.26)
 - open/close 1933 (time 1789068108.79 total 30.01 last 67.08)
 - open/close 2599 (time 1789068118.80 total 40.03 last 66.49)
 - open/close 3242 (time 1789068128.82 total 50.04 last 64.23)
 - open/close 3938 (time 1789068138.82 total 60.05 last 69.57)
total: 4500 open/close in 68.23 seconds: 65.95 ops/second
 - /unlink 1192 (time 1789068161.71 total 10.00 last 119.17)
 - /unlink 2425 (time 1789068171.71 total 20.01 last 123.27)
 - /unlink 3813 (time 1789068181.71 total 30.01 last 138.79)
total: 4500 /unlink in 35.70 seconds: 126.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 649 (time 1789068202.59 total 10.02 last 64.80)
 - open/close 1250 (time 1789068212.61 total 20.03 last 59.99)
 - open/close 1960 (time 1789068222.62 total 30.04 last 70.92)
 - open/close 2672 (time 1789068232.62 total 40.05 last 71.18)
 - open/close 3383 (time 1789068242.63 total 50.05 last 71.06)
 - open/close 4139 (time 1789068252.63 total 60.05 last 75.58)
total: 4500 open/close in 65.42 seconds: 68.78 ops/second
 - /unlink 1139 (time 1789068273.17 total 10.01 last 113.79)
 - /unlink 2137 (time 1789068283.18 total 20.02 last 99.68)
 - /unlink 3040 (time 1789068293.18 total 30.02 last 90.30)
 - /unlink 4088 (time 1789068303.18 total 40.02 last 104.78)
total: 4500 /unlink in 44.52 seconds: 101.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 653 (time 1789068322.61 total 10.00 last 65.29)
 - open/close 1281 (time 1789068332.62 total 20.02 last 62.72)
 - open/close 1947 (time 1789068342.63 total 30.02 last 66.57)
 - open/close 2566 (time 1789068352.63 total 40.03 last 61.86)
 - open/close 3183 (time 1789068362.65 total 50.04 last 61.60)
 - open/close 3827 (time 1789068372.66 total 60.05 last 64.36)
 - open/close 4441 (time 1789068382.67 total 70.06 last 61.31)
total: 4500 open/close in 70.89 seconds: 63.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1347 (time 1789068399.06 total 10.02 last 134.47)
 - /unlink 2653 (time 1789068409.06 total 20.02 last 130.53)
 - /unlink 3905 (time 1789068419.07 total 30.03 last 125.07)
total: 4500 /unlink in 36.51 seconds: 123.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 628 (time 1789068440.17 total 10.01 last 62.75)
 - open/close 1314 (time 1789068450.18 total 20.01 last 68.55)
 - open/close 1950 (time 1789068460.19 total 30.02 last 63.54)
 - open/close 2609 (time 1789068470.21 total 40.05 last 65.76)
 - open/close 3341 (time 1789068480.22 total 50.06 last 73.13)
 - open/close 4001 (time 1789068490.23 total 60.06 last 65.97)
total: 4500 open/close in 67.70 seconds: 66.47 ops/second
 - /unlink 1273 (time 1789068513.20 total 10.01 last 127.12)
 - /unlink 2480 (time 1789068523.20 total 20.02 last 120.68)
 - /unlink 3734 (time 1789068533.20 total 30.02 last 125.36)
total: 4500 /unlink in 36.00 seconds: 124.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 652 (time 1789068554.06 total 10.01 last 65.16)
 - open/close 1335 (time 1789068564.09 total 20.03 last 68.12)
 - open/close 2031 (time 1789068574.10 total 30.05 last 69.50)
 - open/close 2708 (time 1789068584.11 total 40.06 last 67.59)
 - open/close 3356 (time 1789068594.12 total 50.07 last 64.78)
 - open/close 4018 (time 1789068604.12 total 60.07 last 66.17)
total: 4500 open/close in 67.98 seconds: 66.20 ops/second
 - /unlink 1241 (time 1789068626.62 total 10.00 last 124.09)
 - /unlink 2389 (time 1789068636.62 total 20.01 last 114.73)
 - /unlink 3533 (time 1789068646.63 total 30.01 last 114.35)
total: 4500 /unlink in 36.84 seconds: 122.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 627 (time 1789068668.35 total 10.00 last 62.69)
 - open/close 1247 (time 1789068678.36 total 20.01 last 61.94)
 - open/close 1870 (time 1789068688.37 total 30.02 last 62.25)
 - open/close 2555 (time 1789068698.37 total 40.02 last 68.48)
 - open/close 3132 (time 1789068708.38 total 50.03 last 57.63)
 - open/close 3765 (time 1789068718.40 total 60.06 last 63.16)
 - open/close 4355 (time 1789068728.41 total 70.07 last 58.93)
total: 4500 open/close in 72.19 seconds: 62.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1281 (time 1789068744.96 total 10.00 last 128.06)
 - /unlink 2474 (time 1789068754.96 total 20.01 last 119.26)
 - /unlink 3793 (time 1789068764.96 total 30.01 last 131.87)
total: 4500 /unlink in 36.04 seconds: 124.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 748 (time 1789068785.79 total 10.01 last 74.71)
 - open/close 1343 (time 1789068795.80 total 20.02 last 59.42)
 - open/close 2049 (time 1789068805.82 total 30.04 last 70.47)
 - open/close 2750 (time 1789068815.83 total 40.05 last 70.07)
 - open/close 3475 (time 1789068825.84 total 50.06 last 72.40)
 - open/close 4146 (time 1789068835.85 total 60.07 last 67.06)
total: 4500 open/close in 65.54 seconds: 68.66 ops/second
 - /unlink 1209 (time 1789068856.35 total 10.00 last 120.86)
 - /unlink 2436 (time 1789068866.35 total 20.01 last 122.67)
 - /unlink 3427 (time 1789068876.37 total 30.03 last 98.86)
 - /unlink 4454 (time 1789068886.38 total 40.04 last 102.64)
total: 4500 /unlink in 40.92 seconds: 109.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 581 (time 1789068903.48 total 10.02 last 58.00)
 - open/close 1175 (time 1789068913.50 total 20.04 last 59.28)
 - open/close 1811 (time 1789068923.51 total 30.05 last 63.55)
 - open/close 2455 (time 1789068933.51 total 40.05 last 64.40)
 - open/close 3045 (time 1789068943.51 total 50.05 last 58.99)
 - open/close 3704 (time 1789068953.52 total 60.05 last 65.87)
 - open/close 4243 (time 1789068963.52 total 70.05 last 53.90)
total: 4500 open/close in 74.08 seconds: 60.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1141 (time 1789068984.61 total 10.00 last 114.07)
 - /unlink 2352 (time 1789068994.61 total 20.00 last 121.07)
 - /unlink 3456 (time 1789069004.61 total 30.01 last 110.38)
total: 4500 /unlink in 39.93 seconds: 112.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 579 (time 1789069029.65 total 10.01 last 57.85)
 - open/close 1212 (time 1789069039.66 total 20.02 last 63.22)
 - open/close 1862 (time 1789069049.66 total 30.03 last 64.97)
 - open/close 2486 (time 1789069059.67 total 40.04 last 62.33)
 - open/close 3148 (time 1789069069.68 total 50.05 last 66.13)
 - open/close 3759 (time 1789069079.69 total 60.05 last 61.07)
 - open/close 4339 (time 1789069089.70 total 70.06 last 57.93)
total: 4500 open/close in 73.68 seconds: 61.07 ops/second
 - /unlink 1154 (time 1789069109.15 total 10.00 last 115.38)
 - /unlink 2312 (time 1789069119.15 total 20.01 last 115.73)
 - /unlink 3466 (time 1789069129.16 total 30.02 last 115.27)
total: 4500 /unlink in 38.19 seconds: 117.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 617 (time 1789069153.24 total 10.00 last 61.70)
 - open/close 1265 (time 1789069163.26 total 20.03 last 64.62)
 - open/close 1923 (time 1789069173.27 total 30.03 last 65.77)
 - open/close 2564 (time 1789069183.27 total 40.04 last 64.08)
 - open/close 3168 (time 1789069193.30 total 50.06 last 60.24)
 - open/close 3859 (time 1789069203.30 total 60.06 last 69.09)
 - open/close 4352 (time 1789069213.31 total 70.07 last 49.26)
total: 4500 open/close in 72.56 seconds: 62.02 ops/second
 - /unlink 1093 (time 1789069231.09 total 10.01 last 109.19)
 - /unlink 2338 (time 1789069241.10 total 20.01 last 124.47)
 - /unlink 3551 (time 1789069251.10 total 30.02 last 121.26)
total: 4500 /unlink in 38.00 seconds: 118.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 664 (time 1789069274.12 total 10.02 last 66.29)
 - open/close 1297 (time 1789069284.13 total 20.03 last 63.21)
 - open/close 1906 (time 1789069294.15 total 30.04 last 60.82)
 - open/close 2532 (time 1789069304.17 total 40.07 last 62.42)
 - open/close 3137 (time 1789069314.17 total 50.07 last 60.50)
 - open/close 3685 (time 1789069324.19 total 60.09 last 54.71)
 - open/close 4255 (time 1789069334.20 total 70.10 last 56.96)
total: 4500 open/close in 74.12 seconds: 60.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1354 (time 1789069353.55 total 10.01 last 135.31)
 - /unlink 2465 (time 1789069363.55 total 20.01 last 111.06)
 - /unlink 3672 (time 1789069373.57 total 30.02 last 120.55)
 - /unlink 4465 (time 1789069383.58 total 40.03 last 79.23)
total: 4500 /unlink in 40.83 seconds: 110.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 588 (time 1789069400.10 total 10.01 last 58.74)
 - open/close 1164 (time 1789069410.11 total 20.02 last 57.54)
 - open/close 1611 (time 1789069420.13 total 30.04 last 44.62)
 - open/close 2063 (time 1789069430.13 total 40.04 last 45.19)
 - open/close 2570 (time 1789069440.15 total 50.06 last 50.60)
 - open/close 3154 (time 1789069450.15 total 60.07 last 58.38)
 - open/close 3855 (time 1789069460.16 total 70.07 last 70.07)
 - open/close 4498 (time 1789069470.17 total 80.08 last 64.25)
total: 4500 open/close in 80.09 seconds: 56.19 ops/second
 - /unlink 1259 (time 1789069485.34 total 10.00 last 125.89)
 - /unlink 2439 (time 1789069495.34 total 20.00 last 117.98)
 - /unlink 3730 (time 1789069505.34 total 30.00 last 129.08)
total: 4500 /unlink in 36.09 seconds: 124.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 631 (time 1789069526.14 total 10.00 last 63.08)
 - open/close 1178 (time 1789069536.15 total 20.01 last 54.69)
 - open/close 1790 (time 1789069546.17 total 30.03 last 61.06)
 - open/close 2440 (time 1789069556.18 total 40.04 last 64.93)
 - open/close 3166 (time 1789069566.18 total 50.04 last 72.59)
 - open/close 3932 (time 1789069576.20 total 60.06 last 76.45)
total: 4500 open/close in 68.85 seconds: 65.36 ops/second
 - /unlink 1250 (time 1789069599.88 total 10.00 last 124.98)
 - /unlink 2496 (time 1789069609.89 total 20.00 last 124.57)
 - /unlink 3549 (time 1789069619.89 total 30.01 last 105.28)
 - /unlink 4496 (time 1789069629.91 total 40.03 last 94.52)
total: 4500 /unlink in 40.05 seconds: 112.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 587 (time 1789069649.36 total 10.00 last 58.69)
 - open/close 1228 (time 1789069659.37 total 20.01 last 64.07)
 - open/close 1887 (time 1789069669.38 total 30.02 last 65.83)
 - open/close 2491 (time 1789069679.40 total 40.03 last 60.30)
 - open/close 3184 (time 1789069689.41 total 50.05 last 69.18)
 - open/close 3845 (time 1789069699.42 total 60.05 last 66.08)
 - open/close 4483 (time 1789069709.42 total 70.06 last 63.75)
total: 4500 open/close in 70.23 seconds: 64.08 ops/second
 - /unlink 1416 (time 1789069724.81 total 10.01 last 141.50)
 - /unlink 2575 (time 1789069734.81 total 20.01 last 115.83)
 - /unlink 3793 (time 1789069744.82 total 30.02 last 121.71)
total: 4500 /unlink in 36.02 seconds: 124.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 643 (time 1789069765.95 total 10.01 last 64.22)
 - open/close 1267 (time 1789069775.96 total 20.02 last 62.36)
 - open/close 1915 (time 1789069785.97 total 30.03 last 64.75)
 - open/close 2522 (time 1789069795.99 total 40.05 last 60.54)
 - open/close 3137 (time 1789069806.01 total 50.07 last 61.41)
 - open/close 3725 (time 1789069816.02 total 60.08 last 58.70)
 - open/close 4224 (time 1789069826.03 total 70.09 last 49.85)
total: 4500 open/close in 74.36 seconds: 60.51 ops/second
 - /unlink 1253 (time 1789069846.03 total 10.01 last 125.23)
 - /unlink 2524 (time 1789069856.04 total 20.01 last 127.02)
 - /unlink 3778 (time 1789069866.04 total 30.02 last 125.36)
total: 4500 /unlink in 36.17 seconds: 124.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 701 (time 1789069887.63 total 10.02 last 69.99)
 - open/close 1326 (time 1789069897.64 total 20.02 last 62.47)
 - open/close 1991 (time 1789069907.64 total 30.02 last 66.49)
 - open/close 2576 (time 1789069917.64 total 40.03 last 58.46)
 - open/close 3092 (time 1789069927.66 total 50.05 last 51.52)
 - open/close 3731 (time 1789069937.69 total 60.08 last 63.69)
 - open/close 4340 (time 1789069947.69 total 70.08 last 60.90)
total: 4500 open/close in 72.73 seconds: 61.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1174 (time 1789069965.71 total 10.00 last 117.38)
 - /unlink 2216 (time 1789069975.71 total 20.00 last 104.20)
 - /unlink 3293 (time 1789069985.71 total 30.01 last 107.63)
 - /unlink 4269 (time 1789069995.72 total 40.01 last 97.54)
total: 4500 /unlink in 42.33 seconds: 106.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 668 (time 1789070014.46 total 10.01 last 66.71)
 - open/close 1303 (time 1789070024.46 total 20.02 last 63.46)
 - open/close 1962 (time 1789070034.48 total 30.03 last 65.82)
 - open/close 2603 (time 1789070044.50 total 40.06 last 63.94)
 - open/close 3267 (time 1789070054.50 total 50.06 last 66.37)
 - open/close 3924 (time 1789070064.50 total 60.06 last 65.70)
total: 4500 open/close in 69.73 seconds: 64.54 ops/second
 - /unlink 1213 (time 1789070089.41 total 10.01 last 121.19)
 - /unlink 2316 (time 1789070099.41 total 20.01 last 110.28)
 - /unlink 3305 (time 1789070109.42 total 30.02 last 98.78)
 - /unlink 4479 (time 1789070119.42 total 40.03 last 117.36)
total: 4500 /unlink in 40.17 seconds: 112.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 704 (time 1789070134.81 total 10.01 last 70.31)
 - open/close 1324 (time 1789070144.81 total 20.02 last 61.97)
 - open/close 1966 (time 1789070154.83 total 30.03 last 64.10)
 - open/close 2549 (time 1789070164.83 total 40.04 last 58.27)
 - open/close 3219 (time 1789070174.84 total 50.04 last 66.95)
 - open/close 3873 (time 1789070184.85 total 60.06 last 65.33)
total: 4500 open/close in 69.96 seconds: 64.32 ops/second
 - /unlink 1183 (time 1789070210.19 total 10.01 last 118.14)
 - /unlink 2289 (time 1789070220.20 total 20.02 last 110.57)
 - /unlink 3509 (time 1789070230.20 total 30.02 last 121.91)
total: 4500 /unlink in 37.66 seconds: 119.48 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95332 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg446-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg446-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server
stop mds service on oleg446-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server
stop mds service on oleg446-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg446-server
unloading modules via unload_modules_local on: 'oleg446-server'
oleg446-server: oleg446-server.virtnet: executing unload_modules_local
oleg446-server: kmemleak disabled
modules unloaded.