== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:40:42 (1788482442)
Stopping clients: oleg110-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg110-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg110-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg110-server
oleg110-server: oleg110-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg110-server'
oleg110-server: oleg110-server.virtnet: executing load_modules_local
oleg110-server: kmemleak disabled
oleg110-server: Loading modules from /home/green/git/lustre-release/lustre
oleg110-server: detected 4 online CPUs by sysfs
oleg110-server: MODOPTS_LIBCFS=
oleg110-server: Force libcfs to create 2 CPU partitions
oleg110-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg110-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg110-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg110-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg110-client.virtnet:  -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 623 (time 1788482683.17 total 10.01 last 62.27)
 - open/close 1146 (time 1788482693.17 total 20.01 last 52.30)
 - open/close 1622 (time 1788482703.19 total 30.02 last 47.52)
 - open/close 2074 (time 1788482713.19 total 40.03 last 45.18)
 - open/close 2801 (time 1788482723.19 total 50.03 last 72.69)
 - open/close 3375 (time 1788482733.19 total 60.03 last 57.38)
 - open/close 4036 (time 1788482743.20 total 70.04 last 66.03)
total: 4500 open/close in 77.06 seconds: 58.39 ops/second
 - /unlink 1431 (time 1788482765.03 total 10.00 last 143.09)
 - /unlink 2741 (time 1788482775.04 total 20.00 last 130.98)
 - /unlink 4168 (time 1788482785.04 total 30.00 last 142.67)
total: 4500 /unlink in 32.25 seconds: 139.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 770 (time 1788482802.71 total 10.01 last 76.94)
 - open/close 1392 (time 1788482812.72 total 20.02 last 62.14)
 - open/close 2221 (time 1788482822.72 total 30.02 last 82.88)
 - open/close 2955 (time 1788482832.72 total 40.02 last 73.39)
 - open/close 3718 (time 1788482842.72 total 50.02 last 76.30)
 - open/close 4446 (time 1788482852.74 total 60.03 last 72.69)
total: 4500 open/close in 60.63 seconds: 74.22 ops/second
 - /unlink 1256 (time 1788482868.90 total 10.02 last 125.40)
 - /unlink 2476 (time 1788482878.91 total 20.02 last 121.95)
 - /unlink 3841 (time 1788482888.91 total 30.02 last 136.49)
total: 4500 /unlink in 35.90 seconds: 125.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 424 (time 1788482912.07 total 10.00 last 42.39)
 - open/close 880 (time 1788482922.07 total 20.01 last 45.58)
 - open/close 1637 (time 1788482932.08 total 30.01 last 75.68)
 - open/close 2331 (time 1788482942.08 total 40.01 last 69.40)
 - open/close 3022 (time 1788482952.08 total 50.01 last 69.07)
 - open/close 3706 (time 1788482962.08 total 60.02 last 68.38)
 - open/close 4382 (time 1788482972.10 total 70.03 last 67.47)
total: 4500 open/close in 71.94 seconds: 62.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1240 (time 1788482988.86 total 10.01 last 123.94)
 - /unlink 2694 (time 1788482998.86 total 20.01 last 145.36)
 - /unlink 4100 (time 1788483008.86 total 30.01 last 140.60)
total: 4500 /unlink in 32.46 seconds: 138.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 817 (time 1788483025.70 total 10.01 last 81.61)
 - open/close 1396 (time 1788483035.70 total 20.01 last 57.88)
 - open/close 2036 (time 1788483045.71 total 30.02 last 63.95)
 - open/close 2751 (time 1788483055.72 total 40.03 last 71.43)
 - open/close 3457 (time 1788483065.72 total 50.04 last 70.55)
 - open/close 4055 (time 1788483075.76 total 60.07 last 59.61)
total: 4500 open/close in 66.72 seconds: 67.45 ops/second
 - /unlink 1437 (time 1788483097.15 total 10.01 last 143.50)
 - /unlink 2797 (time 1788483107.15 total 20.02 last 135.95)
 - /unlink 4167 (time 1788483117.16 total 30.02 last 136.98)
total: 4500 /unlink in 32.39 seconds: 138.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 749 (time 1788483134.33 total 10.01 last 74.81)
 - open/close 1373 (time 1788483144.34 total 20.03 last 62.31)
 - open/close 2006 (time 1788483154.34 total 30.03 last 63.29)
 - open/close 2712 (time 1788483164.36 total 40.05 last 70.45)
 - open/close 3376 (time 1788483174.36 total 50.05 last 66.40)
 - open/close 4126 (time 1788483184.37 total 60.05 last 74.97)
total: 4500 open/close in 65.65 seconds: 68.55 ops/second
 - /unlink 1331 (time 1788483205.21 total 10.01 last 133.03)
 - /unlink 2330 (time 1788483215.22 total 20.02 last 99.79)
 - /unlink 3290 (time 1788483225.22 total 30.02 last 95.98)
 - /unlink 4381 (time 1788483235.22 total 40.02 last 109.05)
total: 4500 /unlink in 41.21 seconds: 109.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 662 (time 1788483253.63 total 10.01 last 66.11)
 - open/close 1442 (time 1788483263.64 total 20.02 last 77.98)
 - open/close 2174 (time 1788483273.64 total 30.02 last 73.17)
 - open/close 2956 (time 1788483283.66 total 40.04 last 78.03)
 - open/close 3624 (time 1788483293.68 total 50.06 last 66.71)
 - open/close 4290 (time 1788483303.69 total 60.07 last 66.52)
total: 4500 open/close in 62.54 seconds: 71.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1323 (time 1788483321.85 total 10.00 last 132.26)
 - /unlink 2748 (time 1788483331.86 total 20.01 last 142.35)
 - /unlink 4104 (time 1788483341.86 total 30.02 last 135.57)
total: 4500 /unlink in 32.85 seconds: 136.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 815 (time 1788483360.57 total 10.00 last 81.48)
 - open/close 1524 (time 1788483370.58 total 20.01 last 70.85)
 - open/close 2180 (time 1788483380.59 total 30.02 last 65.57)
 - open/close 2944 (time 1788483390.59 total 40.02 last 76.37)
 - open/close 3756 (time 1788483400.60 total 50.03 last 81.12)
 - open/close 4494 (time 1788483410.60 total 60.03 last 73.79)
total: 4500 open/close in 60.16 seconds: 74.81 ops/second
 - /unlink 1377 (time 1788483425.73 total 10.00 last 137.64)
 - /unlink 2499 (time 1788483435.75 total 20.02 last 112.03)
 - /unlink 3639 (time 1788483445.75 total 30.03 last 113.93)
total: 4500 /unlink in 36.25 seconds: 124.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 696 (time 1788483466.83 total 10.01 last 69.50)
 - open/close 1390 (time 1788483476.84 total 20.02 last 69.38)
 - open/close 2040 (time 1788483486.86 total 30.04 last 64.84)
 - open/close 2706 (time 1788483496.87 total 40.05 last 66.54)
 - open/close 3475 (time 1788483506.87 total 50.05 last 76.89)
 - open/close 4128 (time 1788483516.88 total 60.06 last 65.24)
total: 4500 open/close in 63.85 seconds: 70.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1200 (time 1788483534.90 total 10.00 last 120.00)
 - /unlink 2360 (time 1788483544.90 total 20.00 last 115.97)
 - /unlink 3634 (time 1788483554.90 total 30.00 last 127.40)
total: 4500 /unlink in 37.31 seconds: 120.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 619 (time 1788483579.25 total 10.00 last 61.88)
 - open/close 1251 (time 1788483589.25 total 20.00 last 63.20)
 - open/close 2043 (time 1788483599.26 total 30.01 last 79.15)
 - open/close 2803 (time 1788483609.26 total 40.02 last 75.94)
 - open/close 3476 (time 1788483619.27 total 50.02 last 67.26)
 - open/close 3994 (time 1788483629.29 total 60.05 last 51.69)
 - open/close 4412 (time 1788483639.60 total 70.35 last 40.55)
total: 4500 open/close in 73.11 seconds: 61.55 ops/second
 - /unlink 1073 (time 1788483659.39 total 10.00 last 107.27)
 - /unlink 2120 (time 1788483669.39 total 20.00 last 104.69)
 - /unlink 3368 (time 1788483679.39 total 30.00 last 124.79)
total: 4500 /unlink in 38.89 seconds: 115.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 800 (time 1788483704.02 total 10.00 last 79.97)
 - open/close 1540 (time 1788483714.02 total 20.01 last 73.98)
 - open/close 2309 (time 1788483724.02 total 30.01 last 76.88)
 - open/close 2966 (time 1788483734.03 total 40.02 last 65.63)
 - open/close 3750 (time 1788483744.05 total 50.03 last 78.29)
total: 4500 open/close in 58.74 seconds: 76.61 ops/second
 - /unlink 1349 (time 1788483767.59 total 10.00 last 134.85)
 - /unlink 2555 (time 1788483777.59 total 20.01 last 120.59)
 - /unlink 3896 (time 1788483787.59 total 30.01 last 134.05)
total: 4500 /unlink in 34.64 seconds: 129.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 801 (time 1788483807.47 total 10.01 last 80.05)
 - open/close 1355 (time 1788483817.48 total 20.02 last 55.35)
 - open/close 1818 (time 1788483827.50 total 30.03 last 46.23)
 - open/close 2483 (time 1788483837.51 total 40.04 last 66.41)
 - open/close 3093 (time 1788483847.52 total 50.05 last 60.95)
 - open/close 3725 (time 1788483857.53 total 60.06 last 63.15)
 - open/close 4399 (time 1788483867.54 total 70.08 last 67.30)
total: 4500 open/close in 71.88 seconds: 62.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1332 (time 1788483884.31 total 10.00 last 133.17)
 - /unlink 2632 (time 1788483894.32 total 20.01 last 129.93)
 - /unlink 3932 (time 1788483904.32 total 30.01 last 129.96)
total: 4500 /unlink in 35.10 seconds: 128.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 772 (time 1788483924.48 total 10.01 last 77.16)
 - open/close 1323 (time 1788483934.49 total 20.01 last 55.05)
 - open/close 2166 (time 1788483944.49 total 30.01 last 84.29)
 - open/close 2855 (time 1788483954.49 total 40.02 last 68.85)
 - open/close 3465 (time 1788483964.50 total 50.03 last 60.95)
 - open/close 4032 (time 1788483974.52 total 60.05 last 56.60)
total: 4500 open/close in 67.46 seconds: 66.71 ops/second
 - /unlink 1070 (time 1788483998.01 total 10.00 last 106.98)
 - /unlink 2311 (time 1788484008.02 total 20.01 last 123.95)
 - /unlink 3663 (time 1788484018.02 total 30.02 last 135.14)
total: 4500 /unlink in 36.59 seconds: 122.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 678 (time 1788484040.09 total 10.00 last 67.77)
 - open/close 1264 (time 1788484050.11 total 20.02 last 58.49)
 - open/close 1778 (time 1788484060.13 total 30.04 last 51.31)
 - open/close 2210 (time 1788484070.15 total 40.06 last 43.11)
 - open/close 2884 (time 1788484080.16 total 50.07 last 67.36)
 - open/close 3578 (time 1788484090.16 total 60.07 last 69.36)
 - open/close 4047 (time 1788484100.19 total 70.10 last 46.79)
 - open/close 4445 (time 1788484110.19 total 80.10 last 39.77)
total: 4500 open/close in 80.76 seconds: 55.72 ops/second
 - /unlink 1355 (time 1788484126.68 total 10.00 last 135.45)
 - /unlink 2608 (time 1788484136.69 total 20.01 last 125.27)
 - /unlink 4015 (time 1788484146.69 total 30.01 last 140.67)
total: 4500 /unlink in 33.00 seconds: 136.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 620 (time 1788484165.43 total 10.01 last 61.95)
 - open/close 1262 (time 1788484175.43 total 20.01 last 64.17)
 - open/close 1894 (time 1788484185.45 total 30.03 last 63.08)
 - open/close 2597 (time 1788484195.46 total 40.04 last 70.25)
 - open/close 3316 (time 1788484205.47 total 50.04 last 71.86)
 - open/close 4107 (time 1788484215.47 total 60.05 last 79.05)
total: 4500 open/close in 65.45 seconds: 68.75 ops/second
 - /unlink 1215 (time 1788484236.37 total 10.00 last 121.49)
 - /unlink 2386 (time 1788484246.38 total 20.00 last 117.06)
 - /unlink 3554 (time 1788484256.38 total 30.01 last 116.73)
total: 4500 /unlink in 38.08 seconds: 118.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 712 (time 1788484280.78 total 10.02 last 71.09)
 - open/close 1333 (time 1788484290.79 total 20.02 last 62.08)
 - open/close 1928 (time 1788484300.79 total 30.02 last 59.48)
 - open/close 2585 (time 1788484310.79 total 40.03 last 65.68)
 - open/close 3267 (time 1788484320.80 total 50.03 last 68.15)
 - open/close 4030 (time 1788484330.81 total 60.04 last 76.26)
total: 4500 open/close in 67.56 seconds: 66.61 ops/second
 - /unlink 1084 (time 1788484353.78 total 10.00 last 108.38)
 - /unlink 2284 (time 1788484363.79 total 20.02 last 119.81)
 - /unlink 3641 (time 1788484373.80 total 30.03 last 135.58)
total: 4500 /unlink in 37.57 seconds: 119.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 519 (time 1788484398.31 total 10.00 last 51.88)
 - open/close 1023 (time 1788484408.32 total 20.02 last 50.34)
 - open/close 1442 (time 1788484418.32 total 30.02 last 41.89)
 - open/close 1941 (time 1788484428.34 total 40.03 last 49.84)
 - open/close 2620 (time 1788484438.34 total 50.03 last 67.90)
 - open/close 3101 (time 1788484448.37 total 60.06 last 47.94)
 - open/close 3676 (time 1788484458.39 total 70.08 last 57.41)
 - open/close 4214 (time 1788484468.39 total 80.09 last 53.77)
total: 4500 open/close in 84.71 seconds: 53.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1175 (time 1788484488.55 total 10.01 last 117.42)
 - /unlink 2261 (time 1788484498.56 total 20.01 last 108.59)
 - /unlink 3233 (time 1788484508.56 total 30.01 last 97.15)
 - /unlink 4386 (time 1788484518.56 total 40.01 last 115.29)
total: 4500 /unlink in 41.53 seconds: 108.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 502 (time 1788484538.15 total 10.03 last 50.04)
 - open/close 1059 (time 1788484548.16 total 20.04 last 55.65)
 - open/close 1744 (time 1788484558.17 total 30.05 last 68.42)
 - open/close 2420 (time 1788484568.17 total 40.05 last 67.60)
 - open/close 3027 (time 1788484578.17 total 50.06 last 60.69)
 - open/close 3659 (time 1788484588.18 total 60.06 last 63.18)
 - open/close 4276 (time 1788484598.20 total 70.08 last 61.59)
total: 4500 open/close in 73.27 seconds: 61.42 ops/second
 - /unlink 1413 (time 1788484616.35 total 10.00 last 141.29)
 - /unlink 2708 (time 1788484626.35 total 20.00 last 129.49)
 - /unlink 3967 (time 1788484636.36 total 30.01 last 125.84)
total: 4500 /unlink in 35.48 seconds: 126.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 536 (time 1788484657.53 total 10.01 last 53.56)
 - open/close 1240 (time 1788484667.53 total 20.01 last 70.37)
 - open/close 1892 (time 1788484677.54 total 30.02 last 65.15)
 - open/close 2516 (time 1788484687.54 total 40.03 last 62.37)
 - open/close 3137 (time 1788484697.55 total 50.03 last 62.07)
 - open/close 3780 (time 1788484707.55 total 60.03 last 64.29)
total: 4500 open/close in 69.33 seconds: 64.90 ops/second
 - /unlink 1071 (time 1788484731.73 total 10.00 last 107.08)
 - /unlink 2425 (time 1788484741.74 total 20.01 last 135.34)
 - /unlink 3696 (time 1788484751.74 total 30.01 last 127.07)
total: 4500 /unlink in 35.94 seconds: 125.20 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95293 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg110-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server
stop mds service on oleg110-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server
stop mds service on oleg110-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server
unloading modules via unload_modules_local on: 'oleg110-server'
oleg110-server: oleg110-server.virtnet: executing unload_modules_local
oleg110-server: kmemleak disabled
modules unloaded.