== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:20:08 (1789053608)
Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg319-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg319-server
oleg319-server: oleg319-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg319-server'
oleg319-server: oleg319-server.virtnet: executing load_modules_local
oleg319-server: kmemleak disabled
oleg319-server: Loading modules from /home/green/git/lustre-release/lustre
oleg319-server: detected 4 online CPUs by sysfs
oleg319-server: MODOPTS_LIBCFS=
oleg319-server: Force libcfs to create 2 CPU partitions
oleg319-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg319-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all
pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg319-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all
pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg319-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all
pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg319-client.virtnet:  -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 643 (time 1789053824.61 total 10.01 last 64.23)
 - open/close 1226 (time 1789053834.61 total 20.01 last 58.29)
 - open/close 1820 (time 1789053844.62 total 30.02 last 59.34)
 - open/close 2458 (time 1789053854.62 total 40.02 last 63.79)
 - open/close 3087 (time 1789053864.63 total 50.03 last 62.85)
 - open/close 3716 (time 1789053874.64 total 60.04 last 62.84)
 - open/close 4316 (time 1789053884.65 total 70.05 last 59.96)
total: 4500 open/close in 73.50 seconds: 61.23 ops/second
 - /unlink 971 (time 1789053904.01 total 10.01 last 97.00)
 - /unlink 1902 (time 1789053914.01 total 20.01 last 93.08)
 - /unlink 2985 (time 1789053924.02 total 30.02 last 108.23)
 - /unlink 4194 (time 1789053934.03 total 40.03 last 120.78)
total: 4500 /unlink in 42.27 seconds: 106.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 648 (time 1789053950.79 total 10.00 last 64.79)
 - open/close 1255 (time 1789053960.79 total 20.00 last 60.68)
 - open/close 1829 (time 1789053970.81 total 30.02 last 57.30)
 - open/close 2456 (time 1789053980.81 total 40.03 last 62.67)
 - open/close 3042 (time 1789053990.83 total 50.04 last 58.52)
 - open/close 3628 (time 1789054000.83 total 60.04 last 58.60)
 - open/close 4211 (time 1789054010.84 total 70.05 last 58.23)
total: 4500 open/close in 73.95 seconds: 60.85 ops/second
 - /unlink 1064 (time 1789054028.96 total 10.00 last 106.40)
 - /unlink 2003 (time 1789054038.98 total 20.02 last 93.76)
 - /unlink 3061 (time 1789054048.99 total 30.03 last 105.69)
 - /unlink 4033 (time 1789054058.99 total 40.03 last 97.19)
total: 4500 /unlink in 43.64 seconds: 103.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 580 (time 1789054077.83 total 10.00 last 57.98)
 - open/close 1098 (time 1789054087.84 total 20.01 last 51.77)
 - open/close 1644 (time 1789054097.86 total 30.03 last 54.49)
 - open/close 2200 (time 1789054107.86 total 40.03 last 55.59)
 - open/close 2613 (time 1789054117.89 total 50.06 last 41.20)
 - open/close 3234 (time 1789054127.93 total 60.10 last 61.84)
 - open/close 3942 (time 1789054137.94 total 70.11 last 70.73)
total: 4500 open/close in 78.01 seconds: 57.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1234 (time 1789054160.46 total 10.00 last 123.34)
 - /unlink 2385 (time 1789054170.47 total 20.01 last 115.01)
 - /unlink 3635 (time 1789054180.47 total 30.02 last 124.96)
total: 4500 /unlink in 37.28 seconds: 120.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 641 (time 1789054202.51 total 10.00 last 64.08)
 - open/close 1346 (time 1789054212.52 total 20.02 last 70.41)
 - open/close 2073 (time 1789054222.52 total 30.02 last 72.68)
 - open/close 2766 (time 1789054232.53 total 40.02 last 69.27)
 - open/close 3479 (time 1789054242.54 total 50.04 last 71.19)
 - open/close 4152 (time 1789054252.56 total 60.05 last 67.19)
total: 4500 open/close in 64.73 seconds: 69.52 ops/second
 - /unlink 1348 (time 1789054272.08 total 10.01 last 134.72)
 - /unlink 2687 (time 1789054282.08 total 20.01 last 133.86)
 - /unlink 4047 (time 1789054292.08 total 30.01 last 135.97)
total: 4500 /unlink in 33.61 seconds: 133.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 706 (time 1789054311.13 total 10.00 last 70.57)
 - open/close 1407 (time 1789054321.14 total 20.02 last 70.02)
 - open/close 2075 (time 1789054331.14 total 30.02 last 66.76)
 - open/close 2509 (time 1789054341.16 total 40.04 last 43.33)
 - open/close 2946 (time 1789054351.17 total 50.05 last 43.67)
 - open/close 3532 (time 1789054361.17 total 60.05 last 58.58)
 - open/close 4166 (time 1789054371.18 total 70.06 last 63.32)
total: 4500 open/close in 74.68 seconds: 60.25 ops/second
 - /unlink 1294 (time 1789054391.25 total 10.01 last 129.26)
 - /unlink 2567 (time 1789054401.26 total 20.02 last 127.23)
 - /unlink 3935 (time 1789054411.26 total 30.02 last 136.74)
total: 4500 /unlink in 34.01 seconds: 132.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 776 (time 1789054430.12 total 10.01 last 77.55)
 - open/close 1452 (time 1789054440.14 total 20.02 last 67.53)
 - open/close 1976 (time 1789054450.15 total 30.03 last 52.32)
 - open/close 2532 (time 1789054460.16 total 40.04 last 55.54)
 - open/close 3113 (time 1789054470.17 total 50.05 last 58.05)
 - open/close 3736 (time 1789054480.17 total 60.05 last 62.28)
 - open/close 4366 (time 1789054490.18 total 70.06 last 62.98)
total: 4500 open/close in 72.42 seconds: 62.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1175 (time 1789054506.30 total 10.00 last 117.47)
 - /unlink 2138 (time 1789054516.30 total 20.00 last 96.30)
 - /unlink 2954 (time 1789054526.31 total 30.01 last 81.58)
 - /unlink 3968 (time 1789054536.31 total 40.02 last 101.30)
total: 4500 /unlink in 46.70 seconds: 96.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 637 (time 1789054558.60 total 10.02 last 63.58)
 - open/close 1304 (time 1789054568.61 total 20.03 last 66.61)
 - open/close 2049 (time 1789054578.62 total 30.04 last 74.45)
 - open/close 2747 (time 1789054588.62 total 40.04 last 69.80)
 - open/close 3424 (time 1789054598.62 total 50.04 last 67.69)
 - open/close 4061 (time 1789054608.62 total 60.04 last 63.68)
total: 4500 open/close in 66.73 seconds: 67.44 ops/second
 - /unlink 1236 (time 1789054629.74 total 10.00 last 123.57)
 - /unlink 2514 (time 1789054639.74 total 20.00 last 127.78)
 - /unlink 3753 (time 1789054649.74 total 30.01 last 123.87)
total: 4500 /unlink in 36.14 seconds: 124.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 654 (time 1789054670.50 total 10.00 last 65.37)
 - open/close 1344 (time 1789054680.51 total 20.01 last 68.98)
 - open/close 2073 (time 1789054690.51 total 30.01 last 72.90)
 - open/close 2780 (time 1789054700.53 total 40.03 last 70.56)
 - open/close 3456 (time 1789054710.53 total 50.03 last 67.58)
 - open/close 4110 (time 1789054720.54 total 60.04 last 65.36)
total: 4500 open/close in 66.04 seconds: 68.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1233 (time 1789054741.66 total 10.00 last 123.28)
 - /unlink 2491 (time 1789054751.67 total 20.01 last 125.75)
 - /unlink 3640 (time 1789054761.67 total 30.01 last 114.81)
total: 4500 /unlink in 37.28 seconds: 120.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 672 (time 1789054784.53 total 10.01 last 67.11)
 - open/close 1336 (time 1789054794.53 total 20.02 last 66.36)
 - open/close 2002 (time 1789054804.54 total 30.02 last 66.56)
 - open/close 2678 (time 1789054814.58 total 40.06 last 67.35)
 - open/close 3358 (time 1789054824.58 total 50.07 last 67.96)
 - open/close 4066 (time 1789054834.60 total 60.08 last 70.68)
total: 4500 open/close in 65.92 seconds: 68.26 ops/second
 - /unlink 1262 (time 1789054855.32 total 10.00 last 126.18)
 - /unlink 2542 (time 1789054865.33 total 20.00 last 127.99)
 - /unlink 3842 (time 1789054875.33 total 30.00 last 129.99)
total: 4500 /unlink in 35.15 seconds: 128.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 691 (time 1789054894.99 total 10.01 last 69.03)
 - open/close 1357 (time 1789054905.00 total 20.03 last 66.49)
 - open/close 2027 (time 1789054915.00 total 30.03 last 66.99)
 - open/close 2773 (time 1789054925.04 total 40.06 last 74.36)
 - open/close 3430 (time 1789054935.04 total 50.07 last 65.64)
 - open/close 4148 (time 1789054945.06 total 60.08 last 71.70)
total: 4500 open/close in 65.83 seconds: 68.36 ops/second
 - /unlink 1389 (time 1789054965.28 total 10.01 last 138.82)
 - /unlink 2587 (time 1789054975.28 total 20.01 last 119.79)
 - /unlink 3701 (time 1789054985.29 total 30.02 last 111.28)
total: 4500 /unlink in 37.18 seconds: 121.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 449 (time 1789055008.25 total 10.03 last 44.76)
 - open/close 896 (time 1789055018.29 total 20.06 last 44.55)
 - open/close 1425 (time 1789055028.31 total 30.08 last 52.80)
 - open/close 2124 (time 1789055038.31 total 40.09 last 69.85)
 - open/close 2658 (time 1789055048.34 total 50.12 last 53.26)
 - open/close 3182 (time 1789055058.35 total 60.13 last 52.33)
 - open/close 3666 (time 1789055068.36 total 70.14 last 48.35)
 - open/close 4305 (time 1789055078.39 total 80.16 last 63.75)
total: 4500 open/close in 83.44 seconds: 53.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1317 (time 1789055096.34 total 10.01 last 131.58)
 - /unlink 2530 (time 1789055106.34 total 20.01 last 121.26)
 - /unlink 3854 (time 1789055116.35 total 30.02 last 132.34)
total: 4500 /unlink in 35.19 seconds: 127.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 649 (time 1789055135.68 total 10.00 last 64.87)
 - open/close 1341 (time 1789055145.69 total 20.02 last 69.07)
 - open/close 2052 (time 1789055155.71 total 30.04 last 71.01)
 - open/close 2728 (time 1789055165.72 total 40.05 last 67.52)
 - open/close 3456 (time 1789055175.73 total 50.06 last 72.69)
 - open/close 4175 (time 1789055185.74 total 60.07 last 71.87)
total: 4500 open/close in 64.46 seconds: 69.81 ops/second
 - /unlink 1426 (time 1789055204.69 total 10.00 last 142.54)
 - /unlink 2763 (time 1789055214.70 total 20.01 last 133.60)
 - /unlink 4027 (time 1789055224.70 total 30.01 last 126.37)
total: 4500 /unlink in 33.84 seconds: 132.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 743 (time 1789055242.55 total 10.00 last 74.30)
 - open/close 1511 (time 1789055252.56 total 20.00 last 76.78)
 - open/close 2173 (time 1789055262.58 total 30.03 last 66.02)
 - open/close 2807 (time 1789055272.59 total 40.03 last 63.38)
 - open/close 3516 (time 1789055282.59 total 50.04 last 70.88)
 - open/close 4187 (time 1789055292.60 total 60.04 last 67.06)
total: 4500 open/close in 64.51 seconds: 69.76 ops/second
 - /unlink 1290 (time 1789055311.70 total 10.00 last 128.97)
 - /unlink 2490 (time 1789055321.70 total 20.01 last 119.90)
 - /unlink 3713 (time 1789055331.71 total 30.01 last 122.25)
total: 4500 /unlink in 36.95 seconds: 121.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 723 (time 1789055353.49 total 10.01 last 72.22)
 - open/close 1395 (time 1789055363.49 total 20.01 last 67.18)
 - open/close 2112 (time 1789055373.50 total 30.03 last 71.62)
 - open/close 2764 (time 1789055383.52 total 40.04 last 65.10)
 - open/close 3398 (time 1789055393.53 total 50.05 last 63.35)
 - open/close 4038 (time 1789055403.54 total 60.06 last 63.94)
total: 4500 open/close in 67.83 seconds: 66.34 ops/second
 - /unlink 1254 (time 1789055426.78 total 10.00 last 125.35)
 - /unlink 2342 (time 1789055436.78 total 20.01 last 108.77)
 - /unlink 3517 (time 1789055446.79 total 30.01 last 117.43)
total: 4500 /unlink in 38.00 seconds: 118.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 572 (time 1789055471.29 total 10.02 last 57.08)
 - open/close 1098 (time 1789055481.29 total 20.02 last 52.59)
 - open/close 1761 (time 1789055491.32 total 30.06 last 66.09)
 - open/close 2455 (time 1789055501.33 total 40.06 last 69.33)
 - open/close 3114 (time 1789055511.34 total 50.07 last 65.88)
 - open/close 3761 (time 1789055521.34 total 60.07 last 64.66)
 - open/close 4467 (time 1789055531.34 total 70.08 last 70.59)
total: 4500 open/close in 70.51 seconds: 63.82 ops/second
 - /unlink 875 (time 1789055548.00 total 10.00 last 87.46)
 - /unlink 1974 (time 1789055558.00 total 20.01 last 109.88)
 - /unlink 2966 (time 1789055568.00 total 30.01 last 99.16)
 - /unlink 4175 (time 1789055578.00 total 40.01 last 120.90)
total: 4500 /unlink in 42.82 seconds: 105.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 581 (time 1789055596.66 total 10.02 last 58.01)
 - open/close 1005 (time 1789055606.68 total 20.03 last 42.32)
 - open/close 1497 (time 1789055616.69 total 30.04 last 49.15)
 - open/close 1943 (time 1789055626.71 total 40.07 last 44.49)
 - open/close 2522 (time 1789055636.72 total 50.07 last 57.86)
 - open/close 3069 (time 1789055646.73 total 60.08 last 54.65)
 - open/close 3625 (time 1789055656.75 total 70.10 last 55.49)
 - open/close 4052 (time 1789055666.76 total 80.11 last 42.66)
total: 4500 open/close in 88.74 seconds: 50.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1114 (time 1789055691.45 total 10.00 last 111.39)
 - /unlink 2256 (time 1789055701.45 total 20.00 last 114.19)
 - /unlink 3461 (time 1789055711.46 total 30.01 last 120.41)
total: 4500 /unlink in 38.79 seconds: 116.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 571 (time 1789055736.79 total 10.01 last 57.05)
 - open/close 1209 (time 1789055746.81 total 20.02 last 63.69)
 - open/close 1809 (time 1789055756.81 total 30.03 last 59.98)
 - open/close 2449 (time 1789055766.81 total 40.03 last 63.99)
 - open/close 3026 (time 1789055776.82 total 50.04 last 57.64)
 - open/close 3594 (time 1789055786.84 total 60.05 last 56.73)
 - open/close 4186 (time 1789055796.85 total 70.07 last 59.11)
total: 4500 open/close in 74.93 seconds: 60.05 ops/second
 - /unlink 1262 (time 1789055816.31 total 10.00 last 126.15)
 - /unlink 2596 (time 1789055826.31 total 20.01 last 133.36)
 - /unlink 3962 (time 1789055836.31 total 30.01 last 136.55)
total: 4500 /unlink in 34.14 seconds: 131.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 662 (time 1789055854.97 total 10.01 last 66.11)
 - open/close 1338 (time 1789055864.98 total 20.02 last 67.56)
 - open/close 1928 (time 1789055874.98 total 30.02 last 58.99)
 - open/close 2615 (time 1789055884.99 total 40.03 last 68.65)
 - open/close 3260 (time 1789055895.00 total 50.04 last 64.39)
 - open/close 3950 (time 1789055905.02 total 60.06 last 68.92)
total: 4500 open/close in 67.06 seconds: 67.10 ops/second
 - /unlink 1183 (time 1789055927.39 total 10.01 last 118.22)
 - /unlink 2354 (time 1789055937.41 total 20.02 last 116.96)
 - /unlink 3527 (time 1789055947.41 total 30.02 last 117.26)
 - /unlink 4414 (time 1789055957.41 total 40.02 last 88.70)
total: 4500 /unlink in 41.12 seconds: 109.42 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95256 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg319-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg319-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server
stop mds service on oleg319-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server
stop mds service on oleg319-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server
unloading modules via unload_modules_local on: 'oleg319-server'
oleg319-server: oleg319-server.virtnet: executing unload_modules_local
oleg319-server: kmemleak disabled
modules unloaded.