== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:06:23 (1788584783)
Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg207-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg207-server
oleg207-server: oleg207-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg207-server'
oleg207-server: oleg207-server.virtnet: executing load_modules_local
oleg207-server: kmemleak disabled
oleg207-server: Loading modules from /home/green/git/lustre-release/lustre
oleg207-server: detected 4 online CPUs by sysfs
oleg207-server: MODOPTS_LIBCFS=
oleg207-server: Force libcfs to create 2 CPU partitions
oleg207-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg207-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all
pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg207-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all
pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg207-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all
pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg207-client.virtnet:  -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 617 (time 1788585019.02 total 10.01 last 61.65)
 - open/close 1236 (time 1788585029.03 total 20.01 last 61.88)
 - open/close 2047 (time 1788585039.03 total 30.01 last 81.09)
 - open/close 2860 (time 1788585049.04 total 40.02 last 81.25)
 - open/close 3575 (time 1788585059.04 total 50.02 last 71.46)
 - open/close 4365 (time 1788585069.05 total 60.03 last 78.95)
total: 4500 open/close in 62.25 seconds: 72.29 ops/second
 - /unlink 1241 (time 1788585086.24 total 10.01 last 124.03)
 - /unlink 2489 (time 1788585096.24 total 20.01 last 124.73)
 - /unlink 3659 (time 1788585106.24 total 30.01 last 116.99)
total: 4500 /unlink in 36.31 seconds: 123.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 507 (time 1788585126.56 total 10.02 last 50.61)
 - open/close 934 (time 1788585136.57 total 20.03 last 42.66)
 - open/close 1347 (time 1788585146.58 total 30.03 last 41.27)
 - open/close 1807 (time 1788585156.60 total 40.05 last 45.92)
 - open/close 2369 (time 1788585166.63 total 50.08 last 56.03)
 - open/close 3155 (time 1788585176.63 total 60.08 last 78.60)
 - open/close 3976 (time 1788585186.63 total 70.09 last 82.07)
total: 4500 open/close in 76.68 seconds: 58.68 ops/second
 - /unlink 1400 (time 1788585207.81 total 10.01 last 139.86)
 - /unlink 2731 (time 1788585217.81 total 20.01 last 133.10)
 - /unlink 3964 (time 1788585227.81 total 30.01 last 123.30)
total: 4500 /unlink in 33.22 seconds: 135.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 628 (time 1788585246.88 total 10.01 last 62.75)
 - open/close 1331 (time 1788585256.89 total 20.01 last 70.27)
 - open/close 2120 (time 1788585266.91 total 30.03 last 78.75)
 - open/close 2860 (time 1788585276.91 total 40.03 last 73.98)
 - open/close 3484 (time 1788585286.92 total 50.04 last 62.35)
 - open/close 4173 (time 1788585296.92 total 60.05 last 68.87)
total: 4500 open/close in 63.95 seconds: 70.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1427 (time 1788585316.23 total 10.00 last 142.68)
 - /unlink 2543 (time 1788585326.24 total 20.00 last 111.57)
 - /unlink 3580 (time 1788585336.25 total 30.01 last 103.61)
total: 4500 /unlink in 38.78 seconds: 116.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 847 (time 1788585360.32 total 10.00 last 84.67)
 - open/close 1620 (time 1788585370.33 total 20.01 last 77.25)
 - open/close 2196 (time 1788585380.38 total 30.06 last 57.29)
 - open/close 2887 (time 1788585390.38 total 40.07 last 69.06)
 - open/close 3538 (time 1788585400.40 total 50.09 last 64.98)
 - open/close 4332 (time 1788585410.41 total 60.09 last 79.36)
total: 4500 open/close in 62.70 seconds: 71.77 ops/second
 - /unlink 1517 (time 1788585428.01 total 10.00 last 151.69)
 - /unlink 2716 (time 1788585438.02 total 20.00 last 119.87)
 - /unlink 4346 (time 1788585448.02 total 30.01 last 162.96)
total: 4500 /unlink in 30.93 seconds: 145.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 679 (time 1788585464.03 total 10.00 last 67.90)
 - open/close 1398 (time 1788585474.03 total 20.00 last 71.90)
 - open/close 2219 (time 1788585484.04 total 30.02 last 81.98)
 - open/close 3102 (time 1788585494.05 total 40.02 last 88.26)
 - open/close 3810 (time 1788585504.05 total 50.03 last 70.75)
 - open/close 4436 (time 1788585514.06 total 60.04 last 62.54)
total: 4500 open/close in 60.98 seconds: 73.79 ops/second
 - /unlink 1268 (time 1788585529.53 total 10.00 last 126.79)
 - /unlink 2320 (time 1788585539.53 total 20.00 last 105.16)
 - /unlink 3632 (time 1788585549.54 total 30.01 last 131.08)
total: 4500 /unlink in 36.93 seconds: 121.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 764 (time 1788585571.33 total 10.00 last 76.39)
 - open/close 1450 (time 1788585581.34 total 20.01 last 68.52)
 - open/close 2111 (time 1788585591.34 total 30.01 last 66.10)
 - open/close 2957 (time 1788585601.34 total 40.01 last 84.59)
 - open/close 3816 (time 1788585611.34 total 50.02 last 85.89)
total: 4500 open/close in 59.65 seconds: 75.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1279 (time 1788585635.46 total 10.00 last 127.86)
 - /unlink 2575 (time 1788585645.46 total 20.01 last 129.50)
 - /unlink 4043 (time 1788585655.47 total 30.02 last 146.66)
total: 4500 /unlink in 33.23 seconds: 135.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 727 (time 1788585673.79 total 10.01 last 72.64)
 - open/close 1465 (time 1788585683.79 total 20.01 last 73.78)
 - open/close 2234 (time 1788585693.80 total 30.02 last 76.87)
 - open/close 2923 (time 1788585703.80 total 40.02 last 68.89)
 - open/close 3548 (time 1788585713.81 total 50.02 last 62.45)
 - open/close 4250 (time 1788585723.81 total 60.03 last 70.14)
total: 4500 open/close in 63.35 seconds: 71.03 ops/second
 - /unlink 1455 (time 1788585741.58 total 10.01 last 145.40)
 - /unlink 3061 (time 1788585751.59 total 20.01 last 160.57)
 - /unlink 4482 (time 1788585761.59 total 30.01 last 142.03)
total: 4500 /unlink in 30.12 seconds: 149.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 796 (time 1788585776.46 total 10.02 last 79.48)
 - open/close 1555 (time 1788585786.47 total 20.02 last 75.87)
 - open/close 2430 (time 1788585796.47 total 30.02 last 87.47)
 - open/close 2983 (time 1788585806.49 total 40.04 last 55.19)
 - open/close 3679 (time 1788585816.50 total 50.05 last 69.53)
 - open/close 4261 (time 1788585826.51 total 60.06 last 58.15)
total: 4500 open/close in 64.24 seconds: 70.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1192 (time 1788585845.79 total 10.01 last 119.11)
 - /unlink 2467 (time 1788585855.80 total 20.01 last 127.41)
 - /unlink 3645 (time 1788585865.81 total 30.02 last 117.70)
total: 4500 /unlink in 37.67 seconds: 119.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 725 (time 1788585888.97 total 10.00 last 72.49)
 - open/close 1403 (time 1788585898.98 total 20.02 last 67.71)
 - open/close 2231 (time 1788585908.98 total 30.02 last 82.77)
 - open/close 2823 (time 1788585918.99 total 40.03 last 59.16)
 - open/close 3409 (time 1788585929.00 total 50.04 last 58.51)
 - open/close 4104 (time 1788585939.01 total 60.04 last 69.49)
total: 4500 open/close in 68.03 seconds: 66.15 ops/second
 - /unlink 1252 (time 1788585963.27 total 10.01 last 125.09)
 - /unlink 2403 (time 1788585973.28 total 20.02 last 114.97)
 - /unlink 3806 (time 1788585983.28 total 30.02 last 140.25)
total: 4500 /unlink in 35.54 seconds: 126.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 739 (time 1788586003.95 total 10.00 last 73.89)
 - open/close 1532 (time 1788586013.96 total 20.01 last 79.24)
 - open/close 2386 (time 1788586023.97 total 30.02 last 85.33)
 - open/close 3169 (time 1788586033.97 total 40.02 last 78.29)
 - open/close 3991 (time 1788586043.97 total 50.02 last 82.17)
total: 4500 open/close in 56.65 seconds: 79.43 ops/second
 - /unlink 1200 (time 1788586066.60 total 10.03 last 119.66)
 - /unlink 2300 (time 1788586076.61 total 20.03 last 109.93)
 - /unlink 3494 (time 1788586086.61 total 30.04 last 119.38)
total: 4500 /unlink in 37.78 seconds: 119.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 698 (time 1788586109.35 total 10.01 last 69.75)
 - open/close 1306 (time 1788586119.35 total 20.01 last 60.80)
 - open/close 2020 (time 1788586129.36 total 30.01 last 71.36)
 - open/close 2797 (time 1788586139.36 total 40.02 last 77.67)
 - open/close 3589 (time 1788586149.37 total 50.03 last 79.13)
 - open/close 4326 (time 1788586159.37 total 60.03 last 73.66)
total: 4500 open/close in 62.50 seconds: 72.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1426 (time 1788586176.57 total 10.00 last 142.55)
 - /unlink 2958 (time 1788586186.57 total 20.00 last 153.19)
 - /unlink 4374 (time 1788586196.58 total 30.01 last 141.45)
total: 4500 /unlink in 31.27 seconds: 143.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 845 (time 1788586211.91 total 10.01 last 84.44)
 - open/close 1611 (time 1788586221.92 total 20.02 last 76.50)
 - open/close 2341 (time 1788586231.94 total 30.03 last 72.90)
 - open/close 3039 (time 1788586241.94 total 40.04 last 69.76)
 - open/close 3730 (time 1788586251.96 total 50.06 last 68.94)
total: 4500 open/close in 60.05 seconds: 74.94 ops/second
 - /unlink 1323 (time 1788586276.69 total 10.00 last 132.24)
 - /unlink 2591 (time 1788586286.69 total 20.00 last 126.80)
 - /unlink 3795 (time 1788586296.69 total 30.01 last 120.38)
total: 4500 /unlink in 35.18 seconds: 127.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 792 (time 1788586316.84 total 10.02 last 79.05)
 - open/close 1536 (time 1788586326.84 total 20.02 last 74.38)
 - open/close 2267 (time 1788586336.85 total 30.03 last 73.06)
 - open/close 2891 (time 1788586346.85 total 40.03 last 62.38)
 - open/close 3495 (time 1788586356.86 total 50.03 last 60.39)
 - open/close 4095 (time 1788586366.87 total 60.05 last 59.89)
total: 4500 open/close in 66.60 seconds: 67.57 ops/second
 - /unlink 1362 (time 1788586387.97 total 10.00 last 136.20)
 - /unlink 2567 (time 1788586397.97 total 20.00 last 120.48)
 - /unlink 3850 (time 1788586407.97 total 30.00 last 128.30)
total: 4500 /unlink in 34.44 seconds: 130.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 673 (time 1788586427.56 total 10.02 last 67.18)
 - open/close 1448 (time 1788586437.57 total 20.03 last 77.44)
 - open/close 2155 (time 1788586447.57 total 30.03 last 70.68)
 - open/close 2952 (time 1788586457.58 total 40.04 last 79.62)
 - open/close 3671 (time 1788586467.59 total 50.05 last 71.85)
 - open/close 4409 (time 1788586477.60 total 60.05 last 73.73)
total: 4500 open/close in 61.55 seconds: 73.11 ops/second
 - /unlink 1411 (time 1788586493.65 total 10.00 last 141.05)
 - /unlink 2707 (time 1788586503.65 total 20.00 last 129.58)
 - /unlink 4225 (time 1788586513.65 total 30.01 last 151.73)
total: 4500 /unlink in 32.37 seconds: 139.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 679 (time 1788586532.15 total 10.02 last 67.80)
 - open/close 1332 (time 1788586542.16 total 20.02 last 65.27)
 - open/close 1943 (time 1788586552.16 total 30.03 last 61.05)
 - open/close 2648 (time 1788586562.18 total 40.05 last 70.38)
 - open/close 3436 (time 1788586572.19 total 50.05 last 78.76)
 - open/close 4144 (time 1788586582.21 total 60.07 last 70.66)
total: 4500 open/close in 64.88 seconds: 69.36 ops/second
 - /unlink 1464 (time 1788586602.04 total 10.00 last 146.39)
 - /unlink 2751 (time 1788586612.04 total 20.00 last 128.67)
 - /unlink 3957 (time 1788586622.04 total 30.00 last 120.58)
total: 4500 /unlink in 33.56 seconds: 134.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 742 (time 1788586641.50 total 10.02 last 74.04)
 - open/close 1444 (time 1788586651.51 total 20.03 last 70.12)
 - open/close 2253 (time 1788586661.52 total 30.04 last 80.80)
 - open/close 3014 (time 1788586671.52 total 40.05 last 76.09)
 - open/close 3690 (time 1788586681.53 total 50.05 last 67.55)
 - open/close 4396 (time 1788586691.54 total 60.06 last 70.53)
total: 4500 open/close in 61.46 seconds: 73.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1432 (time 1788586708.07 total 10.00 last 143.16)
 - /unlink 2928 (time 1788586718.07 total 20.00 last 149.58)
 - /unlink 4395 (time 1788586728.07 total 30.01 last 146.68)
total: 4500 /unlink in 31.20 seconds: 144.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 771 (time 1788586743.84 total 10.01 last 77.03)
 - open/close 1481 (time 1788586753.85 total 20.02 last 70.92)
 - open/close 2248 (time 1788586763.86 total 30.04 last 76.56)
 - open/close 3045 (time 1788586773.86 total 40.04 last 79.70)
 - open/close 3824 (time 1788586783.88 total 50.05 last 77.80)
total: 4500 open/close in 58.87 seconds: 76.44 ops/second
 - /unlink 1273 (time 1788586806.93 total 10.00 last 127.29)
 - /unlink 2672 (time 1788586816.94 total 20.01 last 139.79)
 - /unlink 3949 (time 1788586826.94 total 30.01 last 127.68)
total: 4500 /unlink in 33.71 seconds: 133.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 703 (time 1788586845.66 total 10.00 last 70.29)
 - open/close 1422 (time 1788586855.66 total 20.00 last 71.89)
 - open/close 2165 (time 1788586865.68 total 30.02 last 74.21)
 - open/close 2903 (time 1788586875.69 total 40.03 last 73.68)
 - open/close 3700 (time 1788586885.69 total 50.03 last 79.69)
 - open/close 4427 (time 1788586895.70 total 60.04 last 72.67)
total: 4500 open/close in 60.95 seconds: 73.83 ops/second
 - /unlink 1425 (time 1788586911.38 total 10.01 last 142.34)
 - /unlink 2940 (time 1788586921.38 total 20.01 last 151.48)
 - /unlink 4286 (time 1788586931.38 total 30.02 last 134.54)
total: 4500 /unlink in 31.90 seconds: 141.07 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95371 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg207-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg207-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server
stop mds service on oleg207-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server
stop mds service on oleg207-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server
unloading modules via unload_modules_local on: 'oleg207-server'
oleg207-server: oleg207-server.virtnet: executing unload_modules_local
oleg207-server: kmemleak disabled
modules unloaded.