== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:46:21 (1789785981)
Stopping clients: oleg628-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg628-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg628-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg628-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg628-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg628-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg628-server
oleg628-server: oleg628-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg628-server'
oleg628-server: oleg628-server.virtnet: executing load_modules_local
oleg628-server: kmemleak disabled
oleg628-server: Loading modules from /home/green/git/lustre-release/lustre
oleg628-server: detected 4 online CPUs by sysfs
oleg628-server: MODOPTS_LIBCFS=
oleg628-server: Force libcfs to create 2 CPU partitions
oleg628-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg628-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg628-server: oleg628-server.virtnet: executing set_default_debug -1 all
pdsh@oleg628-client: oleg628-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg628-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg628-server: oleg628-server.virtnet: executing set_default_debug -1 all
pdsh@oleg628-client: oleg628-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg628-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg628-server: oleg628-server.virtnet: executing set_default_debug -1 all
pdsh@oleg628-client: oleg628-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg628-client.virtnet:  -o user_xattr,flock 192.168.206.128@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 630 (time 1789786180.37 total 10.00 last 62.98)
 - open/close 1383 (time 1789786190.37 total 20.01 last 75.27)
 - open/close 2127 (time 1789786200.38 total 30.01 last 74.35)
 - open/close 2899 (time 1789786210.40 total 40.03 last 77.08)
 - open/close 3701 (time 1789786220.40 total 50.04 last 80.16)
 - open/close 4456 (time 1789786230.40 total 60.04 last 75.47)
total: 4500 open/close in 60.71 seconds: 74.12 ops/second
 - /unlink 1445 (time 1789786245.04 total 10.01 last 144.37)
 - /unlink 2806 (time 1789786255.05 total 20.01 last 136.06)
 - /unlink 4193 (time 1789786265.05 total 30.01 last 138.66)
total: 4500 /unlink in 32.14 seconds: 140.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 673 (time 1789786281.06 total 10.00 last 67.29)
 - open/close 1346 (time 1789786291.07 total 20.01 last 67.26)
 - open/close 2049 (time 1789786301.08 total 30.02 last 70.21)
 - open/close 2785 (time 1789786311.10 total 40.04 last 73.48)
 - open/close 3535 (time 1789786321.10 total 50.04 last 74.98)
 - open/close 4236 (time 1789786331.11 total 60.05 last 70.03)
total: 4500 open/close in 63.40 seconds: 70.98 ops/second
 - /unlink 1316 (time 1789786349.12 total 10.00 last 131.55)
 - /unlink 2603 (time 1789786359.12 total 20.01 last 128.67)
 - /unlink 3942 (time 1789786369.13 total 30.02 last 133.77)
total: 4500 /unlink in 34.16 seconds: 131.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 688 (time 1789786387.77 total 10.00 last 68.78)
 - open/close 1381 (time 1789786397.78 total 20.01 last 69.25)
 - open/close 2157 (time 1789786407.79 total 30.02 last 77.54)
 - open/close 2940 (time 1789786417.80 total 40.03 last 78.19)
 - open/close 3657 (time 1789786427.81 total 50.04 last 71.65)
 - open/close 4402 (time 1789786437.81 total 60.04 last 74.48)
total: 4500 open/close in 61.37 seconds: 73.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1222 (time 1789786454.10 total 10.01 last 122.13)
 - /unlink 2670 (time 1789786464.11 total 20.01 last 144.72)
 - /unlink 4149 (time 1789786474.11 total 30.01 last 147.90)
total: 4500 /unlink in 32.39 seconds: 138.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 826 (time 1789786490.27 total 10.00 last 82.59)
 - open/close 1644 (time 1789786500.27 total 20.00 last 81.79)
 - open/close 2481 (time 1789786510.28 total 30.01 last 83.62)
 - open/close 3240 (time 1789786520.29 total 40.02 last 75.81)
 - open/close 4043 (time 1789786530.30 total 50.03 last 80.26)
total: 4500 open/close in 56.43 seconds: 79.75 ops/second
 - /unlink 1468 (time 1789786551.01 total 10.01 last 146.66)
 - /unlink 3018 (time 1789786561.03 total 20.03 last 154.74)
 - /unlink 4492 (time 1789786571.04 total 30.04 last 147.26)
total: 4500 /unlink in 30.10 seconds: 149.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 763 (time 1789786585.97 total 10.01 last 76.26)
 - open/close 1484 (time 1789786595.97 total 20.01 last 72.10)
 - open/close 2250 (time 1789786605.97 total 30.01 last 76.57)
 - open/close 2971 (time 1789786615.97 total 40.01 last 72.08)
 - open/close 3788 (time 1789786625.98 total 50.01 last 81.67)
total: 4500 open/close in 59.57 seconds: 75.54 ops/second
 - /unlink 1468 (time 1789786650.62 total 10.00 last 146.78)
 - /unlink 2748 (time 1789786660.62 total 20.01 last 127.94)
 - /unlink 4307 (time 1789786670.62 total 30.01 last 155.89)
total: 4500 /unlink in 31.50 seconds: 142.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 810 (time 1789786686.70 total 10.03 last 80.77)
 - open/close 1483 (time 1789786696.70 total 20.03 last 67.27)
 - open/close 2148 (time 1789786706.71 total 30.04 last 66.44)
 - open/close 2902 (time 1789786716.73 total 40.06 last 75.24)
 - open/close 3649 (time 1789786726.74 total 50.07 last 74.67)
 - open/close 4315 (time 1789786736.74 total 60.07 last 66.55)
total: 4500 open/close in 62.49 seconds: 72.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1407 (time 1789786753.20 total 10.01 last 140.53)
 - /unlink 2912 (time 1789786763.20 total 20.01 last 150.48)
 - /unlink 4310 (time 1789786773.21 total 30.02 last 139.65)
total: 4500 /unlink in 31.55 seconds: 142.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 836 (time 1789786788.67 total 10.01 last 83.54)
 - open/close 1634 (time 1789786798.68 total 20.01 last 79.78)
 - open/close 2413 (time 1789786808.68 total 30.01 last 77.88)
 - open/close 3164 (time 1789786818.69 total 40.02 last 75.03)
 - open/close 3921 (time 1789786828.69 total 50.03 last 75.67)
total: 4500 open/close in 58.24 seconds: 77.27 ops/second
 - /unlink 1488 (time 1789786851.23 total 10.00 last 148.74)
 - /unlink 2785 (time 1789786861.24 total 20.02 last 129.50)
 - /unlink 4206 (time 1789786871.25 total 30.02 last 142.06)
total: 4500 /unlink in 32.14 seconds: 140.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 657 (time 1789786887.66 total 10.00 last 65.67)
 - open/close 1371 (time 1789786897.68 total 20.02 last 71.28)
 - open/close 2121 (time 1789786907.68 total 30.02 last 74.99)
 - open/close 2830 (time 1789786917.69 total 40.03 last 70.87)
 - open/close 3544 (time 1789786927.70 total 50.04 last 71.33)
 - open/close 4301 (time 1789786937.71 total 60.05 last 75.63)
total: 4500 open/close in 62.49 seconds: 72.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1312 (time 1789786953.85 total 10.00 last 131.17)
 - /unlink 2728 (time 1789786963.85 total 20.01 last 141.56)
 - /unlink 4123 (time 1789786973.87 total 30.02 last 139.27)
total: 4500 /unlink in 33.68 seconds: 133.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 671 (time 1789786993.07 total 10.01 last 67.04)
 - open/close 1420 (time 1789787003.09 total 20.02 last 74.81)
 - open/close 2164 (time 1789787013.09 total 30.02 last 74.38)
 - open/close 2901 (time 1789787023.09 total 40.03 last 73.66)
 - open/close 3694 (time 1789787033.09 total 50.03 last 79.29)
 - open/close 4465 (time 1789787043.11 total 60.04 last 77.00)
total: 4500 open/close in 60.51 seconds: 74.37 ops/second
 - /unlink 1637 (time 1789787058.11 total 10.00 last 163.67)
 - /unlink 3121 (time 1789787068.11 total 20.00 last 148.40)
 - /unlink 4490 (time 1789787078.12 total 30.01 last 136.80)
total: 4500 /unlink in 30.07 seconds: 149.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 689 (time 1789787093.24 total 10.01 last 68.80)
 - open/close 1420 (time 1789787103.25 total 20.02 last 73.08)
 - open/close 2233 (time 1789787113.26 total 30.03 last 81.21)
 - open/close 2959 (time 1789787123.26 total 40.03 last 72.58)
 - open/close 3731 (time 1789787133.27 total 50.04 last 77.15)
 - open/close 4475 (time 1789787143.27 total 60.04 last 74.37)
total: 4500 open/close in 60.41 seconds: 74.49 ops/second
 - /unlink 1434 (time 1789787157.45 total 10.00 last 143.36)
 - /unlink 2767 (time 1789787167.45 total 20.01 last 133.24)
 - /unlink 4355 (time 1789787177.45 total 30.01 last 158.76)
total: 4500 /unlink in 30.99 seconds: 145.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 762 (time 1789787193.08 total 10.01 last 76.11)
 - open/close 1440 (time 1789787203.08 total 20.01 last 67.80)
 - open/close 2160 (time 1789787213.09 total 30.02 last 71.96)
 - open/close 2911 (time 1789787223.10 total 40.02 last 75.05)
 - open/close 3704 (time 1789787233.10 total 50.03 last 79.27)
total: 4500 open/close in 59.80 seconds: 75.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1437 (time 1789787256.16 total 10.00 last 143.67)
 - /unlink 2837 (time 1789787266.16 total 20.01 last 139.91)
 - /unlink 4116 (time 1789787276.17 total 30.01 last 127.85)
total: 4500 /unlink in 32.70 seconds: 137.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 676 (time 1789787294.99 total 10.02 last 67.49)
 - open/close 1412 (time 1789787305.00 total 20.02 last 73.54)
 - open/close 2155 (time 1789787315.03 total 30.05 last 74.10)
 - open/close 2861 (time 1789787325.03 total 40.06 last 70.56)
 - open/close 3529 (time 1789787335.04 total 50.06 last 66.76)
 - open/close 4208 (time 1789787345.04 total 60.06 last 67.89)
total: 4500 open/close in 64.31 seconds: 69.97 ops/second
 - /unlink 1304 (time 1789787364.35 total 10.00 last 130.40)
 - /unlink 2630 (time 1789787374.36 total 20.01 last 132.51)
 - /unlink 3912 (time 1789787384.38 total 30.02 last 127.99)
total: 4500 /unlink in 34.74 seconds: 129.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 671 (time 1789787404.25 total 10.01 last 67.05)
 - open/close 1285 (time 1789787414.25 total 20.01 last 61.38)
 - open/close 1954 (time 1789787424.27 total 30.03 last 66.79)
 - open/close 2555 (time 1789787434.28 total 40.04 last 60.04)
 - open/close 3134 (time 1789787444.29 total 50.05 last 57.83)
 - open/close 3863 (time 1789787454.31 total 60.06 last 72.81)
total: 4500 open/close in 68.81 seconds: 65.40 ops/second
 - /unlink 1312 (time 1789787479.19 total 10.00 last 131.15)
 - /unlink 2603 (time 1789787489.20 total 20.01 last 129.03)
 - /unlink 3955 (time 1789787499.20 total 30.01 last 135.18)
total: 4500 /unlink in 34.24 seconds: 131.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 683 (time 1789787519.00 total 10.00 last 68.28)
 - open/close 1317 (time 1789787529.02 total 20.02 last 63.31)
 - open/close 2027 (time 1789787539.02 total 30.02 last 70.94)
 - open/close 2698 (time 1789787549.03 total 40.03 last 67.08)
 - open/close 3294 (time 1789787559.04 total 50.04 last 59.53)
 - open/close 3916 (time 1789787569.05 total 60.05 last 62.13)
total: 4500 open/close in 68.93 seconds: 65.29 ops/second
 - /unlink 1321 (time 1789787592.65 total 10.00 last 132.04)
 - /unlink 2666 (time 1789787602.65 total 20.01 last 134.47)
 - /unlink 4107 (time 1789787612.67 total 30.02 last 143.86)
total: 4500 /unlink in 33.09 seconds: 135.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 617 (time 1789787630.74 total 10.01 last 61.65)
 - open/close 1249 (time 1789787640.76 total 20.02 last 63.12)
 - open/close 1964 (time 1789787650.76 total 30.03 last 71.45)
 - open/close 2719 (time 1789787660.77 total 40.03 last 75.46)
 - open/close 3511 (time 1789787670.77 total 50.03 last 79.19)
 - open/close 4263 (time 1789787680.77 total 60.04 last 75.16)
total: 4500 open/close in 63.81 seconds: 70.52 ops/second
 - /unlink 1306 (time 1789787699.10 total 10.00 last 130.54)
 - /unlink 2687 (time 1789787709.11 total 20.02 last 137.92)
 - /unlink 4072 (time 1789787719.12 total 30.02 last 138.42)
total: 4500 /unlink in 33.27 seconds: 135.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 708 (time 1789787737.90 total 10.02 last 70.63)
 - open/close 1520 (time 1789787747.91 total 20.03 last 81.15)
 - open/close 2243 (time 1789787757.91 total 30.04 last 72.26)
 - open/close 2875 (time 1789787767.93 total 40.05 last 63.08)
 - open/close 3480 (time 1789787777.93 total 50.06 last 60.49)
 - open/close 4201 (time 1789787787.93 total 60.06 last 72.09)
total: 4500 open/close in 64.70 seconds: 69.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1392 (time 1789787806.56 total 10.01 last 139.13)
 - /unlink 2704 (time 1789787816.56 total 20.01 last 131.15)
 - /unlink 4063 (time 1789787826.57 total 30.01 last 135.84)
total: 4500 /unlink in 33.85 seconds: 132.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 714 (time 1789787844.73 total 10.01 last 71.36)
 - open/close 1425 (time 1789787854.75 total 20.02 last 70.96)
 - open/close 2134 (time 1789787864.75 total 30.03 last 70.89)
 - open/close 2929 (time 1789787874.75 total 40.03 last 79.45)
 - open/close 3654 (time 1789787884.76 total 50.04 last 72.47)
 - open/close 4272 (time 1789787894.78 total 60.06 last 61.64)
total: 4500 open/close in 64.43 seconds: 69.84 ops/second
 - /unlink 1145 (time 1789787914.47 total 10.03 last 114.17)
 - /unlink 2246 (time 1789787924.48 total 20.03 last 110.06)
 - /unlink 3360 (time 1789787934.49 total 30.05 last 111.24)
total: 4500 /unlink in 38.07 seconds: 118.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 656 (time 1789787958.40 total 10.01 last 65.53)
 - open/close 1260 (time 1789787968.40 total 20.01 last 60.38)
 - open/close 1872 (time 1789787978.41 total 30.02 last 61.14)
 - open/close 2508 (time 1789787988.44 total 40.05 last 63.45)
 - open/close 3207 (time 1789787998.45 total 50.06 last 69.82)
 - open/close 3917 (time 1789788008.45 total 60.06 last 70.96)
total: 4500 open/close in 68.74 seconds: 65.47 ops/second
 - /unlink 1400 (time 1789788031.75 total 10.00 last 139.96)
 - /unlink 2728 (time 1789788041.76 total 20.01 last 132.67)
 - /unlink 3917 (time 1789788051.76 total 30.02 last 118.86)
total: 4500 /unlink in 34.09 seconds: 132.01 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95307 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg628-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg628-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg628-server
stop mds service on oleg628-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg628-server
stop mds service on oleg628-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg628-server
unloading modules via unload_modules_local on: 'oleg628-server'
oleg628-server: oleg628-server.virtnet: executing unload_modules_local
oleg628-server: kmemleak disabled
modules unloaded.