== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:25:58 (1789493158)
Stopping clients: oleg426-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg426-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg426-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg426-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg426-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg426-server
oleg426-server: oleg426-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg426-server'
oleg426-server: oleg426-server.virtnet: executing load_modules_local
oleg426-server: kmemleak disabled
oleg426-server: Loading modules from /home/green/git/lustre-release/lustre
oleg426-server: detected 4 online CPUs by sysfs
oleg426-server: MODOPTS_LIBCFS=
oleg426-server: Force libcfs to create 2 CPU partitions
oleg426-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg426-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all
pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg426-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all
pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg426-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all
pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg426-client.virtnet:  -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 816 (time 1789493357.47 total 10.01 last 81.55)
 - open/close 1598 (time 1789493367.47 total 20.01 last 78.19)
 - open/close 2352 (time 1789493377.48 total 30.02 last 75.31)
 - open/close 3034 (time 1789493387.50 total 40.04 last 68.09)
 - open/close 3762 (time 1789493397.50 total 50.04 last 72.77)
total: 4500 open/close in 59.52 seconds: 75.61 ops/second
 - /unlink 1368 (time 1789493421.93 total 10.01 last 136.66)
 - /unlink 2785 (time 1789493431.93 total 20.01 last 141.69)
 - /unlink 4333 (time 1789493441.94 total 30.02 last 154.68)
total: 4500 /unlink in 31.72 seconds: 141.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 726 (time 1789493458.55 total 10.00 last 72.60)
 - open/close 1531 (time 1789493468.55 total 20.00 last 80.49)
 - open/close 2298 (time 1789493478.57 total 30.02 last 76.55)
 - open/close 3145 (time 1789493488.58 total 40.03 last 84.64)
 - open/close 3969 (time 1789493498.58 total 50.03 last 82.39)
total: 4500 open/close in 57.21 seconds: 78.66 ops/second
 - /unlink 1397 (time 1789493520.77 total 10.00 last 139.69)
 - /unlink 2594 (time 1789493530.77 total 20.00 last 119.68)
 - /unlink 3656 (time 1789493540.78 total 30.01 last 106.15)
total: 4500 /unlink in 36.26 seconds: 124.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 857 (time 1789493563.08 total 10.02 last 85.55)
 - open/close 1612 (time 1789493573.08 total 20.02 last 75.47)
 - open/close 2352 (time 1789493583.09 total 30.03 last 73.93)
 - open/close 3146 (time 1789493593.11 total 40.05 last 79.22)
 - open/close 3936 (time 1789493603.12 total 50.06 last 78.94)
 - open/close 4485 (time 1789493613.12 total 60.06 last 54.88)
total: 4500 open/close in 60.27 seconds: 74.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1382 (time 1789493627.36 total 10.00 last 138.18)
 - /unlink 2669 (time 1789493637.36 total 20.00 last 128.68)
 - /unlink 4258 (time 1789493647.38 total 30.03 last 158.56)
total: 4500 /unlink in 31.70 seconds: 141.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 724 (time 1789493663.73 total 10.01 last 72.30)
 - open/close 1493 (time 1789493673.74 total 20.02 last 76.83)
 - open/close 2266 (time 1789493683.75 total 30.03 last 77.25)
 - open/close 2961 (time 1789493693.77 total 40.05 last 69.32)
 - open/close 3689 (time 1789493703.78 total 50.06 last 72.77)
total: 4500 open/close in 59.33 seconds: 75.85 ops/second
 - /unlink 1600 (time 1789493727.46 total 10.01 last 159.84)
 - /unlink 2980 (time 1789493737.47 total 20.02 last 137.87)
total: 4500 /unlink in 29.68 seconds: 151.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 787 (time 1789493761.78 total 10.01 last 78.61)
 - open/close 1553 (time 1789493771.80 total 20.03 last 76.46)
 - open/close 2306 (time 1789493781.81 total 30.03 last 75.27)
 - open/close 3076 (time 1789493791.81 total 40.03 last 76.99)
 - open/close 3777 (time 1789493801.82 total 50.04 last 70.04)
total: 4500 open/close in 58.81 seconds: 76.52 ops/second
 - /unlink 1367 (time 1789493824.78 total 10.01 last 136.58)
 - /unlink 2795 (time 1789493834.79 total 20.01 last 142.77)
 - /unlink 4174 (time 1789493844.79 total 30.01 last 137.90)
total: 4500 /unlink in 32.26 seconds: 139.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 783 (time 1789493862.30 total 10.01 last 78.20)
 - open/close 1471 (time 1789493872.30 total 20.02 last 68.76)
 - open/close 2096 (time 1789493882.30 total 30.02 last 62.49)
 - open/close 3050 (time 1789493892.32 total 40.04 last 95.25)
 - open/close 3862 (time 1789493902.32 total 50.04 last 81.18)
total: 4500 open/close in 58.08 seconds: 77.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1298 (time 1789493925.42 total 10.00 last 129.77)
 - /unlink 2375 (time 1789493935.42 total 20.00 last 107.69)
 - /unlink 3811 (time 1789493945.42 total 30.01 last 143.56)
total: 4500 /unlink in 35.24 seconds: 127.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 688 (time 1789493965.74 total 10.01 last 68.74)
 - open/close 1393 (time 1789493975.74 total 20.01 last 70.47)
 - open/close 2070 (time 1789493985.75 total 30.02 last 67.69)
 - open/close 2778 (time 1789493995.75 total 40.03 last 70.73)
 - open/close 3501 (time 1789494005.78 total 50.05 last 72.13)
 - open/close 4169 (time 1789494015.80 total 60.07 last 66.66)
total: 4500 open/close in 65.56 seconds: 68.64 ops/second
 - /unlink 1452 (time 1789494036.24 total 10.00 last 145.15)
 - /unlink 2795 (time 1789494046.25 total 20.01 last 134.23)
 - /unlink 4330 (time 1789494056.25 total 30.02 last 153.37)
total: 4500 /unlink in 31.59 seconds: 142.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 767 (time 1789494073.09 total 10.01 last 76.63)
 - open/close 1577 (time 1789494083.09 total 20.01 last 81.00)
 - open/close 2362 (time 1789494093.12 total 30.05 last 78.21)
 - open/close 3076 (time 1789494103.12 total 40.05 last 71.39)
 - open/close 3847 (time 1789494113.13 total 50.05 last 77.06)
total: 4500 open/close in 58.42 seconds: 77.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1303 (time 1789494135.09 total 10.00 last 130.30)
 - /unlink 2766 (time 1789494145.10 total 20.00 last 146.24)
 - /unlink 4134 (time 1789494155.10 total 30.01 last 136.72)
total: 4500 /unlink in 32.55 seconds: 138.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 793 (time 1789494172.49 total 10.01 last 79.24)
 - open/close 1486 (time 1789494182.50 total 20.01 last 69.26)
 - open/close 2273 (time 1789494192.50 total 30.01 last 78.69)
 - open/close 3035 (time 1789494202.50 total 40.01 last 76.20)
 - open/close 3781 (time 1789494212.51 total 50.02 last 74.52)
 - open/close 4487 (time 1789494222.52 total 60.04 last 70.52)
total: 4500 open/close in 60.28 seconds: 74.65 ops/second
 - /unlink 1486 (time 1789494239.62 total 10.00 last 148.54)
 - /unlink 3058 (time 1789494249.62 total 20.00 last 157.19)
total: 4500 /unlink in 29.72 seconds: 151.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 731 (time 1789494274.57 total 10.00 last 73.07)
 - open/close 1401 (time 1789494284.57 total 20.00 last 67.00)
 - open/close 2284 (time 1789494294.57 total 30.01 last 88.29)
 - open/close 3127 (time 1789494304.57 total 40.01 last 84.28)
 - open/close 3914 (time 1789494314.58 total 50.02 last 78.62)
 - open/close 4374 (time 1789494324.62 total 60.06 last 45.84)
total: 4500 open/close in 61.42 seconds: 73.27 ops/second
 - /unlink 1493 (time 1789494341.08 total 10.01 last 149.09)
 - /unlink 2789 (time 1789494351.09 total 20.02 last 129.52)
 - /unlink 4090 (time 1789494361.10 total 30.03 last 130.00)
total: 4500 /unlink in 33.30 seconds: 135.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 699 (time 1789494380.06 total 10.05 last 69.52)
 - open/close 1409 (time 1789494390.06 total 20.06 last 70.96)
 - open/close 2205 (time 1789494400.07 total 30.07 last 79.53)
 - open/close 2940 (time 1789494410.08 total 40.07 last 73.47)
 - open/close 3808 (time 1789494420.08 total 50.08 last 86.77)
total: 4500 open/close in 58.15 seconds: 77.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1739 (time 1789494443.46 total 10.01 last 173.71)
 - /unlink 3209 (time 1789494453.46 total 20.01 last 146.99)
total: 4500 /unlink in 28.70 seconds: 156.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 718 (time 1789494477.14 total 10.01 last 71.75)
 - open/close 1358 (time 1789494487.16 total 20.02 last 63.89)
 - open/close 1901 (time 1789494497.17 total 30.03 last 54.26)
 - open/close 2582 (time 1789494507.17 total 40.04 last 68.07)
 - open/close 3423 (time 1789494517.19 total 50.05 last 83.99)
 - open/close 4251 (time 1789494527.19 total 60.05 last 82.78)
total: 4500 open/close in 62.98 seconds: 71.45 ops/second
 - /unlink 1461 (time 1789494544.55 total 10.00 last 146.03)
 - /unlink 2686 (time 1789494554.55 total 20.00 last 122.50)
 - /unlink 4182 (time 1789494564.56 total 30.02 last 149.44)
total: 4500 /unlink in 32.16 seconds: 139.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 841 (time 1789494581.99 total 10.00 last 84.09)
 - open/close 1602 (time 1789494591.99 total 20.00 last 76.09)
 - open/close 2486 (time 1789494602.00 total 30.01 last 88.33)
 - open/close 3326 (time 1789494612.00 total 40.01 last 83.97)
 - open/close 4070 (time 1789494622.01 total 50.02 last 74.32)
total: 4500 open/close in 55.42 seconds: 81.20 ops/second
 - /unlink 1439 (time 1789494641.40 total 10.01 last 143.83)
 - /unlink 2927 (time 1789494651.40 total 20.01 last 148.77)
 - /unlink 4459 (time 1789494661.40 total 30.01 last 153.19)
total: 4500 /unlink in 30.27 seconds: 148.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 749 (time 1789494676.43 total 10.00 last 74.89)
 - open/close 1488 (time 1789494686.44 total 20.01 last 73.85)
 - open/close 2340 (time 1789494696.45 total 30.01 last 85.15)
 - open/close 3202 (time 1789494706.45 total 40.02 last 86.16)
 - open/close 4057 (time 1789494716.46 total 50.02 last 85.48)
total: 4500 open/close in 55.45 seconds: 81.15 ops/second
 - /unlink 1421 (time 1789494735.90 total 10.01 last 142.03)
 - /unlink 2866 (time 1789494745.90 total 20.01 last 144.41)
 - /unlink 4334 (time 1789494755.91 total 30.02 last 146.71)
total: 4500 /unlink in 31.37 seconds: 143.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 809 (time 1789494771.81 total 10.02 last 80.77)
 - open/close 1607 (time 1789494781.82 total 20.03 last 79.72)
 - open/close 2428 (time 1789494791.84 total 30.04 last 82.00)
 - open/close 3218 (time 1789494801.84 total 40.05 last 78.94)
 - open/close 4143 (time 1789494811.85 total 50.05 last 92.46)
total: 4500 open/close in 54.35 seconds: 82.80 ops/second
 - /unlink 1546 (time 1789494830.99 total 10.00 last 154.53)
 - /unlink 2946 (time 1789494840.99 total 20.01 last 139.95)
 - /unlink 4322 (time 1789494851.00 total 30.01 last 137.56)
total: 4500 /unlink in 32.20 seconds: 139.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 536 (time 1789494870.58 total 10.02 last 53.48)
 - open/close 1090 (time 1789494880.61 total 20.06 last 55.21)
 - open/close 1600 (time 1789494890.66 total 30.10 last 50.79)
 - open/close 1987 (time 1789494900.71 total 40.15 last 38.49)
 - open/close 2350 (time 1789494910.71 total 50.15 last 36.29)
 - open/close 2887 (time 1789494920.72 total 60.16 last 53.63)
 - open/close 3495 (time 1789494930.73 total 70.17 last 60.76)
 - open/close 4226 (time 1789494940.73 total 80.17 last 73.09)
total: 4500 open/close in 83.30 seconds: 54.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1458 (time 1789494959.39 total 10.00 last 145.76)
 - /unlink 2828 (time 1789494969.39 total 20.01 last 136.94)
 - /unlink 4245 (time 1789494979.40 total 30.01 last 141.64)
total: 4500 /unlink in 31.67 seconds: 142.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 843 (time 1789494996.17 total 10.00 last 84.26)
 - open/close 1585 (time 1789495006.18 total 20.01 last 74.18)
 - open/close 2441 (time 1789495016.18 total 30.02 last 85.52)
 - open/close 3305 (time 1789495026.19 total 40.02 last 86.32)
 - open/close 4093 (time 1789495036.21 total 50.04 last 78.68)
total: 4500 open/close in 55.94 seconds: 80.45 ops/second
 - /unlink 1473 (time 1789495057.11 total 10.01 last 147.22)
 - /unlink 2814 (time 1789495067.11 total 20.01 last 134.07)
 - /unlink 4173 (time 1789495077.11 total 30.01 last 135.84)
total: 4500 /unlink in 32.33 seconds: 139.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 855 (time 1789495094.49 total 10.00 last 85.46)
 - open/close 1633 (time 1789495104.49 total 20.01 last 77.76)
 - open/close 2499 (time 1789495114.51 total 30.03 last 86.43)
 - open/close 3463 (time 1789495124.53 total 40.04 last 96.28)
 - open/close 4177 (time 1789495134.54 total 50.06 last 71.31)
total: 4500 open/close in 54.16 seconds: 83.08 ops/second
 - /unlink 1565 (time 1789495153.03 total 10.00 last 156.47)
 - /unlink 3067 (time 1789495163.04 total 20.01 last 150.13)
total: 4500 /unlink in 29.44 seconds: 152.87 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95275 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg426-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg426-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server
stop mds service on oleg426-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg426-server
stop mds service on oleg426-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg426-server
unloading modules via unload_modules_local on: 'oleg426-server'
oleg426-server: oleg426-server.virtnet: executing unload_modules_local
oleg426-server: kmemleak disabled
modules unloaded.