== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:28:39 (1789302519)
Stopping clients: oleg326-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg326-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg326-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg326-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg326-server
oleg326-server: oleg326-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg326-server'
oleg326-server: oleg326-server.virtnet: executing load_modules_local
oleg326-server: kmemleak disabled
oleg326-server: Loading modules from /home/green/git/lustre-release/lustre
oleg326-server: detected 4 online CPUs by sysfs
oleg326-server: MODOPTS_LIBCFS=
oleg326-server: Force libcfs to create 2 CPU partitions
oleg326-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg326-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all
pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg326-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all
pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg326-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all
pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg326-client.virtnet:  -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 651 (time 1789302733.26 total 10.02 last 64.98)
 - open/close 1311 (time 1789302743.28 total 20.04 last 65.89)
 - open/close 2021 (time 1789302753.28 total 30.04 last 70.98)
 - open/close 2723 (time 1789302763.30 total 40.06 last 70.08)
 - open/close 3521 (time 1789302773.31 total 50.07 last 79.72)
 - open/close 4276 (time 1789302783.32 total 60.08 last 75.36)
total: 4500 open/close in 63.34 seconds: 71.04 ops/second
 - /unlink 1293 (time 1789302800.25 total 10.00 last 129.27)
 - /unlink 2358 (time 1789302810.26 total 20.01 last 106.42)
 - /unlink 3419 (time 1789302820.26 total 30.01 last 106.09)
total: 4500 /unlink in 38.15 seconds: 117.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 769 (time 1789302842.96 total 10.01 last 76.85)
 - open/close 1474 (time 1789302852.97 total 20.02 last 70.43)
 - open/close 2157 (time 1789302862.98 total 30.03 last 68.23)
 - open/close 2735 (time 1789302872.99 total 40.04 last 57.72)
 - open/close 3390 (time 1789302882.99 total 50.05 last 65.47)
 - open/close 4120 (time 1789302893.00 total 60.05 last 72.94)
total: 4500 open/close in 66.89 seconds: 67.28 ops/second
 - /unlink 1146 (time 1789302914.82 total 10.00 last 114.58)
 - /unlink 2358 (time 1789302924.82 total 20.00 last 121.18)
 - /unlink 3449 (time 1789302934.83 total 30.01 last 109.05)
total: 4500 /unlink in 37.96 seconds: 118.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 619 (time 1789302958.12 total 10.02 last 61.78)
 - open/close 1253 (time 1789302968.13 total 20.02 last 63.38)
 - open/close 1962 (time 1789302978.14 total 30.03 last 70.83)
 - open/close 2687 (time 1789302988.14 total 40.03 last 72.49)
 - open/close 3326 (time 1789302998.14 total 50.03 last 63.90)
 - open/close 4028 (time 1789303008.15 total 60.05 last 70.10)
total: 4500 open/close in 66.63 seconds: 67.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1331 (time 1789303029.47 total 10.01 last 132.98)
 - /unlink 2715 (time 1789303039.47 total 20.01 last 138.32)
 - /unlink 4049 (time 1789303049.47 total 30.01 last 133.40)
total: 4500 /unlink in 33.40 seconds: 134.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 699 (time 1789303068.22 total 10.01 last 69.86)
 - open/close 1441 (time 1789303078.22 total 20.01 last 74.18)
 - open/close 2114 (time 1789303088.25 total 30.04 last 67.10)
 - open/close 2825 (time 1789303098.25 total 40.04 last 71.09)
 - open/close 3406 (time 1789303108.29 total 50.08 last 57.90)
 - open/close 3968 (time 1789303118.29 total 60.08 last 56.18)
total: 4500 open/close in 67.43 seconds: 66.74 ops/second
 - /unlink 1301 (time 1789303140.91 total 10.00 last 130.08)
 - /unlink 2580 (time 1789303150.91 total 20.00 last 127.88)
 - /unlink 3845 (time 1789303160.91 total 30.00 last 126.48)
total: 4500 /unlink in 34.97 seconds: 128.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 659 (time 1789303180.91 total 10.02 last 65.75)
 - open/close 1373 (time 1789303190.91 total 20.02 last 71.39)
 - open/close 2110 (time 1789303200.92 total 30.03 last 73.65)
 - open/close 2826 (time 1789303210.93 total 40.04 last 71.51)
 - open/close 3603 (time 1789303220.95 total 50.06 last 77.55)
 - open/close 4354 (time 1789303230.95 total 60.07 last 75.07)
total: 4500 open/close in 61.92 seconds: 72.67 ops/second
 - /unlink 1365 (time 1789303247.29 total 10.00 last 136.45)
 - /unlink 2633 (time 1789303257.29 total 20.01 last 126.76)
 - /unlink 3860 (time 1789303267.29 total 30.01 last 122.67)
total: 4500 /unlink in 35.06 seconds: 128.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 743 (time 1789303287.24 total 10.01 last 74.26)
 - open/close 1375 (time 1789303297.27 total 20.03 last 63.03)
 - open/close 1974 (time 1789303307.28 total 30.04 last 59.87)
 - open/close 2495 (time 1789303317.30 total 40.06 last 51.96)
 - open/close 3025 (time 1789303327.32 total 50.08 last 52.89)
 - open/close 3615 (time 1789303337.33 total 60.09 last 58.98)
 - open/close 4267 (time 1789303347.34 total 70.10 last 65.12)
total: 4500 open/close in 73.75 seconds: 61.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1249 (time 1789303366.28 total 10.01 last 124.83)
 - /unlink 2575 (time 1789303376.29 total 20.01 last 132.55)
 - /unlink 3769 (time 1789303386.30 total 30.02 last 119.29)
total: 4500 /unlink in 36.85 seconds: 122.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 657 (time 1789303408.75 total 10.00 last 65.68)
 - open/close 1280 (time 1789303418.75 total 20.01 last 62.28)
 - open/close 1963 (time 1789303428.76 total 30.02 last 68.21)
 - open/close 2516 (time 1789303438.77 total 40.02 last 55.28)
 - open/close 3092 (time 1789303448.79 total 50.04 last 57.50)
 - open/close 3693 (time 1789303458.79 total 60.05 last 60.07)
 - open/close 4316 (time 1789303468.80 total 70.05 last 62.25)
total: 4500 open/close in 72.85 seconds: 61.77 ops/second
 - /unlink 1339 (time 1789303486.35 total 10.00 last 133.90)
 - /unlink 2668 (time 1789303496.36 total 20.01 last 132.78)
 - /unlink 3942 (time 1789303506.37 total 30.02 last 127.28)
total: 4500 /unlink in 34.50 seconds: 130.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 667 (time 1789303526.10 total 10.02 last 66.58)
 - open/close 1363 (time 1789303536.10 total 20.02 last 69.57)
 - open/close 2041 (time 1789303546.12 total 30.05 last 67.65)
 - open/close 2755 (time 1789303556.13 total 40.05 last 71.33)
 - open/close 3447 (time 1789303566.14 total 50.06 last 69.15)
 - open/close 4146 (time 1789303576.14 total 60.07 last 69.86)
total: 4500 open/close in 65.45 seconds: 68.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1363 (time 1789303596.56 total 10.00 last 136.29)
 - /unlink 2681 (time 1789303606.57 total 20.01 last 131.70)
 - /unlink 3980 (time 1789303616.57 total 30.01 last 129.88)
total: 4500 /unlink in 34.21 seconds: 131.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 688 (time 1789303635.57 total 10.02 last 68.63)
 - open/close 1334 (time 1789303645.59 total 20.04 last 64.51)
 - open/close 2003 (time 1789303655.60 total 30.05 last 66.83)
 - open/close 2622 (time 1789303665.60 total 40.05 last 61.89)
 - open/close 3247 (time 1789303675.61 total 50.06 last 62.45)
 - open/close 3979 (time 1789303685.61 total 60.06 last 73.19)
total: 4500 open/close in 67.09 seconds: 67.08 ops/second
 - /unlink 1239 (time 1789303707.72 total 10.00 last 123.86)
 - /unlink 2321 (time 1789303717.72 total 20.01 last 108.16)
 - /unlink 3398 (time 1789303727.74 total 30.03 last 107.50)
 - /unlink 4264 (time 1789303737.75 total 40.04 last 86.50)
total: 4500 /unlink in 42.34 seconds: 106.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 626 (time 1789303756.53 total 10.01 last 62.55)
 - open/close 1325 (time 1789303766.54 total 20.02 last 69.80)
 - open/close 1956 (time 1789303776.54 total 30.03 last 63.08)
 - open/close 2599 (time 1789303786.56 total 40.04 last 64.21)
 - open/close 3285 (time 1789303796.56 total 50.04 last 68.60)
 - open/close 3993 (time 1789303806.57 total 60.05 last 70.71)
total: 4500 open/close in 66.98 seconds: 67.19 ops/second
 - /unlink 1321 (time 1789303827.65 total 10.00 last 132.06)
 - /unlink 2590 (time 1789303837.65 total 20.00 last 126.89)
 - /unlink 3852 (time 1789303847.66 total 30.01 last 126.08)
total: 4500 /unlink in 34.97 seconds: 128.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 693 (time 1789303867.00 total 10.02 last 69.18)
 - open/close 1379 (time 1789303877.00 total 20.02 last 68.60)
 - open/close 2104 (time 1789303887.01 total 30.02 last 72.46)
 - open/close 2810 (time 1789303897.02 total 40.04 last 70.52)
 - open/close 3537 (time 1789303907.03 total 50.05 last 72.62)
 - open/close 4242 (time 1789303917.03 total 60.05 last 70.48)
total: 4500 open/close in 63.62 seconds: 70.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1337 (time 1789303934.48 total 10.00 last 133.66)
 - /unlink 2587 (time 1789303944.48 total 20.01 last 124.91)
 - /unlink 3892 (time 1789303954.50 total 30.02 last 130.33)
total: 4500 /unlink in 34.32 seconds: 131.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 651 (time 1789303974.00 total 10.02 last 64.99)
 - open/close 1356 (time 1789303984.00 total 20.02 last 70.49)
 - open/close 2072 (time 1789303994.01 total 30.03 last 71.53)
 - open/close 2755 (time 1789304004.01 total 40.03 last 68.27)
 - open/close 3395 (time 1789304014.02 total 50.04 last 63.96)
 - open/close 4089 (time 1789304024.02 total 60.04 last 69.36)
total: 4500 open/close in 66.04 seconds: 68.14 ops/second
 - /unlink 1278 (time 1789304044.36 total 10.00 last 127.75)
 - /unlink 2576 (time 1789304054.36 total 20.01 last 129.70)
 - /unlink 3891 (time 1789304064.37 total 30.02 last 131.42)
total: 4500 /unlink in 34.69 seconds: 129.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 657 (time 1789304083.95 total 10.00 last 65.70)
 - open/close 1359 (time 1789304093.96 total 20.01 last 70.13)
 - open/close 1925 (time 1789304103.96 total 30.01 last 56.59)
 - open/close 2519 (time 1789304113.97 total 40.02 last 59.35)
 - open/close 3041 (time 1789304123.98 total 50.03 last 52.14)
 - open/close 3647 (time 1789304133.98 total 60.03 last 60.59)
 - open/close 4270 (time 1789304143.99 total 70.04 last 62.26)
total: 4500 open/close in 73.70 seconds: 61.06 ops/second
 - /unlink 1268 (time 1789304162.81 total 10.01 last 126.72)
 - /unlink 2554 (time 1789304172.82 total 20.01 last 128.53)
 - /unlink 3762 (time 1789304182.82 total 30.01 last 120.77)
total: 4500 /unlink in 36.34 seconds: 123.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 665 (time 1789304204.63 total 10.01 last 66.44)
 - open/close 1323 (time 1789304214.64 total 20.02 last 65.71)
 - open/close 2113 (time 1789304224.65 total 30.04 last 78.91)
 - open/close 2774 (time 1789304234.67 total 40.05 last 65.98)
 - open/close 3469 (time 1789304244.68 total 50.06 last 69.43)
 - open/close 4115 (time 1789304254.70 total 60.08 last 64.50)
total: 4500 open/close in 65.88 seconds: 68.31 ops/second
 - /unlink 1281 (time 1789304276.72 total 10.00 last 128.06)
 - /unlink 2380 (time 1789304286.72 total 20.00 last 109.90)
 - /unlink 3634 (time 1789304296.73 total 30.01 last 125.30)
total: 4500 /unlink in 37.48 seconds: 120.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 704 (time 1789304320.11 total 10.02 last 70.28)
 - open/close 1316 (time 1789304330.11 total 20.02 last 61.18)
 - open/close 1980 (time 1789304340.11 total 30.02 last 66.38)
 - open/close 2673 (time 1789304350.12 total 40.03 last 69.23)
 - open/close 3407 (time 1789304360.13 total 50.04 last 73.37)
 - open/close 4143 (time 1789304370.13 total 60.04 last 73.58)
total: 4500 open/close in 65.01 seconds: 69.22 ops/second
 - /unlink 1297 (time 1789304389.75 total 10.00 last 129.69)
 - /unlink 2616 (time 1789304399.76 total 20.01 last 131.82)
 - /unlink 3836 (time 1789304409.76 total 30.01 last 121.95)
total: 4500 /unlink in 36.08 seconds: 124.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 577 (time 1789304431.14 total 10.00 last 57.69)
 - open/close 1281 (time 1789304441.15 total 20.01 last 70.33)
 - open/close 2012 (time 1789304451.17 total 30.03 last 72.95)
 - open/close 2648 (time 1789304461.18 total 40.04 last 63.51)
 - open/close 3270 (time 1789304471.18 total 50.05 last 62.19)
 - open/close 3956 (time 1789304481.19 total 60.05 last 68.56)
total: 4500 open/close in 67.28 seconds: 66.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1453 (time 1789304502.44 total 10.00 last 145.25)
 - /unlink 2724 (time 1789304512.44 total 20.01 last 127.04)
 - /unlink 4017 (time 1789304522.45 total 30.02 last 129.18)
total: 4500 /unlink in 33.33 seconds: 135.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 664 (time 1789304540.96 total 10.01 last 66.30)
 - open/close 1320 (time 1789304551.01 total 20.06 last 65.28)
 - open/close 1905 (time 1789304561.01 total 30.07 last 58.49)
 - open/close 2647 (time 1789304571.01 total 40.07 last 74.18)
 - open/close 3320 (time 1789304581.02 total 50.08 last 67.23)
 - open/close 4006 (time 1789304591.02 total 60.08 last 68.59)
total: 4500 open/close in 67.20 seconds: 66.96 ops/second
 - /unlink 1303 (time 1789304612.84 total 10.00 last 130.27)
 - /unlink 2579 (time 1789304622.85 total 20.01 last 127.49)
 - /unlink 3864 (time 1789304632.85 total 30.01 last 128.49)
total: 4500 /unlink in 34.73 seconds: 129.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 671 (time 1789304653.08 total 10.02 last 66.98)
 - open/close 1342 (time 1789304663.08 total 20.02 last 67.09)
 - open/close 2033 (time 1789304673.09 total 30.02 last 69.06)
 - open/close 2727 (time 1789304683.10 total 40.03 last 69.34)
 - open/close 3367 (time 1789304693.10 total 50.04 last 63.95)
 - open/close 4072 (time 1789304703.11 total 60.05 last 70.45)
total: 4500 open/close in 66.51 seconds: 67.66 ops/second
 - /unlink 1345 (time 1789304723.82 total 10.00 last 134.47)
 - /unlink 2624 (time 1789304733.82 total 20.00 last 127.88)
 - /unlink 3938 (time 1789304743.82 total 30.01 last 131.35)
total: 4500 /unlink in 34.27 seconds: 131.32 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95297 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg326-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg326-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server
stop mds service on oleg326-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server
stop mds service on oleg326-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg326-server
unloading modules via unload_modules_local on: 'oleg326-server'
oleg326-server: oleg326-server.virtnet: executing unload_modules_local
oleg326-server: kmemleak disabled
modules unloaded.