== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:10:48 (1788945048)
Stopping clients: oleg145-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg145-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg145-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg145-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg145-server
oleg145-server: oleg145-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg145-server'
oleg145-server: oleg145-server.virtnet: executing load_modules_local
oleg145-server: kmemleak disabled
oleg145-server: Loading modules from /home/green/git/lustre-release/lustre
oleg145-server: detected 4 online CPUs by sysfs
oleg145-server: MODOPTS_LIBCFS=
oleg145-server: Force libcfs to create 2 CPU partitions
oleg145-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg145-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all
pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg145-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all
pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg145-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all
pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg145-client.virtnet:  -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 673 (time 1788945266.78 total 10.01 last 67.25)
 - open/close 1285 (time 1788945276.80 total 20.02 last 61.10)
 - open/close 2108 (time 1788945286.81 total 30.03 last 82.24)
 - open/close 2812 (time 1788945296.81 total 40.03 last 70.38)
 - open/close 3679 (time 1788945306.82 total 50.05 last 86.56)
 - open/close 4327 (time 1788945316.83 total 60.06 last 64.76)
total: 4500 open/close in 62.62 seconds: 71.87 ops/second
 - /unlink 1218 (time 1788945334.54 total 10.00 last 121.76)
 - /unlink 2341 (time 1788945344.55 total 20.01 last 112.24)
 - /unlink 3728 (time 1788945354.55 total 30.01 last 138.64)
total: 4500 /unlink in 35.61 seconds: 126.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 734 (time 1788945375.24 total 10.00 last 73.40)
 - open/close 1382 (time 1788945385.25 total 20.01 last 64.75)
 - open/close 2071 (time 1788945395.25 total 30.01 last 68.86)
 - open/close 2770 (time 1788945405.26 total 40.02 last 69.85)
 - open/close 3570 (time 1788945415.27 total 50.03 last 79.94)
 - open/close 4236 (time 1788945425.27 total 60.03 last 66.59)
total: 4500 open/close in 63.68 seconds: 70.67 ops/second
 - /unlink 1219 (time 1788945443.95 total 10.00 last 121.85)
 - /unlink 2606 (time 1788945453.96 total 20.01 last 138.64)
 - /unlink 3811 (time 1788945463.96 total 30.01 last 120.49)
total: 4500 /unlink in 34.65 seconds: 129.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 617 (time 1788945485.26 total 10.01 last 61.67)
 - open/close 1242 (time 1788945495.26 total 20.01 last 62.47)
 - open/close 1867 (time 1788945505.27 total 30.02 last 62.45)
 - open/close 2570 (time 1788945515.28 total 40.02 last 70.28)
 - open/close 3216 (time 1788945525.28 total 50.03 last 64.55)
 - open/close 4001 (time 1788945535.29 total 60.04 last 78.43)
total: 4500 open/close in 66.72 seconds: 67.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1351 (time 1788945556.25 total 10.00 last 135.04)
 - /unlink 2635 (time 1788945566.26 total 20.01 last 128.26)
 - /unlink 3885 (time 1788945576.26 total 30.02 last 124.97)
total: 4500 /unlink in 34.66 seconds: 129.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 658 (time 1788945596.55 total 10.00 last 65.77)
 - open/close 1245 (time 1788945606.56 total 20.01 last 58.66)
 - open/close 1843 (time 1788945616.56 total 30.01 last 59.80)
 - open/close 2495 (time 1788945626.58 total 40.03 last 65.07)
 - open/close 3042 (time 1788945636.60 total 50.06 last 54.57)
 - open/close 3697 (time 1788945646.61 total 60.07 last 65.41)
 - open/close 4361 (time 1788945656.62 total 70.08 last 66.33)
total: 4500 open/close in 72.21 seconds: 62.32 ops/second
 - /unlink 1150 (time 1788945674.32 total 10.00 last 114.97)
 - /unlink 1990 (time 1788945684.33 total 20.01 last 83.94)
 - /unlink 2932 (time 1788945694.33 total 30.01 last 94.17)
 - /unlink 4212 (time 1788945704.33 total 40.01 last 127.99)
total: 4500 /unlink in 41.91 seconds: 107.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 791 (time 1788945723.15 total 10.00 last 79.09)
 - open/close 1434 (time 1788945733.17 total 20.01 last 64.22)
 - open/close 2039 (time 1788945743.20 total 30.05 last 60.29)
 - open/close 2684 (time 1788945753.21 total 40.06 last 64.44)
 - open/close 3326 (time 1788945763.21 total 50.06 last 64.19)
 - open/close 4045 (time 1788945773.22 total 60.07 last 71.85)
total: 4500 open/close in 68.20 seconds: 65.98 ops/second
 - /unlink 1288 (time 1788945795.78 total 10.01 last 128.63)
 - /unlink 2627 (time 1788945805.78 total 20.02 last 133.87)
 - /unlink 3911 (time 1788945815.79 total 30.02 last 128.38)
total: 4500 /unlink in 34.86 seconds: 129.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 512 (time 1788945835.32 total 10.04 last 51.02)
 - open/close 983 (time 1788945845.34 total 20.05 last 47.04)
 - open/close 1615 (time 1788945855.34 total 30.05 last 63.18)
 - open/close 2341 (time 1788945865.35 total 40.06 last 72.54)
 - open/close 3011 (time 1788945875.35 total 50.06 last 66.96)
 - open/close 3659 (time 1788945885.36 total 60.07 last 64.76)
 - open/close 4347 (time 1788945895.37 total 70.08 last 68.74)
total: 4500 open/close in 71.88 seconds: 62.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1527 (time 1788945912.35 total 10.00 last 152.70)
 - /unlink 2771 (time 1788945922.35 total 20.00 last 124.37)
 - /unlink 4019 (time 1788945932.35 total 30.00 last 124.78)
total: 4500 /unlink in 34.51 seconds: 130.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 644 (time 1788945951.79 total 10.00 last 64.39)
 - open/close 1302 (time 1788945961.79 total 20.01 last 65.77)
 - open/close 2019 (time 1788945971.79 total 30.01 last 71.69)
 - open/close 2778 (time 1788945981.80 total 40.02 last 75.84)
 - open/close 3506 (time 1788945991.81 total 50.03 last 72.71)
 - open/close 4159 (time 1788946001.82 total 60.03 last 65.26)
total: 4500 open/close in 65.23 seconds: 68.98 ops/second
 - /unlink 1348 (time 1788946022.06 total 10.00 last 134.75)
 - /unlink 2644 (time 1788946032.06 total 20.01 last 129.57)
 - /unlink 3981 (time 1788946042.06 total 30.01 last 133.69)
total: 4500 /unlink in 33.36 seconds: 134.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 732 (time 1788946060.70 total 10.03 last 72.98)
 - open/close 1327 (time 1788946070.74 total 20.07 last 59.27)
 - open/close 1954 (time 1788946080.74 total 30.07 last 62.70)
 - open/close 2739 (time 1788946090.75 total 40.08 last 78.41)
 - open/close 3369 (time 1788946100.77 total 50.10 last 62.88)
 - open/close 3983 (time 1788946110.80 total 60.14 last 61.18)
total: 4500 open/close in 67.52 seconds: 66.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1113 (time 1788946133.32 total 10.00 last 111.26)
 - /unlink 2384 (time 1788946143.32 total 20.00 last 127.10)
 - /unlink 3598 (time 1788946153.33 total 30.01 last 121.33)
total: 4500 /unlink in 37.10 seconds: 121.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 606 (time 1788946175.97 total 10.02 last 60.46)
 - open/close 1370 (time 1788946185.98 total 20.03 last 76.35)
 - open/close 2174 (time 1788946195.98 total 30.04 last 80.34)
 - open/close 2792 (time 1788946205.99 total 40.04 last 61.76)
 - open/close 3453 (time 1788946216.02 total 50.07 last 65.91)
 - open/close 4027 (time 1788946226.03 total 60.08 last 57.36)
total: 4500 open/close in 66.29 seconds: 67.88 ops/second
 - /unlink 1137 (time 1788946247.02 total 10.00 last 113.69)
 - /unlink 2213 (time 1788946257.02 total 20.00 last 107.58)
 - /unlink 3191 (time 1788946267.02 total 30.01 last 97.77)
total: 4500 /unlink in 39.74 seconds: 113.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 706 (time 1788946292.14 total 10.01 last 70.53)
 - open/close 1323 (time 1788946302.16 total 20.03 last 61.59)
 - open/close 1953 (time 1788946312.17 total 30.03 last 62.96)
 - open/close 2546 (time 1788946322.17 total 40.04 last 59.25)
 - open/close 3286 (time 1788946332.18 total 50.05 last 73.94)
 - open/close 4030 (time 1788946342.19 total 60.05 last 74.37)
total: 4500 open/close in 66.30 seconds: 67.87 ops/second
 - /unlink 1249 (time 1788946363.11 total 10.01 last 124.74)
 - /unlink 2626 (time 1788946373.11 total 20.01 last 137.68)
 - /unlink 3871 (time 1788946383.11 total 30.02 last 124.49)
total: 4500 /unlink in 35.42 seconds: 127.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 705 (time 1788946403.51 total 10.01 last 70.40)
 - open/close 1297 (time 1788946413.52 total 20.03 last 59.12)
 - open/close 1985 (time 1788946423.52 total 30.03 last 68.79)
 - open/close 2537 (time 1788946433.52 total 40.03 last 55.19)
 - open/close 3301 (time 1788946443.53 total 50.04 last 76.38)
 - open/close 4018 (time 1788946453.54 total 60.05 last 71.63)
total: 4500 open/close in 65.67 seconds: 68.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1391 (time 1788946473.90 total 10.00 last 139.09)
 - /unlink 2926 (time 1788946483.90 total 20.01 last 153.43)
 - /unlink 4082 (time 1788946493.90 total 30.01 last 115.57)
total: 4500 /unlink in 32.63 seconds: 137.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 722 (time 1788946511.87 total 10.00 last 72.19)
 - open/close 1455 (time 1788946521.89 total 20.02 last 73.20)
 - open/close 2257 (time 1788946531.90 total 30.03 last 80.09)
 - open/close 3074 (time 1788946541.92 total 40.05 last 81.56)
 - open/close 3892 (time 1788946551.95 total 50.08 last 81.53)
total: 4500 open/close in 58.83 seconds: 76.49 ops/second
 - /unlink 1341 (time 1788946575.70 total 10.00 last 134.06)
 - /unlink 2498 (time 1788946585.70 total 20.00 last 115.69)
 - /unlink 3755 (time 1788946595.70 total 30.01 last 125.66)
total: 4500 /unlink in 36.38 seconds: 123.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 594 (time 1788946618.39 total 10.00 last 59.39)
 - open/close 1296 (time 1788946628.40 total 20.01 last 70.15)
 - open/close 2025 (time 1788946638.40 total 30.01 last 72.89)
 - open/close 2727 (time 1788946648.40 total 40.01 last 70.17)
 - open/close 3456 (time 1788946658.41 total 50.02 last 72.84)
 - open/close 4163 (time 1788946668.44 total 60.05 last 70.53)
total: 4500 open/close in 65.06 seconds: 69.17 ops/second
 - /unlink 1561 (time 1788946688.32 total 10.00 last 156.10)
 - /unlink 2915 (time 1788946698.32 total 20.00 last 135.38)
 - /unlink 4297 (time 1788946708.32 total 30.00 last 138.17)
total: 4500 /unlink in 31.69 seconds: 141.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 637 (time 1788946725.01 total 10.00 last 63.69)
 - open/close 1480 (time 1788946735.03 total 20.02 last 84.15)
 - open/close 2275 (time 1788946745.05 total 30.04 last 79.32)
 - open/close 2906 (time 1788946755.06 total 40.06 last 63.02)
 - open/close 3675 (time 1788946765.07 total 50.07 last 76.81)
 - open/close 4413 (time 1788946775.08 total 60.07 last 73.75)
total: 4500 open/close in 61.50 seconds: 73.17 ops/second
 - /unlink 1415 (time 1788946791.04 total 10.00 last 141.44)
 - /unlink 2809 (time 1788946801.05 total 20.01 last 139.32)
 - /unlink 4204 (time 1788946811.05 total 30.01 last 139.44)
total: 4500 /unlink in 32.57 seconds: 138.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 774 (time 1788946828.28 total 10.00 last 77.39)
 - open/close 1669 (time 1788946838.29 total 20.01 last 89.43)
 - open/close 2375 (time 1788946848.29 total 30.01 last 70.59)
 - open/close 3024 (time 1788946858.30 total 40.02 last 64.84)
 - open/close 3639 (time 1788946868.30 total 50.02 last 61.49)
 - open/close 4301 (time 1788946878.31 total 60.03 last 66.16)
total: 4500 open/close in 62.99 seconds: 71.44 ops/second
 - /unlink 1403 (time 1788946896.38 total 10.00 last 140.30)
 - /unlink 2578 (time 1788946906.39 total 20.00 last 117.46)
 - /unlink 3981 (time 1788946916.39 total 30.01 last 140.22)
total: 4500 /unlink in 34.37 seconds: 130.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 766 (time 1788946936.52 total 10.00 last 76.60)
 - open/close 1474 (time 1788946946.53 total 20.01 last 70.75)
 - open/close 2301 (time 1788946956.54 total 30.02 last 82.59)
 - open/close 2898 (time 1788946966.55 total 40.02 last 59.69)
 - open/close 3495 (time 1788946976.55 total 50.02 last 59.69)
 - open/close 4201 (time 1788946986.55 total 60.03 last 70.57)
total: 4500 open/close in 65.56 seconds: 68.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1465 (time 1788947007.10 total 10.00 last 146.49)
 - /unlink 2599 (time 1788947017.10 total 20.01 last 113.33)
 - /unlink 3955 (time 1788947027.11 total 30.02 last 135.48)
total: 4500 /unlink in 34.42 seconds: 130.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 575 (time 1788947046.33 total 10.01 last 57.45)
 - open/close 1284 (time 1788947056.36 total 20.04 last 70.70)
 - open/close 2006 (time 1788947066.37 total 30.05 last 72.13)
 - open/close 2759 (time 1788947076.38 total 40.06 last 75.23)
 - open/close 3445 (time 1788947086.39 total 50.06 last 68.55)
 - open/close 4078 (time 1788947096.39 total 60.07 last 63.27)
total: 4500 open/close in 65.35 seconds: 68.86 ops/second
 - /unlink 1252 (time 1788947117.09 total 10.02 last 125.01)
 - /unlink 2500 (time 1788947127.10 total 20.02 last 124.69)
 - /unlink 3685 (time 1788947137.10 total 30.03 last 118.45)
total: 4500 /unlink in 36.38 seconds: 123.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 718 (time 1788947158.90 total 10.01 last 71.72)
 - open/close 1410 (time 1788947168.91 total 20.02 last 69.17)
 - open/close 2052 (time 1788947178.91 total 30.02 last 64.19)
 - open/close 2814 (time 1788947188.92 total 40.03 last 76.14)
 - open/close 3542 (time 1788947198.92 total 50.03 last 72.79)
 - open/close 4191 (time 1788947208.93 total 60.04 last 64.84)
total: 4500 open/close in 63.77 seconds: 70.56 ops/second
 - /unlink 1174 (time 1788947228.00 total 10.02 last 117.18)
 - /unlink 2420 (time 1788947238.01 total 20.03 last 124.51)
 - /unlink 3625 (time 1788947248.01 total 30.03 last 120.48)
total: 4500 /unlink in 37.46 seconds: 120.13 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95429 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg145-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg145-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server
stop mds service on oleg145-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server
stop mds service on oleg145-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg145-server
unloading modules via unload_modules_local on: 'oleg145-server'
oleg145-server: oleg145-server.virtnet: executing unload_modules_local
oleg145-server: kmemleak disabled
modules unloaded.