== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:35:59 (1788496559)
Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg414-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg414-server
oleg414-server: oleg414-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg414-server'
oleg414-server: oleg414-server.virtnet: executing load_modules_local
oleg414-server: kmemleak disabled
oleg414-server: Loading modules from /home/green/git/lustre-release/lustre
oleg414-server: detected 4 online CPUs by sysfs
oleg414-server: MODOPTS_LIBCFS=
oleg414-server: Force libcfs to create 2 CPU partitions
oleg414-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg414-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all
pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg414-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all
pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg414-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all
pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg414-client.virtnet:  -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 799 (time 1788496763.64 total 10.01 last 79.81)
 - open/close 1572 (time 1788496773.65 total 20.03 last 77.17)
 - open/close 2352 (time 1788496783.66 total 30.03 last 77.95)
 - open/close 3241 (time 1788496793.66 total 40.04 last 88.88)
 - open/close 3909 (time 1788496803.67 total 50.04 last 66.76)
total: 4500 open/close in 59.87 seconds: 75.16 ops/second
 - /unlink 1650 (time 1788496828.67 total 10.01 last 164.81)
 - /unlink 2857 (time 1788496838.67 total 20.01 last 120.69)
 - /unlink 4498 (time 1788496848.67 total 30.01 last 164.09)
total: 4500 /unlink in 30.02 seconds: 149.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 820 (time 1788496863.54 total 10.01 last 81.95)
 - open/close 1557 (time 1788496873.55 total 20.01 last 73.69)
 - open/close 2310 (time 1788496883.55 total 30.01 last 75.27)
 - open/close 3073 (time 1788496893.56 total 40.02 last 76.25)
 - open/close 3819 (time 1788496903.56 total 50.02 last 74.59)
total: 4500 open/close in 59.79 seconds: 75.27 ops/second
 - /unlink 1372 (time 1788496928.73 total 10.01 last 137.05)
 - /unlink 2407 (time 1788496938.74 total 20.02 last 103.42)
 - /unlink 3606 (time 1788496948.74 total 30.03 last 119.82)
total: 4500 /unlink in 36.04 seconds: 124.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 848 (time 1788496970.32 total 10.02 last 84.63)
 - open/close 1654 (time 1788496980.33 total 20.03 last 80.51)
 - open/close 2472 (time 1788496990.34 total 30.04 last 81.76)
 - open/close 3223 (time 1788497000.37 total 40.06 last 74.88)
 - open/close 3859 (time 1788497010.41 total 50.10 last 63.34)
total: 4500 open/close in 59.07 seconds: 76.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1130 (time 1788497034.66 total 10.00 last 112.97)
 - /unlink 2179 (time 1788497044.66 total 20.01 last 104.81)
 - /unlink 3383 (time 1788497054.67 total 30.02 last 120.33)
total: 4500 /unlink in 38.58 seconds: 116.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 694 (time 1788497078.54 total 10.00 last 69.39)
 - open/close 1362 (time 1788497088.55 total 20.01 last 66.73)
 - open/close 1954 (time 1788497098.58 total 30.04 last 59.02)
 - open/close 2500 (time 1788497108.58 total 40.05 last 54.58)
 - open/close 3126 (time 1788497118.59 total 50.05 last 62.55)
 - open/close 3705 (time 1788497128.59 total 60.06 last 57.90)
 - open/close 4278 (time 1788497138.60 total 70.06 last 57.28)
total: 4500 open/close in 74.13 seconds: 60.70 ops/second
 - /unlink 1502 (time 1788497157.22 total 10.00 last 150.19)
 - /unlink 2870 (time 1788497167.22 total 20.00 last 136.75)
 - /unlink 4474 (time 1788497177.23 total 30.01 last 160.27)
total: 4500 /unlink in 30.30 seconds: 148.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 823 (time 1788497192.36 total 10.01 last 82.22)
 - open/close 1550 (time 1788497202.37 total 20.02 last 72.62)
 - open/close 2354 (time 1788497212.38 total 30.03 last 80.36)
 - open/close 3186 (time 1788497222.38 total 40.03 last 83.14)
 - open/close 3969 (time 1788497232.39 total 50.03 last 78.30)
total: 4500 open/close in 56.88 seconds: 79.11 ops/second
 - /unlink 1367 (time 1788497255.23 total 10.00 last 136.68)
 - /unlink 2809 (time 1788497265.24 total 20.01 last 144.09)
 - /unlink 4208 (time 1788497275.24 total 30.01 last 139.84)
total: 4500 /unlink in 32.23 seconds: 139.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 560 (time 1788497292.90 total 10.03 last 55.81)
 - open/close 931 (time 1788497302.94 total 20.07 last 36.98)
 - open/close 1340 (time 1788497312.97 total 30.11 last 40.75)
 - open/close 1935 (time 1788497322.99 total 40.12 last 59.42)
 - open/close 2746 (time 1788497333.00 total 50.13 last 80.99)
 - open/close 3492 (time 1788497343.01 total 60.14 last 74.55)
 - open/close 4292 (time 1788497353.03 total 70.16 last 79.83)
total: 4500 open/close in 72.75 seconds: 61.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1468 (time 1788497370.09 total 10.01 last 146.72)
 - /unlink 2655 (time 1788497380.10 total 20.01 last 118.60)
 - /unlink 3869 (time 1788497390.10 total 30.02 last 121.36)
total: 4500 /unlink in 34.52 seconds: 130.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 655 (time 1788497410.22 total 10.01 last 65.42)
 - open/close 1191 (time 1788497420.23 total 20.02 last 53.53)
 - open/close 1716 (time 1788497430.26 total 30.05 last 52.36)
 - open/close 2360 (time 1788497440.27 total 40.06 last 64.35)
 - open/close 2980 (time 1788497450.28 total 50.07 last 61.94)
 - open/close 3601 (time 1788497460.28 total 60.07 last 62.07)
 - open/close 4271 (time 1788497470.28 total 70.08 last 66.97)
total: 4500 open/close in 73.48 seconds: 61.24 ops/second
 - /unlink 1300 (time 1788497489.24 total 10.00 last 129.97)
 - /unlink 2692 (time 1788497499.24 total 20.01 last 139.08)
 - /unlink 4040 (time 1788497509.25 total 30.02 last 134.68)
total: 4500 /unlink in 33.69 seconds: 133.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 665 (time 1788497528.37 total 10.02 last 66.37)
 - open/close 1362 (time 1788497538.37 total 20.02 last 69.68)
 - open/close 2097 (time 1788497548.37 total 30.02 last 73.50)
 - open/close 2819 (time 1788497558.38 total 40.03 last 72.15)
 - open/close 3637 (time 1788497568.38 total 50.03 last 81.77)
 - open/close 4454 (time 1788497578.39 total 60.04 last 81.61)
total: 4500 open/close in 60.50 seconds: 74.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1529 (time 1788497593.44 total 10.01 last 152.82)
 - /unlink 2563 (time 1788497603.44 total 20.01 last 103.35)
 - /unlink 3883 (time 1788497613.45 total 30.01 last 131.95)
total: 4500 /unlink in 36.06 seconds: 124.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 857 (time 1788497634.24 total 10.01 last 85.64)
 - open/close 1588 (time 1788497644.25 total 20.02 last 73.00)
 - open/close 2481 (time 1788497654.26 total 30.03 last 89.25)
 - open/close 3342 (time 1788497664.26 total 40.03 last 86.09)
 - open/close 4194 (time 1788497674.27 total 50.04 last 85.12)
total: 4500 open/close in 53.52 seconds: 84.08 ops/second
 - /unlink 1499 (time 1788497692.28 total 10.01 last 149.74)
 - /unlink 3024 (time 1788497702.28 total 20.01 last 152.47)
total: 4500 /unlink in 29.56 seconds: 152.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 803 (time 1788497726.28 total 10.01 last 80.26)
 - open/close 1552 (time 1788497736.29 total 20.01 last 74.85)
 - open/close 2354 (time 1788497746.31 total 30.03 last 80.05)
 - open/close 3108 (time 1788497756.32 total 40.05 last 75.29)
 - open/close 3895 (time 1788497766.33 total 50.05 last 78.63)
total: 4500 open/close in 58.55 seconds: 76.86 ops/second
 - /unlink 1207 (time 1788497790.56 total 10.00 last 120.66)
 - /unlink 2440 (time 1788497800.57 total 20.01 last 123.25)
 - /unlink 3701 (time 1788497810.57 total 30.01 last 126.08)
total: 4500 /unlink in 35.59 seconds: 126.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 631 (time 1788497831.26 total 10.02 last 63.00)
 - open/close 1180 (time 1788497841.26 total 20.03 last 54.85)
 - open/close 1791 (time 1788497851.27 total 30.03 last 61.05)
 - open/close 2389 (time 1788497861.28 total 40.04 last 59.76)
 - open/close 3187 (time 1788497871.29 total 50.05 last 79.75)
 - open/close 3923 (time 1788497881.29 total 60.05 last 73.57)
total: 4500 open/close in 67.62 seconds: 66.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1577 (time 1788497903.20 total 10.01 last 157.57)
 - /unlink 3031 (time 1788497913.20 total 20.01 last 145.36)
 - /unlink 4239 (time 1788497923.21 total 30.01 last 120.76)
total: 4500 /unlink in 31.98 seconds: 140.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 643 (time 1788497940.37 total 10.01 last 64.21)
 - open/close 1339 (time 1788497950.38 total 20.02 last 69.54)
 - open/close 2032 (time 1788497960.38 total 30.02 last 69.29)
 - open/close 2793 (time 1788497970.39 total 40.03 last 76.05)
 - open/close 3447 (time 1788497980.39 total 50.03 last 65.36)
 - open/close 4347 (time 1788497990.40 total 60.04 last 89.93)
total: 4500 open/close in 61.89 seconds: 72.71 ops/second
 - /unlink 1481 (time 1788498006.82 total 10.00 last 148.08)
 - /unlink 2904 (time 1788498016.82 total 20.00 last 142.25)
 - /unlink 4450 (time 1788498026.83 total 30.01 last 154.49)
total: 4500 /unlink in 30.49 seconds: 147.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 946 (time 1788498041.77 total 10.00 last 94.56)
 - open/close 1747 (time 1788498051.78 total 20.01 last 80.04)
 - open/close 2460 (time 1788498061.80 total 30.03 last 71.15)
 - open/close 3246 (time 1788498071.80 total 40.04 last 78.57)
 - open/close 4037 (time 1788498081.82 total 50.05 last 78.97)
total: 4500 open/close in 56.52 seconds: 79.61 ops/second
 - /unlink 1444 (time 1788498103.44 total 10.01 last 144.22)
 - /unlink 2717 (time 1788498113.44 total 20.02 last 127.25)
 - /unlink 4060 (time 1788498123.44 total 30.02 last 134.28)
total: 4500 /unlink in 33.73 seconds: 133.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 737 (time 1788498142.21 total 10.01 last 73.65)
 - open/close 1405 (time 1788498152.21 total 20.01 last 66.77)
 - open/close 2059 (time 1788498162.22 total 30.02 last 65.33)
 - open/close 2808 (time 1788498172.22 total 40.02 last 74.89)
 - open/close 3573 (time 1788498182.26 total 50.06 last 76.20)
 - open/close 4322 (time 1788498192.26 total 60.06 last 74.90)
total: 4500 open/close in 62.28 seconds: 72.25 ops/second
 - /unlink 1425 (time 1788498209.28 total 10.01 last 142.36)
 - /unlink 2679 (time 1788498219.29 total 20.02 last 125.27)
 - /unlink 3880 (time 1788498229.30 total 30.03 last 119.94)
total: 4500 /unlink in 33.97 seconds: 132.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 770 (time 1788498248.65 total 10.01 last 76.94)
 - open/close 1522 (time 1788498258.65 total 20.02 last 75.14)
 - open/close 2295 (time 1788498268.67 total 30.03 last 77.19)
 - open/close 3051 (time 1788498278.67 total 40.03 last 75.57)
 - open/close 3815 (time 1788498288.67 total 50.04 last 76.39)
total: 4500 open/close in 59.67 seconds: 75.41 ops/second
 - /unlink 1249 (time 1788498313.53 total 10.01 last 124.78)
 - /unlink 2224 (time 1788498323.54 total 20.02 last 97.41)
 - /unlink 3395 (time 1788498333.54 total 30.02 last 117.04)
total: 4500 /unlink in 37.81 seconds: 119.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 766 (time 1788498356.66 total 10.01 last 76.53)
 - open/close 1379 (time 1788498366.67 total 20.02 last 61.23)
 - open/close 2007 (time 1788498376.69 total 30.04 last 62.69)
 - open/close 2704 (time 1788498386.69 total 40.04 last 69.68)
 - open/close 3393 (time 1788498396.71 total 50.06 last 68.75)
 - open/close 4022 (time 1788498406.72 total 60.07 last 62.87)
total: 4500 open/close in 67.75 seconds: 66.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1678 (time 1788498428.87 total 10.00 last 167.73)
 - /unlink 2930 (time 1788498438.87 total 20.01 last 125.15)
total: 4500 /unlink in 29.69 seconds: 151.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 799 (time 1788498463.55 total 10.01 last 79.79)
 - open/close 1430 (time 1788498473.55 total 20.02 last 63.08)
 - open/close 2123 (time 1788498483.56 total 30.02 last 69.25)
 - open/close 2948 (time 1788498493.56 total 40.03 last 82.47)
 - open/close 3712 (time 1788498503.59 total 50.06 last 76.13)
total: 4500 open/close in 59.87 seconds: 75.16 ops/second
 - /unlink 1613 (time 1788498527.66 total 10.00 last 161.29)
 - /unlink 2999 (time 1788498537.67 total 20.01 last 138.50)
total: 4500 /unlink in 29.92 seconds: 150.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 680 (time 1788498562.63 total 10.01 last 67.94)
 - open/close 1457 (time 1788498572.63 total 20.01 last 77.67)
 - open/close 2203 (time 1788498582.65 total 30.03 last 74.48)
 - open/close 2967 (time 1788498592.65 total 40.03 last 76.40)
 - open/close 3724 (time 1788498602.65 total 50.03 last 75.67)
total: 4500 open/close in 59.17 seconds: 76.05 ops/second
 - /unlink 1512 (time 1788498627.64 total 10.00 last 151.18)
 - /unlink 2789 (time 1788498637.65 total 20.01 last 127.61)
 - /unlink 3996 (time 1788498647.65 total 30.01 last 120.69)
total: 4500 /unlink in 35.85 seconds: 125.52 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95278 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg414-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg414-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server
stop mds service on oleg414-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server
stop mds service on oleg414-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server
unloading modules via unload_modules_local on: 'oleg414-server'
oleg414-server: oleg414-server.virtnet: executing unload_modules_local
oleg414-server: kmemleak disabled
modules unloaded.