== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:27:07 (1790983627)
Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg260-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg260-server
oleg260-server: oleg260-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg260-server'
oleg260-server: oleg260-server.virtnet: executing load_modules_local
oleg260-server: kmemleak disabled
oleg260-server: Loading modules from /home/green/git/lustre-release/lustre
oleg260-server: detected 4 online CPUs by sysfs
oleg260-server: MODOPTS_LIBCFS=
oleg260-server: Force libcfs to create 2 CPU partitions
oleg260-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg260-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg260-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg260-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg260-client.virtnet:  -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 691 (time 1790983837.02 total 10.00 last 69.07)
 - open/close 1359 (time 1790983847.03 total 20.01 last 66.74)
 - open/close 1945 (time 1790983857.05 total 30.03 last 58.50)
 - open/close 2490 (time 1790983867.07 total 40.05 last 54.41)
 - open/close 3077 (time 1790983877.07 total 50.05 last 58.67)
 - open/close 3704 (time 1790983887.09 total 60.07 last 62.61)
 - open/close 4183 (time 1790983897.09 total 70.07 last 47.87)
total: 4500 open/close in 75.81 seconds: 59.36 ops/second
 - /unlink 1282 (time 1790983918.97 total 10.00 last 128.19)
 - /unlink 2431 (time 1790983928.98 total 20.00 last 114.86)
 - /unlink 3481 (time 1790983938.98 total 30.01 last 104.98)
total: 4500 /unlink in 37.73 seconds: 119.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 651 (time 1790983962.08 total 10.02 last 64.99)
 - open/close 1245 (time 1790983972.08 total 20.02 last 59.40)
 - open/close 1949 (time 1790983982.08 total 30.02 last 70.39)
 - open/close 2667 (time 1790983992.11 total 40.04 last 71.63)
 - open/close 3358 (time 1790984002.11 total 50.05 last 69.05)
 - open/close 3984 (time 1790984012.12 total 60.05 last 62.57)
total: 4500 open/close in 68.09 seconds: 66.09 ops/second
 - /unlink 1280 (time 1790984035.54 total 10.00 last 127.98)
 - /unlink 2426 (time 1790984045.55 total 20.00 last 114.57)
 - /unlink 3701 (time 1790984055.55 total 30.01 last 127.43)
total: 4500 /unlink in 35.52 seconds: 126.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 731 (time 1790984075.67 total 10.01 last 73.03)
 - open/close 1404 (time 1790984085.70 total 20.04 last 67.09)
 - open/close 2024 (time 1790984095.72 total 30.06 last 61.89)
 - open/close 2719 (time 1790984105.73 total 40.07 last 69.43)
 - open/close 3502 (time 1790984115.74 total 50.08 last 78.24)
 - open/close 4051 (time 1790984125.74 total 60.08 last 54.89)
total: 4500 open/close in 65.45 seconds: 68.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1358 (time 1790984145.64 total 10.01 last 135.65)
 - /unlink 2611 (time 1790984155.64 total 20.01 last 125.29)
 - /unlink 3917 (time 1790984165.65 total 30.01 last 130.57)
total: 4500 /unlink in 34.27 seconds: 131.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 735 (time 1790984184.64 total 10.03 last 73.31)
 - open/close 1500 (time 1790984194.65 total 20.04 last 76.40)
 - open/close 2236 (time 1790984204.65 total 30.04 last 73.58)
 - open/close 3025 (time 1790984214.65 total 40.04 last 78.89)
 - open/close 3787 (time 1790984224.66 total 50.04 last 76.20)
total: 4500 open/close in 59.15 seconds: 76.07 ops/second
 - /unlink 1373 (time 1790984248.36 total 10.00 last 137.29)
 - /unlink 2554 (time 1790984258.37 total 20.00 last 118.07)
 - /unlink 3864 (time 1790984268.37 total 30.01 last 130.93)
total: 4500 /unlink in 35.86 seconds: 125.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 606 (time 1790984289.93 total 10.00 last 60.58)
 - open/close 1269 (time 1790984299.94 total 20.01 last 66.28)
 - open/close 1853 (time 1790984309.94 total 30.01 last 58.39)
 - open/close 2333 (time 1790984319.95 total 40.02 last 47.96)
 - open/close 2927 (time 1790984329.95 total 50.02 last 59.36)
 - open/close 3644 (time 1790984339.96 total 60.03 last 71.64)
 - open/close 4435 (time 1790984349.97 total 70.04 last 79.05)
total: 4500 open/close in 70.95 seconds: 63.42 ops/second
 - /unlink 1167 (time 1790984366.82 total 10.00 last 116.68)
 - /unlink 2435 (time 1790984376.83 total 20.01 last 126.74)
 - /unlink 3389 (time 1790984386.83 total 30.01 last 95.38)
total: 4500 /unlink in 39.67 seconds: 113.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 642 (time 1790984412.35 total 10.00 last 64.19)
 - open/close 1275 (time 1790984422.37 total 20.02 last 63.15)
 - open/close 1892 (time 1790984432.38 total 30.03 last 61.66)
 - open/close 2579 (time 1790984442.39 total 40.04 last 68.64)
 - open/close 3188 (time 1790984452.40 total 50.06 last 60.79)
 - open/close 3777 (time 1790984462.42 total 60.07 last 58.82)
 - open/close 4471 (time 1790984472.43 total 70.08 last 69.35)
total: 4500 open/close in 70.55 seconds: 63.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1140 (time 1790984488.74 total 10.01 last 113.91)
 - /unlink 2450 (time 1790984498.74 total 20.01 last 130.99)
 - /unlink 3571 (time 1790984508.74 total 30.01 last 112.10)
total: 4500 /unlink in 36.83 seconds: 122.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 684 (time 1790984531.52 total 10.01 last 68.36)
 - open/close 1278 (time 1790984541.52 total 20.01 last 59.35)
 - open/close 1937 (time 1790984551.54 total 30.03 last 65.80)
 - open/close 2546 (time 1790984561.54 total 40.03 last 60.87)
 - open/close 3241 (time 1790984571.55 total 50.04 last 69.46)
 - open/close 4022 (time 1790984581.56 total 60.05 last 78.04)
total: 4500 open/close in 67.38 seconds: 66.79 ops/second
 - /unlink 1075 (time 1790984603.32 total 10.00 last 107.48)
 - /unlink 2132 (time 1790984613.33 total 20.01 last 105.61)
 - /unlink 3198 (time 1790984623.33 total 30.02 last 106.53)
total: 4500 /unlink in 38.97 seconds: 115.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 594 (time 1790984647.91 total 10.01 last 59.37)
 - open/close 1285 (time 1790984657.91 total 20.01 last 69.10)
 - open/close 2040 (time 1790984667.92 total 30.01 last 75.50)
 - open/close 2736 (time 1790984677.92 total 40.02 last 69.54)
 - open/close 3556 (time 1790984687.93 total 50.03 last 81.92)
 - open/close 4300 (time 1790984697.94 total 60.03 last 74.36)
total: 4500 open/close in 62.35 seconds: 72.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1460 (time 1790984715.72 total 10.00 last 145.99)
 - /unlink 2666 (time 1790984725.73 total 20.00 last 120.58)
 - /unlink 3772 (time 1790984735.73 total 30.00 last 110.59)
total: 4500 /unlink in 35.28 seconds: 127.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 704 (time 1790984756.35 total 10.00 last 70.38)
 - open/close 1346 (time 1790984766.36 total 20.01 last 64.17)
 - open/close 2112 (time 1790984776.36 total 30.01 last 76.60)
 - open/close 2778 (time 1790984786.36 total 40.02 last 66.55)
 - open/close 3483 (time 1790984796.38 total 50.03 last 70.42)
 - open/close 4080 (time 1790984806.39 total 60.05 last 59.59)
total: 4500 open/close in 67.78 seconds: 66.40 ops/second
 - /unlink 1232 (time 1790984830.43 total 10.01 last 123.13)
 - /unlink 2168 (time 1790984840.44 total 20.01 last 93.53)
 - /unlink 3182 (time 1790984850.44 total 30.01 last 101.38)
 - /unlink 4295 (time 1790984860.45 total 40.02 last 111.24)
total: 4500 /unlink in 42.24 seconds: 106.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 717 (time 1790984878.41 total 10.00 last 71.69)
 - open/close 1249 (time 1790984888.43 total 20.02 last 53.09)
 - open/close 1831 (time 1790984898.44 total 30.03 last 58.15)
 - open/close 2494 (time 1790984908.45 total 40.04 last 66.25)
 - open/close 3097 (time 1790984918.46 total 50.05 last 60.24)
 - open/close 3674 (time 1790984928.48 total 60.07 last 57.56)
 - open/close 4313 (time 1790984938.49 total 70.08 last 63.86)
total: 4500 open/close in 72.59 seconds: 62.00 ops/second
 - /unlink 1062 (time 1790984956.58 total 10.01 last 106.14)
 - /unlink 2110 (time 1790984966.58 total 20.01 last 104.73)
 - /unlink 3451 (time 1790984976.58 total 30.01 last 134.09)
total: 4500 /unlink in 38.44 seconds: 117.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 694 (time 1790985001.12 total 10.01 last 69.35)
 - open/close 1357 (time 1790985011.14 total 20.02 last 66.21)
 - open/close 2045 (time 1790985021.16 total 30.05 last 68.63)
 - open/close 2743 (time 1790985031.18 total 40.06 last 69.68)
 - open/close 3276 (time 1790985041.19 total 50.07 last 53.25)
 - open/close 3899 (time 1790985051.20 total 60.08 last 62.25)
total: 4500 open/close in 68.20 seconds: 65.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1116 (time 1790985074.63 total 10.00 last 111.57)
 - /unlink 2437 (time 1790985084.63 total 20.00 last 132.09)
 - /unlink 3681 (time 1790985094.64 total 30.01 last 124.35)
total: 4500 /unlink in 37.96 seconds: 118.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 688 (time 1790985117.98 total 10.03 last 68.57)
 - open/close 1216 (time 1790985127.99 total 20.04 last 52.78)
 - open/close 1815 (time 1790985138.00 total 30.05 last 59.80)
 - open/close 2467 (time 1790985148.01 total 40.06 last 65.18)
 - open/close 3170 (time 1790985158.01 total 50.06 last 70.24)
 - open/close 3811 (time 1790985168.02 total 60.07 last 64.05)
total: 4500 open/close in 69.29 seconds: 64.95 ops/second
 - /unlink 1399 (time 1790985191.79 total 10.01 last 139.83)
 - /unlink 2710 (time 1790985201.80 total 20.02 last 130.88)
 - /unlink 4093 (time 1790985211.81 total 30.03 last 138.24)
total: 4500 /unlink in 33.47 seconds: 134.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 786 (time 1790985230.16 total 10.02 last 78.43)
 - open/close 1632 (time 1790985240.17 total 20.03 last 84.54)
 - open/close 2433 (time 1790985250.17 total 30.03 last 80.09)
 - open/close 3250 (time 1790985260.18 total 40.04 last 81.62)
 - open/close 4053 (time 1790985270.18 total 50.04 last 80.28)
total: 4500 open/close in 55.55 seconds: 81.00 ops/second
 - /unlink 1328 (time 1790985290.03 total 10.01 last 132.69)
 - /unlink 2527 (time 1790985300.03 total 20.01 last 119.84)
 - /unlink 3766 (time 1790985310.04 total 30.02 last 123.88)
total: 4500 /unlink in 34.85 seconds: 129.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 761 (time 1790985330.16 total 10.00 last 76.09)
 - open/close 1351 (time 1790985340.17 total 20.01 last 58.97)
 - open/close 1970 (time 1790985350.17 total 30.01 last 61.89)
 - open/close 2701 (time 1790985360.18 total 40.02 last 73.04)
 - open/close 3513 (time 1790985370.18 total 50.02 last 81.14)
 - open/close 4340 (time 1790985380.21 total 60.05 last 82.50)
total: 4500 open/close in 62.07 seconds: 72.50 ops/second
 - /unlink 1509 (time 1790985396.78 total 10.00 last 150.89)
 - /unlink 2713 (time 1790985406.79 total 20.01 last 120.29)
 - /unlink 4030 (time 1790985416.79 total 30.01 last 131.70)
total: 4500 /unlink in 33.73 seconds: 133.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 802 (time 1790985436.02 total 10.01 last 80.13)
 - open/close 1573 (time 1790985446.02 total 20.01 last 77.07)
 - open/close 2303 (time 1790985456.04 total 30.03 last 72.90)
 - open/close 3018 (time 1790985466.04 total 40.03 last 71.48)
 - open/close 3721 (time 1790985476.04 total 50.03 last 70.28)
 - open/close 4435 (time 1790985486.05 total 60.04 last 71.34)
total: 4500 open/close in 60.89 seconds: 73.90 ops/second
 - /unlink 1376 (time 1790985502.09 total 10.00 last 137.56)
 - /unlink 2858 (time 1790985512.09 total 20.01 last 148.15)
 - /unlink 4142 (time 1790985522.09 total 30.01 last 128.36)
total: 4500 /unlink in 33.21 seconds: 135.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 733 (time 1790985540.95 total 10.00 last 73.27)
 - open/close 1487 (time 1790985550.96 total 20.02 last 75.29)
 - open/close 2177 (time 1790985561.00 total 30.05 last 68.76)
 - open/close 2769 (time 1790985571.00 total 40.06 last 59.17)
 - open/close 3510 (time 1790985581.01 total 50.06 last 74.09)
 - open/close 4212 (time 1790985591.01 total 60.06 last 70.18)
total: 4500 open/close in 64.23 seconds: 70.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1155 (time 1790985610.68 total 10.01 last 115.43)
 - /unlink 2343 (time 1790985620.69 total 20.01 last 118.71)
 - /unlink 3346 (time 1790985630.69 total 30.02 last 100.21)
 - /unlink 4398 (time 1790985640.70 total 40.03 last 105.17)
total: 4500 /unlink in 40.77 seconds: 110.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 697 (time 1790985657.54 total 10.02 last 69.59)
 - open/close 1316 (time 1790985667.55 total 20.02 last 61.85)
 - open/close 1998 (time 1790985677.56 total 30.03 last 68.13)
 - open/close 2555 (time 1790985687.57 total 40.05 last 55.64)
 - open/close 2989 (time 1790985697.57 total 50.05 last 43.39)
 - open/close 3483 (time 1790985707.59 total 60.07 last 49.30)
 - open/close 3923 (time 1790985717.59 total 70.07 last 43.98)
total: 4500 open/close in 79.84 seconds: 56.36 ops/second
 - /unlink 1350 (time 1790985742.78 total 10.01 last 134.90)
 - /unlink 2594 (time 1790985752.78 total 20.01 last 124.37)
 - /unlink 3799 (time 1790985762.79 total 30.02 last 120.42)
total: 4500 /unlink in 36.46 seconds: 123.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 576 (time 1790985785.08 total 10.01 last 57.55)
 - open/close 1229 (time 1790985795.08 total 20.01 last 65.30)
 - open/close 1893 (time 1790985805.09 total 30.02 last 66.34)
 - open/close 2527 (time 1790985815.11 total 40.04 last 63.28)
 - open/close 3272 (time 1790985825.12 total 50.04 last 74.47)
 - open/close 3963 (time 1790985835.15 total 60.07 last 68.91)
total: 4500 open/close in 66.93 seconds: 67.23 ops/second
 - /unlink 1308 (time 1790985856.78 total 10.01 last 130.67)
 - /unlink 2754 (time 1790985866.78 total 20.01 last 144.54)
 - /unlink 4024 (time 1790985876.79 total 30.02 last 126.91)
total: 4500 /unlink in 33.17 seconds: 135.68 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11358: 95558 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg260-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg260-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server
stop mds service on oleg260-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server
stop mds service on oleg260-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server
unloading modules via unload_modules_local on: 'oleg260-server'
oleg260-server: oleg260-server.virtnet: executing unload_modules_local
oleg260-server: kmemleak disabled
modules unloaded.