== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:15:51 (1782224151)
Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg360-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg360-server
oleg360-server: oleg360-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg360-server'
oleg360-server: oleg360-server.virtnet: executing load_modules_local
oleg360-server: Loading modules from /home/green/git/lustre-release/lustre
oleg360-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg360-server: detected 4 online CPUs by sysfs
oleg360-server: MODOPTS_LIBCFS=
oleg360-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg360-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all
pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg360-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all
pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg360-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all
pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg360-client.virtnet:  -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 658 (time 1782224356.38 total 10.01 last 65.76)
 - open/close 1342 (time 1782224366.38 total 20.01 last 68.34)
 - open/close 2053 (time 1782224376.39 total 30.02 last 71.08)
 - open/close 2713 (time 1782224386.40 total 40.03 last 65.92)
 - open/close 3431 (time 1782224396.40 total 50.03 last 71.80)
 - open/close 4083 (time 1782224406.43 total 60.06 last 65.01)
total: 4500 open/close in 66.24 seconds: 67.94 ops/second
 - /unlink 1257 (time 1782224427.26 total 10.00 last 125.67)
 - /unlink 2498 (time 1782224437.27 total 20.01 last 124.05)
 - /unlink 3749 (time 1782224447.27 total 30.01 last 125.06)
total: 4500 /unlink in 36.67 seconds: 122.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 512 (time 1782224469.32 total 10.00 last 51.20)
 - open/close 1176 (time 1782224479.34 total 20.02 last 66.27)
 - open/close 1852 (time 1782224489.34 total 30.02 last 67.59)
 - open/close 2455 (time 1782224499.34 total 40.02 last 60.29)
 - open/close 3114 (time 1782224509.35 total 50.03 last 65.85)
 - open/close 3844 (time 1782224519.36 total 60.05 last 72.88)
total: 4500 open/close in 69.44 seconds: 64.80 ops/second
 - /unlink 1254 (time 1782224542.98 total 10.00 last 125.35)
 - /unlink 2313 (time 1782224552.99 total 20.01 last 105.80)
 - /unlink 3534 (time 1782224562.99 total 30.02 last 122.08)
total: 4500 /unlink in 37.46 seconds: 120.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 672 (time 1782224584.93 total 10.01 last 67.13)
 - open/close 1296 (time 1782224594.94 total 20.02 last 62.35)
 - open/close 2001 (time 1782224604.94 total 30.03 last 70.45)
 - open/close 2642 (time 1782224614.95 total 40.03 last 64.08)
 - open/close 3278 (time 1782224624.96 total 50.04 last 63.53)
 - open/close 3945 (time 1782224634.96 total 60.04 last 66.70)
total: 4500 open/close in 68.44 seconds: 65.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1271 (time 1782224657.69 total 10.01 last 127.03)
 - /unlink 2587 (time 1782224667.69 total 20.01 last 131.53)
 - /unlink 3951 (time 1782224677.70 total 30.02 last 136.30)
total: 4500 /unlink in 33.94 seconds: 132.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 660 (time 1782224696.03 total 10.01 last 65.95)
 - open/close 1400 (time 1782224706.04 total 20.01 last 73.95)
 - open/close 2092 (time 1782224716.05 total 30.03 last 69.11)
 - open/close 2782 (time 1782224726.08 total 40.06 last 68.79)
 - open/close 3505 (time 1782224736.09 total 50.07 last 72.21)
 - open/close 4150 (time 1782224746.10 total 60.07 last 64.48)
total: 4500 open/close in 65.47 seconds: 68.74 ops/second
 - /unlink 1266 (time 1782224766.32 total 10.01 last 126.42)
 - /unlink 2525 (time 1782224776.33 total 20.02 last 125.82)
 - /unlink 3859 (time 1782224786.34 total 30.03 last 133.30)
total: 4500 /unlink in 35.23 seconds: 127.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 580 (time 1782224805.94 total 10.01 last 57.95)
 - open/close 1205 (time 1782224815.96 total 20.03 last 62.36)
 - open/close 1843 (time 1782224825.98 total 30.05 last 63.66)
 - open/close 2440 (time 1782224835.98 total 40.05 last 59.69)
 - open/close 3022 (time 1782224845.98 total 50.06 last 58.19)
 - open/close 3676 (time 1782224856.00 total 60.07 last 65.30)
 - open/close 4366 (time 1782224866.00 total 70.08 last 68.96)
total: 4500 open/close in 72.34 seconds: 62.21 ops/second
 - /unlink 1120 (time 1782224882.98 total 10.00 last 111.98)
 - /unlink 2205 (time 1782224892.99 total 20.01 last 108.36)
 - /unlink 3222 (time 1782224903.00 total 30.02 last 101.64)
 - /unlink 4225 (time 1782224913.01 total 40.03 last 100.21)
total: 4500 /unlink in 43.01 seconds: 104.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 648 (time 1782224931.39 total 10.01 last 64.76)
 - open/close 1297 (time 1782224941.40 total 20.02 last 64.83)
 - open/close 2001 (time 1782224951.40 total 30.02 last 70.36)
 - open/close 2653 (time 1782224961.41 total 40.03 last 65.13)
 - open/close 3315 (time 1782224971.42 total 50.03 last 66.20)
 - open/close 3944 (time 1782224981.42 total 60.04 last 62.86)
total: 4500 open/close in 68.69 seconds: 65.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1147 (time 1782225005.23 total 10.01 last 114.62)
 - /unlink 2225 (time 1782225015.23 total 20.01 last 107.74)
 - /unlink 3319 (time 1782225025.24 total 30.02 last 109.35)
total: 4500 /unlink in 39.59 seconds: 113.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 634 (time 1782225049.57 total 10.01 last 63.36)
 - open/close 1311 (time 1782225059.57 total 20.01 last 67.68)
 - open/close 1962 (time 1782225069.58 total 30.02 last 65.05)
 - open/close 2631 (time 1782225079.59 total 40.03 last 66.84)
 - open/close 3241 (time 1782225089.61 total 50.05 last 60.87)
 - open/close 3865 (time 1782225099.61 total 60.05 last 62.38)
total: 4500 open/close in 69.30 seconds: 64.93 ops/second
 - /unlink 1245 (time 1782225124.47 total 10.00 last 124.50)
 - /unlink 2310 (time 1782225134.48 total 20.00 last 106.47)
 - /unlink 3469 (time 1782225144.48 total 30.01 last 115.83)
total: 4500 /unlink in 38.24 seconds: 117.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 599 (time 1782225168.68 total 10.01 last 59.85)
 - open/close 1262 (time 1782225178.69 total 20.02 last 66.24)
 - open/close 1890 (time 1782225188.69 total 30.02 last 62.78)
 - open/close 2575 (time 1782225198.70 total 40.03 last 68.42)
 - open/close 3219 (time 1782225208.71 total 50.04 last 64.34)
 - open/close 3854 (time 1782225218.71 total 60.04 last 63.48)
 - open/close 4404 (time 1782225228.72 total 70.05 last 54.98)
total: 4500 open/close in 72.28 seconds: 62.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 957 (time 1782225247.49 total 10.01 last 95.64)
 - /unlink 2016 (time 1782225257.50 total 20.01 last 105.84)
 - /unlink 3040 (time 1782225267.50 total 30.01 last 102.38)
 - /unlink 3980 (time 1782225277.51 total 40.02 last 93.93)
total: 4500 /unlink in 44.56 seconds: 101.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 607 (time 1782225297.46 total 10.00 last 60.68)
 - open/close 1212 (time 1782225307.47 total 20.01 last 60.44)
 - open/close 1850 (time 1782225317.48 total 30.02 last 63.77)
 - open/close 2490 (time 1782225327.48 total 40.02 last 63.97)
 - open/close 3175 (time 1782225337.50 total 50.04 last 68.37)
 - open/close 3782 (time 1782225347.52 total 60.06 last 60.61)
 - open/close 4454 (time 1782225357.52 total 70.06 last 67.16)
total: 4500 open/close in 70.68 seconds: 63.66 ops/second
 - /unlink 1244 (time 1782225373.01 total 10.01 last 124.22)
 - /unlink 2438 (time 1782225383.02 total 20.03 last 119.25)
 - /unlink 3580 (time 1782225393.02 total 30.03 last 114.18)
total: 4500 /unlink in 38.65 seconds: 116.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 688 (time 1782225416.75 total 10.00 last 68.80)
 - open/close 1352 (time 1782225426.75 total 20.00 last 66.38)
 - open/close 1933 (time 1782225436.76 total 30.01 last 58.08)
 - open/close 2453 (time 1782225446.77 total 40.03 last 51.90)
 - open/close 3062 (time 1782225456.78 total 50.04 last 60.84)
 - open/close 3724 (time 1782225466.80 total 60.05 last 66.10)
 - open/close 4366 (time 1782225476.82 total 70.07 last 64.09)
total: 4500 open/close in 72.15 seconds: 62.37 ops/second
 - /unlink 1184 (time 1782225493.65 total 10.00 last 118.40)
 - /unlink 2490 (time 1782225503.65 total 20.00 last 130.56)
 - /unlink 3763 (time 1782225513.65 total 30.01 last 127.25)
total: 4500 /unlink in 37.05 seconds: 121.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 649 (time 1782225536.62 total 10.01 last 64.84)
 - open/close 1297 (time 1782225546.64 total 20.03 last 64.66)
 - open/close 1917 (time 1782225556.67 total 30.06 last 61.84)
 - open/close 2580 (time 1782225566.67 total 40.07 last 66.24)
 - open/close 3258 (time 1782225576.69 total 50.08 last 67.69)
 - open/close 3939 (time 1782225586.70 total 60.09 last 68.05)
total: 4500 open/close in 68.36 seconds: 65.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1301 (time 1782225608.69 total 10.01 last 129.95)
 - /unlink 2502 (time 1782225618.70 total 20.01 last 120.08)
 - /unlink 3773 (time 1782225628.70 total 30.02 last 127.04)
total: 4500 /unlink in 35.87 seconds: 125.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 615 (time 1782225648.75 total 10.02 last 61.39)
 - open/close 1234 (time 1782225658.76 total 20.04 last 61.79)
 - open/close 1866 (time 1782225668.77 total 30.04 last 63.15)
 - open/close 2577 (time 1782225678.78 total 40.05 last 71.05)
 - open/close 3221 (time 1782225688.79 total 50.06 last 64.33)
 - open/close 3873 (time 1782225698.80 total 60.07 last 65.15)
total: 4500 open/close in 69.27 seconds: 64.96 ops/second
 - /unlink 1231 (time 1782225722.51 total 10.00 last 123.10)
 - /unlink 2448 (time 1782225732.52 total 20.00 last 121.67)
 - /unlink 3714 (time 1782225742.52 total 30.01 last 126.56)
total: 4500 /unlink in 35.95 seconds: 125.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 751 (time 1782225762.74 total 10.02 last 74.94)
 - open/close 1460 (time 1782225772.74 total 20.02 last 70.89)
 - open/close 2176 (time 1782225782.74 total 30.03 last 71.56)
 - open/close 2871 (time 1782225792.75 total 40.03 last 69.47)
 - open/close 3584 (time 1782225802.75 total 50.04 last 71.26)
 - open/close 4323 (time 1782225812.76 total 60.05 last 73.85)
total: 4500 open/close in 62.52 seconds: 71.97 ops/second
 - /unlink 1261 (time 1782225829.27 total 10.00 last 126.08)
 - /unlink 2443 (time 1782225839.27 total 20.01 last 118.16)
 - /unlink 3687 (time 1782225849.28 total 30.02 last 124.28)
total: 4500 /unlink in 36.48 seconds: 123.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 658 (time 1782225870.62 total 10.00 last 65.78)
 - open/close 1321 (time 1782225880.63 total 20.01 last 66.23)
 - open/close 2009 (time 1782225890.64 total 30.02 last 68.74)
 - open/close 2714 (time 1782225900.65 total 40.03 last 70.44)
 - open/close 3427 (time 1782225910.66 total 50.04 last 71.22)
 - open/close 4117 (time 1782225920.66 total 60.04 last 69.00)
total: 4500 open/close in 65.64 seconds: 68.56 ops/second
 - /unlink 1388 (time 1782225941.13 total 10.00 last 138.77)
 - /unlink 2555 (time 1782225951.13 total 20.00 last 116.68)
 - /unlink 3944 (time 1782225961.13 total 30.01 last 138.87)
total: 4500 /unlink in 34.40 seconds: 130.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 641 (time 1782225980.53 total 10.00 last 64.08)
 - open/close 1324 (time 1782225990.55 total 20.02 last 68.18)
 - open/close 2015 (time 1782226000.55 total 30.02 last 69.08)
 - open/close 2714 (time 1782226010.56 total 40.03 last 69.83)
 - open/close 3395 (time 1782226020.58 total 50.05 last 67.98)
 - open/close 4064 (time 1782226030.59 total 60.06 last 66.85)
total: 4500 open/close in 65.98 seconds: 68.20 ops/second
 - /unlink 1308 (time 1782226051.01 total 10.01 last 130.72)
 - /unlink 2382 (time 1782226061.01 total 20.01 last 107.36)
 - /unlink 3607 (time 1782226071.02 total 30.02 last 122.42)
total: 4500 /unlink in 37.13 seconds: 121.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 667 (time 1782226093.00 total 10.02 last 66.59)
 - open/close 1221 (time 1782226103.02 total 20.03 last 55.32)
 - open/close 1866 (time 1782226113.04 total 30.05 last 64.36)
 - open/close 2385 (time 1782226123.04 total 40.06 last 51.86)
 - open/close 3050 (time 1782226133.05 total 50.07 last 66.44)
 - open/close 3804 (time 1782226143.05 total 60.07 last 75.38)
total: 4500 open/close in 69.44 seconds: 64.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1344 (time 1782226166.52 total 10.01 last 134.32)
 - /unlink 2674 (time 1782226176.52 total 20.01 last 132.97)
 - /unlink 4004 (time 1782226186.53 total 30.02 last 132.86)
total: 4500 /unlink in 33.81 seconds: 133.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 759 (time 1782226205.40 total 10.01 last 75.85)
 - open/close 1484 (time 1782226215.41 total 20.01 last 72.48)
 - open/close 2226 (time 1782226225.41 total 30.01 last 74.17)
 - open/close 3000 (time 1782226235.42 total 40.02 last 77.32)
 - open/close 3727 (time 1782226245.42 total 50.03 last 72.68)
 - open/close 4471 (time 1782226255.43 total 60.03 last 74.35)
total: 4500 open/close in 60.49 seconds: 74.39 ops/second
 - /unlink 1370 (time 1782226270.27 total 10.00 last 136.93)
 - /unlink 2618 (time 1782226280.27 total 20.01 last 124.76)
 - /unlink 3910 (time 1782226290.27 total 30.01 last 129.17)
total: 4500 /unlink in 35.16 seconds: 128.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 721 (time 1782226310.71 total 10.00 last 72.10)
 - open/close 1417 (time 1782226320.71 total 20.00 last 69.58)
 - open/close 2138 (time 1782226330.71 total 30.01 last 72.06)
 - open/close 2836 (time 1782226340.72 total 40.01 last 69.78)
 - open/close 3519 (time 1782226350.72 total 50.02 last 68.24)
 - open/close 4247 (time 1782226360.74 total 60.03 last 72.70)
total: 4500 open/close in 63.76 seconds: 70.58 ops/second
 - /unlink 1246 (time 1782226379.34 total 10.01 last 124.49)
 - /unlink 2426 (time 1782226389.34 total 20.02 last 117.92)
 - /unlink 3700 (time 1782226399.35 total 30.03 last 127.27)
total: 4500 /unlink in 36.00 seconds: 124.98 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94949 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg360-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg360-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server
stop mds service on oleg360-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server
stop mds service on oleg360-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server

unloading modules via unload_modules_local on: 'oleg360-server'
oleg360-server: oleg360-server.virtnet: executing unload_modules_local
oleg360-server: 
modules unloaded.