== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:11:08 (1780330268)
Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg220-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg220-server
oleg220-server: oleg220-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: Loading modules from /home/green/git/lustre-release/lustre
oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg220-server: detected 4 online CPUs by sysfs
oleg220-server: MODOPTS_LIBCFS=
oleg220-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg220-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg220-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg220-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 974 (time 1780330441.34 total 10.00 last 97.36)
 - open/close 1893 (time 1780330451.34 total 20.00 last 91.89)
 - open/close 2890 (time 1780330461.34 total 30.00 last 99.70)
 - open/close 3853 (time 1780330471.34 total 40.01 last 96.26)
total: 4500 open/close in 46.39 seconds: 96.99 ops/second
 - /unlink 2158 (time 1780330490.46 total 10.00 last 215.73)
 - /unlink 4440 (time 1780330500.46 total 20.00 last 228.16)
total: 4500 /unlink in 20.27 seconds: 221.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 1118 (time 1780330513.46 total 10.00 last 111.77)
 - open/close 2283 (time 1780330523.47 total 20.02 last 116.36)
 - open/close 3424 (time 1780330533.47 total 30.02 last 114.07)
total: 4500 open/close in 38.89 seconds: 115.71 ops/second
 - /unlink 1761 (time 1780330554.83 total 10.00 last 176.10)
 - /unlink 2752 (time 1780330564.84 total 20.00 last 99.07)
 - /unlink 3689 (time 1780330574.94 total 30.10 last 92.78)
total: 4500 /unlink in 36.78 seconds: 122.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 533 (time 1780330597.01 total 10.03 last 53.15)
 - open/close 1009 (time 1780330607.03 total 20.05 last 47.51)
 - open/close 1456 (time 1780330617.03 total 30.05 last 44.68)
 - open/close 1886 (time 1780330627.03 total 40.05 last 42.99)
 - open/close 2320 (time 1780330637.04 total 50.06 last 43.36)
 - open/close 2861 (time 1780330647.05 total 60.07 last 54.05)
 - open/close 3510 (time 1780330657.06 total 70.08 last 64.86)
 - open/close 4157 (time 1780330667.06 total 80.08 last 64.68)
total: 4500 open/close in 84.98 seconds: 52.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 994 (time 1780330687.17 total 10.01 last 99.33)
 - /unlink 2296 (time 1780330697.18 total 20.01 last 130.18)
 - /unlink 3577 (time 1780330707.18 total 30.01 last 128.09)
total: 4500 /unlink in 39.52 seconds: 113.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 804 (time 1780330732.92 total 10.00 last 80.39)
 - open/close 1597 (time 1780330742.93 total 20.01 last 79.27)
 - open/close 2185 (time 1780330752.93 total 30.01 last 58.76)
 - open/close 2733 (time 1780330762.97 total 40.04 last 54.62)
 - open/close 3424 (time 1780330772.97 total 50.05 last 69.07)
 - open/close 4093 (time 1780330782.97 total 60.05 last 66.87)
total: 4500 open/close in 67.43 seconds: 66.73 ops/second
 - /unlink 1228 (time 1780330806.10 total 10.01 last 122.74)
 - /unlink 2099 (time 1780330816.12 total 20.02 last 86.95)
 - /unlink 3295 (time 1780330826.13 total 30.03 last 119.49)
 - /unlink 4403 (time 1780330836.13 total 40.03 last 110.76)
total: 4500 /unlink in 40.91 seconds: 109.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 716 (time 1780330852.72 total 10.00 last 71.56)
 - open/close 1414 (time 1780330862.72 total 20.01 last 69.77)
 - open/close 2077 (time 1780330872.72 total 30.01 last 66.28)
 - open/close 2759 (time 1780330882.73 total 40.02 last 68.15)
 - open/close 3353 (time 1780330892.76 total 50.04 last 59.25)
 - open/close 3934 (time 1780330902.77 total 60.06 last 58.01)
 - open/close 4481 (time 1780330912.78 total 70.07 last 54.63)
total: 4500 open/close in 70.30 seconds: 64.01 ops/second
 - /unlink 1113 (time 1780330930.64 total 10.01 last 111.13)
 - /unlink 2316 (time 1780330940.64 total 20.02 last 120.24)
 - /unlink 3675 (time 1780330950.65 total 30.02 last 135.83)
total: 4500 /unlink in 37.44 seconds: 120.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 620 (time 1780330973.44 total 10.02 last 61.86)
 - open/close 1246 (time 1780330983.46 total 20.04 last 62.47)
 - open/close 1789 (time 1780330993.47 total 30.05 last 54.27)
 - open/close 2400 (time 1780331003.47 total 40.05 last 61.06)
 - open/close 3065 (time 1780331013.49 total 50.07 last 66.40)
 - open/close 3686 (time 1780331023.51 total 60.10 last 61.94)
 - open/close 4271 (time 1780331033.53 total 70.11 last 58.41)
total: 4500 open/close in 73.48 seconds: 61.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1297 (time 1780331051.83 total 10.01 last 129.60)
 - /unlink 2293 (time 1780331061.83 total 20.01 last 99.59)
 - /unlink 3474 (time 1780331071.83 total 30.01 last 118.06)
total: 4500 /unlink in 39.94 seconds: 112.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 534 (time 1780331097.81 total 10.01 last 53.37)
 - open/close 1248 (time 1780331107.81 total 20.01 last 71.36)
 - open/close 1788 (time 1780331117.82 total 30.02 last 53.97)
 - open/close 2498 (time 1780331127.83 total 40.03 last 70.93)
 - open/close 3060 (time 1780331137.83 total 50.03 last 56.20)
 - open/close 3642 (time 1780331147.83 total 60.03 last 58.20)
 - open/close 4248 (time 1780331157.85 total 70.05 last 60.47)
total: 4500 open/close in 74.65 seconds: 60.28 ops/second
 - /unlink 1256 (time 1780331178.44 total 10.00 last 125.56)
 - /unlink 2484 (time 1780331188.44 total 20.00 last 122.79)
 - /unlink 3666 (time 1780331198.44 total 30.01 last 118.16)
total: 4500 /unlink in 37.30 seconds: 120.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 612 (time 1780331221.76 total 10.01 last 61.14)
 - open/close 1222 (time 1780331231.77 total 20.01 last 60.99)
 - open/close 1763 (time 1780331241.77 total 30.02 last 54.06)
 - open/close 2403 (time 1780331251.78 total 40.03 last 63.94)
 - open/close 3012 (time 1780331261.81 total 50.06 last 60.73)
 - open/close 3704 (time 1780331271.81 total 60.06 last 69.16)
 - open/close 4394 (time 1780331281.83 total 70.08 last 68.88)
total: 4500 open/close in 71.39 seconds: 63.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1179 (time 1780331298.07 total 10.01 last 117.83)
 - /unlink 2302 (time 1780331308.08 total 20.01 last 112.26)
 - /unlink 3503 (time 1780331318.09 total 30.02 last 119.97)
total: 4500 /unlink in 38.84 seconds: 115.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 590 (time 1780331343.17 total 10.01 last 58.95)
 - open/close 1168 (time 1780331353.18 total 20.02 last 57.73)
 - open/close 1664 (time 1780331363.18 total 30.02 last 49.58)
 - open/close 2171 (time 1780331373.18 total 40.03 last 50.69)
 - open/close 2664 (time 1780331383.20 total 50.04 last 49.22)
 - open/close 3237 (time 1780331393.20 total 60.04 last 57.30)
 - open/close 3953 (time 1780331403.20 total 70.04 last 71.60)
total: 4500 open/close in 78.17 seconds: 57.57 ops/second
 - /unlink 1148 (time 1780331426.91 total 10.00 last 114.74)
 - /unlink 2052 (time 1780331436.92 total 20.01 last 90.36)
 - /unlink 3290 (time 1780331446.92 total 30.01 last 123.75)
total: 4500 /unlink in 38.72 seconds: 116.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 674 (time 1780331470.73 total 10.01 last 67.37)
 - open/close 1267 (time 1780331480.74 total 20.01 last 59.29)
 - open/close 1785 (time 1780331490.76 total 30.03 last 51.70)
 - open/close 2312 (time 1780331500.77 total 40.04 last 52.65)
 - open/close 2949 (time 1780331510.78 total 50.05 last 63.60)
 - open/close 3557 (time 1780331520.82 total 60.09 last 60.59)
 - open/close 3969 (time 1780331530.83 total 70.10 last 41.16)
 - open/close 4401 (time 1780331540.83 total 80.10 last 43.19)
total: 4500 open/close in 82.13 seconds: 54.79 ops/second
 - /unlink 1142 (time 1780331558.15 total 10.00 last 114.16)
 - /unlink 2169 (time 1780331568.16 total 20.01 last 102.65)
 - /unlink 3276 (time 1780331578.16 total 30.01 last 110.67)
 - /unlink 4434 (time 1780331588.17 total 40.02 last 115.73)
total: 4500 /unlink in 40.92 seconds: 109.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 591 (time 1780331605.86 total 10.02 last 58.99)
 - open/close 1145 (time 1780331615.87 total 20.03 last 55.33)
 - open/close 1716 (time 1780331625.88 total 30.04 last 57.07)
 - open/close 2375 (time 1780331635.89 total 40.05 last 65.84)
 - open/close 3104 (time 1780331645.90 total 50.06 last 72.82)
 - open/close 3713 (time 1780331655.91 total 60.07 last 60.84)
 - open/close 4269 (time 1780331665.92 total 70.07 last 55.56)
total: 4500 open/close in 74.38 seconds: 60.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 965 (time 1780331686.43 total 10.01 last 96.45)
 - /unlink 2198 (time 1780331696.43 total 20.01 last 123.29)
 - /unlink 3391 (time 1780331706.43 total 30.01 last 119.29)
total: 4500 /unlink in 37.90 seconds: 118.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 708 (time 1780331730.25 total 10.01 last 70.73)
 - open/close 1365 (time 1780331740.25 total 20.01 last 65.67)
 - open/close 2030 (time 1780331750.27 total 30.03 last 66.40)
 - open/close 2709 (time 1780331760.28 total 40.04 last 67.83)
 - open/close 3316 (time 1780331770.28 total 50.05 last 60.66)
 - open/close 3836 (time 1780331780.29 total 60.05 last 51.97)
 - open/close 4477 (time 1780331790.31 total 70.07 last 63.98)
total: 4500 open/close in 70.37 seconds: 63.94 ops/second
 - /unlink 1309 (time 1780331806.44 total 10.00 last 130.86)
 - /unlink 2400 (time 1780331816.44 total 20.00 last 109.09)
 - /unlink 3688 (time 1780331826.44 total 30.01 last 128.78)
total: 4500 /unlink in 36.90 seconds: 121.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 674 (time 1780331849.33 total 10.01 last 67.36)
 - open/close 1252 (time 1780331859.33 total 20.01 last 57.77)
 - open/close 1919 (time 1780331869.34 total 30.02 last 66.65)
 - open/close 2571 (time 1780331879.35 total 40.03 last 65.14)
 - open/close 3229 (time 1780331889.36 total 50.03 last 65.77)
 - open/close 3846 (time 1780331899.37 total 60.04 last 61.62)
total: 4500 open/close in 69.83 seconds: 64.45 ops/second
 - /unlink 1199 (time 1780331924.97 total 10.01 last 119.72)
 - /unlink 2398 (time 1780331934.97 total 20.02 last 119.86)
 - /unlink 3520 (time 1780331944.97 total 30.02 last 112.16)
total: 4500 /unlink in 39.44 seconds: 114.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 651 (time 1780331970.77 total 10.01 last 65.00)
 - open/close 1246 (time 1780331980.78 total 20.02 last 59.44)
 - open/close 1924 (time 1780331990.78 total 30.03 last 67.75)
 - open/close 2650 (time 1780332000.79 total 40.04 last 72.54)
 - open/close 3275 (time 1780332010.81 total 50.06 last 62.40)
 - open/close 4042 (time 1780332020.81 total 60.06 last 76.64)
total: 4500 open/close in 66.14 seconds: 68.03 ops/second
 - /unlink 1169 (time 1780332042.90 total 10.01 last 116.82)
 - /unlink 2491 (time 1780332052.90 total 20.01 last 132.14)
 - /unlink 3591 (time 1780332062.91 total 30.02 last 109.89)
total: 4500 /unlink in 38.98 seconds: 115.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 605 (time 1780332088.07 total 10.01 last 60.46)
 - open/close 1233 (time 1780332098.10 total 20.03 last 62.63)
 - open/close 1980 (time 1780332108.11 total 30.04 last 74.66)
 - open/close 2685 (time 1780332118.12 total 40.05 last 70.44)
 - open/close 3383 (time 1780332128.12 total 50.06 last 69.75)
 - open/close 4058 (time 1780332138.13 total 60.07 last 67.42)
total: 4500 open/close in 67.58 seconds: 66.59 ops/second
 - /unlink 1181 (time 1780332161.05 total 10.00 last 118.07)
 - /unlink 2361 (time 1780332171.06 total 20.01 last 117.88)
 - /unlink 3608 (time 1780332181.06 total 30.01 last 124.69)
total: 4500 /unlink in 37.36 seconds: 120.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 634 (time 1780332204.45 total 10.00 last 63.39)
 - open/close 1325 (time 1780332214.45 total 20.00 last 69.08)
 - open/close 1961 (time 1780332224.48 total 30.03 last 63.42)
 - open/close 2601 (time 1780332234.49 total 40.04 last 63.96)
 - open/close 3144 (time 1780332244.49 total 50.04 last 54.30)
 - open/close 3784 (time 1780332254.49 total 60.04 last 63.98)
 - open/close 4374 (time 1780332264.50 total 70.06 last 58.92)
total: 4500 open/close in 72.40 seconds: 62.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 870 (time 1780332282.93 total 10.00 last 86.98)
 - /unlink 1973 (time 1780332292.94 total 20.01 last 110.18)
 - /unlink 2982 (time 1780332302.95 total 30.03 last 100.78)
 - /unlink 4031 (time 1780332312.95 total 40.03 last 104.85)
total: 4500 /unlink in 43.75 seconds: 102.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 606 (time 1780332332.32 total 10.01 last 60.55)
 - open/close 1152 (time 1780332342.33 total 20.01 last 54.58)
 - open/close 1824 (time 1780332352.33 total 30.02 last 67.18)
 - open/close 2448 (time 1780332362.34 total 40.03 last 62.32)
 - open/close 3171 (time 1780332372.36 total 50.04 last 72.19)
 - open/close 3917 (time 1780332382.36 total 60.05 last 74.55)
total: 4500 open/close in 67.62 seconds: 66.55 ops/second
 - /unlink 1253 (time 1780332404.58 total 10.00 last 125.25)
 - /unlink 2398 (time 1780332414.59 total 20.01 last 114.40)
 - /unlink 3701 (time 1780332424.59 total 30.02 last 130.26)
total: 4500 /unlink in 36.20 seconds: 124.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 499 (time 1780332446.36 total 10.02 last 49.81)
 - open/close 1174 (time 1780332456.36 total 20.02 last 67.48)
 - open/close 1899 (time 1780332466.37 total 30.03 last 72.44)
 - open/close 2604 (time 1780332476.38 total 40.04 last 70.47)
 - open/close 3214 (time 1780332486.42 total 50.08 last 60.73)
 - open/close 3719 (time 1780332496.45 total 60.11 last 50.37)
 - open/close 4227 (time 1780332506.46 total 70.12 last 50.72)
total: 4500 open/close in 74.84 seconds: 60.12 ops/second
 - /unlink 1133 (time 1780332525.96 total 10.03 last 112.94)
 - /unlink 2475 (time 1780332535.96 total 20.03 last 134.15)
 - /unlink 3471 (time 1780332545.97 total 30.04 last 99.58)
total: 4500 /unlink in 39.64 seconds: 113.53 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 93792 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server
stop mds service on oleg220-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server
stop mds service on oleg220-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server

unloading modules via unload_modules_local on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing unload_modules_local
oleg220-server: 
modules unloaded.