== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:27:36 (1789147656)
Stopping clients: oleg336-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg336-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg336-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg336-server
oleg336-server: oleg336-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg336-server'
oleg336-server: oleg336-server.virtnet: executing load_modules_local
oleg336-server: Loading modules from /home/green/git/lustre-release/lustre
oleg336-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg336-server: detected 4 online CPUs by sysfs
oleg336-server: MODOPTS_LIBCFS=
oleg336-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg336-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all
pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg336-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all
pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg336-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all
pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg336-client.virtnet:  -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 671 (time 1789147883.66 total 10.01 last 67.03)
 - open/close 1345 (time 1789147893.66 total 20.01 last 67.39)
 - open/close 1908 (time 1789147903.66 total 30.02 last 56.27)
 - open/close 2500 (time 1789147913.66 total 40.02 last 59.19)
 - open/close 3072 (time 1789147923.66 total 50.02 last 57.20)
 - open/close 3729 (time 1789147933.69 total 60.04 last 65.56)
 - open/close 4427 (time 1789147943.69 total 70.04 last 69.78)
total: 4500 open/close in 71.23 seconds: 63.18 ops/second
 - /unlink 1245 (time 1789147959.82 total 10.01 last 124.38)
 - /unlink 2434 (time 1789147969.83 total 20.02 last 118.79)
 - /unlink 3874 (time 1789147979.84 total 30.03 last 143.83)
total: 4500 /unlink in 35.44 seconds: 126.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 575 (time 1789148000.29 total 10.01 last 57.46)
 - open/close 1251 (time 1789148010.30 total 20.02 last 67.52)
 - open/close 1934 (time 1789148020.31 total 30.03 last 68.24)
 - open/close 2699 (time 1789148030.31 total 40.03 last 76.48)
 - open/close 3398 (time 1789148040.32 total 50.04 last 69.86)
 - open/close 4132 (time 1789148050.33 total 60.04 last 73.34)
total: 4500 open/close in 65.51 seconds: 68.70 ops/second
 - /unlink 1220 (time 1789148071.27 total 10.00 last 122.00)
 - /unlink 2306 (time 1789148081.28 total 20.01 last 108.47)
 - /unlink 3429 (time 1789148091.28 total 30.02 last 112.24)
total: 4500 /unlink in 39.46 seconds: 114.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 738 (time 1789148116.50 total 10.02 last 73.66)
 - open/close 1397 (time 1789148126.51 total 20.03 last 65.85)
 - open/close 2073 (time 1789148136.52 total 30.03 last 67.56)
 - open/close 2717 (time 1789148146.52 total 40.03 last 64.40)
 - open/close 3322 (time 1789148156.52 total 50.04 last 60.48)
 - open/close 3978 (time 1789148166.55 total 60.06 last 65.43)
total: 4500 open/close in 67.30 seconds: 66.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1266 (time 1789148188.64 total 10.01 last 126.50)
 - /unlink 2557 (time 1789148198.64 total 20.01 last 129.06)
 - /unlink 3772 (time 1789148208.64 total 30.01 last 121.45)
total: 4500 /unlink in 34.84 seconds: 129.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 621 (time 1789148229.19 total 10.00 last 62.09)
 - open/close 1298 (time 1789148239.21 total 20.02 last 67.60)
 - open/close 1994 (time 1789148249.22 total 30.03 last 69.52)
 - open/close 2706 (time 1789148259.23 total 40.03 last 71.15)
 - open/close 3409 (time 1789148269.24 total 50.05 last 70.22)
 - open/close 4165 (time 1789148279.25 total 60.06 last 75.53)
total: 4500 open/close in 65.14 seconds: 69.08 ops/second
 - /unlink 1334 (time 1789148298.95 total 10.00 last 133.40)
 - /unlink 2562 (time 1789148308.95 total 20.00 last 122.75)
 - /unlink 3839 (time 1789148318.95 total 30.01 last 127.67)
total: 4500 /unlink in 35.52 seconds: 126.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 582 (time 1789148340.47 total 10.02 last 58.11)
 - open/close 1243 (time 1789148350.47 total 20.02 last 66.07)
 - open/close 1926 (time 1789148360.47 total 30.02 last 68.29)
 - open/close 2647 (time 1789148370.48 total 40.03 last 72.05)
 - open/close 3349 (time 1789148380.49 total 50.04 last 70.15)
 - open/close 4065 (time 1789148390.49 total 60.04 last 71.56)
total: 4500 open/close in 66.55 seconds: 67.61 ops/second
 - /unlink 1353 (time 1789148411.89 total 10.00 last 135.29)
 - /unlink 2645 (time 1789148421.90 total 20.00 last 129.18)
 - /unlink 3971 (time 1789148431.90 total 30.01 last 132.51)
total: 4500 /unlink in 34.32 seconds: 131.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 719 (time 1789148451.08 total 10.00 last 71.87)
 - open/close 1389 (time 1789148461.09 total 20.01 last 66.95)
 - open/close 2071 (time 1789148471.09 total 30.02 last 68.15)
 - open/close 2748 (time 1789148481.10 total 40.02 last 67.66)
 - open/close 3401 (time 1789148491.12 total 50.04 last 65.19)
 - open/close 4053 (time 1789148501.12 total 60.05 last 65.16)
total: 4500 open/close in 66.83 seconds: 67.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1289 (time 1789148522.98 total 10.00 last 128.89)
 - /unlink 2563 (time 1789148532.99 total 20.01 last 127.32)
 - /unlink 3947 (time 1789148543.00 total 30.01 last 138.30)
total: 4500 /unlink in 33.89 seconds: 132.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 574 (time 1789148562.90 total 10.00 last 57.40)
 - open/close 1125 (time 1789148572.90 total 20.00 last 55.09)
 - open/close 1666 (time 1789148582.91 total 30.02 last 54.03)
 - open/close 2227 (time 1789148592.92 total 40.03 last 56.03)
 - open/close 2882 (time 1789148602.93 total 50.04 last 65.44)
 - open/close 3597 (time 1789148612.94 total 60.04 last 71.47)
 - open/close 4205 (time 1789148622.95 total 70.05 last 60.75)
total: 4500 open/close in 74.22 seconds: 60.63 ops/second
 - /unlink 1349 (time 1789148641.42 total 10.00 last 134.89)
 - /unlink 2682 (time 1789148651.42 total 20.00 last 133.28)
 - /unlink 4072 (time 1789148661.42 total 30.00 last 138.98)
total: 4500 /unlink in 33.40 seconds: 134.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 701 (time 1789148680.29 total 10.01 last 70.02)
 - open/close 1380 (time 1789148690.29 total 20.01 last 67.90)
 - open/close 2101 (time 1789148700.30 total 30.02 last 72.05)
 - open/close 2898 (time 1789148710.31 total 40.03 last 79.63)
 - open/close 3674 (time 1789148720.32 total 50.04 last 77.52)
 - open/close 4319 (time 1789148730.33 total 60.05 last 64.40)
total: 4500 open/close in 62.43 seconds: 72.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1267 (time 1789148747.61 total 10.00 last 126.69)
 - /unlink 2456 (time 1789148757.61 total 20.00 last 118.87)
 - /unlink 3608 (time 1789148767.61 total 30.01 last 115.17)
total: 4500 /unlink in 37.10 seconds: 121.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 690 (time 1789148789.32 total 10.00 last 68.97)
 - open/close 1231 (time 1789148799.32 total 20.01 last 54.08)
 - open/close 1815 (time 1789148809.32 total 30.01 last 58.39)
 - open/close 2392 (time 1789148819.33 total 40.02 last 57.65)
 - open/close 3034 (time 1789148829.34 total 50.03 last 64.14)
 - open/close 3678 (time 1789148839.38 total 60.06 last 64.17)
 - open/close 4368 (time 1789148849.39 total 70.08 last 68.90)
total: 4500 open/close in 71.94 seconds: 62.55 ops/second
 - /unlink 1248 (time 1789148866.02 total 10.01 last 124.70)
 - /unlink 2304 (time 1789148876.02 total 20.01 last 105.58)
 - /unlink 3531 (time 1789148886.02 total 30.02 last 122.64)
total: 4500 /unlink in 37.14 seconds: 121.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 707 (time 1789148908.25 total 10.01 last 70.64)
 - open/close 1310 (time 1789148918.25 total 20.01 last 60.29)
 - open/close 1681 (time 1789148928.28 total 30.04 last 36.99)
 - open/close 2058 (time 1789148938.32 total 40.08 last 37.56)
 - open/close 2503 (time 1789148948.32 total 50.08 last 44.49)
 - open/close 3120 (time 1789148958.32 total 60.08 last 61.69)
 - open/close 3876 (time 1789148968.33 total 70.09 last 75.54)
total: 4500 open/close in 78.19 seconds: 57.55 ops/second
 - /unlink 1189 (time 1789148990.54 total 10.01 last 118.83)
 - /unlink 2314 (time 1789149000.54 total 20.01 last 112.44)
 - /unlink 3472 (time 1789149010.55 total 30.01 last 115.76)
total: 4500 /unlink in 37.70 seconds: 119.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 620 (time 1789149033.04 total 10.02 last 61.87)
 - open/close 1142 (time 1789149043.05 total 20.03 last 52.17)
 - open/close 1774 (time 1789149053.07 total 30.05 last 63.07)
 - open/close 2465 (time 1789149063.08 total 40.06 last 68.99)
 - open/close 3124 (time 1789149073.09 total 50.07 last 65.87)
 - open/close 3789 (time 1789149083.09 total 60.07 last 66.46)
 - open/close 4447 (time 1789149093.10 total 70.08 last 65.77)
total: 4500 open/close in 70.89 seconds: 63.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1128 (time 1789149109.14 total 10.00 last 112.79)
 - /unlink 2185 (time 1789149119.15 total 20.01 last 105.63)
 - /unlink 3166 (time 1789149129.15 total 30.01 last 98.09)
 - /unlink 4345 (time 1789149139.15 total 40.01 last 117.87)
total: 4500 /unlink in 41.37 seconds: 108.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 644 (time 1789149154.90 total 10.02 last 64.30)
 - open/close 1398 (time 1789149164.91 total 20.02 last 75.38)
 - open/close 2108 (time 1789149174.92 total 30.03 last 70.91)
 - open/close 2756 (time 1789149184.95 total 40.06 last 64.63)
 - open/close 3305 (time 1789149194.95 total 50.06 last 54.90)
 - open/close 3907 (time 1789149204.96 total 60.08 last 60.10)
total: 4500 open/close in 69.73 seconds: 64.53 ops/second
 - /unlink 1009 (time 1789149229.73 total 10.01 last 100.84)
 - /unlink 1954 (time 1789149239.74 total 20.01 last 94.42)
 - /unlink 3125 (time 1789149249.74 total 30.02 last 117.04)
 - /unlink 4351 (time 1789149259.75 total 40.02 last 122.55)
total: 4500 /unlink in 41.61 seconds: 108.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 656 (time 1789149276.58 total 10.00 last 65.57)
 - open/close 1249 (time 1789149286.59 total 20.02 last 59.19)
 - open/close 1839 (time 1789149296.63 total 30.06 last 58.77)
 - open/close 2421 (time 1789149306.64 total 40.06 last 58.18)
 - open/close 3039 (time 1789149316.64 total 50.07 last 61.79)
 - open/close 3680 (time 1789149326.66 total 60.09 last 63.94)
 - open/close 4298 (time 1789149336.67 total 70.10 last 61.76)
total: 4500 open/close in 73.63 seconds: 61.12 ops/second
 - /unlink 1270 (time 1789149354.43 total 10.00 last 127.00)
 - /unlink 2385 (time 1789149364.45 total 20.02 last 111.29)
 - /unlink 3609 (time 1789149374.45 total 30.02 last 122.38)
total: 4500 /unlink in 38.44 seconds: 117.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 557 (time 1789149398.38 total 10.00 last 55.70)
 - open/close 1103 (time 1789149408.42 total 20.04 last 54.40)
 - open/close 1763 (time 1789149418.45 total 30.07 last 65.80)
 - open/close 2348 (time 1789149428.46 total 40.08 last 58.42)
 - open/close 2989 (time 1789149438.47 total 50.09 last 64.05)
 - open/close 3599 (time 1789149448.47 total 60.09 last 61.00)
 - open/close 4178 (time 1789149458.48 total 70.10 last 57.86)
total: 4500 open/close in 74.75 seconds: 60.20 ops/second
 - /unlink 1141 (time 1789149478.54 total 10.00 last 114.09)
 - /unlink 2277 (time 1789149488.55 total 20.01 last 113.53)
 - /unlink 3434 (time 1789149498.55 total 30.01 last 115.66)
total: 4500 /unlink in 38.57 seconds: 116.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 621 (time 1789149522.53 total 10.02 last 61.96)
 - open/close 1187 (time 1789149532.54 total 20.03 last 56.54)
 - open/close 1772 (time 1789149542.54 total 30.04 last 58.49)
 - open/close 2452 (time 1789149552.55 total 40.04 last 67.96)
 - open/close 3089 (time 1789149562.55 total 50.04 last 63.69)
 - open/close 3805 (time 1789149572.55 total 60.04 last 71.60)
 - open/close 4367 (time 1789149582.56 total 70.05 last 56.14)
total: 4500 open/close in 72.24 seconds: 62.29 ops/second
 - /unlink 1311 (time 1789149600.25 total 10.01 last 131.03)
 - /unlink 2495 (time 1789149610.26 total 20.01 last 118.32)
 - /unlink 3546 (time 1789149620.27 total 30.02 last 105.02)
total: 4500 /unlink in 37.80 seconds: 119.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 600 (time 1789149644.95 total 10.00 last 60.00)
 - open/close 1189 (time 1789149654.96 total 20.01 last 58.85)
 - open/close 1808 (time 1789149664.99 total 30.04 last 61.70)
 - open/close 2381 (time 1789149675.02 total 40.07 last 57.13)
 - open/close 3001 (time 1789149685.04 total 50.09 last 61.87)
 - open/close 3692 (time 1789149695.05 total 60.10 last 69.04)
 - open/close 4403 (time 1789149705.05 total 70.10 last 71.08)
total: 4500 open/close in 71.51 seconds: 62.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1343 (time 1789149721.11 total 10.00 last 134.30)
 - /unlink 2669 (time 1789149731.11 total 20.00 last 132.56)
 - /unlink 4009 (time 1789149741.12 total 30.01 last 133.96)
total: 4500 /unlink in 34.10 seconds: 131.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 688 (time 1789149759.97 total 10.00 last 68.77)
 - open/close 1354 (time 1789149769.97 total 20.01 last 66.57)
 - open/close 2115 (time 1789149780.00 total 30.03 last 75.93)
 - open/close 2783 (time 1789149790.01 total 40.04 last 66.72)
 - open/close 3451 (time 1789149800.03 total 50.06 last 66.68)
 - open/close 4141 (time 1789149810.03 total 60.06 last 69.00)
total: 4500 open/close in 65.95 seconds: 68.24 ops/second
 - /unlink 1313 (time 1789149830.66 total 10.01 last 131.12)
 - /unlink 2581 (time 1789149840.67 total 20.02 last 126.68)
 - /unlink 3625 (time 1789149850.67 total 30.03 last 104.34)
total: 4500 /unlink in 38.89 seconds: 115.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 485 (time 1789149874.46 total 10.02 last 48.39)
 - open/close 949 (time 1789149884.48 total 20.04 last 46.31)
 - open/close 1533 (time 1789149894.49 total 30.06 last 58.32)
 - open/close 2177 (time 1789149904.50 total 40.06 last 64.37)
 - open/close 2764 (time 1789149914.51 total 50.08 last 58.61)
 - open/close 3494 (time 1789149924.52 total 60.09 last 72.95)
 - open/close 4106 (time 1789149934.53 total 70.10 last 61.13)
total: 4500 open/close in 76.29 seconds: 58.99 ops/second
 - /unlink 1219 (time 1789149955.29 total 10.02 last 121.65)
 - /unlink 2513 (time 1789149965.30 total 20.02 last 129.34)
 - /unlink 3738 (time 1789149975.30 total 30.03 last 122.47)
total: 4500 /unlink in 36.31 seconds: 123.93 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95361 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg336-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg336-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server
stop mds service on oleg336-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server
stop mds service on oleg336-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server

unloading modules via unload_modules_local on: 'oleg336-server'
oleg336-server: oleg336-server.virtnet: executing unload_modules_local
oleg336-server: 
modules unloaded.