== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:35:23 (1786048523)
Stopping clients: oleg602-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg602-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg602-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg602-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg602-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg602-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg602-server
oleg602-server: oleg602-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg602-server'
oleg602-server: oleg602-server.virtnet: executing load_modules_local
oleg602-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg602-server: Loading modules from /home/green/git/lustre-release/lustre
oleg602-server: detected 4 online CPUs by sysfs
oleg602-server: MODOPTS_LIBCFS=
oleg602-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg602-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all
pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg602-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all
pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg602-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all
pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg602-client.virtnet:  -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 590 (time 1786048710.89 total 10.00 last 58.99)
 - open/close 1195 (time 1786048720.90 total 20.02 last 60.40)
 - open/close 1886 (time 1786048730.91 total 30.03 last 69.03)
 - open/close 2566 (time 1786048740.93 total 40.05 last 67.88)
 - open/close 3255 (time 1786048750.94 total 50.05 last 68.84)
 - open/close 3917 (time 1786048760.95 total 60.07 last 66.11)
total: 4500 open/close in 68.05 seconds: 66.12 ops/second
 - /unlink 1206 (time 1786048784.20 total 10.00 last 120.60)
 - /unlink 2480 (time 1786048794.20 total 20.00 last 127.35)
 - /unlink 3643 (time 1786048804.21 total 30.01 last 116.21)
total: 4500 /unlink in 36.70 seconds: 122.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 672 (time 1786048825.51 total 10.01 last 67.15)
 - open/close 1302 (time 1786048835.52 total 20.01 last 62.96)
 - open/close 1973 (time 1786048845.52 total 30.02 last 67.08)
 - open/close 2766 (time 1786048855.53 total 40.03 last 79.22)
 - open/close 3502 (time 1786048865.54 total 50.03 last 73.56)
 - open/close 4137 (time 1786048875.56 total 60.05 last 63.39)
total: 4500 open/close in 65.20 seconds: 69.02 ops/second
 - /unlink 1298 (time 1786048894.62 total 10.00 last 129.75)
 - /unlink 2532 (time 1786048904.62 total 20.01 last 123.34)
 - /unlink 3853 (time 1786048914.62 total 30.01 last 132.08)
total: 4500 /unlink in 34.86 seconds: 129.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 729 (time 1786048934.80 total 10.01 last 72.85)
 - open/close 1433 (time 1786048944.81 total 20.02 last 70.34)
 - open/close 2146 (time 1786048954.83 total 30.03 last 71.19)
 - open/close 2856 (time 1786048964.83 total 40.04 last 70.95)
 - open/close 3585 (time 1786048974.84 total 50.05 last 72.82)
 - open/close 4380 (time 1786048984.86 total 60.07 last 79.34)
total: 4500 open/close in 61.85 seconds: 72.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1370 (time 1786049000.55 total 10.01 last 136.81)
 - /unlink 2751 (time 1786049010.55 total 20.02 last 138.09)
 - /unlink 4147 (time 1786049020.56 total 30.02 last 139.56)
total: 4500 /unlink in 33.13 seconds: 135.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 717 (time 1786049038.48 total 10.00 last 71.67)
 - open/close 1307 (time 1786049048.50 total 20.03 last 58.87)
 - open/close 1816 (time 1786049058.52 total 30.04 last 50.81)
 - open/close 2439 (time 1786049068.53 total 40.05 last 62.28)
 - open/close 3004 (time 1786049078.54 total 50.06 last 56.43)
 - open/close 3698 (time 1786049088.54 total 60.06 last 69.38)
 - open/close 4417 (time 1786049098.55 total 70.07 last 71.87)
total: 4500 open/close in 71.08 seconds: 63.31 ops/second
 - /unlink 1272 (time 1786049114.90 total 10.00 last 127.16)
 - /unlink 2600 (time 1786049124.90 total 20.00 last 132.80)
 - /unlink 3865 (time 1786049134.91 total 30.01 last 126.45)
total: 4500 /unlink in 34.77 seconds: 129.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 706 (time 1786049155.15 total 10.01 last 70.51)
 - open/close 1398 (time 1786049165.15 total 20.02 last 69.17)
 - open/close 2134 (time 1786049175.17 total 30.04 last 73.47)
 - open/close 2800 (time 1786049185.18 total 40.04 last 66.54)
 - open/close 3412 (time 1786049195.19 total 50.06 last 61.12)
 - open/close 4090 (time 1786049205.20 total 60.07 last 67.74)
total: 4500 open/close in 65.71 seconds: 68.48 ops/second
 - /unlink 1206 (time 1786049225.91 total 10.00 last 120.60)
 - /unlink 2417 (time 1786049235.92 total 20.01 last 121.03)
 - /unlink 3629 (time 1786049245.92 total 30.01 last 121.20)
total: 4500 /unlink in 38.13 seconds: 118.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 617 (time 1786049269.32 total 10.02 last 61.58)
 - open/close 1201 (time 1786049279.35 total 20.04 last 58.27)
 - open/close 1792 (time 1786049289.35 total 30.05 last 59.06)
 - open/close 2394 (time 1786049299.39 total 40.09 last 59.99)
 - open/close 3041 (time 1786049309.39 total 50.09 last 64.68)
 - open/close 3711 (time 1786049319.41 total 60.10 last 66.89)
 - open/close 4399 (time 1786049329.42 total 70.11 last 68.73)
total: 4500 open/close in 71.96 seconds: 62.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1257 (time 1786049346.58 total 10.00 last 125.70)
 - /unlink 2485 (time 1786049356.58 total 20.00 last 122.77)
 - /unlink 3785 (time 1786049366.59 total 30.01 last 129.85)
total: 4500 /unlink in 36.78 seconds: 122.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 682 (time 1786049388.75 total 10.02 last 68.04)
 - open/close 1359 (time 1786049398.76 total 20.03 last 67.67)
 - open/close 1976 (time 1786049408.77 total 30.04 last 61.63)
 - open/close 2554 (time 1786049418.80 total 40.07 last 57.65)
 - open/close 3149 (time 1786049428.80 total 50.07 last 59.48)
 - open/close 3766 (time 1786049438.82 total 60.09 last 61.54)
 - open/close 4404 (time 1786049448.83 total 70.10 last 63.78)
total: 4500 open/close in 71.41 seconds: 63.02 ops/second
 - /unlink 1202 (time 1786049465.22 total 10.00 last 120.15)
 - /unlink 2477 (time 1786049475.22 total 20.01 last 127.48)
 - /unlink 3637 (time 1786049485.22 total 30.01 last 115.99)
total: 4500 /unlink in 37.92 seconds: 118.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 586 (time 1786049508.73 total 10.01 last 58.52)
 - open/close 1250 (time 1786049518.75 total 20.04 last 66.23)
 - open/close 1878 (time 1786049528.77 total 30.05 last 62.71)
 - open/close 2581 (time 1786049538.77 total 40.06 last 70.28)
 - open/close 3279 (time 1786049548.78 total 50.06 last 69.77)
 - open/close 4003 (time 1786049558.79 total 60.07 last 72.32)
total: 4500 open/close in 67.31 seconds: 66.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1116 (time 1786049580.60 total 10.02 last 111.37)
 - /unlink 2237 (time 1786049590.61 total 20.03 last 112.04)
 - /unlink 3292 (time 1786049600.61 total 30.03 last 105.46)
 - /unlink 4438 (time 1786049610.61 total 40.03 last 114.54)
total: 4500 /unlink in 40.75 seconds: 110.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 570 (time 1786049626.06 total 10.01 last 56.96)
 - open/close 1159 (time 1786049636.07 total 20.01 last 58.87)
 - open/close 1775 (time 1786049646.08 total 30.02 last 61.54)
 - open/close 2372 (time 1786049656.08 total 40.03 last 59.67)
 - open/close 3030 (time 1786049666.09 total 50.03 last 65.77)
 - open/close 3772 (time 1786049676.09 total 60.04 last 74.15)
 - open/close 4481 (time 1786049686.12 total 70.06 last 70.73)
total: 4500 open/close in 70.46 seconds: 63.86 ops/second
 - /unlink 1239 (time 1786049701.62 total 10.00 last 123.90)
 - /unlink 2432 (time 1786049711.63 total 20.01 last 119.24)
 - /unlink 3719 (time 1786049721.63 total 30.01 last 128.68)
total: 4500 /unlink in 36.97 seconds: 121.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 662 (time 1786049743.99 total 10.00 last 66.17)
 - open/close 1256 (time 1786049753.99 total 20.01 last 59.38)
 - open/close 1859 (time 1786049764.00 total 30.02 last 60.24)
 - open/close 2512 (time 1786049774.01 total 40.03 last 65.25)
 - open/close 3176 (time 1786049784.02 total 50.04 last 66.33)
 - open/close 3831 (time 1786049794.03 total 60.04 last 65.45)
total: 4500 open/close in 69.95 seconds: 64.33 ops/second
 - /unlink 1117 (time 1786049819.01 total 10.00 last 111.67)
 - /unlink 2146 (time 1786049829.02 total 20.01 last 102.84)
 - /unlink 3204 (time 1786049839.02 total 30.02 last 105.73)
 - /unlink 4288 (time 1786049849.02 total 40.02 last 108.39)
total: 4500 /unlink in 41.97 seconds: 107.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 622 (time 1786049866.78 total 10.01 last 62.14)
 - open/close 1252 (time 1786049876.78 total 20.01 last 62.98)
 - open/close 1938 (time 1786049886.79 total 30.02 last 68.53)
 - open/close 2559 (time 1786049896.80 total 40.03 last 62.04)
 - open/close 3254 (time 1786049906.81 total 50.04 last 69.46)
 - open/close 4047 (time 1786049916.82 total 60.05 last 79.23)
total: 4500 open/close in 66.84 seconds: 67.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1186 (time 1786049939.05 total 10.01 last 118.52)
 - /unlink 2250 (time 1786049949.05 total 20.01 last 106.38)
 - /unlink 3421 (time 1786049959.06 total 30.02 last 117.01)
 - /unlink 4394 (time 1786049969.06 total 40.02 last 97.29)
total: 4500 /unlink in 41.07 seconds: 109.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 476 (time 1786049986.80 total 10.05 last 47.34)
 - open/close 983 (time 1786049996.80 total 20.05 last 50.70)
 - open/close 1554 (time 1786050006.82 total 30.07 last 56.99)
 - open/close 2185 (time 1786050016.82 total 40.08 last 63.09)
 - open/close 2840 (time 1786050026.83 total 50.08 last 65.48)
 - open/close 3512 (time 1786050036.83 total 60.08 last 67.19)
 - open/close 4165 (time 1786050046.84 total 70.09 last 65.24)
total: 4500 open/close in 74.85 seconds: 60.12 ops/second
 - /unlink 1210 (time 1786050067.14 total 10.01 last 120.91)
 - /unlink 2400 (time 1786050077.14 total 20.01 last 118.96)
 - /unlink 3631 (time 1786050087.15 total 30.02 last 123.01)
total: 4500 /unlink in 38.12 seconds: 118.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 556 (time 1786050112.95 total 10.01 last 55.57)
 - open/close 1275 (time 1786050122.96 total 20.01 last 71.84)
 - open/close 1900 (time 1786050132.96 total 30.02 last 62.48)
 - open/close 2547 (time 1786050143.00 total 40.05 last 64.48)
 - open/close 3145 (time 1786050153.00 total 50.06 last 59.77)
 - open/close 3695 (time 1786050163.01 total 60.06 last 54.99)
 - open/close 4264 (time 1786050173.01 total 70.06 last 56.87)
total: 4500 open/close in 74.07 seconds: 60.75 ops/second
 - /unlink 1094 (time 1786050192.01 total 10.02 last 109.13)
 - /unlink 2285 (time 1786050202.01 total 20.03 last 119.08)
 - /unlink 3561 (time 1786050212.02 total 30.03 last 127.49)
total: 4500 /unlink in 38.23 seconds: 117.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 604 (time 1786050236.90 total 10.00 last 60.38)
 - open/close 1207 (time 1786050246.91 total 20.01 last 60.27)
 - open/close 1880 (time 1786050256.91 total 30.01 last 67.28)
 - open/close 2551 (time 1786050266.93 total 40.03 last 66.95)
 - open/close 3220 (time 1786050276.93 total 50.04 last 66.88)
 - open/close 3903 (time 1786050286.94 total 60.04 last 68.30)
total: 4500 open/close in 69.75 seconds: 64.52 ops/second
 - /unlink 1209 (time 1786050311.27 total 10.00 last 120.89)
 - /unlink 2189 (time 1786050321.29 total 20.01 last 97.89)
 - /unlink 3103 (time 1786050331.29 total 30.01 last 91.40)
 - /unlink 3992 (time 1786050341.29 total 40.01 last 88.90)
total: 4500 /unlink in 43.87 seconds: 102.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 591 (time 1786050361.56 total 10.01 last 59.06)
 - open/close 1176 (time 1786050371.56 total 20.01 last 58.48)
 - open/close 1742 (time 1786050381.56 total 30.01 last 56.59)
 - open/close 2355 (time 1786050391.58 total 40.03 last 61.19)
 - open/close 2977 (time 1786050401.58 total 50.03 last 62.19)
 - open/close 3637 (time 1786050411.59 total 60.04 last 65.95)
 - open/close 4286 (time 1786050421.60 total 70.05 last 64.86)
total: 4500 open/close in 72.71 seconds: 61.89 ops/second
 - /unlink 1209 (time 1786050438.54 total 10.00 last 120.87)
 - /unlink 2452 (time 1786050448.55 total 20.01 last 124.21)
 - /unlink 3667 (time 1786050458.55 total 30.01 last 121.49)
total: 4500 /unlink in 38.81 seconds: 115.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 676 (time 1786050484.11 total 10.01 last 67.54)
 - open/close 1332 (time 1786050494.12 total 20.02 last 65.55)
 - open/close 2036 (time 1786050504.12 total 30.02 last 70.40)
 - open/close 2686 (time 1786050514.13 total 40.02 last 64.95)
 - open/close 3321 (time 1786050524.14 total 50.03 last 63.44)
 - open/close 3932 (time 1786050534.15 total 60.04 last 61.05)
total: 4500 open/close in 67.93 seconds: 66.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1156 (time 1786050557.11 total 10.00 last 115.59)
 - /unlink 2347 (time 1786050567.12 total 20.00 last 119.07)
 - /unlink 3642 (time 1786050577.13 total 30.02 last 129.32)
total: 4500 /unlink in 36.84 seconds: 122.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 647 (time 1786050599.30 total 10.01 last 64.63)
 - open/close 1312 (time 1786050609.30 total 20.01 last 66.48)
 - open/close 1855 (time 1786050619.31 total 30.01 last 54.30)
 - open/close 2443 (time 1786050629.33 total 40.04 last 58.64)
 - open/close 2993 (time 1786050639.35 total 50.05 last 54.93)
 - open/close 3579 (time 1786050649.35 total 60.06 last 58.58)
 - open/close 4232 (time 1786050659.35 total 70.06 last 65.27)
total: 4500 open/close in 74.51 seconds: 60.39 ops/second
 - /unlink 1197 (time 1786050677.91 total 10.00 last 119.69)
 - /unlink 2206 (time 1786050687.91 total 20.00 last 100.90)
 - /unlink 3437 (time 1786050697.92 total 30.01 last 122.97)
total: 4500 /unlink in 38.87 seconds: 115.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 700 (time 1786050722.62 total 10.00 last 69.98)
 - open/close 1384 (time 1786050732.62 total 20.01 last 68.34)
 - open/close 2059 (time 1786050742.62 total 30.01 last 67.49)
 - open/close 2750 (time 1786050752.63 total 40.02 last 69.06)
 - open/close 3448 (time 1786050762.63 total 50.02 last 69.78)
 - open/close 4170 (time 1786050772.64 total 60.03 last 72.14)
total: 4500 open/close in 65.21 seconds: 69.00 ops/second
 - /unlink 1388 (time 1786050792.45 total 10.00 last 138.77)
 - /unlink 2549 (time 1786050802.45 total 20.01 last 116.06)
 - /unlink 3910 (time 1786050812.46 total 30.01 last 136.00)
total: 4500 /unlink in 34.40 seconds: 130.80 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95231 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg602-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg602-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg602-server
stop mds service on oleg602-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg602-server
stop mds service on oleg602-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg602-server

unloading modules via unload_modules_local on: 'oleg602-server'
oleg602-server: oleg602-server.virtnet: executing unload_modules_local
oleg602-server: 
modules unloaded.