== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:43:09 (1775767389)
Stopping clients: oleg614-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg614-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg614-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg614-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg614-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg614-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg614-server
oleg614-server: oleg614-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg614-server'
oleg614-server: oleg614-server.virtnet: executing load_modules_local
oleg614-server: Loading modules from /home/green/git/lustre-release/lustre
oleg614-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg614-server: detected 4 online CPUs by sysfs
oleg614-server: MODOPTS_LIBCFS=
oleg614-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg614-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all
pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg614-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all
pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg614-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all
pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg614-client.virtnet:  -o user_xattr,flock 192.168.206.114@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 650 (time 1775767597.28 total 10.02 last 64.86)
 - open/close 1241 (time 1775767607.28 total 20.02 last 59.09)
 - open/close 1804 (time 1775767617.30 total 30.04 last 56.19)
 - open/close 2358 (time 1775767627.32 total 40.05 last 55.34)
 - open/close 2936 (time 1775767637.32 total 50.06 last 57.74)
 - open/close 3464 (time 1775767647.33 total 60.07 last 52.79)
 - open/close 4026 (time 1775767657.33 total 70.07 last 56.16)
total: 4500 open/close in 78.76 seconds: 57.13 ops/second
 - /unlink 926 (time 1775767681.06 total 10.01 last 92.50)
 - /unlink 1929 (time 1775767691.07 total 20.02 last 100.17)
 - /unlink 3020 (time 1775767701.08 total 30.03 last 109.01)
 - /unlink 4086 (time 1775767711.09 total 40.04 last 106.46)
total: 4500 /unlink in 43.66 seconds: 103.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 460 (time 1775767730.40 total 10.02 last 45.92)
 - open/close 973 (time 1775767740.42 total 20.03 last 51.21)
 - open/close 1542 (time 1775767750.43 total 30.05 last 56.83)
 - open/close 2179 (time 1775767760.45 total 40.07 last 63.56)
 - open/close 2735 (time 1775767770.48 total 50.09 last 55.47)
 - open/close 3361 (time 1775767780.48 total 60.10 last 62.55)
 - open/close 3938 (time 1775767790.49 total 70.11 last 57.64)
 - open/close 4483 (time 1775767800.50 total 80.11 last 54.48)
total: 4500 open/close in 80.33 seconds: 56.02 ops/second
 - /unlink 973 (time 1775767816.89 total 10.01 last 97.24)
 - /unlink 1960 (time 1775767826.91 total 20.02 last 98.53)
 - /unlink 2983 (time 1775767836.91 total 30.02 last 102.29)
 - /unlink 3922 (time 1775767846.92 total 40.04 last 93.79)
total: 4500 /unlink in 46.31 seconds: 97.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 571 (time 1775767868.55 total 10.03 last 56.92)
 - open/close 1110 (time 1775767878.56 total 20.04 last 53.84)
 - open/close 1529 (time 1775767888.57 total 30.05 last 41.86)
 - open/close 2137 (time 1775767898.57 total 40.06 last 60.79)
 - open/close 2626 (time 1775767908.59 total 50.08 last 48.79)
 - open/close 3129 (time 1775767918.60 total 60.09 last 50.25)
 - open/close 3645 (time 1775767928.61 total 70.10 last 51.56)
 - open/close 4236 (time 1775767938.62 total 80.11 last 59.03)
total: 4500 open/close in 84.18 seconds: 53.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1079 (time 1775767957.65 total 10.00 last 107.88)
 - /unlink 2130 (time 1775767967.65 total 20.01 last 105.05)
 - /unlink 3175 (time 1775767977.67 total 30.03 last 104.26)
 - /unlink 4163 (time 1775767987.67 total 40.03 last 98.78)
total: 4500 /unlink in 43.24 seconds: 104.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 598 (time 1775768006.85 total 10.02 last 59.70)
 - open/close 1220 (time 1775768016.86 total 20.02 last 62.16)
 - open/close 1783 (time 1775768026.87 total 30.03 last 56.27)
 - open/close 2345 (time 1775768036.89 total 40.05 last 56.10)
 - open/close 2912 (time 1775768046.89 total 50.05 last 56.67)
 - open/close 3515 (time 1775768056.90 total 60.06 last 60.25)
 - open/close 4128 (time 1775768066.90 total 70.07 last 61.27)
total: 4500 open/close in 76.18 seconds: 59.07 ops/second
 - /unlink 1127 (time 1775768088.23 total 10.00 last 112.68)
 - /unlink 2201 (time 1775768098.24 total 20.01 last 107.35)
 - /unlink 3234 (time 1775768108.24 total 30.01 last 103.27)
 - /unlink 4310 (time 1775768118.24 total 40.01 last 107.56)
total: 4500 /unlink in 42.11 seconds: 106.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 548 (time 1775768135.92 total 10.00 last 54.79)
 - open/close 1130 (time 1775768145.92 total 20.00 last 58.19)
 - open/close 1616 (time 1775768155.93 total 30.01 last 48.56)
 - open/close 2124 (time 1775768165.94 total 40.02 last 50.76)
 - open/close 2680 (time 1775768175.95 total 50.03 last 55.54)
 - open/close 3279 (time 1775768185.96 total 60.04 last 59.84)
 - open/close 3894 (time 1775768195.96 total 70.04 last 61.50)
 - open/close 4394 (time 1775768205.96 total 80.04 last 49.99)
total: 4500 open/close in 81.75 seconds: 55.04 ops/second
 - /unlink 900 (time 1775768222.89 total 10.01 last 89.88)
 - /unlink 2007 (time 1775768232.90 total 20.02 last 110.63)
 - /unlink 2986 (time 1775768242.90 total 30.03 last 97.83)
 - /unlink 4062 (time 1775768252.92 total 40.04 last 107.46)
total: 4500 /unlink in 44.18 seconds: 101.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 544 (time 1775768272.13 total 10.01 last 54.36)
 - open/close 1057 (time 1775768282.14 total 20.01 last 51.27)
 - open/close 1584 (time 1775768292.15 total 30.03 last 52.63)
 - open/close 2139 (time 1775768302.17 total 40.04 last 55.41)
 - open/close 2680 (time 1775768312.18 total 50.05 last 54.04)
 - open/close 3283 (time 1775768322.19 total 60.06 last 60.25)
 - open/close 3734 (time 1775768332.20 total 70.07 last 45.05)
 - open/close 4289 (time 1775768342.20 total 80.07 last 55.50)
total: 4500 open/close in 83.81 seconds: 53.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1068 (time 1775768360.94 total 10.01 last 106.68)
 - /unlink 1976 (time 1775768370.95 total 20.01 last 90.78)
 - /unlink 3043 (time 1775768380.95 total 30.01 last 106.69)
 - /unlink 4008 (time 1775768390.95 total 40.02 last 96.49)
total: 4500 /unlink in 44.94 seconds: 100.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 458 (time 1775768411.59 total 10.02 last 45.72)
 - open/close 993 (time 1775768421.60 total 20.03 last 53.44)
 - open/close 1611 (time 1775768431.61 total 30.04 last 61.75)
 - open/close 2210 (time 1775768441.63 total 40.05 last 59.81)
 - open/close 2823 (time 1775768451.64 total 50.07 last 61.22)
 - open/close 3385 (time 1775768461.65 total 60.07 last 56.16)
 - open/close 3982 (time 1775768471.67 total 70.10 last 59.55)
total: 4500 open/close in 78.63 seconds: 57.23 ops/second
 - /unlink 1433 (time 1775768494.90 total 10.01 last 143.23)
 - /unlink 2684 (time 1775768504.90 total 20.01 last 125.10)
 - /unlink 4081 (time 1775768514.90 total 30.01 last 139.64)
total: 4500 /unlink in 33.14 seconds: 135.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 661 (time 1775768531.52 total 10.00 last 66.09)
 - open/close 1607 (time 1775768541.52 total 20.00 last 94.57)
 - open/close 2434 (time 1775768551.52 total 30.01 last 82.67)
 - open/close 3274 (time 1775768561.52 total 40.01 last 83.99)
 - open/close 4048 (time 1775768571.53 total 50.02 last 77.32)
total: 4500 open/close in 57.56 seconds: 78.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1579 (time 1775768594.23 total 10.00 last 157.88)
 - /unlink 2755 (time 1775768604.23 total 20.00 last 117.56)
 - /unlink 3748 (time 1775768614.23 total 30.01 last 99.26)
total: 4500 /unlink in 36.88 seconds: 122.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 662 (time 1775768637.30 total 10.02 last 66.09)
 - open/close 1358 (time 1775768647.31 total 20.02 last 69.57)
 - open/close 1962 (time 1775768657.32 total 30.03 last 60.33)
 - open/close 2556 (time 1775768667.33 total 40.04 last 59.35)
 - open/close 3148 (time 1775768677.33 total 50.05 last 59.17)
 - open/close 3685 (time 1775768687.34 total 60.05 last 53.66)
 - open/close 4225 (time 1775768697.36 total 70.08 last 53.88)
total: 4500 open/close in 75.19 seconds: 59.85 ops/second
 - /unlink 1013 (time 1775768718.75 total 10.01 last 101.17)
 - /unlink 1975 (time 1775768728.76 total 20.02 last 96.16)
 - /unlink 2987 (time 1775768738.76 total 30.02 last 101.14)
 - /unlink 3958 (time 1775768748.76 total 40.02 last 97.10)
total: 4500 /unlink in 46.67 seconds: 96.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 557 (time 1775768771.15 total 10.01 last 55.65)
 - open/close 1160 (time 1775768781.15 total 20.01 last 60.29)
 - open/close 1714 (time 1775768791.17 total 30.03 last 55.29)
 - open/close 2356 (time 1775768801.18 total 40.04 last 64.13)
 - open/close 2945 (time 1775768811.19 total 50.05 last 58.86)
 - open/close 3541 (time 1775768821.20 total 60.06 last 59.52)
 - open/close 4128 (time 1775768831.21 total 70.07 last 58.64)
total: 4500 open/close in 76.29 seconds: 58.98 ops/second
 - /unlink 1073 (time 1775768853.35 total 10.00 last 107.26)
 - /unlink 2080 (time 1775768863.35 total 20.00 last 100.69)
 - /unlink 3094 (time 1775768873.36 total 30.01 last 101.34)
 - /unlink 4148 (time 1775768883.37 total 40.02 last 105.28)
total: 4500 /unlink in 43.36 seconds: 103.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 621 (time 1775768901.68 total 10.01 last 62.04)
 - open/close 1260 (time 1775768911.69 total 20.01 last 63.87)
 - open/close 1827 (time 1775768921.69 total 30.02 last 56.68)
 - open/close 2358 (time 1775768931.70 total 40.02 last 53.07)
 - open/close 2889 (time 1775768941.71 total 50.04 last 53.04)
 - open/close 3412 (time 1775768951.72 total 60.04 last 52.26)
 - open/close 3991 (time 1775768961.73 total 70.06 last 57.83)
total: 4500 open/close in 78.63 seconds: 57.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1044 (time 1775768984.92 total 10.00 last 104.38)
 - /unlink 2148 (time 1775768994.92 total 20.01 last 110.36)
 - /unlink 3326 (time 1775769004.92 total 30.01 last 117.80)
 - /unlink 4439 (time 1775769014.93 total 40.01 last 111.22)
total: 4500 /unlink in 40.67 seconds: 110.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 591 (time 1775769030.66 total 10.00 last 59.08)
 - open/close 1145 (time 1775769040.67 total 20.02 last 55.33)
 - open/close 1710 (time 1775769050.67 total 30.02 last 56.48)
 - open/close 2391 (time 1775769060.69 total 40.04 last 67.97)
 - open/close 3022 (time 1775769070.70 total 50.05 last 63.05)
 - open/close 3562 (time 1775769080.71 total 60.05 last 53.97)
 - open/close 4157 (time 1775769090.74 total 70.08 last 59.33)
total: 4500 open/close in 76.14 seconds: 59.10 ops/second
 - /unlink 1208 (time 1775769111.97 total 10.01 last 120.65)
 - /unlink 2203 (time 1775769121.97 total 20.01 last 99.49)
 - /unlink 3305 (time 1775769131.97 total 30.01 last 110.20)
 - /unlink 4401 (time 1775769141.99 total 40.03 last 109.44)
total: 4500 /unlink in 40.90 seconds: 110.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 577 (time 1775769157.70 total 10.01 last 57.64)
 - open/close 1194 (time 1775769167.71 total 20.02 last 61.64)
 - open/close 1764 (time 1775769177.73 total 30.04 last 56.91)
 - open/close 2412 (time 1775769187.74 total 40.04 last 64.76)
 - open/close 3029 (time 1775769197.74 total 50.05 last 61.65)
 - open/close 3627 (time 1775769207.74 total 60.05 last 59.80)
 - open/close 4205 (time 1775769217.75 total 70.05 last 57.79)
total: 4500 open/close in 76.51 seconds: 58.82 ops/second
 - /unlink 996 (time 1775769239.47 total 10.00 last 99.58)
 - /unlink 2139 (time 1775769249.47 total 20.01 last 114.25)
 - /unlink 3219 (time 1775769259.48 total 30.02 last 107.87)
 - /unlink 4247 (time 1775769269.50 total 40.04 last 102.61)
total: 4500 /unlink in 42.54 seconds: 105.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 652 (time 1775769286.31 total 10.02 last 65.10)
 - open/close 1261 (time 1775769296.31 total 20.02 last 60.89)
 - open/close 1858 (time 1775769306.32 total 30.03 last 59.64)
 - open/close 2462 (time 1775769316.34 total 40.04 last 60.29)
 - open/close 3126 (time 1775769326.35 total 50.06 last 66.30)
 - open/close 3778 (time 1775769336.36 total 60.07 last 65.16)
 - open/close 4431 (time 1775769346.38 total 70.08 last 65.21)
total: 4500 open/close in 71.26 seconds: 63.15 ops/second
 - /unlink 1095 (time 1775769362.19 total 10.00 last 109.45)
 - /unlink 2373 (time 1775769372.19 total 20.01 last 127.78)
 - /unlink 3486 (time 1775769382.20 total 30.01 last 111.23)
total: 4500 /unlink in 38.81 seconds: 115.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 699 (time 1775769405.73 total 10.00 last 69.90)
 - open/close 1382 (time 1775769415.73 total 20.00 last 68.28)
 - open/close 2037 (time 1775769425.73 total 30.01 last 65.49)
 - open/close 2702 (time 1775769435.74 total 40.01 last 66.46)
 - open/close 3355 (time 1775769445.75 total 50.02 last 65.23)
 - open/close 3964 (time 1775769455.76 total 60.04 last 60.82)
total: 4500 open/close in 68.88 seconds: 65.33 ops/second
 - /unlink 1093 (time 1775769478.48 total 10.01 last 109.18)
 - /unlink 2222 (time 1775769488.48 total 20.01 last 112.89)
 - /unlink 3487 (time 1775769498.50 total 30.03 last 126.25)
total: 4500 /unlink in 38.22 seconds: 117.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 692 (time 1775769520.38 total 10.01 last 69.10)
 - open/close 1329 (time 1775769530.39 total 20.02 last 63.64)
 - open/close 1953 (time 1775769540.40 total 30.04 last 62.32)
 - open/close 2615 (time 1775769550.42 total 40.05 last 66.09)
 - open/close 3275 (time 1775769560.43 total 50.06 last 65.97)
 - open/close 3953 (time 1775769570.44 total 60.07 last 67.73)
total: 4500 open/close in 68.82 seconds: 65.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1170 (time 1775769594.10 total 10.00 last 116.99)
 - /unlink 2259 (time 1775769604.11 total 20.00 last 108.89)
 - /unlink 3487 (time 1775769614.11 total 30.01 last 122.74)
total: 4500 /unlink in 39.13 seconds: 115.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 582 (time 1775769638.15 total 10.04 last 57.96)
 - open/close 1179 (time 1775769648.16 total 20.04 last 59.69)
 - open/close 1775 (time 1775769658.16 total 30.05 last 59.58)
 - open/close 2350 (time 1775769668.17 total 40.05 last 57.45)
 - open/close 2966 (time 1775769678.18 total 50.06 last 61.55)
 - open/close 3565 (time 1775769688.19 total 60.08 last 59.80)
 - open/close 4254 (time 1775769698.19 total 70.08 last 68.89)
total: 4500 open/close in 73.79 seconds: 60.98 ops/second
 - /unlink 1194 (time 1775769715.96 total 10.02 last 119.20)
 - /unlink 2225 (time 1775769725.97 total 20.03 last 102.96)
 - /unlink 3179 (time 1775769735.98 total 30.04 last 95.29)
 - /unlink 4237 (time 1775769745.99 total 40.05 last 105.68)
total: 4500 /unlink in 42.60 seconds: 105.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 607 (time 1775769763.72 total 10.01 last 60.65)
 - open/close 1209 (time 1775769773.74 total 20.02 last 60.11)
 - open/close 1834 (time 1775769783.74 total 30.03 last 62.47)
 - open/close 2553 (time 1775769793.75 total 40.03 last 71.87)
 - open/close 3099 (time 1775769803.76 total 50.05 last 54.51)
 - open/close 3527 (time 1775769813.77 total 60.06 last 42.76)
 - open/close 3937 (time 1775769823.77 total 70.06 last 41.00)
 - open/close 4454 (time 1775769833.78 total 80.06 last 51.70)
total: 4500 open/close in 81.05 seconds: 55.52 ops/second
 - /unlink 854 (time 1775769850.29 total 10.01 last 85.32)
 - /unlink 1955 (time 1775769860.29 total 20.01 last 110.09)
 - /unlink 3167 (time 1775769870.29 total 30.01 last 121.18)
 - /unlink 4313 (time 1775769880.30 total 40.02 last 114.55)
total: 4500 /unlink in 41.78 seconds: 107.71 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11184: 94233 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg614-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg614-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg614-server
stop mds service on oleg614-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg614-server
stop mds service on oleg614-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg614-server

unloading modules via unload_modules_local on: 'oleg614-server'
oleg614-server: oleg614-server.virtnet: executing unload_modules_local
oleg614-server: 
modules unloaded.