== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:22:54 (1787872974)
Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg417-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg417-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg417-server
oleg417-server: oleg417-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg417-server'
oleg417-server: oleg417-server.virtnet: executing load_modules_local
oleg417-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg417-server: Loading modules from /home/green/git/lustre-release/lustre
oleg417-server: detected 4 online CPUs by sysfs
oleg417-server: MODOPTS_LIBCFS=
oleg417-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg417-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all
pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg417-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all
pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg417-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all
pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg417-client.virtnet:  -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 559 (time 1787873216.03 total 10.02 last 55.80)
 - open/close 1092 (time 1787873226.05 total 20.04 last 53.17)
 - open/close 1606 (time 1787873236.06 total 30.05 last 51.37)
 - open/close 2192 (time 1787873246.07 total 40.06 last 58.55)
 - open/close 2692 (time 1787873256.07 total 50.06 last 49.97)
 - open/close 3270 (time 1787873266.08 total 60.07 last 57.76)
 - open/close 3816 (time 1787873276.08 total 70.07 last 54.57)
 - open/close 4333 (time 1787873286.10 total 80.09 last 51.60)
total: 4500 open/close in 83.66 seconds: 53.79 ops/second
 - /unlink 1004 (time 1787873306.38 total 10.00 last 100.38)
 - /unlink 2020 (time 1787873316.39 total 20.01 last 101.51)
 - /unlink 2969 (time 1787873326.40 total 30.02 last 94.80)
 - /unlink 4103 (time 1787873336.40 total 40.02 last 113.39)
total: 4500 /unlink in 44.27 seconds: 101.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 540 (time 1787873356.58 total 10.02 last 53.91)
 - open/close 994 (time 1787873366.60 total 20.03 last 45.34)
 - open/close 1470 (time 1787873376.61 total 30.05 last 47.51)
 - open/close 2076 (time 1787873386.64 total 40.07 last 60.47)
 - open/close 2685 (time 1787873396.64 total 50.08 last 60.86)
 - open/close 3262 (time 1787873406.67 total 60.10 last 57.56)
 - open/close 3856 (time 1787873416.67 total 70.11 last 59.37)
 - open/close 4434 (time 1787873426.68 total 80.11 last 57.77)
total: 4500 open/close in 81.19 seconds: 55.42 ops/second
 - /unlink 1081 (time 1787873443.73 total 10.00 last 108.08)
 - /unlink 2168 (time 1787873453.74 total 20.00 last 108.67)
 - /unlink 3265 (time 1787873463.74 total 30.01 last 109.63)
 - /unlink 4333 (time 1787873473.76 total 40.02 last 106.65)
total: 4500 /unlink in 41.51 seconds: 108.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 626 (time 1787873491.48 total 10.02 last 62.48)
 - open/close 1178 (time 1787873501.49 total 20.03 last 55.14)
 - open/close 1712 (time 1787873511.51 total 30.04 last 53.33)
 - open/close 2377 (time 1787873521.51 total 40.04 last 66.49)
 - open/close 2882 (time 1787873531.51 total 50.05 last 50.49)
 - open/close 3386 (time 1787873541.51 total 60.05 last 50.37)
 - open/close 3912 (time 1787873551.52 total 70.06 last 52.56)
 - open/close 4489 (time 1787873561.53 total 80.07 last 57.63)
total: 4500 open/close in 80.21 seconds: 56.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1082 (time 1787873576.80 total 10.01 last 108.11)
 - /unlink 2089 (time 1787873586.81 total 20.01 last 100.64)
 - /unlink 3037 (time 1787873596.81 total 30.02 last 94.75)
 - /unlink 3923 (time 1787873606.81 total 40.02 last 88.58)
total: 4500 /unlink in 46.65 seconds: 96.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 536 (time 1787873629.87 total 10.01 last 53.56)
 - open/close 1062 (time 1787873639.89 total 20.03 last 52.49)
 - open/close 1604 (time 1787873649.89 total 30.03 last 54.19)
 - open/close 2166 (time 1787873659.89 total 40.03 last 56.18)
 - open/close 2776 (time 1787873669.94 total 50.07 last 60.75)
 - open/close 3279 (time 1787873679.94 total 60.08 last 50.28)
 - open/close 3839 (time 1787873689.95 total 70.09 last 55.96)
 - open/close 4485 (time 1787873699.95 total 80.09 last 64.60)
total: 4500 open/close in 80.38 seconds: 55.98 ops/second
 - /unlink 1098 (time 1787873715.12 total 10.00 last 109.80)
 - /unlink 2258 (time 1787873725.13 total 20.00 last 115.94)
 - /unlink 3187 (time 1787873735.13 total 30.01 last 92.89)
 - /unlink 4196 (time 1787873745.13 total 40.01 last 100.85)
total: 4500 /unlink in 42.63 seconds: 105.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 637 (time 1787873763.37 total 10.00 last 63.69)
 - open/close 1202 (time 1787873773.37 total 20.01 last 56.48)
 - open/close 1839 (time 1787873783.39 total 30.02 last 63.62)
 - open/close 2489 (time 1787873793.40 total 40.03 last 64.91)
 - open/close 3091 (time 1787873803.42 total 50.05 last 60.07)
 - open/close 3717 (time 1787873813.43 total 60.06 last 62.57)
 - open/close 4410 (time 1787873823.43 total 70.06 last 69.29)
total: 4500 open/close in 71.42 seconds: 63.01 ops/second
 - /unlink 1171 (time 1787873839.89 total 10.00 last 117.10)
 - /unlink 2234 (time 1787873849.90 total 20.01 last 106.20)
 - /unlink 3087 (time 1787873859.94 total 30.05 last 84.95)
 - /unlink 4160 (time 1787873869.95 total 40.06 last 107.24)
total: 4500 /unlink in 43.30 seconds: 103.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 643 (time 1787873888.68 total 10.01 last 64.23)
 - open/close 1268 (time 1787873898.68 total 20.01 last 62.49)
 - open/close 1848 (time 1787873908.70 total 30.02 last 57.94)
 - open/close 2469 (time 1787873918.71 total 40.04 last 62.02)
 - open/close 3071 (time 1787873928.72 total 50.05 last 60.12)
 - open/close 3683 (time 1787873938.73 total 60.06 last 61.15)
 - open/close 4307 (time 1787873948.73 total 70.06 last 62.38)
total: 4500 open/close in 73.42 seconds: 61.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1260 (time 1787873966.75 total 10.00 last 125.95)
 - /unlink 2497 (time 1787873976.75 total 20.01 last 123.67)
 - /unlink 3773 (time 1787873986.76 total 30.01 last 127.57)
total: 4500 /unlink in 35.92 seconds: 125.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 672 (time 1787874007.23 total 10.01 last 67.15)
 - open/close 1317 (time 1787874017.24 total 20.02 last 64.44)
 - open/close 1973 (time 1787874027.24 total 30.02 last 65.57)
 - open/close 2528 (time 1787874037.25 total 40.03 last 55.47)
 - open/close 3089 (time 1787874047.25 total 50.03 last 56.10)
 - open/close 3803 (time 1787874057.25 total 60.03 last 71.38)
 - open/close 4450 (time 1787874067.27 total 70.04 last 64.60)
total: 4500 open/close in 70.90 seconds: 63.47 ops/second
 - /unlink 1200 (time 1787874083.18 total 10.01 last 119.93)
 - /unlink 2335 (time 1787874093.19 total 20.01 last 113.46)
 - /unlink 3586 (time 1787874103.20 total 30.02 last 124.95)
total: 4500 /unlink in 37.39 seconds: 120.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 653 (time 1787874126.16 total 10.03 last 65.11)
 - open/close 1249 (time 1787874136.18 total 20.04 last 59.53)
 - open/close 1858 (time 1787874146.18 total 30.05 last 60.85)
 - open/close 2450 (time 1787874156.19 total 40.06 last 59.15)
 - open/close 3042 (time 1787874166.19 total 50.06 last 59.19)
 - open/close 3695 (time 1787874176.20 total 60.06 last 65.26)
 - open/close 4402 (time 1787874186.21 total 70.08 last 70.60)
total: 4500 open/close in 71.54 seconds: 62.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1149 (time 1787874202.23 total 10.02 last 114.73)
 - /unlink 2362 (time 1787874212.23 total 20.02 last 121.30)
 - /unlink 3511 (time 1787874222.23 total 30.02 last 114.89)
total: 4500 /unlink in 39.45 seconds: 114.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 664 (time 1787874246.53 total 10.01 last 66.31)
 - open/close 1299 (time 1787874256.53 total 20.02 last 63.47)
 - open/close 1934 (time 1787874266.54 total 30.03 last 63.44)
 - open/close 2548 (time 1787874276.56 total 40.05 last 61.27)
 - open/close 3119 (time 1787874286.58 total 50.06 last 57.02)
 - open/close 3776 (time 1787874296.60 total 60.08 last 65.56)
 - open/close 4373 (time 1787874306.60 total 70.09 last 59.67)
total: 4500 open/close in 71.91 seconds: 62.58 ops/second
 - /unlink 1285 (time 1787874323.66 total 10.01 last 128.41)
 - /unlink 2464 (time 1787874333.67 total 20.01 last 117.82)
 - /unlink 3652 (time 1787874343.68 total 30.02 last 118.70)
total: 4500 /unlink in 37.49 seconds: 120.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 509 (time 1787874366.39 total 10.04 last 50.68)
 - open/close 1047 (time 1787874376.39 total 20.05 last 53.75)
 - open/close 1577 (time 1787874386.41 total 30.07 last 52.91)
 - open/close 2107 (time 1787874396.42 total 40.08 last 52.94)
 - open/close 2611 (time 1787874406.44 total 50.09 last 50.34)
 - open/close 3241 (time 1787874416.44 total 60.10 last 62.96)
 - open/close 3796 (time 1787874426.44 total 70.10 last 55.49)
 - open/close 4400 (time 1787874436.45 total 80.11 last 60.36)
total: 4500 open/close in 81.52 seconds: 55.20 ops/second
 - /unlink 1026 (time 1787874453.36 total 10.00 last 102.58)
 - /unlink 2104 (time 1787874463.36 total 20.00 last 107.79)
 - /unlink 3214 (time 1787874473.36 total 30.01 last 110.94)
 - /unlink 4445 (time 1787874483.37 total 40.02 last 122.96)
total: 4500 /unlink in 40.60 seconds: 110.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 651 (time 1787874498.52 total 10.01 last 65.07)
 - open/close 1255 (time 1787874508.53 total 20.01 last 60.35)
 - open/close 1905 (time 1787874518.53 total 30.02 last 64.98)
 - open/close 2557 (time 1787874528.54 total 40.02 last 65.17)
 - open/close 3190 (time 1787874538.54 total 50.03 last 63.25)
 - open/close 3855 (time 1787874548.56 total 60.04 last 66.41)
total: 4500 open/close in 69.98 seconds: 64.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1307 (time 1787874573.81 total 10.01 last 130.61)
 - /unlink 2485 (time 1787874583.81 total 20.01 last 117.79)
 - /unlink 3719 (time 1787874593.82 total 30.01 last 123.32)
total: 4500 /unlink in 36.73 seconds: 122.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 650 (time 1787874615.89 total 10.01 last 64.91)
 - open/close 1316 (time 1787874625.91 total 20.03 last 66.47)
 - open/close 1966 (time 1787874635.92 total 30.04 last 64.97)
 - open/close 2632 (time 1787874645.93 total 40.05 last 66.50)
 - open/close 3251 (time 1787874655.94 total 50.06 last 61.83)
 - open/close 3832 (time 1787874665.96 total 60.08 last 58.03)
total: 4500 open/close in 69.82 seconds: 64.46 ops/second
 - /unlink 1271 (time 1787874691.40 total 10.00 last 127.08)
 - /unlink 2564 (time 1787874701.40 total 20.00 last 129.26)
 - /unlink 3693 (time 1787874711.40 total 30.01 last 112.87)
total: 4500 /unlink in 36.39 seconds: 123.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 530 (time 1787874733.25 total 10.00 last 52.99)
 - open/close 1107 (time 1787874743.25 total 20.01 last 57.68)
 - open/close 1699 (time 1787874753.26 total 30.02 last 59.12)
 - open/close 2305 (time 1787874763.28 total 40.04 last 60.48)
 - open/close 2900 (time 1787874773.31 total 50.06 last 59.36)
 - open/close 3505 (time 1787874783.32 total 60.08 last 60.42)
 - open/close 4042 (time 1787874793.33 total 70.08 last 53.67)
total: 4500 open/close in 79.49 seconds: 56.61 ops/second
 - /unlink 1203 (time 1787874818.85 total 10.00 last 120.26)
 - /unlink 2206 (time 1787874828.85 total 20.01 last 100.27)
 - /unlink 3448 (time 1787874838.85 total 30.01 last 124.16)
total: 4500 /unlink in 39.46 seconds: 114.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 661 (time 1787874863.83 total 10.00 last 66.07)
 - open/close 1257 (time 1787874873.83 total 20.01 last 59.59)
 - open/close 1874 (time 1787874883.84 total 30.01 last 61.65)
 - open/close 2536 (time 1787874893.85 total 40.03 last 66.12)
 - open/close 3176 (time 1787874903.87 total 50.04 last 63.91)
 - open/close 3903 (time 1787874913.88 total 60.06 last 72.61)
total: 4500 open/close in 69.40 seconds: 64.84 ops/second
 - /unlink 1225 (time 1787874938.11 total 10.00 last 122.45)
 - /unlink 2451 (time 1787874948.11 total 20.00 last 122.60)
 - /unlink 3773 (time 1787874958.11 total 30.01 last 132.12)
total: 4500 /unlink in 35.73 seconds: 125.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 606 (time 1787874978.11 total 10.00 last 60.58)
 - open/close 1272 (time 1787874988.11 total 20.01 last 66.56)
 - open/close 1921 (time 1787874998.12 total 30.02 last 64.82)
 - open/close 2581 (time 1787875008.14 total 40.04 last 65.87)
 - open/close 3171 (time 1787875018.15 total 50.05 last 58.96)
 - open/close 3855 (time 1787875028.17 total 60.07 last 68.26)
total: 4500 open/close in 70.05 seconds: 64.24 ops/second
 - /unlink 1198 (time 1787875052.50 total 10.00 last 119.74)
 - /unlink 2477 (time 1787875062.51 total 20.01 last 127.85)
 - /unlink 3829 (time 1787875072.52 total 30.02 last 135.05)
total: 4500 /unlink in 35.43 seconds: 127.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 644 (time 1787875092.81 total 10.01 last 64.32)
 - open/close 1272 (time 1787875102.81 total 20.02 last 62.78)
 - open/close 1824 (time 1787875112.82 total 30.02 last 55.17)
 - open/close 2428 (time 1787875122.84 total 40.04 last 60.28)
 - open/close 3083 (time 1787875132.85 total 50.05 last 65.45)
 - open/close 3765 (time 1787875142.86 total 60.06 last 68.10)
 - open/close 4493 (time 1787875152.87 total 70.07 last 72.76)
total: 4500 open/close in 70.19 seconds: 64.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1307 (time 1787875167.72 total 10.00 last 130.68)
 - /unlink 2696 (time 1787875177.72 total 20.00 last 138.86)
 - /unlink 3901 (time 1787875187.72 total 30.01 last 120.48)
total: 4500 /unlink in 34.26 seconds: 131.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 661 (time 1787875206.88 total 10.01 last 66.05)
 - open/close 1279 (time 1787875216.89 total 20.02 last 61.74)
 - open/close 1920 (time 1787875226.95 total 30.08 last 63.71)
 - open/close 2549 (time 1787875236.96 total 40.08 last 62.85)
 - open/close 3331 (time 1787875246.96 total 50.09 last 78.19)
 - open/close 3878 (time 1787875256.97 total 60.09 last 54.67)
 - open/close 4175 (time 1787875267.07 total 70.19 last 29.40)
 - open/close 4426 (time 1787875277.08 total 80.21 last 25.06)
total: 4500 open/close in 83.70 seconds: 53.76 ops/second
 conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error()
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11712:test_135()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7932:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7995:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7794:run_test()
  = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11718:main()
Dumping lctl log to /tmp/testlogs//conf-sanity.test_135.*.1787875296.log
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11382: 95393 Terminated              coproc COPROC $LFS changelog --follow $service
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11382: 95397 Terminated              __test_135_file_thread "$service"
rsync: chown "/tmp/testlogs/.conf-sanity.test_135.debug_log.oleg417-server.1787875296.log.wPULqK" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.conf-sanity.test_135.dmesg.oleg417-server.1787875296.log.FfNPFr" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg417-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg417-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server
stop mds service on oleg417-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server
stop mds service on oleg417-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg417-server

unloading modules via unload_modules_local on: 'oleg417-server'
oleg417-server: oleg417-server.virtnet: executing unload_modules_local
oleg417-server: 
modules unloaded.