== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:03:11 (1768550591)
Stopping clients: oleg258-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg258-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg258-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg258-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg258-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg258-server
oleg258-server: oleg258-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg258-server'
oleg258-server: oleg258-server.virtnet: executing load_modules_local
oleg258-server: Loading modules from /home/green/git/lustre-release/lustre
oleg258-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg258-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg258-server: detected 4 online CPUs by sysfs
oleg258-server: MODOPTS_LIBCFS=
oleg258-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg258-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all
pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg258-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all
pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg258-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all
pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg258-client.virtnet:  -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 634 (time 1768550774.01 total 10.00 last 63.38)
 - open/close 1373 (time 1768550784.02 total 20.01 last 73.85)
 - open/close 2100 (time 1768550794.02 total 30.01 last 72.68)
 - open/close 3040 (time 1768550804.03 total 40.03 last 93.88)
 - open/close 3883 (time 1768550814.04 total 50.03 last 84.26)
total: 4500 open/close in 57.98 seconds: 77.62 ops/second
 - /unlink 1291 (time 1768550836.54 total 10.00 last 129.09)
 - /unlink 2778 (time 1768550846.55 total 20.01 last 148.63)
 - /unlink 4155 (time 1768550856.55 total 30.01 last 137.65)
total: 4500 /unlink in 32.46 seconds: 138.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 732 (time 1768550873.89 total 10.01 last 73.15)
 - open/close 1446 (time 1768550883.91 total 20.03 last 71.23)
 - open/close 2104 (time 1768550893.91 total 30.03 last 65.79)
 - open/close 2811 (time 1768550903.96 total 40.07 last 70.40)
 - open/close 3574 (time 1768550913.99 total 50.10 last 76.08)
 - open/close 4381 (time 1768550923.99 total 60.11 last 80.67)
total: 4500 open/close in 61.35 seconds: 73.35 ops/second
 - /unlink 1167 (time 1768550939.76 total 10.01 last 116.63)
 - /unlink 2214 (time 1768550949.77 total 20.01 last 104.65)
 - /unlink 3400 (time 1768550959.77 total 30.01 last 118.55)
total: 4500 /unlink in 39.23 seconds: 114.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 696 (time 1768550984.90 total 10.01 last 69.50)
 - open/close 1408 (time 1768550994.91 total 20.02 last 71.14)
 - open/close 2002 (time 1768551004.92 total 30.03 last 59.36)
 - open/close 2679 (time 1768551014.93 total 40.04 last 67.64)
 - open/close 3329 (time 1768551024.93 total 50.04 last 65.00)
 - open/close 4056 (time 1768551034.93 total 60.04 last 72.66)
total: 4500 open/close in 68.22 seconds: 65.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1169 (time 1768551058.44 total 10.00 last 116.84)
 - /unlink 2327 (time 1768551068.44 total 20.01 last 115.74)
 - /unlink 3701 (time 1768551078.45 total 30.01 last 137.35)
total: 4500 /unlink in 36.19 seconds: 124.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 661 (time 1768551099.42 total 10.03 last 65.92)
 - open/close 1361 (time 1768551109.43 total 20.04 last 69.92)
 - open/close 2133 (time 1768551119.43 total 30.04 last 77.18)
 - open/close 3017 (time 1768551129.45 total 40.06 last 88.24)
 - open/close 3753 (time 1768551139.46 total 50.06 last 73.57)
 - open/close 4379 (time 1768551149.50 total 60.11 last 62.33)
total: 4500 open/close in 62.20 seconds: 72.35 ops/second
 - /unlink 700 (time 1768551169.54 total 10.00 last 69.99)
 - /unlink 2009 (time 1768551179.54 total 20.00 last 130.89)
 - /unlink 3121 (time 1768551189.54 total 30.01 last 111.16)
 - /unlink 4325 (time 1768551199.57 total 40.03 last 120.08)
total: 4500 /unlink in 41.88 seconds: 107.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 759 (time 1768551216.86 total 10.00 last 75.90)
 - open/close 1423 (time 1768551226.86 total 20.00 last 66.38)
 - open/close 2167 (time 1768551236.87 total 30.01 last 74.36)
 - open/close 2943 (time 1768551246.88 total 40.01 last 77.56)
 - open/close 3712 (time 1768551256.89 total 50.03 last 76.81)
 - open/close 4435 (time 1768551266.89 total 60.03 last 72.27)
total: 4500 open/close in 60.78 seconds: 74.04 ops/second
 - /unlink 935 (time 1768551283.28 total 10.00 last 93.47)
 - /unlink 2015 (time 1768551293.29 total 20.01 last 107.95)
 - /unlink 3234 (time 1768551303.29 total 30.01 last 121.87)
 - /unlink 4474 (time 1768551313.29 total 40.01 last 124.00)
total: 4500 /unlink in 40.42 seconds: 111.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 608 (time 1768551329.27 total 10.00 last 60.80)
 - open/close 1218 (time 1768551339.27 total 20.01 last 60.95)
 - open/close 1937 (time 1768551349.28 total 30.01 last 71.88)
 - open/close 2650 (time 1768551359.28 total 40.02 last 71.26)
 - open/close 3278 (time 1768551369.28 total 50.02 last 62.78)
 - open/close 3996 (time 1768551379.29 total 60.03 last 71.75)
total: 4500 open/close in 68.67 seconds: 65.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 974 (time 1768551403.21 total 10.01 last 97.30)
 - /unlink 2156 (time 1768551413.21 total 20.01 last 118.18)
 - /unlink 3427 (time 1768551423.24 total 30.04 last 126.81)
total: 4500 /unlink in 39.33 seconds: 114.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 646 (time 1768551448.16 total 10.01 last 64.56)
 - open/close 1397 (time 1768551458.17 total 20.02 last 75.03)
 - open/close 2104 (time 1768551468.17 total 30.02 last 70.68)
 - open/close 2903 (time 1768551478.18 total 40.03 last 79.82)
 - open/close 3505 (time 1768551488.19 total 50.04 last 60.13)
 - open/close 4154 (time 1768551498.22 total 60.07 last 64.73)
total: 4500 open/close in 66.21 seconds: 67.96 ops/second
 - /unlink 919 (time 1768551521.77 total 10.00 last 91.89)
 - /unlink 1882 (time 1768551531.78 total 20.01 last 96.21)
 - /unlink 3012 (time 1768551541.78 total 30.01 last 112.98)
 - /unlink 4196 (time 1768551551.79 total 40.02 last 118.34)
total: 4500 /unlink in 42.61 seconds: 105.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 560 (time 1768551571.05 total 10.00 last 55.99)
 - open/close 1245 (time 1768551581.05 total 20.00 last 68.49)
 - open/close 1903 (time 1768551591.05 total 30.01 last 65.77)
 - open/close 2652 (time 1768551601.07 total 40.02 last 74.80)
 - open/close 3319 (time 1768551611.07 total 50.02 last 66.68)
 - open/close 4026 (time 1768551621.09 total 60.04 last 70.56)
total: 4500 open/close in 66.68 seconds: 67.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1227 (time 1768551642.47 total 10.00 last 122.70)
 - /unlink 2539 (time 1768551652.47 total 20.01 last 131.12)
 - /unlink 3873 (time 1768551662.48 total 30.01 last 133.37)
total: 4500 /unlink in 34.27 seconds: 131.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 889 (time 1768551682.08 total 10.00 last 88.87)
 - open/close 1647 (time 1768551692.09 total 20.01 last 75.73)
 - open/close 2216 (time 1768551702.10 total 30.03 last 56.83)
 - open/close 2788 (time 1768551712.12 total 40.04 last 57.10)
 - open/close 3416 (time 1768551722.12 total 50.05 last 62.79)
 - open/close 4102 (time 1768551732.13 total 60.06 last 68.52)
total: 4500 open/close in 66.11 seconds: 68.06 ops/second
 - /unlink 1199 (time 1768551753.48 total 10.00 last 119.86)
 - /unlink 2476 (time 1768551763.48 total 20.00 last 127.68)
 - /unlink 3892 (time 1768551773.48 total 30.01 last 141.56)
total: 4500 /unlink in 34.47 seconds: 130.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 684 (time 1768551793.14 total 10.00 last 68.39)
 - open/close 1347 (time 1768551803.15 total 20.01 last 66.25)
 - open/close 1943 (time 1768551813.15 total 30.02 last 59.56)
 - open/close 2575 (time 1768551823.16 total 40.02 last 63.15)
 - open/close 3194 (time 1768551833.17 total 50.04 last 61.83)
 - open/close 3922 (time 1768551843.18 total 60.04 last 72.75)
total: 4500 open/close in 69.05 seconds: 65.17 ops/second
 - /unlink 1265 (time 1768551866.86 total 10.00 last 126.48)
 - /unlink 2672 (time 1768551876.86 total 20.01 last 140.64)
 - /unlink 3942 (time 1768551886.87 total 30.01 last 126.93)
total: 4500 /unlink in 33.39 seconds: 134.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 711 (time 1768551905.70 total 10.03 last 70.92)
 - open/close 1424 (time 1768551915.71 total 20.03 last 71.23)
 - open/close 2097 (time 1768551925.72 total 30.04 last 67.24)
 - open/close 2703 (time 1768551935.73 total 40.05 last 60.54)
 - open/close 3461 (time 1768551945.73 total 50.06 last 75.78)
 - open/close 4161 (time 1768551955.74 total 60.07 last 69.93)
total: 4500 open/close in 65.39 seconds: 68.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1280 (time 1768551975.70 total 10.00 last 127.94)
 - /unlink 2480 (time 1768551985.71 total 20.01 last 119.97)
 - /unlink 3941 (time 1768551995.71 total 30.01 last 146.05)
total: 4500 /unlink in 33.85 seconds: 132.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 695 (time 1768552015.22 total 10.02 last 69.39)
 - open/close 1399 (time 1768552025.23 total 20.02 last 70.37)
 - open/close 2174 (time 1768552035.24 total 30.03 last 77.40)
 - open/close 2807 (time 1768552045.24 total 40.03 last 63.29)
 - open/close 3482 (time 1768552055.25 total 50.04 last 67.46)
 - open/close 4284 (time 1768552065.26 total 60.06 last 80.05)
total: 4500 open/close in 62.81 seconds: 71.65 ops/second
 - /unlink 1364 (time 1768552082.01 total 10.00 last 136.36)
 - /unlink 2617 (time 1768552092.02 total 20.01 last 125.20)
 - /unlink 4065 (time 1768552102.02 total 30.01 last 144.76)
total: 4500 /unlink in 33.63 seconds: 133.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 720 (time 1768552120.49 total 10.01 last 71.93)
 - open/close 1414 (time 1768552130.49 total 20.01 last 69.39)
 - open/close 2148 (time 1768552140.50 total 30.02 last 73.32)
 - open/close 2908 (time 1768552150.50 total 40.03 last 75.97)
 - open/close 3754 (time 1768552160.50 total 50.03 last 84.58)
 - open/close 4364 (time 1768552170.52 total 60.04 last 60.92)
total: 4500 open/close in 61.94 seconds: 72.65 ops/second
 - /unlink 1538 (time 1768552187.87 total 10.01 last 153.72)
 - /unlink 2650 (time 1768552197.88 total 20.01 last 111.11)
 - /unlink 4076 (time 1768552207.88 total 30.02 last 142.56)
total: 4500 /unlink in 32.85 seconds: 136.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 818 (time 1768552225.43 total 10.03 last 81.56)
 - open/close 1550 (time 1768552235.43 total 20.04 last 73.14)
 - open/close 2155 (time 1768552245.45 total 30.05 last 60.40)
 - open/close 2693 (time 1768552255.48 total 40.09 last 53.63)
 - open/close 3310 (time 1768552265.49 total 50.09 last 61.67)
 - open/close 4029 (time 1768552275.49 total 60.10 last 71.86)
total: 4500 open/close in 66.91 seconds: 67.26 ops/second
 - /unlink 1295 (time 1768552297.33 total 10.00 last 129.45)
 - /unlink 2509 (time 1768552307.33 total 20.01 last 121.36)
 - /unlink 3734 (time 1768552317.34 total 30.02 last 122.38)
total: 4500 /unlink in 36.12 seconds: 124.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 856 (time 1768552338.41 total 10.00 last 85.57)
 - open/close 1472 (time 1768552348.42 total 20.01 last 61.57)
 - open/close 2138 (time 1768552358.42 total 30.01 last 66.57)
 - open/close 2795 (time 1768552368.43 total 40.03 last 65.62)
 - open/close 3308 (time 1768552378.45 total 50.05 last 51.20)
 - open/close 3985 (time 1768552388.46 total 60.05 last 67.68)
total: 4500 open/close in 66.24 seconds: 67.93 ops/second
 - /unlink 1193 (time 1768552409.72 total 10.01 last 119.20)
 - /unlink 2486 (time 1768552419.73 total 20.02 last 129.15)
 - /unlink 4035 (time 1768552429.74 total 30.02 last 154.89)
total: 4500 /unlink in 33.05 seconds: 136.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 716 (time 1768552449.00 total 10.01 last 71.51)
 - open/close 1331 (time 1768552459.03 total 20.04 last 61.32)
 - open/close 1851 (time 1768552469.03 total 30.04 last 51.99)
 - open/close 2475 (time 1768552479.03 total 40.05 last 62.36)
 - open/close 3266 (time 1768552489.04 total 50.06 last 79.02)
 - open/close 3975 (time 1768552499.05 total 60.06 last 70.89)
total: 4500 open/close in 67.62 seconds: 66.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1320 (time 1768552520.98 total 10.00 last 131.99)
 - /unlink 2588 (time 1768552530.99 total 20.01 last 126.72)
 - /unlink 3775 (time 1768552540.99 total 30.01 last 118.64)
total: 4500 /unlink in 35.47 seconds: 126.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 630 (time 1768552561.06 total 10.00 last 62.97)
 - open/close 1331 (time 1768552571.07 total 20.02 last 70.01)
 - open/close 2093 (time 1768552581.07 total 30.02 last 76.18)
 - open/close 2902 (time 1768552591.07 total 40.02 last 80.88)
 - open/close 3653 (time 1768552601.08 total 50.03 last 75.06)
 - open/close 4447 (time 1768552611.08 total 60.03 last 79.39)
total: 4500 open/close in 60.65 seconds: 74.19 ops/second
 - /unlink 1072 (time 1768552626.34 total 10.01 last 107.13)
 - /unlink 2294 (time 1768552636.34 total 20.01 last 122.18)
 - /unlink 3534 (time 1768552646.35 total 30.01 last 123.98)
total: 4500 /unlink in 39.63 seconds: 113.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 706 (time 1768552671.85 total 10.00 last 70.59)
 - open/close 1377 (time 1768552681.85 total 20.00 last 67.10)
 - open/close 1979 (time 1768552691.85 total 30.00 last 60.19)
 - open/close 2571 (time 1768552701.85 total 40.01 last 59.19)
 - open/close 3253 (time 1768552711.86 total 50.01 last 68.16)
 - open/close 3953 (time 1768552721.87 total 60.02 last 69.91)
total: 4500 open/close in 68.36 seconds: 65.83 ops/second
 - /unlink 1177 (time 1768552745.42 total 10.01 last 117.64)
 - /unlink 2429 (time 1768552755.43 total 20.01 last 125.13)
 - /unlink 3629 (time 1768552765.43 total 30.01 last 119.97)
total: 4500 /unlink in 37.18 seconds: 121.05 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11101: 94166 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg258-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg258-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server
stop mds service on oleg258-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg258-server
stop mds service on oleg258-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg258-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg258-server'
oleg258-server: oleg258-server.virtnet: executing unload_modules_local
oleg258-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg258-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.