== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:22:50 (1787844170)
Stopping clients: oleg428-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg428-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg428-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg428-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg428-server
oleg428-server: oleg428-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg428-server'
oleg428-server: oleg428-server.virtnet: executing load_modules_local
oleg428-server: Loading modules from /home/green/git/lustre-release/lustre
oleg428-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg428-server: detected 4 online CPUs by sysfs
oleg428-server: MODOPTS_LIBCFS=
oleg428-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg428-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all
pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg428-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all
pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg428-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all
pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg428-client.virtnet:  -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 650 (time 1787844382.44 total 10.01 last 64.97)
 - open/close 1335 (time 1787844392.44 total 20.01 last 68.47)
 - open/close 1970 (time 1787844402.45 total 30.01 last 63.48)
 - open/close 2448 (time 1787844412.49 total 40.05 last 47.61)
 - open/close 2995 (time 1787844422.49 total 50.05 last 54.69)
 - open/close 3585 (time 1787844432.51 total 60.08 last 58.84)
 - open/close 4133 (time 1787844442.52 total 70.09 last 54.77)
total: 4500 open/close in 75.71 seconds: 59.44 ops/second
 - /unlink 1316 (time 1787844463.46 total 10.01 last 131.51)
 - /unlink 2525 (time 1787844473.46 total 20.01 last 120.85)
 - /unlink 3795 (time 1787844483.46 total 30.01 last 126.99)
total: 4500 /unlink in 34.90 seconds: 128.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 669 (time 1787844503.38 total 10.00 last 66.90)
 - open/close 1253 (time 1787844513.38 total 20.00 last 58.40)
 - open/close 1997 (time 1787844523.39 total 30.01 last 74.34)
 - open/close 2548 (time 1787844533.40 total 40.03 last 55.00)
 - open/close 2991 (time 1787844543.41 total 50.03 last 44.28)
 - open/close 3526 (time 1787844553.41 total 60.04 last 53.47)
 - open/close 4109 (time 1787844563.42 total 70.05 last 58.24)
total: 4500 open/close in 77.18 seconds: 58.30 ops/second
 - /unlink 1222 (time 1787844585.06 total 10.00 last 122.18)
 - /unlink 2376 (time 1787844595.06 total 20.00 last 115.36)
 - /unlink 3506 (time 1787844605.07 total 30.01 last 112.95)
total: 4500 /unlink in 38.39 seconds: 117.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 639 (time 1787844629.53 total 10.00 last 63.90)
 - open/close 1272 (time 1787844639.54 total 20.00 last 63.29)
 - open/close 1898 (time 1787844649.55 total 30.02 last 62.51)
 - open/close 2473 (time 1787844659.57 total 40.04 last 57.37)
 - open/close 3072 (time 1787844669.58 total 50.05 last 59.85)
 - open/close 3669 (time 1787844679.60 total 60.06 last 59.61)
 - open/close 4261 (time 1787844689.61 total 70.07 last 59.14)
total: 4500 open/close in 73.68 seconds: 61.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1138 (time 1787844707.97 total 10.01 last 113.74)
 - /unlink 2208 (time 1787844717.98 total 20.01 last 106.96)
 - /unlink 3488 (time 1787844727.98 total 30.01 last 127.98)
total: 4500 /unlink in 38.94 seconds: 115.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 689 (time 1787844752.38 total 10.01 last 68.85)
 - open/close 1358 (time 1787844762.38 total 20.02 last 66.85)
 - open/close 2093 (time 1787844772.39 total 30.02 last 73.45)
 - open/close 2621 (time 1787844782.40 total 40.03 last 52.76)
 - open/close 3337 (time 1787844792.40 total 50.03 last 71.57)
 - open/close 4089 (time 1787844802.40 total 60.04 last 75.19)
total: 4500 open/close in 66.28 seconds: 67.89 ops/second
 - /unlink 1212 (time 1787844824.32 total 10.00 last 121.16)
 - /unlink 2498 (time 1787844834.32 total 20.00 last 128.59)
 - /unlink 3683 (time 1787844844.32 total 30.00 last 118.49)
total: 4500 /unlink in 36.62 seconds: 122.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 658 (time 1787844866.72 total 10.00 last 65.78)
 - open/close 1382 (time 1787844876.73 total 20.01 last 72.32)
 - open/close 2031 (time 1787844886.75 total 30.03 last 64.80)
 - open/close 2595 (time 1787844896.76 total 40.04 last 56.35)
 - open/close 3082 (time 1787844906.77 total 50.05 last 48.65)
 - open/close 3589 (time 1787844916.77 total 60.05 last 50.67)
 - open/close 4125 (time 1787844926.80 total 70.08 last 53.47)
total: 4500 open/close in 76.67 seconds: 58.69 ops/second
 - /unlink 1148 (time 1787844948.77 total 10.00 last 114.77)
 - /unlink 2389 (time 1787844958.79 total 20.02 last 123.92)
 - /unlink 3436 (time 1787844968.81 total 30.04 last 104.52)
total: 4500 /unlink in 39.20 seconds: 114.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 598 (time 1787844993.04 total 10.01 last 59.73)
 - open/close 1277 (time 1787845003.06 total 20.03 last 67.79)
 - open/close 1985 (time 1787845013.06 total 30.03 last 70.77)
 - open/close 2696 (time 1787845023.06 total 40.03 last 71.10)
 - open/close 3408 (time 1787845033.09 total 50.06 last 71.00)
 - open/close 4063 (time 1787845043.11 total 60.07 last 65.42)
total: 4500 open/close in 67.02 seconds: 67.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1152 (time 1787845066.42 total 10.01 last 115.12)
 - /unlink 2417 (time 1787845076.42 total 20.01 last 126.49)
 - /unlink 3702 (time 1787845086.43 total 30.01 last 128.42)
total: 4500 /unlink in 36.58 seconds: 123.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 547 (time 1787845109.23 total 10.00 last 54.68)
 - open/close 1189 (time 1787845119.28 total 20.05 last 63.92)
 - open/close 1820 (time 1787845129.28 total 30.05 last 63.06)
 - open/close 2578 (time 1787845139.29 total 40.06 last 75.77)
 - open/close 3243 (time 1787845149.30 total 50.07 last 66.43)
 - open/close 3883 (time 1787845159.30 total 60.07 last 63.98)
total: 4500 open/close in 68.49 seconds: 65.70 ops/second
 - /unlink 1273 (time 1787845182.17 total 10.01 last 127.24)
 - /unlink 2535 (time 1787845192.17 total 20.01 last 126.11)
 - /unlink 3672 (time 1787845202.19 total 30.03 last 113.47)
total: 4500 /unlink in 37.10 seconds: 121.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 583 (time 1787845225.65 total 10.01 last 58.26)
 - open/close 1251 (time 1787845235.67 total 20.03 last 66.62)
 - open/close 1886 (time 1787845245.67 total 30.04 last 63.49)
 - open/close 2545 (time 1787845255.68 total 40.04 last 65.85)
 - open/close 3184 (time 1787845265.68 total 50.05 last 63.88)
 - open/close 3811 (time 1787845275.69 total 60.05 last 62.69)
 - open/close 4446 (time 1787845285.69 total 70.05 last 63.46)
total: 4500 open/close in 70.67 seconds: 63.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1235 (time 1787845301.51 total 10.01 last 123.38)
 - /unlink 2353 (time 1787845311.51 total 20.01 last 111.79)
 - /unlink 3599 (time 1787845321.52 total 30.02 last 124.48)
total: 4500 /unlink in 37.07 seconds: 121.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 685 (time 1787845343.43 total 10.01 last 68.44)
 - open/close 1315 (time 1787845353.43 total 20.01 last 62.98)
 - open/close 1985 (time 1787845363.43 total 30.02 last 66.98)
 - open/close 2662 (time 1787845373.44 total 40.02 last 67.64)
 - open/close 3366 (time 1787845383.45 total 50.03 last 70.34)
 - open/close 4061 (time 1787845393.46 total 60.04 last 69.46)
total: 4500 open/close in 66.26 seconds: 67.91 ops/second
 - /unlink 1287 (time 1787845413.92 total 10.01 last 128.56)
 - /unlink 2529 (time 1787845423.93 total 20.02 last 124.14)
 - /unlink 3750 (time 1787845433.94 total 30.03 last 121.95)
total: 4500 /unlink in 35.75 seconds: 125.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 664 (time 1787845453.82 total 10.01 last 66.32)
 - open/close 1327 (time 1787845463.83 total 20.02 last 66.23)
 - open/close 1960 (time 1787845473.84 total 30.03 last 63.28)
 - open/close 2584 (time 1787845483.84 total 40.03 last 62.35)
 - open/close 3214 (time 1787845493.87 total 50.06 last 62.85)
 - open/close 3881 (time 1787845503.87 total 60.06 last 66.68)
total: 4500 open/close in 70.02 seconds: 64.27 ops/second
 - /unlink 1238 (time 1787845529.12 total 10.00 last 123.80)
 - /unlink 2235 (time 1787845539.13 total 20.00 last 99.66)
 - /unlink 3553 (time 1787845549.13 total 30.01 last 131.77)
total: 4500 /unlink in 38.23 seconds: 117.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 665 (time 1787845572.01 total 10.00 last 66.48)
 - open/close 1266 (time 1787845582.02 total 20.01 last 60.06)
 - open/close 1941 (time 1787845592.03 total 30.02 last 67.42)
 - open/close 2575 (time 1787845602.03 total 40.03 last 63.38)
 - open/close 3222 (time 1787845612.04 total 50.04 last 64.63)
 - open/close 3874 (time 1787845622.05 total 60.04 last 65.16)
total: 4500 open/close in 69.56 seconds: 64.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1148 (time 1787845647.02 total 10.03 last 114.43)
 - /unlink 2146 (time 1787845657.03 total 20.04 last 99.74)
 - /unlink 3198 (time 1787845667.04 total 30.05 last 105.10)
 - /unlink 4282 (time 1787845677.04 total 40.05 last 108.37)
total: 4500 /unlink in 42.29 seconds: 106.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 568 (time 1787845696.37 total 10.04 last 56.57)
 - open/close 1017 (time 1787845706.38 total 20.05 last 44.86)
 - open/close 1544 (time 1787845716.42 total 30.09 last 52.51)
 - open/close 2082 (time 1787845726.43 total 40.10 last 53.70)
 - open/close 2742 (time 1787845736.44 total 50.11 last 65.93)
 - open/close 3405 (time 1787845746.46 total 60.13 last 66.18)
 - open/close 4125 (time 1787845756.46 total 70.13 last 72.00)
total: 4500 open/close in 75.78 seconds: 59.38 ops/second
 - /unlink 1231 (time 1787845777.09 total 10.01 last 123.03)
 - /unlink 2342 (time 1787845787.10 total 20.01 last 111.07)
 - /unlink 3643 (time 1787845797.10 total 30.01 last 130.08)
total: 4500 /unlink in 36.75 seconds: 122.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 615 (time 1787845819.35 total 10.00 last 61.49)
 - open/close 1190 (time 1787845829.36 total 20.01 last 57.45)
 - open/close 1812 (time 1787845839.38 total 30.03 last 62.08)
 - open/close 2435 (time 1787845849.39 total 40.04 last 62.22)
 - open/close 3033 (time 1787845859.40 total 50.06 last 59.72)
 - open/close 3702 (time 1787845869.41 total 60.06 last 66.85)
 - open/close 4322 (time 1787845879.43 total 70.08 last 61.88)
total: 4500 open/close in 73.11 seconds: 61.55 ops/second
 - /unlink 1294 (time 1787845897.47 total 10.00 last 129.37)
 - /unlink 2355 (time 1787845907.47 total 20.00 last 106.09)
 - /unlink 3423 (time 1787845917.48 total 30.01 last 106.77)
total: 4500 /unlink in 39.19 seconds: 114.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 627 (time 1787845941.70 total 10.01 last 62.65)
 - open/close 1226 (time 1787845951.70 total 20.01 last 59.89)
 - open/close 1859 (time 1787845961.72 total 30.03 last 63.15)
 - open/close 2524 (time 1787845971.74 total 40.05 last 66.42)
 - open/close 3110 (time 1787845981.75 total 50.06 last 58.54)
 - open/close 3757 (time 1787845991.76 total 60.07 last 64.64)
 - open/close 4383 (time 1787846001.78 total 70.09 last 62.47)
total: 4500 open/close in 72.20 seconds: 62.33 ops/second
 - /unlink 1092 (time 1787846020.32 total 10.00 last 109.16)
 - /unlink 2243 (time 1787846030.33 total 20.01 last 115.05)
 - /unlink 3388 (time 1787846040.33 total 30.01 last 114.46)
 - /unlink 4410 (time 1787846050.34 total 40.02 last 102.15)
total: 4500 /unlink in 40.95 seconds: 109.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 541 (time 1787846066.16 total 10.00 last 54.10)
 - open/close 1064 (time 1787846076.17 total 20.01 last 52.27)
 - open/close 1609 (time 1787846086.19 total 30.03 last 54.40)
 - open/close 2208 (time 1787846096.19 total 40.03 last 59.89)
 - open/close 2841 (time 1787846106.21 total 50.05 last 63.17)
 - open/close 3420 (time 1787846116.22 total 60.06 last 57.86)
 - open/close 3998 (time 1787846126.24 total 70.08 last 57.64)
total: 4500 open/close in 78.31 seconds: 57.46 ops/second
 - /unlink 1107 (time 1787846149.58 total 10.00 last 110.66)
 - /unlink 2226 (time 1787846159.58 total 20.01 last 111.79)
 - /unlink 3295 (time 1787846169.59 total 30.02 last 106.86)
 - /unlink 4328 (time 1787846179.59 total 40.02 last 103.28)
total: 4500 /unlink in 41.70 seconds: 107.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 605 (time 1787846197.18 total 10.00 last 60.48)
 - open/close 1171 (time 1787846207.19 total 20.01 last 56.55)
 - open/close 1681 (time 1787846217.19 total 30.02 last 50.98)
 - open/close 2261 (time 1787846227.20 total 40.03 last 57.94)
 - open/close 2944 (time 1787846237.21 total 50.03 last 68.27)
 - open/close 3637 (time 1787846247.22 total 60.04 last 69.23)
 - open/close 4293 (time 1787846257.22 total 70.04 last 65.58)
total: 4500 open/close in 73.52 seconds: 61.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1139 (time 1787846275.71 total 10.00 last 113.88)
 - /unlink 2284 (time 1787846285.71 total 20.01 last 114.42)
 - /unlink 3436 (time 1787846295.72 total 30.02 last 115.10)
total: 4500 /unlink in 38.42 seconds: 117.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 655 (time 1787846319.26 total 10.01 last 65.45)
 - open/close 1330 (time 1787846329.26 total 20.01 last 67.49)
 - open/close 1939 (time 1787846339.27 total 30.01 last 60.87)
 - open/close 2549 (time 1787846349.27 total 40.02 last 60.96)
 - open/close 3104 (time 1787846359.29 total 50.03 last 55.42)
 - open/close 3682 (time 1787846369.30 total 60.04 last 57.75)
 - open/close 4263 (time 1787846379.30 total 70.05 last 58.06)
total: 4500 open/close in 73.52 seconds: 61.21 ops/second
 - /unlink 1189 (time 1787846398.49 total 10.00 last 118.90)
 - /unlink 2321 (time 1787846408.49 total 20.00 last 113.19)
 - /unlink 3361 (time 1787846418.49 total 30.01 last 103.93)
 - /unlink 4273 (time 1787846428.49 total 40.01 last 91.20)
total: 4500 /unlink in 41.75 seconds: 107.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 446 (time 1787846448.58 total 10.00 last 44.59)
 - open/close 1013 (time 1787846458.62 total 20.04 last 56.46)
 - open/close 1547 (time 1787846468.64 total 30.07 last 53.25)
 - open/close 2218 (time 1787846478.66 total 40.09 last 66.99)
 - open/close 2924 (time 1787846488.66 total 50.09 last 70.60)
 - open/close 3601 (time 1787846498.68 total 60.11 last 67.57)
 - open/close 4270 (time 1787846508.69 total 70.12 last 66.81)
total: 4500 open/close in 74.01 seconds: 60.81 ops/second
 - /unlink 1174 (time 1787846527.48 total 10.00 last 117.34)
 - /unlink 2363 (time 1787846537.49 total 20.02 last 118.78)
 - /unlink 3526 (time 1787846547.50 total 30.02 last 116.22)
total: 4500 /unlink in 38.04 seconds: 118.31 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11369: 95467 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg428-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg428-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server
stop mds service on oleg428-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server
stop mds service on oleg428-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg428-server

unloading modules via unload_modules_local on: 'oleg428-server'
oleg428-server: oleg428-server.virtnet: executing unload_modules_local
oleg428-server: 
modules unloaded.