== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:26:45 (1786980405)
Stopping clients: oleg236-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg236-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg236-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg236-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg236-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg236-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg236-server
oleg236-server: oleg236-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg236-server'
oleg236-server: oleg236-server.virtnet: executing load_modules_local
oleg236-server: Loading modules from /home/green/git/lustre-release/lustre
oleg236-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg236-server: detected 4 online CPUs by sysfs
oleg236-server: MODOPTS_LIBCFS=
oleg236-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg236-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all
pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg236-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all
pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg236-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all
pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg236-client.virtnet:  -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 729 (time 1786980615.19 total 10.01 last 72.86)
 - open/close 1426 (time 1786980625.19 total 20.01 last 69.69)
 - open/close 2067 (time 1786980635.20 total 30.01 last 64.08)
 - open/close 2868 (time 1786980645.23 total 40.04 last 79.87)
 - open/close 3497 (time 1786980655.23 total 50.04 last 62.86)
 - open/close 4328 (time 1786980665.25 total 60.07 last 82.91)
total: 4500 open/close in 62.70 seconds: 71.77 ops/second
 - /unlink 1505 (time 1786980682.15 total 10.00 last 150.44)
 - /unlink 2769 (time 1786980692.15 total 20.01 last 126.38)
 - /unlink 4171 (time 1786980702.16 total 30.01 last 140.14)
total: 4500 /unlink in 32.53 seconds: 138.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 798 (time 1786980719.51 total 10.00 last 79.78)
 - open/close 1509 (time 1786980729.56 total 20.05 last 70.77)
 - open/close 2230 (time 1786980739.57 total 30.05 last 72.07)
 - open/close 2718 (time 1786980749.58 total 40.07 last 48.73)
 - open/close 3370 (time 1786980759.58 total 50.07 last 65.17)
 - open/close 4060 (time 1786980769.59 total 60.08 last 68.93)
total: 4500 open/close in 65.83 seconds: 68.36 ops/second
 - /unlink 1468 (time 1786980789.61 total 10.01 last 146.71)
 - /unlink 2786 (time 1786980799.61 total 20.01 last 131.76)
 - /unlink 4097 (time 1786980809.61 total 30.01 last 131.03)
total: 4500 /unlink in 33.10 seconds: 135.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 710 (time 1786980827.31 total 10.00 last 70.98)
 - open/close 1406 (time 1786980837.32 total 20.02 last 69.51)
 - open/close 2240 (time 1786980847.32 total 30.02 last 83.38)
 - open/close 3035 (time 1786980857.33 total 40.02 last 79.45)
 - open/close 3849 (time 1786980867.35 total 50.04 last 81.28)
total: 4500 open/close in 57.49 seconds: 78.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1571 (time 1786980888.68 total 10.00 last 157.03)
 - /unlink 2933 (time 1786980898.68 total 20.01 last 136.17)
 - /unlink 4239 (time 1786980908.69 total 30.01 last 130.57)
total: 4500 /unlink in 31.78 seconds: 141.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 779 (time 1786980925.11 total 10.01 last 77.80)
 - open/close 1575 (time 1786980935.11 total 20.02 last 79.55)
 - open/close 2350 (time 1786980945.11 total 30.02 last 77.50)
 - open/close 3104 (time 1786980955.11 total 40.02 last 75.40)
 - open/close 3887 (time 1786980965.14 total 50.05 last 78.08)
total: 4500 open/close in 58.91 seconds: 76.39 ops/second
 - /unlink 1549 (time 1786980988.65 total 10.00 last 154.88)
 - /unlink 3094 (time 1786980998.65 total 20.00 last 154.49)
total: 4500 /unlink in 28.51 seconds: 157.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 824 (time 1786981022.03 total 10.01 last 82.32)
 - open/close 1554 (time 1786981032.03 total 20.01 last 73.00)
 - open/close 2265 (time 1786981042.04 total 30.02 last 71.05)
 - open/close 3031 (time 1786981052.05 total 40.03 last 76.53)
 - open/close 3771 (time 1786981062.05 total 50.03 last 74.00)
total: 4500 open/close in 59.74 seconds: 75.33 ops/second
 - /unlink 1355 (time 1786981087.04 total 10.00 last 135.44)
 - /unlink 2701 (time 1786981097.05 total 20.02 last 134.46)
 - /unlink 3991 (time 1786981107.05 total 30.02 last 128.97)
total: 4500 /unlink in 33.35 seconds: 134.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 771 (time 1786981125.02 total 10.01 last 77.05)
 - open/close 1444 (time 1786981135.02 total 20.01 last 67.29)
 - open/close 2070 (time 1786981145.03 total 30.02 last 62.52)
 - open/close 2672 (time 1786981155.04 total 40.03 last 60.16)
 - open/close 3348 (time 1786981165.04 total 50.03 last 67.56)
 - open/close 4096 (time 1786981175.05 total 60.03 last 74.79)
total: 4500 open/close in 65.35 seconds: 68.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1398 (time 1786981195.24 total 10.00 last 139.78)
 - /unlink 2713 (time 1786981205.24 total 20.00 last 131.48)
 - /unlink 4059 (time 1786981215.24 total 30.01 last 134.57)
total: 4500 /unlink in 33.08 seconds: 136.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 838 (time 1786981233.29 total 10.01 last 83.72)
 - open/close 1569 (time 1786981243.29 total 20.01 last 73.09)
 - open/close 2411 (time 1786981253.30 total 30.03 last 84.07)
 - open/close 3139 (time 1786981263.30 total 40.03 last 72.79)
 - open/close 3833 (time 1786981273.31 total 50.03 last 69.38)
total: 4500 open/close in 59.81 seconds: 75.23 ops/second
 - /unlink 1483 (time 1786981298.24 total 10.00 last 148.26)
 - /unlink 2829 (time 1786981308.25 total 20.01 last 134.51)
 - /unlink 3970 (time 1786981318.26 total 30.02 last 113.99)
total: 4500 /unlink in 33.80 seconds: 133.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 728 (time 1786981338.29 total 10.00 last 72.78)
 - open/close 1496 (time 1786981348.31 total 20.02 last 76.64)
 - open/close 2326 (time 1786981358.31 total 30.02 last 83.00)
 - open/close 3020 (time 1786981368.33 total 40.04 last 69.28)
 - open/close 3715 (time 1786981378.33 total 50.04 last 69.48)
 - open/close 4332 (time 1786981388.35 total 60.07 last 61.54)
total: 4500 open/close in 62.83 seconds: 71.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1370 (time 1786981407.51 total 10.00 last 136.99)
 - /unlink 2460 (time 1786981417.51 total 20.01 last 108.94)
 - /unlink 3781 (time 1786981427.51 total 30.01 last 132.08)
total: 4500 /unlink in 35.42 seconds: 127.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 666 (time 1786981447.90 total 10.01 last 66.53)
 - open/close 1197 (time 1786981457.90 total 20.01 last 53.10)
 - open/close 1872 (time 1786981467.90 total 30.02 last 67.47)
 - open/close 2730 (time 1786981477.91 total 40.03 last 85.70)
 - open/close 3387 (time 1786981487.96 total 50.07 last 65.42)
 - open/close 3973 (time 1786981497.96 total 60.08 last 58.57)
total: 4500 open/close in 67.75 seconds: 66.42 ops/second
 - /unlink 1380 (time 1786981519.82 total 10.01 last 137.91)
 - /unlink 2663 (time 1786981529.83 total 20.01 last 128.23)
 - /unlink 3856 (time 1786981539.83 total 30.01 last 119.30)
total: 4500 /unlink in 35.57 seconds: 126.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 516 (time 1786981561.74 total 10.01 last 51.55)
 - open/close 1157 (time 1786981571.75 total 20.02 last 64.04)
 - open/close 1967 (time 1786981581.76 total 30.03 last 80.93)
 - open/close 2778 (time 1786981591.76 total 40.03 last 81.09)
 - open/close 3571 (time 1786981601.78 total 50.04 last 79.21)
 - open/close 4249 (time 1786981611.78 total 60.05 last 67.75)
total: 4500 open/close in 63.69 seconds: 70.66 ops/second
 - /unlink 1273 (time 1786981630.41 total 10.01 last 127.21)
 - /unlink 2611 (time 1786981640.41 total 20.01 last 133.76)
 - /unlink 4094 (time 1786981650.41 total 30.01 last 148.28)
total: 4500 /unlink in 32.92 seconds: 136.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 723 (time 1786981669.73 total 10.01 last 72.22)
 - open/close 1531 (time 1786981679.74 total 20.02 last 80.74)
 - open/close 2281 (time 1786981689.77 total 30.05 last 74.73)
 - open/close 3061 (time 1786981699.78 total 40.06 last 77.93)
 - open/close 3689 (time 1786981709.82 total 50.10 last 62.57)
 - open/close 4467 (time 1786981719.82 total 60.10 last 77.79)
total: 4500 open/close in 60.45 seconds: 74.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1291 (time 1786981735.07 total 10.01 last 129.01)
 - /unlink 2533 (time 1786981745.07 total 20.01 last 124.19)
 - /unlink 3723 (time 1786981755.07 total 30.01 last 118.99)
total: 4500 /unlink in 36.80 seconds: 122.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 715 (time 1786981776.47 total 10.01 last 71.45)
 - open/close 1456 (time 1786981786.47 total 20.02 last 74.04)
 - open/close 2249 (time 1786981796.48 total 30.03 last 79.21)
 - open/close 2996 (time 1786981806.49 total 40.04 last 74.63)
 - open/close 3808 (time 1786981816.50 total 50.04 last 81.17)
total: 4500 open/close in 58.50 seconds: 76.92 ops/second
 - /unlink 1255 (time 1786981839.79 total 10.00 last 125.48)
 - /unlink 2491 (time 1786981849.80 total 20.00 last 123.56)
 - /unlink 3953 (time 1786981859.80 total 30.01 last 146.12)
total: 4500 /unlink in 33.73 seconds: 133.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 767 (time 1786981878.99 total 10.01 last 76.65)
 - open/close 1547 (time 1786981889.01 total 20.02 last 77.88)
 - open/close 2228 (time 1786981899.01 total 30.03 last 68.06)
 - open/close 2916 (time 1786981909.01 total 40.03 last 68.79)
 - open/close 3682 (time 1786981919.02 total 50.03 last 76.59)
 - open/close 4425 (time 1786981929.02 total 60.03 last 74.27)
total: 4500 open/close in 60.98 seconds: 73.80 ops/second
 - /unlink 1070 (time 1786981945.09 total 10.00 last 106.99)
 - /unlink 2237 (time 1786981955.10 total 20.01 last 116.65)
 - /unlink 3414 (time 1786981965.10 total 30.01 last 117.62)
total: 4500 /unlink in 39.01 seconds: 115.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 609 (time 1786981989.16 total 10.00 last 60.87)
 - open/close 1188 (time 1786981999.16 total 20.01 last 57.89)
 - open/close 1869 (time 1786982009.20 total 30.04 last 67.84)
 - open/close 2537 (time 1786982019.22 total 40.06 last 66.71)
 - open/close 3142 (time 1786982029.22 total 50.06 last 60.49)
 - open/close 3798 (time 1786982039.22 total 60.07 last 65.55)
 - open/close 4490 (time 1786982049.23 total 70.07 last 69.18)
total: 4500 open/close in 70.23 seconds: 64.07 ops/second
 - /unlink 1010 (time 1786982065.59 total 10.01 last 100.85)
 - /unlink 1914 (time 1786982075.59 total 20.02 last 90.35)
 - /unlink 2993 (time 1786982085.60 total 30.02 last 107.88)
 - /unlink 4271 (time 1786982095.60 total 40.02 last 127.78)
total: 4500 /unlink in 41.86 seconds: 107.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 714 (time 1786982111.91 total 10.02 last 71.27)
 - open/close 1450 (time 1786982121.91 total 20.02 last 73.58)
 - open/close 2090 (time 1786982131.91 total 30.02 last 64.00)
 - open/close 2597 (time 1786982141.92 total 40.03 last 50.65)
 - open/close 3359 (time 1786982151.93 total 50.04 last 76.14)
 - open/close 4078 (time 1786982161.94 total 60.05 last 71.82)
total: 4500 open/close in 66.38 seconds: 67.79 ops/second
 - /unlink 942 (time 1786982184.24 total 10.00 last 94.20)
 - /unlink 1706 (time 1786982194.24 total 20.00 last 76.40)
 - /unlink 2779 (time 1786982204.25 total 30.01 last 107.23)
 - /unlink 4239 (time 1786982214.25 total 40.01 last 145.93)
total: 4500 /unlink in 42.13 seconds: 106.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 659 (time 1786982230.70 total 10.01 last 65.85)
 - open/close 1351 (time 1786982240.71 total 20.02 last 69.15)
 - open/close 2076 (time 1786982250.71 total 30.02 last 72.45)
 - open/close 2837 (time 1786982260.72 total 40.03 last 76.07)
 - open/close 3571 (time 1786982270.72 total 50.03 last 73.37)
 - open/close 4289 (time 1786982280.73 total 60.03 last 71.78)
total: 4500 open/close in 63.14 seconds: 71.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1384 (time 1786982298.48 total 10.01 last 138.32)
 - /unlink 2675 (time 1786982308.48 total 20.01 last 129.09)
 - /unlink 4142 (time 1786982318.49 total 30.02 last 146.54)
total: 4500 /unlink in 32.50 seconds: 138.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 807 (time 1786982335.37 total 10.01 last 80.62)
 - open/close 1498 (time 1786982345.37 total 20.02 last 69.04)
 - open/close 2305 (time 1786982355.38 total 30.02 last 80.66)
 - open/close 3118 (time 1786982365.38 total 40.02 last 81.29)
 - open/close 4009 (time 1786982375.39 total 50.03 last 89.02)
total: 4500 open/close in 56.33 seconds: 79.89 ops/second
 - /unlink 1178 (time 1786982396.30 total 10.00 last 117.78)
 - /unlink 2538 (time 1786982406.30 total 20.01 last 135.95)
 - /unlink 3984 (time 1786982416.31 total 30.01 last 144.51)
total: 4500 /unlink in 34.02 seconds: 132.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 786 (time 1786982435.61 total 10.01 last 78.52)
 - open/close 1670 (time 1786982445.62 total 20.01 last 88.37)
 - open/close 2505 (time 1786982455.62 total 30.02 last 83.47)
 - open/close 3091 (time 1786982465.62 total 40.02 last 58.60)
 - open/close 3564 (time 1786982475.63 total 50.02 last 47.27)
 - open/close 4180 (time 1786982485.64 total 60.03 last 61.54)
total: 4500 open/close in 64.64 seconds: 69.61 ops/second
 - /unlink 1186 (time 1786982504.65 total 10.00 last 118.58)
 - /unlink 2491 (time 1786982514.65 total 20.01 last 130.45)
 - /unlink 3225 (time 1786982524.67 total 30.02 last 73.27)
 - /unlink 4033 (time 1786982534.68 total 40.03 last 80.77)
total: 4500 /unlink in 43.66 seconds: 103.06 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95215 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg236-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg236-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg236-server
stop mds service on oleg236-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg236-server
stop mds service on oleg236-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg236-server

unloading modules via unload_modules_local on: 'oleg236-server'
oleg236-server: oleg236-server.virtnet: executing unload_modules_local
oleg236-server: 
modules unloaded.