== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:42:25 (1787499745)
Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg430-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg430-server
oleg430-server: oleg430-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg430-server'
oleg430-server: oleg430-server.virtnet: executing load_modules_local
oleg430-server: Loading modules from /home/green/git/lustre-release/lustre
oleg430-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg430-server: detected 4 online CPUs by sysfs
oleg430-server: MODOPTS_LIBCFS=
oleg430-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg430-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all
pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg430-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all
pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg430-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all
pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg430-client.virtnet:  -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 747 (time 1787499973.55 total 10.00 last 74.69)
 - open/close 1390 (time 1787499983.57 total 20.02 last 64.20)
 - open/close 2181 (time 1787499993.57 total 30.02 last 79.05)
 - open/close 2930 (time 1787500003.58 total 40.03 last 74.85)
 - open/close 3738 (time 1787500013.59 total 50.04 last 80.72)
total: 4500 open/close in 60.00 seconds: 75.01 ops/second
 - /unlink 1538 (time 1787500037.64 total 10.00 last 153.76)
 - /unlink 2967 (time 1787500047.64 total 20.01 last 142.84)
 - /unlink 4234 (time 1787500057.64 total 30.01 last 126.65)
total: 4500 /unlink in 31.99 seconds: 140.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 739 (time 1787500074.50 total 10.01 last 73.85)
 - open/close 1353 (time 1787500084.51 total 20.01 last 61.36)
 - open/close 2063 (time 1787500094.51 total 30.02 last 70.98)
 - open/close 2778 (time 1787500104.52 total 40.02 last 71.47)
 - open/close 3542 (time 1787500114.52 total 50.02 last 76.37)
 - open/close 4223 (time 1787500124.52 total 60.03 last 68.08)
total: 4500 open/close in 63.77 seconds: 70.56 ops/second
 - /unlink 1534 (time 1787500143.03 total 10.01 last 153.30)
 - /unlink 2853 (time 1787500153.04 total 20.01 last 131.87)
 - /unlink 3965 (time 1787500163.04 total 30.01 last 111.18)
total: 4500 /unlink in 35.52 seconds: 126.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 807 (time 1787500182.84 total 10.01 last 80.62)
 - open/close 1497 (time 1787500192.85 total 20.02 last 68.95)
 - open/close 2417 (time 1787500202.86 total 30.03 last 91.92)
 - open/close 3223 (time 1787500212.86 total 40.03 last 80.55)
 - open/close 3981 (time 1787500222.87 total 50.04 last 75.75)
total: 4500 open/close in 57.22 seconds: 78.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1175 (time 1787500245.68 total 10.01 last 117.42)
 - /unlink 2319 (time 1787500255.69 total 20.01 last 114.39)
 - /unlink 3578 (time 1787500265.69 total 30.01 last 125.86)
total: 4500 /unlink in 37.99 seconds: 118.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 646 (time 1787500291.86 total 10.02 last 64.49)
 - open/close 1204 (time 1787500301.87 total 20.03 last 55.75)
 - open/close 1763 (time 1787500311.88 total 30.04 last 55.85)
 - open/close 2510 (time 1787500321.89 total 40.05 last 74.63)
 - open/close 3213 (time 1787500331.89 total 50.05 last 70.26)
 - open/close 3783 (time 1787500341.91 total 60.07 last 56.88)
 - open/close 4381 (time 1787500351.91 total 70.07 last 59.80)
total: 4500 open/close in 72.68 seconds: 61.91 ops/second
 - /unlink 1272 (time 1787500370.29 total 10.00 last 127.16)
 - /unlink 2268 (time 1787500380.29 total 20.01 last 99.58)
 - /unlink 3502 (time 1787500390.29 total 30.01 last 123.35)
total: 4500 /unlink in 39.71 seconds: 113.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 654 (time 1787500414.90 total 10.01 last 65.35)
 - open/close 1328 (time 1787500424.91 total 20.02 last 67.34)
 - open/close 1911 (time 1787500434.93 total 30.03 last 58.21)
 - open/close 2544 (time 1787500444.93 total 40.04 last 63.28)
 - open/close 3172 (time 1787500454.95 total 50.05 last 62.70)
 - open/close 3812 (time 1787500464.97 total 60.07 last 63.89)
 - open/close 4429 (time 1787500474.98 total 70.08 last 61.62)
total: 4500 open/close in 71.77 seconds: 62.70 ops/second
 - /unlink 962 (time 1787500492.58 total 10.00 last 96.19)
 - /unlink 2042 (time 1787500502.58 total 20.00 last 107.96)
 - /unlink 3112 (time 1787500512.59 total 30.01 last 106.93)
 - /unlink 4132 (time 1787500522.59 total 40.02 last 101.94)
total: 4500 /unlink in 43.24 seconds: 104.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 612 (time 1787500541.33 total 10.00 last 61.20)
 - open/close 1244 (time 1787500551.34 total 20.01 last 63.16)
 - open/close 1855 (time 1787500561.35 total 30.02 last 61.04)
 - open/close 2426 (time 1787500571.35 total 40.02 last 57.10)
 - open/close 3144 (time 1787500581.36 total 50.03 last 71.70)
 - open/close 3818 (time 1787500591.37 total 60.04 last 67.36)
 - open/close 4479 (time 1787500601.40 total 70.07 last 65.89)
total: 4500 open/close in 70.42 seconds: 63.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1234 (time 1787500616.35 total 10.00 last 123.40)
 - /unlink 2396 (time 1787500626.35 total 20.00 last 116.15)
 - /unlink 3617 (time 1787500636.35 total 30.01 last 122.08)
total: 4500 /unlink in 37.35 seconds: 120.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 715 (time 1787500659.23 total 10.00 last 71.48)
 - open/close 1289 (time 1787500669.24 total 20.01 last 57.33)
 - open/close 1905 (time 1787500679.26 total 30.03 last 61.49)
 - open/close 2504 (time 1787500689.27 total 40.04 last 59.83)
 - open/close 3026 (time 1787500699.29 total 50.06 last 52.13)
 - open/close 3595 (time 1787500709.29 total 60.06 last 56.85)
 - open/close 4198 (time 1787500719.30 total 70.07 last 60.25)
total: 4500 open/close in 75.18 seconds: 59.85 ops/second
 - /unlink 1348 (time 1787500739.57 total 10.01 last 134.70)
 - /unlink 2663 (time 1787500749.59 total 20.02 last 131.35)
 - /unlink 3905 (time 1787500759.59 total 30.02 last 124.13)
total: 4500 /unlink in 35.08 seconds: 128.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 681 (time 1787500779.53 total 10.01 last 68.02)
 - open/close 1364 (time 1787500789.54 total 20.03 last 68.19)
 - open/close 2005 (time 1787500799.55 total 30.03 last 64.08)
 - open/close 2768 (time 1787500809.57 total 40.05 last 76.15)
 - open/close 3383 (time 1787500819.57 total 50.06 last 61.45)
 - open/close 4021 (time 1787500829.58 total 60.07 last 63.74)
total: 4500 open/close in 69.07 seconds: 65.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 962 (time 1787500853.94 total 10.00 last 96.15)
 - /unlink 1712 (time 1787500863.95 total 20.01 last 74.97)
 - /unlink 2549 (time 1787500873.95 total 30.01 last 83.66)
 - /unlink 3473 (time 1787500883.95 total 40.02 last 92.37)
 - /unlink 4477 (time 1787500893.95 total 50.02 last 100.39)
total: 4500 /unlink in 50.27 seconds: 89.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 487 (time 1787500909.97 total 10.02 last 48.58)
 - open/close 1033 (time 1787500919.98 total 20.03 last 54.55)
 - open/close 1650 (time 1787500929.98 total 30.04 last 61.69)
 - open/close 2292 (time 1787500939.99 total 40.05 last 64.14)
 - open/close 2899 (time 1787500950.01 total 50.07 last 60.58)
 - open/close 3556 (time 1787500960.02 total 60.07 last 65.65)
 - open/close 4177 (time 1787500970.02 total 70.08 last 62.07)
total: 4500 open/close in 74.69 seconds: 60.25 ops/second
 - /unlink 1315 (time 1787500989.63 total 10.00 last 131.44)
 - /unlink 2429 (time 1787500999.63 total 20.01 last 111.36)
 - /unlink 3591 (time 1787501009.65 total 30.02 last 116.01)
total: 4500 /unlink in 35.93 seconds: 125.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 679 (time 1787501030.39 total 10.00 last 67.88)
 - open/close 1306 (time 1787501040.41 total 20.02 last 62.57)
 - open/close 1956 (time 1787501050.42 total 30.03 last 64.96)
 - open/close 2601 (time 1787501060.42 total 40.04 last 64.46)
 - open/close 3260 (time 1787501070.43 total 50.04 last 65.88)
 - open/close 3927 (time 1787501080.45 total 60.06 last 66.54)
total: 4500 open/close in 69.38 seconds: 64.86 ops/second
 - /unlink 1424 (time 1787501105.33 total 10.02 last 142.15)
 - /unlink 2737 (time 1787501115.33 total 20.02 last 131.30)
 - /unlink 4245 (time 1787501125.33 total 30.02 last 150.78)
total: 4500 /unlink in 31.89 seconds: 141.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 630 (time 1787501141.92 total 10.01 last 62.91)
 - open/close 1308 (time 1787501151.94 total 20.03 last 67.68)
 - open/close 2030 (time 1787501161.94 total 30.04 last 72.15)
 - open/close 2673 (time 1787501171.95 total 40.04 last 64.28)
 - open/close 3344 (time 1787501181.96 total 50.05 last 67.03)
 - open/close 4106 (time 1787501191.96 total 60.05 last 76.20)
total: 4500 open/close in 65.36 seconds: 68.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1282 (time 1787501211.53 total 10.01 last 128.12)
 - /unlink 2629 (time 1787501221.53 total 20.01 last 134.68)
 - /unlink 4037 (time 1787501231.53 total 30.01 last 140.79)
total: 4500 /unlink in 33.33 seconds: 134.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 665 (time 1787501249.51 total 10.00 last 66.48)
 - open/close 1496 (time 1787501259.52 total 20.01 last 83.05)
 - open/close 2192 (time 1787501269.54 total 30.02 last 69.49)
 - open/close 2841 (time 1787501279.55 total 40.04 last 64.78)
 - open/close 3392 (time 1787501289.56 total 50.05 last 55.07)
 - open/close 3981 (time 1787501299.56 total 60.05 last 58.88)
total: 4500 open/close in 67.37 seconds: 66.79 ops/second
 - /unlink 1433 (time 1787501322.52 total 10.01 last 143.13)
 - /unlink 2759 (time 1787501332.53 total 20.02 last 132.51)
 - /unlink 3812 (time 1787501342.54 total 30.02 last 105.25)
total: 4500 /unlink in 36.08 seconds: 124.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 708 (time 1787501363.40 total 10.00 last 70.80)
 - open/close 1480 (time 1787501373.40 total 20.00 last 77.19)
 - open/close 2230 (time 1787501383.40 total 30.00 last 74.98)
 - open/close 2976 (time 1787501393.41 total 40.01 last 74.51)
 - open/close 3687 (time 1787501403.42 total 50.02 last 71.04)
 - open/close 4411 (time 1787501413.44 total 60.04 last 72.25)
total: 4500 open/close in 61.07 seconds: 73.69 ops/second
 - /unlink 1478 (time 1787501429.21 total 10.01 last 147.72)
 - /unlink 2775 (time 1787501439.22 total 20.01 last 129.62)
 - /unlink 4097 (time 1787501449.23 total 30.02 last 132.06)
total: 4500 /unlink in 34.28 seconds: 131.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 733 (time 1787501468.46 total 10.01 last 73.23)
 - open/close 1545 (time 1787501478.47 total 20.01 last 81.17)
 - open/close 2350 (time 1787501488.47 total 30.01 last 80.49)
 - open/close 3177 (time 1787501498.47 total 40.02 last 82.63)
 - open/close 3934 (time 1787501508.48 total 50.03 last 75.63)
total: 4500 open/close in 56.76 seconds: 79.28 ops/second
 - /unlink 1531 (time 1787501529.68 total 10.00 last 153.10)
 - /unlink 2954 (time 1787501539.69 total 20.01 last 142.13)
 - /unlink 4304 (time 1787501549.70 total 30.02 last 134.95)
total: 4500 /unlink in 31.55 seconds: 142.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 740 (time 1787501566.60 total 10.01 last 73.95)
 - open/close 1407 (time 1787501576.61 total 20.01 last 66.69)
 - open/close 2097 (time 1787501586.61 total 30.01 last 68.98)
 - open/close 2782 (time 1787501596.62 total 40.02 last 68.43)
 - open/close 3395 (time 1787501606.62 total 50.02 last 61.29)
 - open/close 4091 (time 1787501616.62 total 60.03 last 69.58)
total: 4500 open/close in 65.57 seconds: 68.63 ops/second
 - /unlink 1366 (time 1787501637.09 total 10.00 last 136.58)
 - /unlink 2647 (time 1787501647.10 total 20.02 last 127.92)
 - /unlink 4022 (time 1787501657.11 total 30.02 last 137.44)
total: 4500 /unlink in 34.39 seconds: 130.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 686 (time 1787501676.40 total 10.00 last 68.58)
 - open/close 1352 (time 1787501686.41 total 20.01 last 66.56)
 - open/close 2075 (time 1787501696.41 total 30.01 last 72.30)
 - open/close 2716 (time 1787501706.42 total 40.02 last 64.01)
 - open/close 3348 (time 1787501716.46 total 50.06 last 62.96)
 - open/close 3990 (time 1787501726.53 total 60.13 last 63.76)
total: 4500 open/close in 67.97 seconds: 66.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1182 (time 1787501749.44 total 10.00 last 118.18)
 - /unlink 2475 (time 1787501759.45 total 20.01 last 129.17)
 - /unlink 3766 (time 1787501769.46 total 30.02 last 128.97)
total: 4500 /unlink in 35.43 seconds: 126.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 717 (time 1787501789.71 total 10.01 last 71.66)
 - open/close 1351 (time 1787501799.71 total 20.01 last 63.37)
 - open/close 1834 (time 1787501809.74 total 30.04 last 48.16)
 - open/close 2491 (time 1787501819.75 total 40.05 last 65.64)
 - open/close 3139 (time 1787501829.77 total 50.07 last 64.68)
 - open/close 3845 (time 1787501839.78 total 60.08 last 70.51)
total: 4500 open/close in 69.01 seconds: 65.21 ops/second
 - /unlink 1208 (time 1787501862.84 total 10.02 last 120.61)
 - /unlink 2421 (time 1787501872.84 total 20.02 last 121.29)
 - /unlink 3543 (time 1787501882.84 total 30.02 last 112.18)
total: 4500 /unlink in 38.75 seconds: 116.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 680 (time 1787501907.05 total 10.02 last 67.89)
 - open/close 1223 (time 1787501917.07 total 20.04 last 54.20)
 - open/close 1757 (time 1787501927.07 total 30.04 last 53.39)
 - open/close 2358 (time 1787501937.08 total 40.04 last 60.07)
 - open/close 3085 (time 1787501947.09 total 50.05 last 72.62)
 - open/close 3695 (time 1787501957.10 total 60.06 last 60.95)
 - open/close 4343 (time 1787501967.10 total 70.07 last 64.76)
total: 4500 open/close in 72.58 seconds: 62.00 ops/second
 - /unlink 1277 (time 1787501985.36 total 10.00 last 127.66)
 - /unlink 2365 (time 1787501995.36 total 20.01 last 108.75)
 - /unlink 3519 (time 1787502005.37 total 30.01 last 115.34)
total: 4500 /unlink in 38.08 seconds: 118.18 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95298 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg430-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg430-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server
stop mds service on oleg430-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server
stop mds service on oleg430-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg430-server

unloading modules via unload_modules_local on: 'oleg430-server'
oleg430-server: oleg430-server.virtnet: executing unload_modules_local
oleg430-server: 
modules unloaded.