== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:08:35 (1768558115)
Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg242-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg242-server
oleg242-server: oleg242-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg242-server'
oleg242-server: oleg242-server.virtnet: executing load_modules_local
oleg242-server: Loading modules from /home/green/git/lustre-release/lustre
oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg242-server: detected 4 online CPUs by sysfs
oleg242-server: MODOPTS_LIBCFS=
oleg242-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg242-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all
pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg242-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all
pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg242-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all
pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg242-client.virtnet:  -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 657 (time 1768558309.34 total 10.02 last 65.57)
 - open/close 1348 (time 1768558319.34 total 20.02 last 69.09)
 - open/close 2124 (time 1768558329.35 total 30.03 last 77.53)
 - open/close 2801 (time 1768558339.35 total 40.04 last 67.65)
 - open/close 3742 (time 1768558349.36 total 50.04 last 94.08)
total: 4500 open/close in 59.49 seconds: 75.64 ops/second
 - /unlink 1334 (time 1768558372.55 total 10.00 last 133.36)
 - /unlink 2677 (time 1768558382.56 total 20.01 last 134.14)
 - /unlink 4020 (time 1768558392.57 total 30.02 last 134.26)
total: 4500 /unlink in 32.50 seconds: 138.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 690 (time 1768558409.66 total 10.03 last 68.77)
 - open/close 1408 (time 1768558419.67 total 20.04 last 71.75)
 - open/close 2126 (time 1768558429.67 total 30.04 last 71.77)
 - open/close 2801 (time 1768558439.67 total 40.05 last 67.49)
 - open/close 3524 (time 1768558449.68 total 50.05 last 72.26)
 - open/close 4067 (time 1768558459.68 total 60.05 last 54.30)
total: 4500 open/close in 65.87 seconds: 68.32 ops/second
 - /unlink 1116 (time 1768558480.66 total 10.00 last 111.55)
 - /unlink 2393 (time 1768558490.67 total 20.01 last 127.63)
 - /unlink 3742 (time 1768558500.67 total 30.01 last 134.86)
total: 4500 /unlink in 36.47 seconds: 123.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 624 (time 1768558521.55 total 10.01 last 62.31)
 - open/close 1208 (time 1768558531.56 total 20.02 last 58.36)
 - open/close 1882 (time 1768558541.57 total 30.03 last 67.31)
 - open/close 2416 (time 1768558551.58 total 40.04 last 53.34)
 - open/close 3092 (time 1768558561.61 total 50.07 last 67.41)
 - open/close 3736 (time 1768558571.61 total 60.07 last 64.40)
 - open/close 4293 (time 1768558581.62 total 70.08 last 55.65)
total: 4500 open/close in 73.79 seconds: 60.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 940 (time 1768558602.91 total 10.00 last 93.98)
 - /unlink 1917 (time 1768558612.92 total 20.01 last 97.64)
 - /unlink 3177 (time 1768558622.92 total 30.01 last 125.96)
 - /unlink 4132 (time 1768558632.92 total 40.01 last 95.49)
total: 4500 /unlink in 43.49 seconds: 103.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 547 (time 1768558654.32 total 10.00 last 54.68)
 - open/close 1064 (time 1768558664.33 total 20.01 last 51.67)
 - open/close 1745 (time 1768558674.34 total 30.02 last 68.03)
 - open/close 2435 (time 1768558684.34 total 40.02 last 68.99)
 - open/close 3204 (time 1768558694.35 total 50.03 last 76.84)
 - open/close 3948 (time 1768558704.35 total 60.04 last 74.34)
total: 4500 open/close in 69.09 seconds: 65.13 ops/second
 - /unlink 1011 (time 1768558728.48 total 10.01 last 100.99)
 - /unlink 2106 (time 1768558738.49 total 20.02 last 109.43)
 - /unlink 3393 (time 1768558748.49 total 30.02 last 128.64)
total: 4500 /unlink in 39.70 seconds: 113.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 768 (time 1768558773.10 total 10.01 last 76.75)
 - open/close 1529 (time 1768558783.10 total 20.01 last 76.05)
 - open/close 2202 (time 1768558793.10 total 30.02 last 67.29)
 - open/close 2852 (time 1768558803.11 total 40.02 last 64.94)
 - open/close 3370 (time 1768558813.13 total 50.04 last 51.72)
 - open/close 3877 (time 1768558823.14 total 60.05 last 50.64)
 - open/close 4475 (time 1768558833.17 total 70.09 last 59.59)
total: 4500 open/close in 70.48 seconds: 63.85 ops/second
 - /unlink 1284 (time 1768558849.93 total 10.00 last 128.37)
 - /unlink 2578 (time 1768558859.93 total 20.00 last 129.40)
 - /unlink 3861 (time 1768558869.94 total 30.02 last 128.12)
total: 4500 /unlink in 35.45 seconds: 126.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 593 (time 1768558889.90 total 10.00 last 59.27)
 - open/close 1245 (time 1768558899.91 total 20.01 last 65.15)
 - open/close 1941 (time 1768558909.92 total 30.02 last 69.53)
 - open/close 2612 (time 1768558919.93 total 40.03 last 67.03)
 - open/close 3316 (time 1768558929.93 total 50.04 last 70.38)
 - open/close 3970 (time 1768558939.93 total 60.04 last 65.38)
total: 4500 open/close in 68.99 seconds: 65.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1019 (time 1768558964.98 total 10.02 last 101.66)
 - /unlink 2106 (time 1768558974.99 total 20.03 last 108.60)
 - /unlink 3339 (time 1768558984.99 total 30.04 last 123.25)
total: 4500 /unlink in 39.59 seconds: 113.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 628 (time 1768559010.23 total 10.02 last 62.67)
 - open/close 1225 (time 1768559020.24 total 20.03 last 59.66)
 - open/close 1918 (time 1768559030.24 total 30.03 last 69.30)
 - open/close 2547 (time 1768559040.24 total 40.03 last 62.88)
 - open/close 3257 (time 1768559050.24 total 50.03 last 70.98)
 - open/close 3874 (time 1768559060.25 total 60.04 last 61.64)
total: 4500 open/close in 69.35 seconds: 64.89 ops/second
 - /unlink 1037 (time 1768559084.92 total 10.01 last 103.64)
 - /unlink 2407 (time 1768559094.92 total 20.01 last 137.00)
 - /unlink 3613 (time 1768559104.93 total 30.02 last 120.47)
total: 4500 /unlink in 37.40 seconds: 120.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 671 (time 1768559128.00 total 10.01 last 67.05)
 - open/close 1365 (time 1768559138.03 total 20.03 last 69.25)
 - open/close 2196 (time 1768559148.03 total 30.03 last 83.08)
 - open/close 2931 (time 1768559158.03 total 40.04 last 73.46)
 - open/close 3526 (time 1768559168.03 total 50.04 last 59.50)
 - open/close 3977 (time 1768559178.03 total 60.04 last 45.09)
 - open/close 4452 (time 1768559188.08 total 70.08 last 47.30)
total: 4500 open/close in 71.28 seconds: 63.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1301 (time 1768559204.01 total 10.00 last 130.07)
 - /unlink 2277 (time 1768559214.02 total 20.01 last 97.55)
 - /unlink 3545 (time 1768559224.02 total 30.01 last 126.75)
total: 4500 /unlink in 38.29 seconds: 117.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 647 (time 1768559248.61 total 10.01 last 64.63)
 - open/close 1350 (time 1768559258.61 total 20.01 last 70.28)
 - open/close 1888 (time 1768559268.64 total 30.04 last 53.66)
 - open/close 2493 (time 1768559278.64 total 40.04 last 60.49)
 - open/close 3118 (time 1768559288.64 total 50.05 last 62.46)
 - open/close 3718 (time 1768559298.65 total 60.05 last 59.99)
 - open/close 4412 (time 1768559308.65 total 70.06 last 69.36)
total: 4500 open/close in 71.51 seconds: 62.93 ops/second
 - /unlink 1188 (time 1768559324.76 total 10.02 last 118.53)
 - /unlink 2480 (time 1768559334.76 total 20.02 last 129.19)
 - /unlink 3643 (time 1768559344.76 total 30.03 last 116.26)
total: 4500 /unlink in 37.45 seconds: 120.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 693 (time 1768559366.86 total 10.00 last 69.28)
 - open/close 1476 (time 1768559376.86 total 20.01 last 78.25)
 - open/close 2196 (time 1768559386.87 total 30.02 last 71.93)
 - open/close 2811 (time 1768559396.89 total 40.04 last 61.39)
 - open/close 3412 (time 1768559406.90 total 50.05 last 60.06)
 - open/close 4114 (time 1768559416.90 total 60.05 last 70.16)
total: 4500 open/close in 65.84 seconds: 68.35 ops/second
 - /unlink 1293 (time 1768559436.81 total 10.00 last 129.27)
 - /unlink 2383 (time 1768559446.81 total 20.00 last 108.99)
 - /unlink 3598 (time 1768559456.81 total 30.01 last 121.47)
total: 4500 /unlink in 37.48 seconds: 120.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 821 (time 1768559479.58 total 10.01 last 82.02)
 - open/close 1576 (time 1768559489.60 total 20.02 last 75.41)
 - open/close 2190 (time 1768559499.61 total 30.03 last 61.33)
 - open/close 2866 (time 1768559509.61 total 40.04 last 67.57)
 - open/close 3504 (time 1768559519.61 total 50.04 last 63.79)
 - open/close 4236 (time 1768559529.65 total 60.08 last 72.91)
total: 4500 open/close in 64.00 seconds: 70.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1187 (time 1768559547.64 total 10.01 last 118.64)
 - /unlink 2469 (time 1768559557.65 total 20.01 last 128.12)
 - /unlink 3691 (time 1768559567.65 total 30.02 last 122.15)
total: 4500 /unlink in 36.90 seconds: 121.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 593 (time 1768559589.24 total 10.02 last 59.21)
 - open/close 1120 (time 1768559599.26 total 20.04 last 52.58)
 - open/close 1702 (time 1768559609.27 total 30.05 last 58.14)
 - open/close 2393 (time 1768559619.28 total 40.05 last 69.07)
 - open/close 3086 (time 1768559629.29 total 50.06 last 69.23)
 - open/close 3677 (time 1768559639.31 total 60.09 last 58.95)
 - open/close 4436 (time 1768559649.33 total 70.10 last 75.79)
total: 4500 open/close in 71.38 seconds: 63.04 ops/second
 - /unlink 1070 (time 1768559665.43 total 10.01 last 106.93)
 - /unlink 2172 (time 1768559675.44 total 20.01 last 110.15)
 - /unlink 3328 (time 1768559685.44 total 30.01 last 115.59)
 - /unlink 4482 (time 1768559695.44 total 40.01 last 115.37)
total: 4500 /unlink in 40.19 seconds: 111.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 649 (time 1768559710.46 total 10.02 last 64.76)
 - open/close 1277 (time 1768559720.47 total 20.02 last 62.80)
 - open/close 1870 (time 1768559730.48 total 30.03 last 59.23)
 - open/close 2554 (time 1768559740.48 total 40.04 last 68.38)
 - open/close 3136 (time 1768559750.49 total 50.05 last 58.14)
 - open/close 3762 (time 1768559760.49 total 60.05 last 62.60)
 - open/close 4293 (time 1768559770.51 total 70.07 last 53.01)
total: 4500 open/close in 73.05 seconds: 61.60 ops/second
 - /unlink 1182 (time 1768559789.22 total 10.00 last 118.18)
 - /unlink 2424 (time 1768559799.22 total 20.00 last 124.18)
 - /unlink 3628 (time 1768559809.22 total 30.01 last 120.36)
total: 4500 /unlink in 37.10 seconds: 121.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 757 (time 1768559832.14 total 10.01 last 75.62)
 - open/close 1361 (time 1768559842.15 total 20.02 last 60.35)
 - open/close 1970 (time 1768559852.16 total 30.02 last 60.86)
 - open/close 2631 (time 1768559862.16 total 40.03 last 66.05)
 - open/close 3296 (time 1768559872.18 total 50.05 last 66.41)
 - open/close 3878 (time 1768559882.18 total 60.05 last 58.17)
 - open/close 4455 (time 1768559892.21 total 70.08 last 57.52)
total: 4500 open/close in 70.81 seconds: 63.55 ops/second
 - /unlink 1331 (time 1768559908.22 total 10.00 last 133.09)
 - /unlink 2358 (time 1768559918.23 total 20.01 last 102.57)
 - /unlink 3471 (time 1768559928.23 total 30.02 last 111.24)
total: 4500 /unlink in 37.80 seconds: 119.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 599 (time 1768559952.34 total 10.01 last 59.86)
 - open/close 1099 (time 1768559962.36 total 20.03 last 49.89)
 - open/close 1697 (time 1768559972.36 total 30.03 last 59.78)
 - open/close 2395 (time 1768559982.37 total 40.04 last 69.73)
 - open/close 3065 (time 1768559992.38 total 50.05 last 66.91)
 - open/close 3694 (time 1768560002.39 total 60.06 last 62.84)
 - open/close 4384 (time 1768560012.41 total 70.08 last 68.90)
total: 4500 open/close in 71.44 seconds: 62.99 ops/second
 - /unlink 1313 (time 1768560029.66 total 10.01 last 131.19)
 - /unlink 2529 (time 1768560039.67 total 20.01 last 121.57)
 - /unlink 3899 (time 1768560049.67 total 30.02 last 136.94)
total: 4500 /unlink in 34.52 seconds: 130.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 656 (time 1768560069.67 total 10.01 last 65.53)
 - open/close 1238 (time 1768560079.68 total 20.02 last 58.13)
 - open/close 1918 (time 1768560089.70 total 30.04 last 67.87)
 - open/close 2665 (time 1768560099.70 total 40.04 last 74.70)
 - open/close 3422 (time 1768560109.70 total 50.04 last 75.68)
 - open/close 4055 (time 1768560119.71 total 60.05 last 63.25)
total: 4500 open/close in 66.79 seconds: 67.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1278 (time 1768560141.44 total 10.00 last 127.78)
 - /unlink 2569 (time 1768560151.44 total 20.00 last 129.07)
 - /unlink 3676 (time 1768560161.45 total 30.01 last 110.68)
total: 4500 /unlink in 36.50 seconds: 123.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 695 (time 1768560182.84 total 10.00 last 69.48)
 - open/close 1369 (time 1768560192.85 total 20.02 last 67.30)
 - open/close 2109 (time 1768560202.86 total 30.02 last 73.98)
 - open/close 2754 (time 1768560212.86 total 40.02 last 64.49)
 - open/close 3389 (time 1768560222.87 total 50.04 last 63.39)
 - open/close 4022 (time 1768560232.89 total 60.06 last 63.18)
total: 4500 open/close in 68.76 seconds: 65.45 ops/second
 - /unlink 1311 (time 1768560256.76 total 10.00 last 131.07)
 - /unlink 2799 (time 1768560266.76 total 20.01 last 148.73)
 - /unlink 4122 (time 1768560276.77 total 30.01 last 132.25)
total: 4500 /unlink in 32.57 seconds: 138.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 774 (time 1768560294.21 total 10.00 last 77.38)
 - open/close 1461 (time 1768560304.22 total 20.01 last 68.63)
 - open/close 2052 (time 1768560314.24 total 30.03 last 59.00)
 - open/close 2812 (time 1768560324.25 total 40.04 last 75.95)
 - open/close 3650 (time 1768560334.26 total 50.04 last 83.74)
 - open/close 4314 (time 1768560344.26 total 60.05 last 66.38)
total: 4500 open/close in 63.05 seconds: 71.38 ops/second
 - /unlink 1476 (time 1768560362.66 total 10.00 last 147.57)
 - /unlink 2747 (time 1768560372.70 total 20.04 last 126.66)
 - /unlink 4116 (time 1768560382.70 total 30.04 last 136.89)
total: 4500 /unlink in 32.58 seconds: 138.13 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11101: 94202 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg242-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg242-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server
stop mds service on oleg242-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server
stop mds service on oleg242-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg242-server'
oleg242-server: oleg242-server.virtnet: executing unload_modules_local
oleg242-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg242-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.