== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:11:24 (1787857884)
Stopping clients: oleg624-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg624-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg624-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg624-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg624-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg624-server
oleg624-server: oleg624-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg624-server'
oleg624-server: oleg624-server.virtnet: executing load_modules_local
oleg624-server: Loading modules from /home/green/git/lustre-release/lustre
oleg624-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg624-server: detected 4 online CPUs by sysfs
oleg624-server: MODOPTS_LIBCFS=
oleg624-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg624-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all
pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg624-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all
pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg624-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all
pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg624-client.virtnet:  -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 713 (time 1787858093.85 total 10.01 last 71.21)
 - open/close 1475 (time 1787858103.86 total 20.02 last 76.11)
 - open/close 2304 (time 1787858113.87 total 30.03 last 82.83)
 - open/close 3053 (time 1787858123.87 total 40.03 last 74.90)
 - open/close 3807 (time 1787858133.87 total 50.04 last 75.39)
total: 4500 open/close in 59.53 seconds: 75.59 ops/second
 - /unlink 1451 (time 1787858157.24 total 10.00 last 145.06)
 - /unlink 2826 (time 1787858167.24 total 20.00 last 137.49)
 - /unlink 4215 (time 1787858177.24 total 30.00 last 138.88)
total: 4500 /unlink in 31.85 seconds: 141.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 761 (time 1787858193.34 total 10.01 last 76.03)
 - open/close 1435 (time 1787858203.34 total 20.01 last 67.40)
 - open/close 2173 (time 1787858213.35 total 30.02 last 73.74)
 - open/close 2965 (time 1787858223.35 total 40.02 last 79.19)
 - open/close 3738 (time 1787858233.36 total 50.03 last 77.20)
total: 4500 open/close in 59.81 seconds: 75.24 ops/second
 - /unlink 1529 (time 1787858257.11 total 10.00 last 152.88)
 - /unlink 2787 (time 1787858267.12 total 20.01 last 125.70)
 - /unlink 4042 (time 1787858277.12 total 30.01 last 125.44)
total: 4500 /unlink in 33.71 seconds: 133.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 729 (time 1787858295.86 total 10.00 last 72.88)
 - open/close 1414 (time 1787858305.88 total 20.02 last 68.41)
 - open/close 2208 (time 1787858315.88 total 30.02 last 79.33)
 - open/close 2984 (time 1787858325.90 total 40.04 last 77.51)
 - open/close 3763 (time 1787858335.90 total 50.04 last 77.89)
 - open/close 4454 (time 1787858345.91 total 60.05 last 69.02)
total: 4500 open/close in 60.75 seconds: 74.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1475 (time 1787858361.13 total 10.01 last 147.35)
 - /unlink 2787 (time 1787858371.14 total 20.01 last 131.18)
 - /unlink 4226 (time 1787858381.14 total 30.02 last 143.85)
total: 4500 /unlink in 32.06 seconds: 140.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 786 (time 1787858398.06 total 10.01 last 78.53)
 - open/close 1537 (time 1787858408.07 total 20.01 last 75.08)
 - open/close 2281 (time 1787858418.07 total 30.01 last 74.39)
 - open/close 2964 (time 1787858428.07 total 40.01 last 68.29)
 - open/close 3680 (time 1787858438.08 total 50.02 last 71.55)
 - open/close 4176 (time 1787858448.11 total 60.05 last 49.44)
total: 4500 open/close in 66.47 seconds: 67.70 ops/second
 - /unlink 1262 (time 1787858470.04 total 10.00 last 126.15)
 - /unlink 2544 (time 1787858480.05 total 20.01 last 128.09)
 - /unlink 3845 (time 1787858490.05 total 30.01 last 130.09)
total: 4500 /unlink in 35.76 seconds: 125.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 642 (time 1787858510.87 total 10.01 last 64.16)
 - open/close 1246 (time 1787858520.88 total 20.02 last 60.34)
 - open/close 1936 (time 1787858530.90 total 30.03 last 68.89)
 - open/close 2659 (time 1787858540.91 total 40.05 last 72.18)
 - open/close 3390 (time 1787858550.92 total 50.05 last 73.08)
 - open/close 4074 (time 1787858560.92 total 60.06 last 68.37)
total: 4500 open/close in 66.41 seconds: 67.76 ops/second
 - /unlink 1370 (time 1787858581.76 total 10.00 last 136.99)
 - /unlink 2768 (time 1787858591.77 total 20.01 last 139.74)
 - /unlink 4094 (time 1787858601.77 total 30.01 last 132.55)
total: 4500 /unlink in 33.19 seconds: 135.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 661 (time 1787858620.40 total 10.01 last 66.05)
 - open/close 1390 (time 1787858630.41 total 20.01 last 72.86)
 - open/close 2138 (time 1787858640.41 total 30.01 last 74.79)
 - open/close 2827 (time 1787858650.43 total 40.03 last 68.76)
 - open/close 3535 (time 1787858660.43 total 50.04 last 70.79)
 - open/close 4244 (time 1787858670.44 total 60.04 last 70.86)
total: 4500 open/close in 63.34 seconds: 71.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1409 (time 1787858688.02 total 10.00 last 140.86)
 - /unlink 2762 (time 1787858698.02 total 20.01 last 135.27)
 - /unlink 4185 (time 1787858708.03 total 30.02 last 142.08)
total: 4500 /unlink in 32.52 seconds: 138.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 707 (time 1787858726.22 total 10.01 last 70.66)
 - open/close 1468 (time 1787858736.23 total 20.01 last 76.03)
 - open/close 2244 (time 1787858746.23 total 30.02 last 77.56)
 - open/close 2990 (time 1787858756.24 total 40.02 last 74.57)
 - open/close 3739 (time 1787858766.25 total 50.04 last 74.80)
 - open/close 4479 (time 1787858776.30 total 60.08 last 73.66)
total: 4500 open/close in 60.57 seconds: 74.30 ops/second
 - /unlink 1280 (time 1787858791.57 total 10.00 last 127.99)
 - /unlink 2383 (time 1787858801.58 total 20.01 last 110.21)
 - /unlink 3651 (time 1787858811.58 total 30.01 last 126.77)
total: 4500 /unlink in 36.47 seconds: 123.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 657 (time 1787858833.07 total 10.03 last 65.53)
 - open/close 1324 (time 1787858843.10 total 20.05 last 66.56)
 - open/close 2067 (time 1787858853.10 total 30.05 last 74.29)
 - open/close 2830 (time 1787858863.10 total 40.05 last 76.28)
 - open/close 3551 (time 1787858873.10 total 50.05 last 72.08)
 - open/close 4268 (time 1787858883.12 total 60.07 last 71.60)
total: 4500 open/close in 63.17 seconds: 71.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1348 (time 1787858900.44 total 10.00 last 134.76)
 - /unlink 2750 (time 1787858910.44 total 20.01 last 140.14)
 - /unlink 4008 (time 1787858920.45 total 30.01 last 125.76)
total: 4500 /unlink in 33.94 seconds: 132.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 767 (time 1787858939.13 total 10.00 last 76.70)
 - open/close 1483 (time 1787858949.14 total 20.01 last 71.51)
 - open/close 2207 (time 1787858959.17 total 30.04 last 72.19)
 - open/close 2912 (time 1787858969.17 total 40.05 last 70.47)
 - open/close 3641 (time 1787858979.18 total 50.06 last 72.84)
 - open/close 4426 (time 1787858989.18 total 60.06 last 78.49)
total: 4500 open/close in 61.00 seconds: 73.78 ops/second
 - /unlink 1393 (time 1787859004.01 total 10.00 last 139.28)
 - /unlink 2745 (time 1787859014.01 total 20.00 last 135.19)
 - /unlink 4246 (time 1787859024.02 total 30.01 last 149.92)
total: 4500 /unlink in 32.22 seconds: 139.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 702 (time 1787859040.81 total 10.00 last 70.20)
 - open/close 1417 (time 1787859050.82 total 20.01 last 71.47)
 - open/close 2197 (time 1787859060.84 total 30.03 last 77.83)
 - open/close 2916 (time 1787859070.84 total 40.03 last 71.88)
 - open/close 3648 (time 1787859080.86 total 50.05 last 73.07)
 - open/close 4416 (time 1787859090.86 total 60.05 last 76.79)
total: 4500 open/close in 61.06 seconds: 73.70 ops/second
 - /unlink 1343 (time 1787859105.86 total 10.00 last 134.24)
 - /unlink 2696 (time 1787859115.86 total 20.01 last 135.25)
 - /unlink 4101 (time 1787859125.87 total 30.02 last 140.39)
total: 4500 /unlink in 32.84 seconds: 137.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 749 (time 1787859143.27 total 10.03 last 74.69)
 - open/close 1403 (time 1787859153.30 total 20.06 last 65.20)
 - open/close 2112 (time 1787859163.31 total 30.08 last 70.78)
 - open/close 2869 (time 1787859173.32 total 40.08 last 75.64)
 - open/close 3525 (time 1787859183.33 total 50.09 last 65.57)
 - open/close 4244 (time 1787859193.33 total 60.09 last 71.88)
total: 4500 open/close in 64.20 seconds: 70.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1262 (time 1787859211.90 total 10.00 last 126.16)
 - /unlink 2383 (time 1787859221.91 total 20.01 last 112.07)
 - /unlink 3430 (time 1787859231.91 total 30.01 last 104.68)
 - /unlink 4395 (time 1787859241.91 total 40.01 last 96.49)
total: 4500 /unlink in 41.01 seconds: 109.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 556 (time 1787859257.89 total 10.00 last 55.58)
 - open/close 1247 (time 1787859267.91 total 20.03 last 68.94)
 - open/close 1962 (time 1787859277.92 total 30.03 last 71.49)
 - open/close 2502 (time 1787859287.95 total 40.06 last 53.82)
 - open/close 3136 (time 1787859297.96 total 50.07 last 63.35)
 - open/close 3813 (time 1787859307.96 total 60.07 last 67.69)
 - open/close 4451 (time 1787859317.97 total 70.09 last 63.69)
total: 4500 open/close in 70.87 seconds: 63.50 ops/second
 - /unlink 1219 (time 1787859333.87 total 10.00 last 121.85)
 - /unlink 2388 (time 1787859343.87 total 20.00 last 116.90)
 - /unlink 3499 (time 1787859353.87 total 30.01 last 111.06)
total: 4500 /unlink in 37.03 seconds: 121.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 623 (time 1787859375.89 total 10.00 last 62.28)
 - open/close 1281 (time 1787859385.89 total 20.01 last 65.75)
 - open/close 1906 (time 1787859395.90 total 30.02 last 62.47)
 - open/close 2565 (time 1787859405.91 total 40.03 last 65.82)
 - open/close 3178 (time 1787859415.92 total 50.03 last 61.27)
 - open/close 3747 (time 1787859425.93 total 60.05 last 56.81)
 - open/close 4305 (time 1787859435.95 total 70.07 last 55.70)
total: 4500 open/close in 72.70 seconds: 61.90 ops/second
 - /unlink 1243 (time 1787859453.70 total 10.00 last 124.27)
 - /unlink 2423 (time 1787859463.71 total 20.02 last 117.79)
 - /unlink 3574 (time 1787859473.71 total 30.02 last 115.09)
total: 4500 /unlink in 38.53 seconds: 116.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 632 (time 1787859497.03 total 10.01 last 63.14)
 - open/close 1217 (time 1787859507.03 total 20.01 last 58.48)
 - open/close 1858 (time 1787859517.04 total 30.02 last 64.05)
 - open/close 2385 (time 1787859527.05 total 40.03 last 52.63)
 - open/close 2918 (time 1787859537.07 total 50.05 last 53.24)
 - open/close 3412 (time 1787859547.08 total 60.06 last 49.34)
 - open/close 3963 (time 1787859557.11 total 70.09 last 54.91)
 - open/close 4451 (time 1787859567.12 total 80.10 last 48.76)
total: 4500 open/close in 80.72 seconds: 55.75 ops/second
 - /unlink 1126 (time 1787859583.04 total 10.01 last 112.51)
 - /unlink 2294 (time 1787859593.04 total 20.01 last 116.78)
 - /unlink 3212 (time 1787859603.05 total 30.02 last 91.75)
 - /unlink 4280 (time 1787859613.06 total 40.03 last 106.63)
total: 4500 /unlink in 42.10 seconds: 106.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 583 (time 1787859630.42 total 10.01 last 58.25)
 - open/close 1189 (time 1787859640.44 total 20.02 last 60.52)
 - open/close 1865 (time 1787859650.44 total 30.02 last 67.59)
 - open/close 2540 (time 1787859660.45 total 40.03 last 67.44)
 - open/close 3238 (time 1787859670.46 total 50.04 last 69.72)
 - open/close 3832 (time 1787859680.51 total 60.10 last 59.08)
 - open/close 4473 (time 1787859690.53 total 70.11 last 64.00)
total: 4500 open/close in 70.46 seconds: 63.86 ops/second
 - /unlink 1281 (time 1787859705.66 total 10.01 last 127.99)
 - /unlink 2492 (time 1787859715.66 total 20.01 last 121.08)
 - /unlink 3738 (time 1787859725.66 total 30.01 last 124.59)
total: 4500 /unlink in 36.22 seconds: 124.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 660 (time 1787859746.26 total 10.04 last 65.74)
 - open/close 1284 (time 1787859756.27 total 20.05 last 62.33)
 - open/close 1877 (time 1787859766.28 total 30.05 last 59.28)
 - open/close 2484 (time 1787859776.29 total 40.07 last 60.63)
 - open/close 3066 (time 1787859786.32 total 50.10 last 58.01)
 - open/close 3684 (time 1787859796.33 total 60.11 last 61.76)
 - open/close 4249 (time 1787859806.34 total 70.11 last 56.45)
total: 4500 open/close in 73.93 seconds: 60.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1230 (time 1787859825.10 total 10.03 last 122.68)
 - /unlink 2433 (time 1787859835.11 total 20.03 last 120.26)
 - /unlink 3671 (time 1787859845.11 total 30.04 last 123.71)
total: 4500 /unlink in 37.60 seconds: 119.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 657 (time 1787859867.28 total 10.01 last 65.66)
 - open/close 1311 (time 1787859877.29 total 20.02 last 65.29)
 - open/close 1993 (time 1787859887.31 total 30.04 last 68.08)
 - open/close 2712 (time 1787859897.32 total 40.05 last 71.85)
 - open/close 3376 (time 1787859907.33 total 50.06 last 66.31)
 - open/close 3962 (time 1787859917.34 total 60.07 last 58.53)
total: 4500 open/close in 67.78 seconds: 66.39 ops/second
 - /unlink 1343 (time 1787859939.82 total 10.00 last 134.26)
 - /unlink 2559 (time 1787859949.82 total 20.00 last 121.60)
 - /unlink 3821 (time 1787859959.83 total 30.01 last 126.09)
total: 4500 /unlink in 35.87 seconds: 125.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 679 (time 1787859981.06 total 10.01 last 67.86)
 - open/close 1254 (time 1787859991.08 total 20.02 last 57.40)
 - open/close 1932 (time 1787860001.08 total 30.02 last 67.80)
 - open/close 2595 (time 1787860011.08 total 40.03 last 66.27)
 - open/close 3287 (time 1787860021.09 total 50.04 last 69.12)
 - open/close 3994 (time 1787860031.10 total 60.05 last 70.66)
total: 4500 open/close in 66.55 seconds: 67.62 ops/second
 - /unlink 1418 (time 1787860051.14 total 10.01 last 141.72)
 - /unlink 2657 (time 1787860061.14 total 20.01 last 123.87)
 - /unlink 4012 (time 1787860071.15 total 30.01 last 135.46)
total: 4500 /unlink in 33.62 seconds: 133.84 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95206 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg624-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg624-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server
stop mds service on oleg624-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg624-server
stop mds service on oleg624-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg624-server

unloading modules via unload_modules_local on: 'oleg624-server'
oleg624-server: oleg624-server.virtnet: executing unload_modules_local
oleg624-server: 
modules unloaded.