== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:49:20 (1786736960)
Stopping clients: oleg607-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg607-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg607-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg607-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg607-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg607-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg607-server
oleg607-server: oleg607-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg607-server'
oleg607-server: oleg607-server.virtnet: executing load_modules_local
oleg607-server: Loading modules from /home/green/git/lustre-release/lustre
oleg607-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg607-server: detected 4 online CPUs by sysfs
oleg607-server: MODOPTS_LIBCFS=
oleg607-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg607-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all
pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg607-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all
pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg607-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all
pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg607-client.virtnet:  -o user_xattr,flock 192.168.206.107@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 706 (time 1786737156.70 total 10.01 last 70.54)
 - open/close 1387 (time 1786737166.71 total 20.02 last 68.03)
 - open/close 2051 (time 1786737176.72 total 30.02 last 66.37)
 - open/close 2749 (time 1786737186.72 total 40.02 last 69.79)
 - open/close 3469 (time 1786737196.74 total 50.04 last 71.87)
 - open/close 4156 (time 1786737206.75 total 60.06 last 68.57)
total: 4500 open/close in 65.05 seconds: 69.18 ops/second
 - /unlink 1311 (time 1786737225.93 total 10.00 last 131.09)
 - /unlink 2804 (time 1786737235.94 total 20.01 last 149.16)
 - /unlink 4280 (time 1786737245.94 total 30.01 last 147.57)
total: 4500 /unlink in 32.07 seconds: 140.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 783 (time 1786737262.70 total 10.02 last 78.17)
 - open/close 1594 (time 1786737272.72 total 20.03 last 80.97)
 - open/close 2351 (time 1786737282.74 total 30.05 last 75.57)
 - open/close 3074 (time 1786737292.74 total 40.05 last 72.29)
 - open/close 3760 (time 1786737302.75 total 50.06 last 68.54)
 - open/close 4442 (time 1786737312.76 total 60.07 last 68.15)
total: 4500 open/close in 60.84 seconds: 73.96 ops/second
 - /unlink 1459 (time 1786737328.20 total 10.00 last 145.85)
 - /unlink 2691 (time 1786737338.21 total 20.01 last 123.17)
 - /unlink 3990 (time 1786737348.21 total 30.01 last 129.88)
total: 4500 /unlink in 34.23 seconds: 131.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 762 (time 1786737367.07 total 10.01 last 76.11)
 - open/close 1608 (time 1786737377.07 total 20.02 last 84.57)
 - open/close 2403 (time 1786737387.09 total 30.04 last 79.34)
 - open/close 3182 (time 1786737397.11 total 40.05 last 77.77)
 - open/close 3937 (time 1786737407.12 total 50.06 last 75.42)
total: 4500 open/close in 57.79 seconds: 77.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1373 (time 1786737428.42 total 10.00 last 137.30)
 - /unlink 2707 (time 1786737438.42 total 20.00 last 133.34)
 - /unlink 4055 (time 1786737448.43 total 30.01 last 134.71)
total: 4500 /unlink in 33.31 seconds: 135.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 646 (time 1786737465.96 total 10.01 last 64.54)
 - open/close 1385 (time 1786737475.97 total 20.02 last 73.83)
 - open/close 2141 (time 1786737485.98 total 30.02 last 75.57)
 - open/close 2879 (time 1786737495.98 total 40.03 last 73.77)
 - open/close 3614 (time 1786737505.99 total 50.03 last 73.46)
 - open/close 4386 (time 1786737515.99 total 60.04 last 77.16)
total: 4500 open/close in 61.69 seconds: 72.94 ops/second
 - /unlink 1335 (time 1786737531.67 total 10.00 last 133.50)
 - /unlink 2567 (time 1786737541.68 total 20.01 last 123.12)
 - /unlink 3955 (time 1786737551.68 total 30.01 last 138.79)
total: 4500 /unlink in 34.37 seconds: 130.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 733 (time 1786737570.86 total 10.02 last 73.18)
 - open/close 1452 (time 1786737580.87 total 20.03 last 71.77)
 - open/close 2147 (time 1786737590.87 total 30.03 last 69.50)
 - open/close 2849 (time 1786737600.88 total 40.04 last 70.19)
 - open/close 3471 (time 1786737610.90 total 50.06 last 62.03)
 - open/close 4180 (time 1786737620.90 total 60.06 last 70.89)
total: 4500 open/close in 64.26 seconds: 70.03 ops/second
 - /unlink 1383 (time 1786737639.12 total 10.01 last 138.15)
 - /unlink 2522 (time 1786737649.13 total 20.02 last 113.85)
 - /unlink 3738 (time 1786737659.13 total 30.02 last 121.55)
total: 4500 /unlink in 35.85 seconds: 125.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 765 (time 1786737679.54 total 10.01 last 76.46)
 - open/close 1516 (time 1786737689.55 total 20.02 last 75.00)
 - open/close 2294 (time 1786737699.56 total 30.03 last 77.74)
 - open/close 3089 (time 1786737709.56 total 40.03 last 79.48)
 - open/close 3847 (time 1786737719.57 total 50.04 last 75.72)
 - open/close 4479 (time 1786737729.59 total 60.06 last 63.07)
total: 4500 open/close in 60.45 seconds: 74.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1409 (time 1786737744.28 total 10.00 last 140.87)
 - /unlink 2749 (time 1786737754.29 total 20.02 last 133.80)
 - /unlink 4094 (time 1786737764.29 total 30.02 last 134.49)
total: 4500 /unlink in 33.19 seconds: 135.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 728 (time 1786737782.03 total 10.02 last 72.65)
 - open/close 1445 (time 1786737792.04 total 20.02 last 71.67)
 - open/close 2086 (time 1786737802.04 total 30.03 last 64.06)
 - open/close 2694 (time 1786737812.05 total 40.04 last 60.73)
 - open/close 3282 (time 1786737822.06 total 50.05 last 58.75)
 - open/close 3902 (time 1786737832.08 total 60.07 last 61.89)
 - open/close 4426 (time 1786737842.09 total 70.07 last 52.37)
total: 4500 open/close in 71.22 seconds: 63.18 ops/second
 - /unlink 1187 (time 1786737859.12 total 10.00 last 118.67)
 - /unlink 2282 (time 1786737869.12 total 20.00 last 109.49)
 - /unlink 3395 (time 1786737879.13 total 30.01 last 111.23)
total: 4500 /unlink in 39.67 seconds: 113.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 568 (time 1786737904.67 total 10.01 last 56.76)
 - open/close 1123 (time 1786737914.68 total 20.02 last 55.45)
 - open/close 1807 (time 1786737924.69 total 30.03 last 68.32)
 - open/close 2505 (time 1786737934.70 total 40.03 last 69.76)
 - open/close 3276 (time 1786737944.71 total 50.05 last 77.01)
 - open/close 4018 (time 1786737954.73 total 60.06 last 74.10)
total: 4500 open/close in 67.21 seconds: 66.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1287 (time 1786737976.31 total 10.01 last 128.62)
 - /unlink 2644 (time 1786737986.31 total 20.01 last 135.66)
 - /unlink 3897 (time 1786737996.32 total 30.02 last 125.19)
total: 4500 /unlink in 35.15 seconds: 128.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 818 (time 1786738017.35 total 10.01 last 81.70)
 - open/close 1500 (time 1786738027.36 total 20.02 last 68.17)
 - open/close 2104 (time 1786738037.38 total 30.04 last 60.25)
 - open/close 2786 (time 1786738047.38 total 40.04 last 68.20)
 - open/close 3531 (time 1786738057.39 total 50.05 last 74.46)
 - open/close 4226 (time 1786738067.40 total 60.06 last 69.43)
total: 4500 open/close in 65.26 seconds: 68.96 ops/second
 - /unlink 1333 (time 1786738087.89 total 10.01 last 133.22)
 - /unlink 2493 (time 1786738097.89 total 20.01 last 115.97)
 - /unlink 3860 (time 1786738107.90 total 30.01 last 136.67)
total: 4500 /unlink in 34.62 seconds: 129.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 712 (time 1786738128.58 total 10.01 last 71.16)
 - open/close 1432 (time 1786738138.63 total 20.06 last 71.64)
 - open/close 2174 (time 1786738148.64 total 30.07 last 74.12)
 - open/close 2862 (time 1786738158.64 total 40.07 last 68.79)
 - open/close 3598 (time 1786738168.65 total 50.07 last 73.55)
 - open/close 4347 (time 1786738178.65 total 60.08 last 74.86)
total: 4500 open/close in 62.16 seconds: 72.40 ops/second
 - /unlink 1185 (time 1786738195.08 total 10.00 last 118.49)
 - /unlink 2300 (time 1786738205.09 total 20.01 last 111.35)
 - /unlink 3422 (time 1786738215.09 total 30.02 last 112.20)
total: 4500 /unlink in 37.59 seconds: 119.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 733 (time 1786738237.51 total 10.01 last 73.20)
 - open/close 1425 (time 1786738247.54 total 20.04 last 69.03)
 - open/close 2119 (time 1786738257.54 total 30.04 last 69.37)
 - open/close 2824 (time 1786738267.58 total 40.08 last 70.25)
 - open/close 3576 (time 1786738277.58 total 50.08 last 75.19)
 - open/close 4298 (time 1786738287.59 total 60.09 last 72.14)
total: 4500 open/close in 62.70 seconds: 71.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1204 (time 1786738305.14 total 10.00 last 120.39)
 - /unlink 2336 (time 1786738315.14 total 20.00 last 113.19)
 - /unlink 3602 (time 1786738325.16 total 30.02 last 126.39)
total: 4500 /unlink in 37.91 seconds: 118.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 678 (time 1786738348.97 total 10.00 last 67.79)
 - open/close 1327 (time 1786738358.97 total 20.01 last 64.87)
 - open/close 1919 (time 1786738368.99 total 30.03 last 59.07)
 - open/close 2614 (time 1786738379.01 total 40.04 last 69.39)
 - open/close 3314 (time 1786738389.02 total 50.05 last 69.95)
 - open/close 4014 (time 1786738399.03 total 60.06 last 69.93)
total: 4500 open/close in 66.14 seconds: 68.04 ops/second
 - /unlink 1362 (time 1786738419.53 total 10.00 last 136.17)
 - /unlink 2670 (time 1786738429.53 total 20.01 last 130.73)
 - /unlink 4016 (time 1786738439.54 total 30.01 last 134.56)
total: 4500 /unlink in 33.82 seconds: 133.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 742 (time 1786738457.86 total 10.01 last 74.11)
 - open/close 1460 (time 1786738467.86 total 20.02 last 71.73)
 - open/close 2179 (time 1786738477.87 total 30.03 last 71.84)
 - open/close 2892 (time 1786738487.87 total 40.03 last 71.29)
 - open/close 3635 (time 1786738497.90 total 50.06 last 74.09)
 - open/close 4375 (time 1786738507.91 total 60.06 last 73.96)
total: 4500 open/close in 61.82 seconds: 72.79 ops/second
 - /unlink 1453 (time 1786738523.95 total 10.02 last 145.05)
 - /unlink 2823 (time 1786738533.96 total 20.02 last 136.93)
 - /unlink 4076 (time 1786738543.96 total 30.02 last 125.28)
total: 4500 /unlink in 33.67 seconds: 133.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 652 (time 1786738563.84 total 10.01 last 65.15)
 - open/close 1415 (time 1786738573.84 total 20.01 last 76.29)
 - open/close 2106 (time 1786738583.86 total 30.03 last 68.95)
 - open/close 2883 (time 1786738593.87 total 40.04 last 77.64)
 - open/close 3644 (time 1786738603.88 total 50.04 last 76.06)
 - open/close 4282 (time 1786738613.90 total 60.07 last 63.65)
total: 4500 open/close in 63.33 seconds: 71.06 ops/second
 - /unlink 1199 (time 1786738632.31 total 10.01 last 119.80)
 - /unlink 2288 (time 1786738642.31 total 20.01 last 108.88)
 - /unlink 3517 (time 1786738652.31 total 30.01 last 122.89)
total: 4500 /unlink in 38.64 seconds: 116.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 662 (time 1786738676.53 total 10.03 last 66.02)
 - open/close 1269 (time 1786738686.55 total 20.05 last 60.59)
 - open/close 1865 (time 1786738696.55 total 30.05 last 59.59)
 - open/close 2587 (time 1786738706.57 total 40.06 last 72.09)
 - open/close 3246 (time 1786738716.58 total 50.08 last 65.81)
 - open/close 3844 (time 1786738726.59 total 60.09 last 59.72)
 - open/close 4479 (time 1786738736.61 total 70.10 last 63.42)
total: 4500 open/close in 70.49 seconds: 63.84 ops/second
 - /unlink 1151 (time 1786738752.71 total 10.01 last 115.02)
 - /unlink 2324 (time 1786738762.71 total 20.01 last 117.26)
 - /unlink 3607 (time 1786738772.71 total 30.01 last 128.29)
total: 4500 /unlink in 37.31 seconds: 120.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 644 (time 1786738795.82 total 10.01 last 64.31)
 - open/close 1296 (time 1786738805.84 total 20.03 last 65.11)
 - open/close 1952 (time 1786738815.85 total 30.04 last 65.53)
 - open/close 2586 (time 1786738825.86 total 40.05 last 63.30)
 - open/close 3214 (time 1786738835.88 total 50.07 last 62.69)
 - open/close 3812 (time 1786738845.89 total 60.08 last 59.77)
 - open/close 4331 (time 1786738855.89 total 70.08 last 51.90)
total: 4500 open/close in 72.42 seconds: 62.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1150 (time 1786738872.67 total 10.01 last 114.94)
 - /unlink 2251 (time 1786738882.68 total 20.01 last 110.04)
 - /unlink 3511 (time 1786738892.68 total 30.02 last 125.94)
total: 4500 /unlink in 38.25 seconds: 117.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 681 (time 1786738916.79 total 10.04 last 67.83)
 - open/close 1348 (time 1786738926.81 total 20.05 last 66.60)
 - open/close 1861 (time 1786738936.82 total 30.07 last 51.20)
 - open/close 2525 (time 1786738946.83 total 40.08 last 66.38)
 - open/close 3202 (time 1786738956.83 total 50.08 last 67.69)
 - open/close 3902 (time 1786738966.85 total 60.10 last 69.85)
total: 4500 open/close in 68.89 seconds: 65.33 ops/second
 - /unlink 1198 (time 1786738990.09 total 10.01 last 119.68)
 - /unlink 2453 (time 1786739000.09 total 20.01 last 125.46)
 - /unlink 3649 (time 1786739010.09 total 30.02 last 119.54)
total: 4500 /unlink in 37.60 seconds: 119.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 733 (time 1786739032.72 total 10.01 last 73.21)
 - open/close 1400 (time 1786739042.72 total 20.02 last 66.67)
 - open/close 2106 (time 1786739052.74 total 30.03 last 70.50)
 - open/close 2824 (time 1786739062.76 total 40.05 last 71.67)
 - open/close 3511 (time 1786739072.76 total 50.05 last 68.67)
 - open/close 4178 (time 1786739082.76 total 60.05 last 66.70)
total: 4500 open/close in 64.65 seconds: 69.60 ops/second
 - /unlink 1285 (time 1786739101.72 total 10.00 last 128.44)
 - /unlink 2452 (time 1786739111.72 total 20.01 last 116.63)
 - /unlink 3632 (time 1786739121.73 total 30.01 last 117.97)
total: 4500 /unlink in 37.53 seconds: 119.92 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95276 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg607-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg607-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg607-server
stop mds service on oleg607-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg607-server
stop mds service on oleg607-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg607-server

unloading modules via unload_modules_local on: 'oleg607-server'
oleg607-server: oleg607-server.virtnet: executing unload_modules_local
oleg607-server: 
modules unloaded.