== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:30:42 (1786663842)
Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg159-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg159-server
oleg159-server: oleg159-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg159-server'
oleg159-server: oleg159-server.virtnet: executing load_modules_local
oleg159-server: Loading modules from /home/green/git/lustre-release/lustre
oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg159-server: detected 4 online CPUs by sysfs
oleg159-server: MODOPTS_LIBCFS=
oleg159-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg159-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all
pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg159-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all
pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg159-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all
pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg159-client.virtnet:  -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 739 (time 1786664082.81 total 10.06 last 73.48)
 - open/close 1498 (time 1786664092.81 total 20.06 last 75.90)
 - open/close 2257 (time 1786664102.83 total 30.07 last 75.80)
 - open/close 3022 (time 1786664112.83 total 40.07 last 76.49)
 - open/close 3770 (time 1786664122.85 total 50.09 last 74.67)
total: 4500 open/close in 60.02 seconds: 74.97 ops/second
 - /unlink 1308 (time 1786664147.68 total 10.00 last 130.75)
 - /unlink 2631 (time 1786664157.68 total 20.01 last 132.28)
 - /unlink 4119 (time 1786664167.68 total 30.01 last 148.76)
total: 4500 /unlink in 33.44 seconds: 134.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 837 (time 1786664186.34 total 10.04 last 83.39)
 - open/close 1742 (time 1786664196.34 total 20.04 last 90.49)
 - open/close 2647 (time 1786664206.35 total 30.05 last 90.42)
 - open/close 3438 (time 1786664216.36 total 40.05 last 79.05)
 - open/close 4267 (time 1786664226.37 total 50.06 last 82.84)
total: 4500 open/close in 53.31 seconds: 84.41 ops/second
 - /unlink 1124 (time 1786664244.48 total 10.01 last 112.25)
 - /unlink 2430 (time 1786664254.49 total 20.02 last 130.54)
 - /unlink 3599 (time 1786664264.49 total 30.02 last 116.90)
total: 4500 /unlink in 37.35 seconds: 120.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 649 (time 1786664286.90 total 10.03 last 64.73)
 - open/close 1251 (time 1786664296.90 total 20.03 last 60.18)
 - open/close 1977 (time 1786664306.91 total 30.03 last 72.58)
 - open/close 2718 (time 1786664316.92 total 40.04 last 74.04)
 - open/close 3493 (time 1786664326.95 total 50.07 last 77.25)
 - open/close 4174 (time 1786664336.95 total 60.08 last 68.08)
total: 4500 open/close in 64.85 seconds: 69.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1410 (time 1786664356.85 total 10.01 last 140.87)
 - /unlink 3015 (time 1786664366.85 total 20.01 last 160.46)
total: 4500 /unlink in 29.15 seconds: 154.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 797 (time 1786664391.48 total 10.01 last 79.65)
 - open/close 1448 (time 1786664401.52 total 20.05 last 64.84)
 - open/close 2113 (time 1786664411.53 total 30.05 last 66.48)
 - open/close 2761 (time 1786664421.53 total 40.05 last 64.77)
 - open/close 3462 (time 1786664431.54 total 50.07 last 70.00)
 - open/close 4154 (time 1786664441.57 total 60.09 last 69.05)
total: 4500 open/close in 65.30 seconds: 68.92 ops/second
 - /unlink 1249 (time 1786664462.34 total 10.01 last 124.78)
 - /unlink 2608 (time 1786664472.35 total 20.01 last 135.90)
 - /unlink 3712 (time 1786664482.35 total 30.01 last 110.39)
total: 4500 /unlink in 36.37 seconds: 123.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 682 (time 1786664504.21 total 10.00 last 68.19)
 - open/close 1349 (time 1786664514.21 total 20.00 last 66.70)
 - open/close 2047 (time 1786664524.25 total 30.03 last 69.57)
 - open/close 2648 (time 1786664534.25 total 40.04 last 60.06)
 - open/close 3279 (time 1786664544.25 total 50.04 last 63.09)
 - open/close 3923 (time 1786664554.25 total 60.04 last 64.40)
total: 4500 open/close in 70.01 seconds: 64.28 ops/second
 - /unlink 897 (time 1786664579.31 total 10.02 last 89.56)
 - /unlink 1775 (time 1786664589.32 total 20.02 last 87.76)
 - /unlink 2951 (time 1786664599.32 total 30.02 last 117.58)
 - /unlink 4289 (time 1786664609.32 total 40.02 last 133.79)
total: 4500 /unlink in 41.80 seconds: 107.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 604 (time 1786664625.62 total 10.02 last 60.26)
 - open/close 1405 (time 1786664635.64 total 20.05 last 79.91)
 - open/close 2258 (time 1786664645.66 total 30.06 last 85.19)
 - open/close 3016 (time 1786664655.67 total 40.08 last 75.67)
 - open/close 3761 (time 1786664665.68 total 50.08 last 74.49)
total: 4500 open/close in 58.91 seconds: 76.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1246 (time 1786664689.16 total 10.00 last 124.58)
 - /unlink 2288 (time 1786664699.17 total 20.00 last 104.17)
 - /unlink 3476 (time 1786664709.17 total 30.01 last 118.79)
total: 4500 /unlink in 37.68 seconds: 119.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 708 (time 1786664731.31 total 10.02 last 70.68)
 - open/close 1355 (time 1786664741.31 total 20.02 last 64.69)
 - open/close 2071 (time 1786664751.31 total 30.02 last 71.56)
 - open/close 2673 (time 1786664761.34 total 40.05 last 60.04)
 - open/close 3407 (time 1786664771.35 total 50.06 last 73.33)
 - open/close 4068 (time 1786664781.36 total 60.07 last 66.06)
total: 4500 open/close in 66.16 seconds: 68.01 ops/second
 - /unlink 1272 (time 1786664802.16 total 10.00 last 127.18)
 - /unlink 2474 (time 1786664812.16 total 20.00 last 120.18)
 - /unlink 3786 (time 1786664822.16 total 30.00 last 131.19)
total: 4500 /unlink in 34.85 seconds: 129.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 772 (time 1786664842.09 total 10.00 last 77.17)
 - open/close 1553 (time 1786664852.09 total 20.01 last 78.07)
 - open/close 2197 (time 1786664862.12 total 30.04 last 64.22)
 - open/close 2892 (time 1786664872.13 total 40.05 last 69.41)
 - open/close 3569 (time 1786664882.14 total 50.05 last 67.67)
 - open/close 4208 (time 1786664892.14 total 60.05 last 63.89)
total: 4500 open/close in 64.44 seconds: 69.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1126 (time 1786664912.61 total 10.01 last 112.54)
 - /unlink 2271 (time 1786664922.61 total 20.01 last 114.49)
 - /unlink 3627 (time 1786664932.62 total 30.01 last 135.55)
total: 4500 /unlink in 36.87 seconds: 122.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 718 (time 1786664954.81 total 10.02 last 71.62)
 - open/close 1375 (time 1786664964.83 total 20.04 last 65.57)
 - open/close 2170 (time 1786664974.85 total 30.06 last 79.34)
 - open/close 2838 (time 1786664984.86 total 40.07 last 66.76)
 - open/close 3581 (time 1786664994.86 total 50.07 last 74.28)
 - open/close 4266 (time 1786665004.86 total 60.08 last 68.48)
total: 4500 open/close in 63.49 seconds: 70.87 ops/second
 - /unlink 1155 (time 1786665023.33 total 10.00 last 115.47)
 - /unlink 2401 (time 1786665033.34 total 20.01 last 124.54)
 - /unlink 3669 (time 1786665043.35 total 30.02 last 126.69)
total: 4500 /unlink in 36.59 seconds: 122.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 749 (time 1786665064.67 total 10.02 last 74.74)
 - open/close 1494 (time 1786665074.68 total 20.03 last 74.40)
 - open/close 2300 (time 1786665084.69 total 30.04 last 80.55)
 - open/close 2940 (time 1786665094.71 total 40.06 last 63.87)
 - open/close 3599 (time 1786665104.72 total 50.07 last 65.86)
 - open/close 4269 (time 1786665114.72 total 60.07 last 66.97)
total: 4500 open/close in 62.83 seconds: 71.62 ops/second
 - /unlink 1132 (time 1786665132.65 total 10.00 last 113.16)
 - /unlink 2362 (time 1786665142.66 total 20.01 last 122.96)
 - /unlink 3739 (time 1786665152.66 total 30.01 last 137.65)
total: 4500 /unlink in 36.13 seconds: 124.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 713 (time 1786665173.75 total 10.01 last 71.19)
 - open/close 1444 (time 1786665183.75 total 20.02 last 73.04)
 - open/close 2214 (time 1786665193.76 total 30.03 last 76.95)
 - open/close 2970 (time 1786665203.76 total 40.03 last 75.57)
 - open/close 3676 (time 1786665213.77 total 50.03 last 70.59)
 - open/close 4321 (time 1786665223.77 total 60.04 last 64.45)
total: 4500 open/close in 62.99 seconds: 71.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1269 (time 1786665241.60 total 10.00 last 126.85)
 - /unlink 2521 (time 1786665251.61 total 20.01 last 125.09)
 - /unlink 3818 (time 1786665261.61 total 30.01 last 129.68)
total: 4500 /unlink in 36.10 seconds: 124.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 672 (time 1786665283.32 total 10.02 last 67.06)
 - open/close 1451 (time 1786665293.33 total 20.03 last 77.84)
 - open/close 2286 (time 1786665303.35 total 30.04 last 83.39)
 - open/close 3120 (time 1786665313.35 total 40.05 last 83.33)
 - open/close 3856 (time 1786665323.36 total 50.06 last 73.52)
total: 4500 open/close in 58.64 seconds: 76.74 ops/second
 - /unlink 1348 (time 1786665346.68 total 10.00 last 134.79)
 - /unlink 2684 (time 1786665356.68 total 20.01 last 133.53)
 - /unlink 4069 (time 1786665366.69 total 30.01 last 138.45)
total: 4500 /unlink in 33.34 seconds: 134.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 832 (time 1786665385.30 total 10.00 last 83.16)
 - open/close 1627 (time 1786665395.31 total 20.01 last 79.42)
 - open/close 2388 (time 1786665405.32 total 30.02 last 76.03)
 - open/close 3029 (time 1786665415.32 total 40.03 last 64.07)
 - open/close 3762 (time 1786665425.33 total 50.04 last 73.25)
 - open/close 4496 (time 1786665435.34 total 60.04 last 73.35)
total: 4500 open/close in 60.07 seconds: 74.91 ops/second
 - /unlink 1272 (time 1786665450.16 total 10.00 last 127.20)
 - /unlink 2701 (time 1786665460.17 total 20.00 last 142.84)
 - /unlink 4059 (time 1786665470.17 total 30.01 last 135.75)
total: 4500 /unlink in 32.99 seconds: 136.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 690 (time 1786665488.49 total 10.00 last 68.97)
 - open/close 1267 (time 1786665498.49 total 20.01 last 57.68)
 - open/close 1869 (time 1786665508.50 total 30.01 last 60.18)
 - open/close 2466 (time 1786665518.50 total 40.02 last 59.67)
 - open/close 3205 (time 1786665528.51 total 50.02 last 73.85)
 - open/close 3929 (time 1786665538.51 total 60.02 last 72.39)
total: 4500 open/close in 67.82 seconds: 66.35 ops/second
 - /unlink 1405 (time 1786665561.60 total 10.00 last 140.47)
 - /unlink 2723 (time 1786665571.60 total 20.00 last 131.80)
 - /unlink 3839 (time 1786665581.62 total 30.02 last 111.43)
total: 4500 /unlink in 35.22 seconds: 127.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 528 (time 1786665601.70 total 10.01 last 52.76)
 - open/close 1317 (time 1786665611.72 total 20.02 last 78.81)
 - open/close 2053 (time 1786665621.73 total 30.04 last 73.47)
 - open/close 2685 (time 1786665631.73 total 40.04 last 63.20)
 - open/close 3314 (time 1786665641.74 total 50.05 last 62.84)
 - open/close 3956 (time 1786665651.75 total 60.06 last 64.15)
total: 4500 open/close in 66.80 seconds: 67.36 ops/second
 - /unlink 1363 (time 1786665673.64 total 10.00 last 136.29)
 - /unlink 2579 (time 1786665683.65 total 20.00 last 121.56)
 - /unlink 3723 (time 1786665693.65 total 30.01 last 114.36)
total: 4500 /unlink in 36.12 seconds: 124.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 854 (time 1786665714.14 total 10.01 last 85.35)
 - open/close 1544 (time 1786665724.16 total 20.03 last 68.85)
 - open/close 2343 (time 1786665734.18 total 30.05 last 79.68)
 - open/close 3165 (time 1786665744.20 total 40.07 last 82.09)
 - open/close 3881 (time 1786665754.21 total 50.08 last 71.55)
total: 4500 open/close in 57.05 seconds: 78.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1354 (time 1786665775.98 total 10.01 last 135.22)
 - /unlink 2690 (time 1786665785.99 total 20.03 last 133.44)
 - /unlink 3983 (time 1786665796.00 total 30.03 last 129.24)
total: 4500 /unlink in 33.90 seconds: 132.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 693 (time 1786665816.20 total 10.01 last 69.20)
 - open/close 1414 (time 1786665826.21 total 20.02 last 72.07)
 - open/close 2064 (time 1786665836.21 total 30.02 last 64.98)
 - open/close 2803 (time 1786665846.21 total 40.02 last 73.88)
 - open/close 3506 (time 1786665856.23 total 50.04 last 70.21)
 - open/close 4156 (time 1786665866.23 total 60.04 last 64.97)
total: 4500 open/close in 64.40 seconds: 69.87 ops/second
 - /unlink 1504 (time 1786665885.23 total 10.01 last 150.28)
 - /unlink 2845 (time 1786665895.23 total 20.01 last 134.09)
 - /unlink 3953 (time 1786665905.24 total 30.02 last 110.72)
total: 4500 /unlink in 34.50 seconds: 130.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 611 (time 1786665925.25 total 10.01 last 61.05)
 - open/close 1297 (time 1786665935.25 total 20.01 last 68.59)
 - open/close 2136 (time 1786665945.26 total 30.02 last 83.84)
 - open/close 2947 (time 1786665955.26 total 40.02 last 81.05)
 - open/close 3744 (time 1786665965.27 total 50.03 last 79.66)
 - open/close 4488 (time 1786665975.27 total 60.03 last 74.37)
total: 4500 open/close in 60.25 seconds: 74.69 ops/second
 - /unlink 1345 (time 1786665990.42 total 10.00 last 134.50)
 - /unlink 2626 (time 1786666000.42 total 20.00 last 128.10)
 - /unlink 3913 (time 1786666010.43 total 30.00 last 128.65)
total: 4500 /unlink in 34.87 seconds: 129.06 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95296 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg159-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg159-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server
stop mds service on oleg159-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server
stop mds service on oleg159-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server

unloading modules via unload_modules_local on: 'oleg159-server'
oleg159-server: oleg159-server.virtnet: executing unload_modules_local
oleg159-server: 
modules unloaded.