== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:10:33 (1787397033)
Stopping clients: oleg218-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg218-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg218-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg218-server
oleg218-server: oleg218-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg218-server'
oleg218-server: oleg218-server.virtnet: executing load_modules_local
oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg218-server: Loading modules from /home/green/git/lustre-release/lustre
oleg218-server: detected 4 online CPUs by sysfs
oleg218-server: MODOPTS_LIBCFS=
oleg218-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg218-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all
pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg218-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all
pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg218-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all
pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg218-client.virtnet:  -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 590 (time 1787397247.78 total 10.01 last 58.94)
 - open/close 1256 (time 1787397257.78 total 20.01 last 66.57)
 - open/close 1888 (time 1787397267.81 total 30.04 last 63.04)
 - open/close 2487 (time 1787397277.83 total 40.07 last 59.74)
 - open/close 3209 (time 1787397287.84 total 50.07 last 72.16)
 - open/close 3879 (time 1787397297.85 total 60.08 last 66.94)
total: 4500 open/close in 68.23 seconds: 65.95 ops/second
 - /unlink 1179 (time 1787397320.64 total 10.00 last 117.90)
 - /unlink 2456 (time 1787397330.64 total 20.01 last 127.64)
 - /unlink 3796 (time 1787397340.65 total 30.01 last 133.88)
total: 4500 /unlink in 36.24 seconds: 124.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 802 (time 1787397362.16 total 10.00 last 80.18)
 - open/close 1582 (time 1787397372.17 total 20.01 last 77.92)
 - open/close 2268 (time 1787397382.19 total 30.03 last 68.47)
 - open/close 2968 (time 1787397392.19 total 40.04 last 69.97)
 - open/close 3725 (time 1787397402.22 total 50.06 last 75.49)
 - open/close 4435 (time 1787397412.24 total 60.08 last 70.88)
total: 4500 open/close in 60.99 seconds: 73.78 ops/second
 - /unlink 1204 (time 1787397428.82 total 10.00 last 120.35)
 - /unlink 2395 (time 1787397438.82 total 20.01 last 119.07)
 - /unlink 3568 (time 1787397448.84 total 30.02 last 117.14)
total: 4500 /unlink in 38.29 seconds: 117.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 666 (time 1787397472.95 total 10.00 last 66.57)
 - open/close 1178 (time 1787397482.96 total 20.01 last 51.15)
 - open/close 1814 (time 1787397492.96 total 30.02 last 63.59)
 - open/close 2565 (time 1787397502.97 total 40.02 last 75.04)
 - open/close 3347 (time 1787397512.97 total 50.03 last 78.16)
 - open/close 4122 (time 1787397522.98 total 60.04 last 77.44)
total: 4500 open/close in 65.33 seconds: 68.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1283 (time 1787397542.96 total 10.00 last 128.29)
 - /unlink 2454 (time 1787397552.96 total 20.00 last 117.07)
 - /unlink 3555 (time 1787397562.96 total 30.01 last 110.04)
total: 4500 /unlink in 37.01 seconds: 121.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 707 (time 1787397585.10 total 10.03 last 70.50)
 - open/close 1190 (time 1787397595.10 total 20.03 last 48.28)
 - open/close 1872 (time 1787397605.10 total 30.03 last 68.19)
 - open/close 2407 (time 1787397615.13 total 40.06 last 53.34)
 - open/close 2770 (time 1787397625.14 total 50.07 last 36.27)
 - open/close 3370 (time 1787397635.16 total 60.09 last 59.90)
 - open/close 4023 (time 1787397645.16 total 70.09 last 65.28)
total: 4500 open/close in 77.03 seconds: 58.42 ops/second
 - /unlink 1231 (time 1787397667.57 total 10.00 last 123.10)
 - /unlink 2301 (time 1787397677.57 total 20.00 last 106.98)
 - /unlink 3702 (time 1787397687.59 total 30.01 last 139.95)
total: 4500 /unlink in 36.68 seconds: 122.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 690 (time 1787397709.59 total 10.01 last 68.91)
 - open/close 1264 (time 1787397719.60 total 20.02 last 57.36)
 - open/close 1784 (time 1787397729.63 total 30.05 last 51.83)
 - open/close 2628 (time 1787397739.64 total 40.06 last 84.38)
 - open/close 3449 (time 1787397749.64 total 50.06 last 82.05)
 - open/close 4181 (time 1787397759.64 total 60.06 last 73.20)
total: 4500 open/close in 64.84 seconds: 69.41 ops/second
 - /unlink 1208 (time 1787397779.76 total 10.00 last 120.76)
 - /unlink 2325 (time 1787397789.76 total 20.01 last 111.67)
 - /unlink 3500 (time 1787397799.77 total 30.01 last 117.47)
total: 4500 /unlink in 38.87 seconds: 115.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 646 (time 1787397824.72 total 10.02 last 64.49)
 - open/close 1286 (time 1787397834.72 total 20.02 last 63.99)
 - open/close 2073 (time 1787397844.73 total 30.03 last 78.63)
 - open/close 2777 (time 1787397854.74 total 40.04 last 70.35)
 - open/close 3427 (time 1787397864.76 total 50.05 last 64.88)
 - open/close 4161 (time 1787397874.77 total 60.07 last 73.26)
total: 4500 open/close in 64.87 seconds: 69.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1018 (time 1787397894.51 total 10.00 last 101.76)
 - /unlink 2488 (time 1787397904.51 total 20.00 last 146.99)
 - /unlink 3977 (time 1787397914.52 total 30.01 last 148.86)
total: 4500 /unlink in 34.22 seconds: 131.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 576 (time 1787397935.13 total 10.03 last 57.46)
 - open/close 1298 (time 1787397945.14 total 20.03 last 72.16)
 - open/close 1991 (time 1787397955.16 total 30.06 last 69.13)
 - open/close 2798 (time 1787397965.18 total 40.07 last 80.54)
 - open/close 3543 (time 1787397975.18 total 50.08 last 74.50)
 - open/close 4250 (time 1787397985.20 total 60.09 last 70.57)
total: 4500 open/close in 63.72 seconds: 70.62 ops/second
 - /unlink 1306 (time 1787398003.94 total 10.00 last 130.56)
 - /unlink 2576 (time 1787398013.95 total 20.00 last 126.98)
 - /unlink 3934 (time 1787398023.95 total 30.01 last 135.76)
total: 4500 /unlink in 36.15 seconds: 124.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 622 (time 1787398044.48 total 10.01 last 62.11)
 - open/close 1149 (time 1787398054.49 total 20.03 last 52.63)
 - open/close 1992 (time 1787398064.50 total 30.03 last 84.28)
 - open/close 2715 (time 1787398074.50 total 40.04 last 72.26)
 - open/close 3430 (time 1787398084.52 total 50.05 last 71.39)
 - open/close 4160 (time 1787398094.52 total 60.06 last 72.96)
total: 4500 open/close in 63.82 seconds: 70.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 912 (time 1787398113.26 total 10.00 last 91.20)
 - /unlink 2158 (time 1787398123.26 total 20.00 last 124.55)
 - /unlink 3446 (time 1787398133.26 total 30.01 last 128.75)
total: 4500 /unlink in 38.45 seconds: 117.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 714 (time 1787398157.76 total 10.00 last 71.38)
 - open/close 1361 (time 1787398167.77 total 20.01 last 64.65)
 - open/close 1955 (time 1787398177.77 total 30.01 last 59.37)
 - open/close 2666 (time 1787398187.78 total 40.02 last 71.04)
 - open/close 3386 (time 1787398197.78 total 50.03 last 71.97)
 - open/close 4213 (time 1787398207.80 total 60.04 last 82.56)
total: 4500 open/close in 64.54 seconds: 69.72 ops/second
 - /unlink 1054 (time 1787398227.48 total 10.00 last 105.36)
 - /unlink 2339 (time 1787398237.48 total 20.01 last 128.45)
 - /unlink 3708 (time 1787398247.48 total 30.01 last 136.90)
total: 4500 /unlink in 36.17 seconds: 124.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 807 (time 1787398269.21 total 10.01 last 80.65)
 - open/close 1579 (time 1787398279.24 total 20.04 last 76.95)
 - open/close 2134 (time 1787398289.24 total 30.04 last 55.50)
 - open/close 2590 (time 1787398299.25 total 40.05 last 45.55)
 - open/close 3232 (time 1787398309.26 total 50.05 last 64.18)
 - open/close 4054 (time 1787398319.26 total 60.06 last 82.18)
total: 4500 open/close in 65.53 seconds: 68.68 ops/second
 - /unlink 1190 (time 1787398339.80 total 10.02 last 118.79)
 - /unlink 2571 (time 1787398349.81 total 20.03 last 137.97)
 - /unlink 3893 (time 1787398359.81 total 30.03 last 132.16)
total: 4500 /unlink in 34.65 seconds: 129.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 614 (time 1787398380.40 total 10.03 last 61.24)
 - open/close 1292 (time 1787398390.40 total 20.03 last 67.77)
 - open/close 1965 (time 1787398400.41 total 30.04 last 67.24)
 - open/close 2731 (time 1787398410.43 total 40.05 last 76.51)
 - open/close 3465 (time 1787398420.43 total 50.06 last 73.37)
 - open/close 4154 (time 1787398430.44 total 60.07 last 68.82)
total: 4500 open/close in 64.50 seconds: 69.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1362 (time 1787398449.85 total 10.00 last 136.17)
 - /unlink 2556 (time 1787398459.89 total 20.03 last 119.01)
 - /unlink 3769 (time 1787398469.89 total 30.04 last 121.22)
total: 4500 /unlink in 36.19 seconds: 124.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 730 (time 1787398491.44 total 10.03 last 72.81)
 - open/close 1570 (time 1787398501.45 total 20.03 last 83.95)
 - open/close 2385 (time 1787398511.45 total 30.04 last 81.46)
 - open/close 3167 (time 1787398521.46 total 40.04 last 78.17)
 - open/close 3869 (time 1787398531.47 total 50.06 last 70.08)
total: 4500 open/close in 59.65 seconds: 75.44 ops/second
 - /unlink 1391 (time 1787398556.45 total 10.01 last 139.01)
 - /unlink 2568 (time 1787398566.45 total 20.01 last 117.65)
 - /unlink 3936 (time 1787398576.45 total 30.01 last 136.79)
total: 4500 /unlink in 34.45 seconds: 130.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 715 (time 1787398596.67 total 10.00 last 71.47)
 - open/close 1491 (time 1787398606.68 total 20.01 last 77.57)
 - open/close 2191 (time 1787398616.68 total 30.02 last 69.94)
 - open/close 2911 (time 1787398626.70 total 40.03 last 71.90)
 - open/close 3610 (time 1787398636.70 total 50.03 last 69.89)
 - open/close 4355 (time 1787398646.71 total 60.05 last 74.39)
total: 4500 open/close in 62.16 seconds: 72.40 ops/second
 - /unlink 1293 (time 1787398664.44 total 10.00 last 129.25)
 - /unlink 2510 (time 1787398674.45 total 20.01 last 121.64)
 - /unlink 3840 (time 1787398684.45 total 30.01 last 132.98)
total: 4500 /unlink in 34.57 seconds: 130.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 572 (time 1787398704.33 total 10.01 last 57.15)
 - open/close 1314 (time 1787398714.34 total 20.02 last 74.15)
 - open/close 1992 (time 1787398724.37 total 30.05 last 67.59)
 - open/close 2748 (time 1787398734.38 total 40.06 last 75.53)
 - open/close 3489 (time 1787398744.38 total 50.06 last 74.09)
 - open/close 4222 (time 1787398754.39 total 60.07 last 73.21)
total: 4500 open/close in 63.66 seconds: 70.69 ops/second
 - /unlink 1425 (time 1787398772.70 total 10.01 last 142.42)
 - /unlink 2634 (time 1787398782.70 total 20.01 last 120.88)
 - /unlink 3885 (time 1787398792.70 total 30.01 last 125.03)
total: 4500 /unlink in 37.51 seconds: 119.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 545 (time 1787398817.60 total 10.01 last 54.45)
 - open/close 1013 (time 1787398827.65 total 20.06 last 46.54)
 - open/close 1557 (time 1787398837.66 total 30.07 last 54.38)
 - open/close 2251 (time 1787398847.66 total 40.07 last 69.40)
 - open/close 2955 (time 1787398857.66 total 50.07 last 70.39)
 - open/close 3650 (time 1787398867.66 total 60.07 last 69.48)
 - open/close 4373 (time 1787398877.66 total 70.07 last 72.29)
total: 4500 open/close in 71.97 seconds: 62.52 ops/second
 - /unlink 1207 (time 1787398895.54 total 10.01 last 120.64)
 - /unlink 2340 (time 1787398905.56 total 20.02 last 113.13)
 - /unlink 3727 (time 1787398915.56 total 30.02 last 138.69)
total: 4500 /unlink in 36.57 seconds: 123.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 613 (time 1787398937.05 total 10.00 last 61.30)
 - open/close 1408 (time 1787398947.06 total 20.02 last 79.35)
 - open/close 2232 (time 1787398957.07 total 30.03 last 82.34)
 - open/close 2980 (time 1787398967.08 total 40.04 last 74.71)
 - open/close 3685 (time 1787398977.09 total 50.05 last 70.45)
 - open/close 4261 (time 1787398987.13 total 60.09 last 57.37)
total: 4500 open/close in 64.25 seconds: 70.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1204 (time 1787399007.16 total 10.01 last 120.29)
 - /unlink 2436 (time 1787399017.17 total 20.01 last 123.15)
 - /unlink 3571 (time 1787399027.17 total 30.02 last 113.47)
total: 4500 /unlink in 36.70 seconds: 122.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 761 (time 1787399049.36 total 10.03 last 75.90)
 - open/close 1468 (time 1787399059.38 total 20.04 last 70.61)
 - open/close 2257 (time 1787399069.39 total 30.05 last 78.81)
 - open/close 2904 (time 1787399079.40 total 40.06 last 64.65)
 - open/close 3730 (time 1787399089.41 total 50.07 last 82.51)
 - open/close 4450 (time 1787399099.41 total 60.07 last 71.98)
total: 4500 open/close in 60.87 seconds: 73.93 ops/second
 - /unlink 1072 (time 1787399116.11 total 10.02 last 107.01)
 - /unlink 2380 (time 1787399126.11 total 20.02 last 130.79)
 - /unlink 3432 (time 1787399136.11 total 30.02 last 105.20)
total: 4500 /unlink in 39.58 seconds: 113.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 702 (time 1787399161.62 total 10.01 last 70.11)
 - open/close 1414 (time 1787399171.63 total 20.02 last 71.17)
 - open/close 2191 (time 1787399181.66 total 30.05 last 77.46)
 - open/close 2892 (time 1787399191.66 total 40.05 last 70.07)
 - open/close 3664 (time 1787399201.70 total 50.08 last 76.96)
 - open/close 4412 (time 1787399211.70 total 60.09 last 74.78)
total: 4500 open/close in 61.42 seconds: 73.27 ops/second
 - /unlink 1350 (time 1787399228.23 total 10.00 last 134.96)
 - /unlink 2735 (time 1787399238.24 total 20.01 last 138.47)
 - /unlink 4043 (time 1787399248.24 total 30.01 last 130.74)
total: 4500 /unlink in 33.08 seconds: 136.04 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95166 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user 'cl1': success
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user 'cl1': success

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg218-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg218-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server
stop mds service on oleg218-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server
stop mds service on oleg218-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server

unloading modules via unload_modules_local on: 'oleg218-server'
oleg218-server: oleg218-server.virtnet: executing unload_modules_local
oleg218-server: 
modules unloaded.