== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:37:18 (1787387838)
Stopping clients: oleg619-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg619-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg619-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg619-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg619-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg619-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg619-server
oleg619-server: oleg619-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg619-server'
oleg619-server: oleg619-server.virtnet: executing load_modules_local
oleg619-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg619-server: Loading modules from /home/green/git/lustre-release/lustre
oleg619-server: detected 4 online CPUs by sysfs
oleg619-server: MODOPTS_LIBCFS=
oleg619-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg619-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all
pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg619-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all
pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg619-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all
pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg619-client.virtnet:  -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 655 (time 1787388021.86 total 10.02 last 65.40)
 - open/close 1371 (time 1787388031.87 total 20.03 last 71.52)
 - open/close 2015 (time 1787388041.88 total 30.03 last 64.38)
 - open/close 2725 (time 1787388051.89 total 40.04 last 70.91)
 - open/close 3260 (time 1787388061.90 total 50.05 last 53.44)
 - open/close 3776 (time 1787388071.94 total 60.10 last 51.38)
 - open/close 4357 (time 1787388081.97 total 70.13 last 57.93)
total: 4500 open/close in 72.35 seconds: 62.20 ops/second
 - /unlink 1239 (time 1787388100.27 total 10.01 last 123.72)
 - /unlink 2558 (time 1787388110.27 total 20.02 last 131.85)
 - /unlink 3863 (time 1787388120.28 total 30.03 last 130.39)
total: 4500 /unlink in 34.91 seconds: 128.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 701 (time 1787388140.02 total 10.01 last 70.03)
 - open/close 1427 (time 1787388150.03 total 20.02 last 72.52)
 - open/close 2110 (time 1787388160.03 total 30.02 last 68.27)
 - open/close 2865 (time 1787388170.04 total 40.03 last 75.44)
 - open/close 3649 (time 1787388180.05 total 50.04 last 78.34)
 - open/close 4373 (time 1787388190.05 total 60.05 last 72.36)
total: 4500 open/close in 61.93 seconds: 72.66 ops/second
 - /unlink 1211 (time 1787388206.75 total 10.00 last 121.09)
 - /unlink 2260 (time 1787388216.75 total 20.01 last 104.83)
 - /unlink 3436 (time 1787388226.75 total 30.01 last 117.60)
total: 4500 /unlink in 37.75 seconds: 119.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 772 (time 1787388249.35 total 10.00 last 77.17)
 - open/close 1402 (time 1787388259.36 total 20.02 last 62.91)
 - open/close 2119 (time 1787388269.37 total 30.02 last 71.68)
 - open/close 2808 (time 1787388279.37 total 40.02 last 68.88)
 - open/close 3520 (time 1787388289.40 total 50.06 last 70.97)
 - open/close 4224 (time 1787388299.40 total 60.06 last 70.39)
total: 4500 open/close in 64.02 seconds: 70.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1297 (time 1787388317.54 total 10.01 last 129.57)
 - /unlink 2702 (time 1787388327.56 total 20.02 last 140.30)
 - /unlink 4074 (time 1787388337.56 total 30.03 last 137.15)
total: 4500 /unlink in 33.27 seconds: 135.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 693 (time 1787388355.78 total 10.01 last 69.21)
 - open/close 1366 (time 1787388365.79 total 20.02 last 67.23)
 - open/close 1955 (time 1787388375.80 total 30.04 last 58.83)
 - open/close 2694 (time 1787388385.81 total 40.04 last 73.86)
 - open/close 3396 (time 1787388395.82 total 50.05 last 70.11)
 - open/close 4022 (time 1787388405.82 total 60.06 last 62.60)
total: 4500 open/close in 67.30 seconds: 66.87 ops/second
 - /unlink 1410 (time 1787388427.71 total 10.00 last 140.94)
 - /unlink 2745 (time 1787388437.72 total 20.01 last 133.42)
 - /unlink 4073 (time 1787388447.72 total 30.01 last 132.75)
total: 4500 /unlink in 32.89 seconds: 136.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 857 (time 1787388465.36 total 10.01 last 85.59)
 - open/close 1650 (time 1787388475.37 total 20.02 last 79.24)
 - open/close 2400 (time 1787388485.38 total 30.03 last 74.95)
 - open/close 3209 (time 1787388495.39 total 40.05 last 80.76)
 - open/close 3910 (time 1787388505.40 total 50.05 last 70.08)
total: 4500 open/close in 58.35 seconds: 77.13 ops/second
 - /unlink 1188 (time 1787388528.05 total 10.00 last 118.79)
 - /unlink 2416 (time 1787388538.06 total 20.01 last 122.73)
 - /unlink 3788 (time 1787388548.06 total 30.01 last 137.10)
total: 4500 /unlink in 35.27 seconds: 127.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 677 (time 1787388567.88 total 10.02 last 67.59)
 - open/close 1321 (time 1787388577.88 total 20.03 last 64.34)
 - open/close 2076 (time 1787388587.89 total 30.03 last 75.49)
 - open/close 2838 (time 1787388597.89 total 40.04 last 76.13)
 - open/close 3634 (time 1787388607.91 total 50.05 last 79.48)
 - open/close 4418 (time 1787388617.91 total 60.06 last 78.35)
total: 4500 open/close in 61.33 seconds: 73.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1387 (time 1787388633.58 total 10.00 last 138.63)
 - /unlink 2728 (time 1787388643.58 total 20.01 last 134.07)
 - /unlink 4211 (time 1787388653.59 total 30.01 last 148.24)
total: 4500 /unlink in 32.37 seconds: 139.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 720 (time 1787388669.93 total 10.00 last 71.99)
 - open/close 1440 (time 1787388679.93 total 20.01 last 71.97)
 - open/close 2138 (time 1787388689.95 total 30.02 last 69.71)
 - open/close 2745 (time 1787388699.96 total 40.03 last 60.64)
 - open/close 3488 (time 1787388709.97 total 50.04 last 74.22)
 - open/close 4283 (time 1787388719.97 total 60.04 last 79.49)
total: 4500 open/close in 62.53 seconds: 71.97 ops/second
 - /unlink 1554 (time 1787388736.86 total 10.00 last 155.36)
 - /unlink 2951 (time 1787388746.86 total 20.00 last 139.68)
 - /unlink 4373 (time 1787388756.87 total 30.01 last 142.11)
total: 4500 /unlink in 31.09 seconds: 144.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 652 (time 1787388773.19 total 10.01 last 65.14)
 - open/close 1347 (time 1787388783.20 total 20.02 last 69.43)
 - open/close 2130 (time 1787388793.21 total 30.03 last 78.20)
 - open/close 2927 (time 1787388803.22 total 40.04 last 79.63)
 - open/close 3751 (time 1787388813.23 total 50.05 last 82.31)
total: 4500 open/close in 59.56 seconds: 75.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1409 (time 1787388837.16 total 10.00 last 140.88)
 - /unlink 2889 (time 1787388847.16 total 20.00 last 147.98)
 - /unlink 4370 (time 1787388857.16 total 30.00 last 148.09)
total: 4500 /unlink in 30.92 seconds: 145.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 910 (time 1787388871.82 total 10.01 last 90.91)
 - open/close 1731 (time 1787388881.83 total 20.02 last 82.01)
 - open/close 2468 (time 1787388891.84 total 30.03 last 73.64)
 - open/close 3223 (time 1787388901.84 total 40.03 last 75.45)
 - open/close 4046 (time 1787388911.84 total 50.04 last 82.28)
total: 4500 open/close in 55.57 seconds: 80.99 ops/second
 - /unlink 1580 (time 1787388930.89 total 10.00 last 157.98)
 - /unlink 3027 (time 1787388940.90 total 20.02 last 144.45)
 - /unlink 4493 (time 1787388950.90 total 30.02 last 146.60)
total: 4500 /unlink in 30.07 seconds: 149.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 837 (time 1787388964.83 total 10.00 last 83.68)
 - open/close 1669 (time 1787388974.84 total 20.01 last 83.17)
 - open/close 2500 (time 1787388984.85 total 30.02 last 83.02)
 - open/close 3261 (time 1787388994.86 total 40.03 last 75.98)
 - open/close 4031 (time 1787389004.87 total 50.03 last 76.98)
total: 4500 open/close in 55.59 seconds: 80.95 ops/second
 - /unlink 1358 (time 1787389024.75 total 10.01 last 135.69)
 - /unlink 2627 (time 1787389034.75 total 20.01 last 126.85)
 - /unlink 4034 (time 1787389044.75 total 30.01 last 140.69)
total: 4500 /unlink in 33.51 seconds: 134.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 706 (time 1787389063.45 total 10.02 last 70.43)
 - open/close 1464 (time 1787389073.46 total 20.03 last 75.76)
 - open/close 2249 (time 1787389083.46 total 30.03 last 78.48)
 - open/close 3024 (time 1787389093.47 total 40.04 last 77.46)
 - open/close 3773 (time 1787389103.47 total 50.04 last 74.86)
total: 4500 open/close in 59.74 seconds: 75.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1382 (time 1787389127.64 total 10.00 last 138.17)
 - /unlink 2761 (time 1787389137.64 total 20.01 last 137.81)
 - /unlink 4237 (time 1787389147.65 total 30.01 last 147.53)
total: 4500 /unlink in 31.91 seconds: 141.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 800 (time 1787389164.14 total 10.00 last 79.98)
 - open/close 1581 (time 1787389174.16 total 20.02 last 77.98)
 - open/close 2405 (time 1787389184.17 total 30.03 last 82.31)
 - open/close 3214 (time 1787389194.17 total 40.03 last 80.87)
 - open/close 3911 (time 1787389204.17 total 50.04 last 69.67)
total: 4500 open/close in 58.19 seconds: 77.34 ops/second
 - /unlink 1283 (time 1787389227.85 total 10.00 last 128.26)
 - /unlink 2651 (time 1787389237.86 total 20.02 last 136.62)
 - /unlink 3879 (time 1787389247.87 total 30.02 last 122.73)
total: 4500 /unlink in 34.58 seconds: 130.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 810 (time 1787389266.50 total 10.01 last 80.92)
 - open/close 1545 (time 1787389276.50 total 20.01 last 73.47)
 - open/close 2385 (time 1787389286.51 total 30.02 last 83.92)
 - open/close 3150 (time 1787389296.51 total 40.03 last 76.46)
 - open/close 3968 (time 1787389306.52 total 50.03 last 81.77)
total: 4500 open/close in 56.99 seconds: 78.97 ops/second
 - /unlink 1401 (time 1787389327.09 total 10.01 last 139.91)
 - /unlink 2805 (time 1787389337.09 total 20.02 last 140.32)
 - /unlink 4259 (time 1787389347.10 total 30.02 last 145.34)
total: 4500 /unlink in 31.98 seconds: 140.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 768 (time 1787389362.87 total 10.01 last 76.74)
 - open/close 1482 (time 1787389372.89 total 20.03 last 71.25)
 - open/close 2263 (time 1787389382.90 total 30.04 last 78.03)
 - open/close 3032 (time 1787389392.90 total 40.04 last 76.85)
 - open/close 3753 (time 1787389402.92 total 50.06 last 72.01)
 - open/close 4483 (time 1787389412.92 total 60.06 last 72.95)
total: 4500 open/close in 60.34 seconds: 74.58 ops/second
 - /unlink 1325 (time 1787389426.72 total 10.00 last 132.48)
 - /unlink 2662 (time 1787389436.72 total 20.00 last 133.68)
 - /unlink 4132 (time 1787389446.72 total 30.01 last 146.90)
total: 4500 /unlink in 32.81 seconds: 137.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 769 (time 1787389463.66 total 10.01 last 76.84)
 - open/close 1408 (time 1787389473.66 total 20.01 last 63.89)
 - open/close 2067 (time 1787389483.66 total 30.01 last 65.90)
 - open/close 2782 (time 1787389493.66 total 40.01 last 71.47)
 - open/close 3542 (time 1787389503.68 total 50.02 last 75.91)
 - open/close 4346 (time 1787389513.68 total 60.03 last 80.34)
total: 4500 open/close in 61.89 seconds: 72.71 ops/second
 - /unlink 1548 (time 1787389529.76 total 10.01 last 154.70)
 - /unlink 2941 (time 1787389539.76 total 20.01 last 139.19)
 - /unlink 4251 (time 1787389549.76 total 30.01 last 130.99)
total: 4500 /unlink in 31.74 seconds: 141.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 747 (time 1787389566.20 total 10.01 last 74.64)
 - open/close 1479 (time 1787389576.20 total 20.01 last 73.19)
 - open/close 2154 (time 1787389586.20 total 30.01 last 67.48)
 - open/close 2875 (time 1787389596.22 total 40.03 last 72.00)
 - open/close 3576 (time 1787389606.22 total 50.03 last 70.06)
 - open/close 4317 (time 1787389616.24 total 60.05 last 73.97)
total: 4500 open/close in 62.29 seconds: 72.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1408 (time 1787389632.74 total 10.00 last 140.79)
 - /unlink 2734 (time 1787389642.74 total 20.00 last 132.59)
 - /unlink 4103 (time 1787389652.74 total 30.00 last 136.87)
total: 4500 /unlink in 33.51 seconds: 134.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 822 (time 1787389670.64 total 10.01 last 82.12)
 - open/close 1580 (time 1787389680.64 total 20.01 last 75.80)
 - open/close 2337 (time 1787389690.65 total 30.01 last 75.68)
 - open/close 3075 (time 1787389700.66 total 40.03 last 73.67)
 - open/close 3808 (time 1787389710.67 total 50.04 last 73.25)
total: 4500 open/close in 59.20 seconds: 76.01 ops/second
 - /unlink 1473 (time 1787389734.46 total 10.01 last 147.19)
 - /unlink 2874 (time 1787389744.46 total 20.01 last 140.07)
 - /unlink 4304 (time 1787389754.47 total 30.02 last 142.83)
total: 4500 /unlink in 31.47 seconds: 143.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 779 (time 1787389769.92 total 10.01 last 77.85)
 - open/close 1549 (time 1787389779.93 total 20.01 last 76.94)
 - open/close 2320 (time 1787389789.94 total 30.03 last 77.01)
 - open/close 3115 (time 1787389799.96 total 40.04 last 79.35)
 - open/close 3914 (time 1787389809.97 total 50.06 last 79.74)
total: 4500 open/close in 57.29 seconds: 78.54 ops/second
 - /unlink 1423 (time 1787389831.21 total 10.00 last 142.27)
 - /unlink 2858 (time 1787389841.21 total 20.00 last 143.48)
 - /unlink 4384 (time 1787389851.21 total 30.00 last 152.58)
total: 4500 /unlink in 31.08 seconds: 144.81 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94915 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user 'cl1': success
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user 'cl1': success

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg619-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg619-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg619-server
stop mds service on oleg619-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg619-server
stop mds service on oleg619-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg619-server

unloading modules via unload_modules_local on: 'oleg619-server'
oleg619-server: oleg619-server.virtnet: executing unload_modules_local
oleg619-server: 
modules unloaded.