== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:10:07 (1779693007)
Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg250-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg250-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg250-server
oleg250-server: oleg250-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg250-server'
oleg250-server: oleg250-server.virtnet: executing load_modules_local
oleg250-server: Loading modules from /home/green/git/lustre-release/lustre
oleg250-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg250-server: detected 4 online CPUs by sysfs
oleg250-server: MODOPTS_LIBCFS=
oleg250-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg250-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all
pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg250-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all
pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg250-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all
pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg250-client.virtnet:  -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 743 (time 1779693224.34 total 10.01 last 74.23)
 - open/close 1304 (time 1779693234.36 total 20.02 last 56.03)
 - open/close 1941 (time 1779693244.36 total 30.03 last 63.68)
 - open/close 2512 (time 1779693254.36 total 40.03 last 57.08)
 - open/close 3197 (time 1779693264.37 total 50.04 last 68.42)
 - open/close 3876 (time 1779693274.37 total 60.04 last 67.90)
total: 4500 open/close in 68.39 seconds: 65.80 ops/second
 - /unlink 1209 (time 1779693297.92 total 10.00 last 120.89)
 - /unlink 2585 (time 1779693307.93 total 20.00 last 137.58)
 - /unlink 3912 (time 1779693317.93 total 30.01 last 132.65)
total: 4500 /unlink in 34.59 seconds: 130.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 576 (time 1779693338.46 total 10.03 last 57.42)
 - open/close 1241 (time 1779693348.46 total 20.04 last 66.45)
 - open/close 1896 (time 1779693358.51 total 30.08 last 65.21)
 - open/close 2556 (time 1779693368.52 total 40.10 last 65.91)
 - open/close 3334 (time 1779693378.57 total 50.14 last 77.44)
 - open/close 3911 (time 1779693388.58 total 60.16 last 57.63)
 - open/close 4391 (time 1779693398.60 total 70.18 last 47.91)
total: 4500 open/close in 71.97 seconds: 62.52 ops/second
 - /unlink 1288 (time 1779693416.29 total 10.00 last 128.78)
 - /unlink 2629 (time 1779693426.30 total 20.01 last 134.03)
 - /unlink 3896 (time 1779693436.30 total 30.01 last 126.69)
total: 4500 /unlink in 34.94 seconds: 128.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 613 (time 1779693456.07 total 10.01 last 61.22)
 - open/close 1139 (time 1779693466.11 total 20.05 last 52.41)
 - open/close 1799 (time 1779693476.13 total 30.07 last 65.85)
 - open/close 2565 (time 1779693486.14 total 40.08 last 76.55)
 - open/close 3349 (time 1779693496.15 total 50.09 last 78.31)
 - open/close 4033 (time 1779693506.16 total 60.10 last 68.35)
total: 4500 open/close in 68.15 seconds: 66.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1136 (time 1779693529.13 total 10.01 last 113.53)
 - /unlink 2404 (time 1779693539.13 total 20.01 last 126.72)
 - /unlink 3493 (time 1779693549.13 total 30.01 last 108.88)
total: 4500 /unlink in 38.91 seconds: 115.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 681 (time 1779693574.01 total 10.00 last 68.08)
 - open/close 1295 (time 1779693584.02 total 20.01 last 61.33)
 - open/close 1903 (time 1779693594.03 total 30.03 last 60.73)
 - open/close 2549 (time 1779693604.04 total 40.03 last 64.56)
 - open/close 3231 (time 1779693614.05 total 50.04 last 68.14)
 - open/close 3948 (time 1779693624.06 total 60.05 last 71.63)
total: 4500 open/close in 67.69 seconds: 66.48 ops/second
 - /unlink 1026 (time 1779693647.81 total 10.00 last 102.55)
 - /unlink 2167 (time 1779693657.81 total 20.01 last 114.08)
 - /unlink 3388 (time 1779693667.82 total 30.02 last 121.89)
total: 4500 /unlink in 38.53 seconds: 116.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 613 (time 1779693691.73 total 10.03 last 61.11)
 - open/close 1140 (time 1779693701.74 total 20.04 last 52.65)
 - open/close 1767 (time 1779693711.75 total 30.05 last 62.61)
 - open/close 2392 (time 1779693721.76 total 40.06 last 62.49)
 - open/close 3124 (time 1779693731.76 total 50.06 last 73.17)
 - open/close 3768 (time 1779693741.77 total 60.07 last 64.37)
 - open/close 4363 (time 1779693751.79 total 70.09 last 59.38)
total: 4500 open/close in 71.90 seconds: 62.59 ops/second
 - /unlink 1383 (time 1779693768.48 total 10.00 last 138.26)
 - /unlink 2669 (time 1779693778.49 total 20.01 last 128.48)
 - /unlink 4098 (time 1779693788.50 total 30.02 last 142.84)
total: 4500 /unlink in 33.37 seconds: 134.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 782 (time 1779693806.64 total 10.00 last 78.18)
 - open/close 1516 (time 1779693816.64 total 20.01 last 73.38)
 - open/close 2261 (time 1779693826.66 total 30.02 last 74.41)
 - open/close 2962 (time 1779693836.66 total 40.03 last 70.05)
 - open/close 3675 (time 1779693846.66 total 50.03 last 71.30)
 - open/close 4396 (time 1779693856.67 total 60.03 last 72.07)
total: 4500 open/close in 61.32 seconds: 73.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1331 (time 1779693872.67 total 10.01 last 133.00)
 - /unlink 2623 (time 1779693882.67 total 20.01 last 129.19)
 - /unlink 3878 (time 1779693892.67 total 30.01 last 125.44)
total: 4500 /unlink in 35.22 seconds: 127.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 672 (time 1779693913.18 total 10.02 last 67.03)
 - open/close 1346 (time 1779693923.18 total 20.03 last 67.39)
 - open/close 1978 (time 1779693933.19 total 30.03 last 63.15)
 - open/close 2719 (time 1779693943.19 total 40.04 last 74.09)
 - open/close 3408 (time 1779693953.20 total 50.05 last 68.79)
 - open/close 4165 (time 1779693963.23 total 60.07 last 75.53)
total: 4500 open/close in 64.35 seconds: 69.93 ops/second
 - /unlink 1258 (time 1779693982.48 total 10.00 last 125.78)
 - /unlink 2532 (time 1779693992.48 total 20.00 last 127.36)
 - /unlink 3928 (time 1779694002.49 total 30.01 last 139.48)
total: 4500 /unlink in 34.58 seconds: 130.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 664 (time 1779694023.05 total 10.00 last 66.39)
 - open/close 1286 (time 1779694033.06 total 20.01 last 62.15)
 - open/close 1963 (time 1779694043.07 total 30.02 last 67.62)
 - open/close 2687 (time 1779694053.08 total 40.03 last 72.34)
 - open/close 3322 (time 1779694063.08 total 50.03 last 63.49)
 - open/close 4022 (time 1779694073.10 total 60.04 last 69.92)
total: 4500 open/close in 65.66 seconds: 68.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1114 (time 1779694093.50 total 10.01 last 111.26)
 - /unlink 2304 (time 1779694103.50 total 20.02 last 118.95)
 - /unlink 3463 (time 1779694113.51 total 30.02 last 115.81)
total: 4500 /unlink in 39.81 seconds: 113.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 551 (time 1779694139.65 total 10.01 last 55.03)
 - open/close 1221 (time 1779694149.68 total 20.04 last 66.83)
 - open/close 1918 (time 1779694159.69 total 30.05 last 69.62)
 - open/close 2691 (time 1779694169.69 total 40.05 last 77.27)
 - open/close 3346 (time 1779694179.73 total 50.09 last 65.26)
 - open/close 4077 (time 1779694189.74 total 60.10 last 73.06)
total: 4500 open/close in 66.07 seconds: 68.11 ops/second
 - /unlink 1292 (time 1779694210.38 total 10.00 last 129.17)
 - /unlink 2477 (time 1779694220.39 total 20.01 last 118.41)
 - /unlink 3765 (time 1779694230.39 total 30.01 last 128.78)
total: 4500 /unlink in 35.64 seconds: 126.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 762 (time 1779694250.49 total 10.00 last 76.16)
 - open/close 1531 (time 1779694260.49 total 20.01 last 76.87)
 - open/close 2324 (time 1779694270.49 total 30.01 last 79.28)
 - open/close 3028 (time 1779694280.52 total 40.04 last 70.20)
 - open/close 3753 (time 1779694290.52 total 50.04 last 72.49)
 - open/close 4499 (time 1779694300.52 total 60.04 last 74.59)
total: 4500 open/close in 60.04 seconds: 74.95 ops/second
 - /unlink 976 (time 1779694314.58 total 10.01 last 97.54)
 - /unlink 2165 (time 1779694324.59 total 20.01 last 118.81)
 - /unlink 3555 (time 1779694334.60 total 30.02 last 138.88)
total: 4500 /unlink in 37.44 seconds: 120.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 733 (time 1779694356.67 total 10.02 last 73.17)
 - open/close 1476 (time 1779694366.68 total 20.02 last 74.27)
 - open/close 2188 (time 1779694376.68 total 30.03 last 71.18)
 - open/close 3009 (time 1779694386.70 total 40.04 last 81.94)
 - open/close 3707 (time 1779694396.70 total 50.05 last 69.78)
 - open/close 4411 (time 1779694406.72 total 60.06 last 70.30)
total: 4500 open/close in 61.19 seconds: 73.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1187 (time 1779694422.57 total 10.00 last 118.66)
 - /unlink 2419 (time 1779694432.57 total 20.01 last 123.16)
 - /unlink 3876 (time 1779694442.58 total 30.01 last 145.58)
total: 4500 /unlink in 34.02 seconds: 132.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 844 (time 1779694461.43 total 10.00 last 84.40)
 - open/close 1630 (time 1779694471.44 total 20.01 last 78.49)
 - open/close 2396 (time 1779694481.44 total 30.02 last 76.57)
 - open/close 3054 (time 1779694491.46 total 40.04 last 65.68)
 - open/close 3747 (time 1779694501.47 total 50.04 last 69.25)
total: 4500 open/close in 59.08 seconds: 76.17 ops/second
 - /unlink 1284 (time 1779694525.23 total 10.01 last 128.26)
 - /unlink 2643 (time 1779694535.23 total 20.02 last 135.81)
 - /unlink 3882 (time 1779694545.24 total 30.02 last 123.82)
total: 4500 /unlink in 34.90 seconds: 128.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 695 (time 1779694564.10 total 10.01 last 69.42)
 - open/close 1475 (time 1779694574.11 total 20.02 last 77.92)
 - open/close 2258 (time 1779694584.11 total 30.02 last 78.28)
 - open/close 2982 (time 1779694594.12 total 40.03 last 72.34)
 - open/close 3655 (time 1779694604.12 total 50.04 last 67.29)
 - open/close 4416 (time 1779694614.14 total 60.05 last 75.98)
total: 4500 open/close in 61.38 seconds: 73.31 ops/second
 - /unlink 1289 (time 1779694629.98 total 10.01 last 128.79)
 - /unlink 2577 (time 1779694639.98 total 20.01 last 128.72)
 - /unlink 4003 (time 1779694649.99 total 30.02 last 142.54)
total: 4500 /unlink in 33.73 seconds: 133.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 816 (time 1779694668.83 total 10.00 last 81.60)
 - open/close 1622 (time 1779694678.84 total 20.01 last 80.55)
 - open/close 2296 (time 1779694688.85 total 30.01 last 67.35)
 - open/close 2964 (time 1779694698.87 total 40.03 last 66.67)
 - open/close 3621 (time 1779694708.87 total 50.04 last 65.67)
 - open/close 4437 (time 1779694718.89 total 60.06 last 81.46)
total: 4500 open/close in 60.88 seconds: 73.91 ops/second
 - /unlink 1471 (time 1779694733.38 total 10.01 last 147.02)
 - /unlink 2781 (time 1779694743.38 total 20.01 last 130.96)
 - /unlink 4028 (time 1779694753.40 total 30.03 last 124.47)
total: 4500 /unlink in 33.19 seconds: 135.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 702 (time 1779694771.66 total 10.01 last 70.16)
 - open/close 1413 (time 1779694781.66 total 20.01 last 71.08)
 - open/close 2290 (time 1779694791.66 total 30.01 last 87.69)
 - open/close 2978 (time 1779694801.67 total 40.01 last 68.77)
 - open/close 3725 (time 1779694811.68 total 50.02 last 74.65)
 - open/close 4469 (time 1779694821.69 total 60.03 last 74.34)
total: 4500 open/close in 60.42 seconds: 74.48 ops/second
 - /unlink 1407 (time 1779694836.32 total 10.00 last 140.64)
 - /unlink 2817 (time 1779694846.32 total 20.01 last 140.92)
 - /unlink 4338 (time 1779694856.33 total 30.02 last 152.01)
total: 4500 /unlink in 31.47 seconds: 142.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 684 (time 1779694872.33 total 10.00 last 68.39)
 - open/close 1332 (time 1779694882.33 total 20.00 last 64.79)
 - open/close 2102 (time 1779694892.33 total 30.00 last 77.00)
 - open/close 2930 (time 1779694902.33 total 40.01 last 82.75)
 - open/close 3713 (time 1779694912.34 total 50.02 last 78.23)
 - open/close 4482 (time 1779694922.36 total 60.04 last 76.74)
total: 4500 open/close in 60.25 seconds: 74.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1409 (time 1779694937.08 total 10.02 last 140.63)
 - /unlink 2953 (time 1779694947.08 total 20.02 last 154.38)
 - /unlink 4223 (time 1779694957.08 total 30.02 last 126.99)
total: 4500 /unlink in 31.76 seconds: 141.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 798 (time 1779694972.86 total 10.00 last 79.80)
 - open/close 1587 (time 1779694982.88 total 20.01 last 78.80)
 - open/close 2325 (time 1779694992.88 total 30.02 last 73.76)
 - open/close 3097 (time 1779695002.89 total 40.02 last 77.15)
 - open/close 3805 (time 1779695012.90 total 50.04 last 70.72)
total: 4500 open/close in 58.53 seconds: 76.89 ops/second
 - /unlink 1513 (time 1779695035.21 total 10.00 last 151.29)
 - /unlink 2852 (time 1779695045.22 total 20.01 last 133.82)
 - /unlink 4202 (time 1779695055.22 total 30.01 last 134.99)
total: 4500 /unlink in 32.54 seconds: 138.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 727 (time 1779695072.04 total 10.02 last 72.58)
 - open/close 1448 (time 1779695082.04 total 20.02 last 72.06)
 - open/close 2223 (time 1779695092.05 total 30.03 last 77.46)
 - open/close 3014 (time 1779695102.05 total 40.03 last 79.06)
 - open/close 3700 (time 1779695112.06 total 50.04 last 68.58)
total: 4500 open/close in 59.54 seconds: 75.58 ops/second
 - /unlink 1501 (time 1779695135.22 total 10.00 last 150.05)
 - /unlink 2763 (time 1779695145.22 total 20.01 last 126.17)
 - /unlink 3957 (time 1779695155.23 total 30.01 last 119.29)
total: 4500 /unlink in 34.66 seconds: 129.83 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 93883 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg250-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg250-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server
stop mds service on oleg250-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server
stop mds service on oleg250-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg250-server

unloading modules via unload_modules_local on: 'oleg250-server'
oleg250-server: oleg250-server.virtnet: executing unload_modules_local
oleg250-server: 
modules unloaded.