== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:16:06 (1786691766)
Stopping clients: oleg215-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg215-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg215-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg215-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg215-server
oleg215-server: oleg215-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg215-server'
oleg215-server: oleg215-server.virtnet: executing load_modules_local
oleg215-server: Loading modules from /home/green/git/lustre-release/lustre
oleg215-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg215-server: detected 4 online CPUs by sysfs
oleg215-server: MODOPTS_LIBCFS=
oleg215-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg215-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg215-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg215-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg215-client.virtnet:  -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 491 (time 1786692019.25 total 10.02 last 49.02)
 - open/close 1205 (time 1786692029.25 total 20.02 last 71.39)
 - open/close 1901 (time 1786692039.25 total 30.02 last 69.59)
 - open/close 2427 (time 1786692049.26 total 40.02 last 52.59)
 - open/close 3134 (time 1786692059.26 total 50.03 last 70.67)
 - open/close 3844 (time 1786692069.26 total 60.03 last 70.99)
 - open/close 4474 (time 1786692079.26 total 70.03 last 62.99)
total: 4500 open/close in 70.32 seconds: 64.00 ops/second
 - /unlink 1220 (time 1786692094.89 total 10.00 last 121.98)
 - /unlink 2731 (time 1786692104.89 total 20.00 last 151.09)
 - /unlink 3818 (time 1786692114.90 total 30.01 last 108.67)
total: 4500 /unlink in 35.98 seconds: 125.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 795 (time 1786692136.56 total 10.00 last 79.49)
 - open/close 1551 (time 1786692146.57 total 20.00 last 75.57)
 - open/close 2385 (time 1786692156.57 total 30.01 last 83.37)
 - open/close 3060 (time 1786692166.57 total 40.01 last 67.49)
 - open/close 3793 (time 1786692176.59 total 50.02 last 73.20)
total: 4500 open/close in 59.20 seconds: 76.02 ops/second
 - /unlink 1297 (time 1786692201.40 total 10.00 last 129.69)
 - /unlink 2495 (time 1786692211.40 total 20.00 last 119.79)
 - /unlink 3782 (time 1786692221.40 total 30.00 last 128.68)
total: 4500 /unlink in 35.18 seconds: 127.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 762 (time 1786692242.29 total 10.01 last 76.16)
 - open/close 1490 (time 1786692252.29 total 20.01 last 72.78)
 - open/close 2221 (time 1786692262.30 total 30.01 last 73.07)
 - open/close 2840 (time 1786692272.31 total 40.02 last 61.85)
 - open/close 3478 (time 1786692282.31 total 50.03 last 63.74)
 - open/close 4009 (time 1786692292.33 total 60.05 last 53.01)
total: 4500 open/close in 69.27 seconds: 64.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1412 (time 1786692317.63 total 10.00 last 141.15)
 - /unlink 2614 (time 1786692327.63 total 20.01 last 120.14)
 - /unlink 3781 (time 1786692337.64 total 30.01 last 116.65)
total: 4500 /unlink in 36.07 seconds: 124.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 624 (time 1786692358.72 total 10.00 last 62.38)
 - open/close 1414 (time 1786692368.74 total 20.02 last 78.85)
 - open/close 2021 (time 1786692378.75 total 30.03 last 60.67)
 - open/close 2823 (time 1786692388.76 total 40.05 last 80.06)
 - open/close 3444 (time 1786692398.77 total 50.05 last 62.08)
 - open/close 4187 (time 1786692408.79 total 60.07 last 74.14)
total: 4500 open/close in 64.57 seconds: 69.69 ops/second
 - /unlink 1444 (time 1786692428.18 total 10.01 last 144.32)
 - /unlink 3058 (time 1786692438.19 total 20.01 last 161.34)
total: 4500 /unlink in 28.91 seconds: 155.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 715 (time 1786692462.20 total 10.01 last 71.45)
 - open/close 1525 (time 1786692472.20 total 20.01 last 80.99)
 - open/close 2230 (time 1786692482.20 total 30.02 last 70.46)
 - open/close 2793 (time 1786692492.23 total 40.04 last 56.14)
 - open/close 3622 (time 1786692502.24 total 50.05 last 82.84)
 - open/close 4432 (time 1786692512.25 total 60.06 last 80.93)
total: 4500 open/close in 60.67 seconds: 74.17 ops/second
 - /unlink 1614 (time 1786692527.16 total 10.01 last 161.30)
 - /unlink 3111 (time 1786692537.16 total 20.01 last 149.67)
total: 4500 /unlink in 28.84 seconds: 156.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 932 (time 1786692560.35 total 10.00 last 93.17)
 - open/close 1760 (time 1786692570.37 total 20.03 last 82.58)
 - open/close 2422 (time 1786692580.38 total 30.04 last 66.16)
 - open/close 3285 (time 1786692590.39 total 40.05 last 86.20)
 - open/close 4069 (time 1786692600.43 total 50.09 last 78.10)
total: 4500 open/close in 56.26 seconds: 79.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1379 (time 1786692621.39 total 10.00 last 137.86)
 - /unlink 2949 (time 1786692631.40 total 20.01 last 156.84)
 - /unlink 4283 (time 1786692641.41 total 30.02 last 133.33)
total: 4500 /unlink in 31.45 seconds: 143.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 833 (time 1786692658.66 total 10.00 last 83.26)
 - open/close 1707 (time 1786692668.67 total 20.01 last 87.36)
 - open/close 2620 (time 1786692678.67 total 30.02 last 91.24)
 - open/close 3397 (time 1786692688.69 total 40.03 last 77.60)
 - open/close 4103 (time 1786692698.69 total 50.03 last 70.56)
total: 4500 open/close in 54.79 seconds: 82.14 ops/second
 - /unlink 1418 (time 1786692718.59 total 10.01 last 141.70)
 - /unlink 2780 (time 1786692728.59 total 20.01 last 136.19)
 - /unlink 4095 (time 1786692738.59 total 30.01 last 131.44)
total: 4500 /unlink in 33.01 seconds: 136.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 678 (time 1786692758.22 total 10.00 last 67.78)
 - open/close 1545 (time 1786692768.23 total 20.02 last 86.59)
 - open/close 2375 (time 1786692778.27 total 30.05 last 82.71)
 - open/close 3053 (time 1786692788.27 total 40.06 last 67.77)
 - open/close 3723 (time 1786692798.29 total 50.07 last 66.87)
 - open/close 4432 (time 1786692808.30 total 60.08 last 70.84)
total: 4500 open/close in 61.34 seconds: 73.36 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1539 (time 1786692824.91 total 10.00 last 153.86)
 - /unlink 2875 (time 1786692834.92 total 20.01 last 133.52)
 - /unlink 4209 (time 1786692844.93 total 30.02 last 133.30)
total: 4500 /unlink in 32.51 seconds: 138.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 795 (time 1786692862.96 total 10.00 last 79.46)
 - open/close 1551 (time 1786692872.99 total 20.03 last 75.39)
 - open/close 2279 (time 1786692883.01 total 30.06 last 72.64)
 - open/close 3132 (time 1786692893.03 total 40.07 last 85.16)
 - open/close 3889 (time 1786692903.03 total 50.08 last 75.67)
total: 4500 open/close in 57.92 seconds: 77.69 ops/second
 - /unlink 1514 (time 1786692925.44 total 10.00 last 151.39)
 - /unlink 2917 (time 1786692935.47 total 20.02 last 140.00)
 - /unlink 4254 (time 1786692945.47 total 30.02 last 133.67)
total: 4500 /unlink in 31.83 seconds: 141.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 641 (time 1786692961.64 total 10.00 last 64.08)
 - open/close 1226 (time 1786692971.64 total 20.01 last 58.49)
 - open/close 2160 (time 1786692981.66 total 30.02 last 93.28)
 - open/close 3110 (time 1786692991.67 total 40.03 last 94.91)
 - open/close 3933 (time 1786693001.67 total 50.03 last 82.27)
total: 4500 open/close in 57.28 seconds: 78.57 ops/second
 - /unlink 1249 (time 1786693024.93 total 10.00 last 124.85)
 - /unlink 2418 (time 1786693034.93 total 20.01 last 116.88)
 - /unlink 3726 (time 1786693044.93 total 30.01 last 130.78)
total: 4500 /unlink in 34.93 seconds: 128.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 655 (time 1786693065.74 total 10.01 last 65.46)
 - open/close 1514 (time 1786693075.76 total 20.02 last 85.75)
 - open/close 2469 (time 1786693085.77 total 30.03 last 95.46)
 - open/close 3208 (time 1786693095.80 total 40.06 last 73.67)
 - open/close 3925 (time 1786693105.82 total 50.08 last 71.54)
total: 4500 open/close in 59.06 seconds: 76.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1472 (time 1786693129.19 total 10.00 last 147.19)
 - /unlink 2976 (time 1786693139.19 total 20.00 last 150.36)
total: 4500 /unlink in 29.28 seconds: 153.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 736 (time 1786693162.87 total 10.00 last 73.59)
 - open/close 1285 (time 1786693172.88 total 20.01 last 54.87)
 - open/close 1760 (time 1786693182.88 total 30.02 last 47.46)
 - open/close 2328 (time 1786693192.89 total 40.02 last 56.75)
 - open/close 3037 (time 1786693202.91 total 50.04 last 70.80)
 - open/close 3749 (time 1786693212.92 total 60.05 last 71.09)
 - open/close 4464 (time 1786693222.93 total 70.06 last 71.46)
total: 4500 open/close in 70.69 seconds: 63.66 ops/second
 - /unlink 1145 (time 1786693238.35 total 10.00 last 114.49)
 - /unlink 2359 (time 1786693248.36 total 20.01 last 121.32)
 - /unlink 3543 (time 1786693258.36 total 30.01 last 118.36)
total: 4500 /unlink in 36.73 seconds: 122.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 754 (time 1786693279.96 total 10.00 last 75.38)
 - open/close 1381 (time 1786693289.97 total 20.01 last 62.65)
 - open/close 2200 (time 1786693299.97 total 30.01 last 81.89)
 - open/close 3016 (time 1786693309.98 total 40.02 last 81.53)
 - open/close 3753 (time 1786693319.99 total 50.03 last 73.61)
 - open/close 4462 (time 1786693330.01 total 60.06 last 70.74)
total: 4500 open/close in 60.76 seconds: 74.06 ops/second
 - /unlink 1268 (time 1786693346.99 total 10.00 last 126.78)
 - /unlink 2507 (time 1786693356.99 total 20.00 last 123.87)
 - /unlink 3505 (time 1786693367.00 total 30.02 last 99.65)
total: 4500 /unlink in 37.28 seconds: 120.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 780 (time 1786693389.33 total 10.01 last 77.92)
 - open/close 1611 (time 1786693399.33 total 20.02 last 83.03)
 - open/close 2261 (time 1786693409.34 total 30.03 last 64.95)
 - open/close 2828 (time 1786693419.34 total 40.03 last 56.69)
 - open/close 3508 (time 1786693429.37 total 50.05 last 67.82)
 - open/close 4091 (time 1786693439.38 total 60.06 last 58.25)
total: 4500 open/close in 66.26 seconds: 67.91 ops/second
 - /unlink 1301 (time 1786693460.70 total 10.00 last 130.05)
 - /unlink 2494 (time 1786693470.71 total 20.01 last 119.21)
 - /unlink 3844 (time 1786693480.71 total 30.01 last 134.98)
total: 4500 /unlink in 34.58 seconds: 130.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 775 (time 1786693500.21 total 10.01 last 77.46)
 - open/close 1554 (time 1786693510.22 total 20.01 last 77.84)
 - open/close 2352 (time 1786693520.23 total 30.02 last 79.73)
 - open/close 3076 (time 1786693530.23 total 40.03 last 72.34)
 - open/close 3577 (time 1786693540.25 total 50.04 last 50.03)
 - open/close 4285 (time 1786693550.25 total 60.04 last 70.80)
total: 4500 open/close in 63.28 seconds: 71.12 ops/second
 - /unlink 1332 (time 1786693567.97 total 10.01 last 133.13)
 - /unlink 2903 (time 1786693577.98 total 20.01 last 157.03)
total: 4500 /unlink in 29.89 seconds: 150.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 800 (time 1786693602.58 total 10.01 last 79.95)
 - open/close 1613 (time 1786693612.58 total 20.01 last 81.28)
 - open/close 2447 (time 1786693622.58 total 30.01 last 83.39)
 - open/close 3225 (time 1786693632.59 total 40.01 last 77.77)
 - open/close 4004 (time 1786693642.59 total 50.02 last 77.87)
total: 4500 open/close in 57.59 seconds: 78.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1236 (time 1786693665.17 total 10.00 last 123.55)
 - /unlink 2669 (time 1786693675.17 total 20.01 last 143.26)
 - /unlink 4168 (time 1786693685.18 total 30.02 last 149.75)
total: 4500 /unlink in 33.03 seconds: 136.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 865 (time 1786693703.66 total 10.00 last 86.47)
 - open/close 1794 (time 1786693713.68 total 20.02 last 92.72)
 - open/close 2483 (time 1786693723.68 total 30.03 last 68.87)
 - open/close 3214 (time 1786693733.71 total 40.05 last 72.92)
 - open/close 3788 (time 1786693743.71 total 50.05 last 57.39)
total: 4500 open/close in 59.63 seconds: 75.47 ops/second
 - /unlink 1262 (time 1786693767.86 total 10.00 last 126.20)
 - /unlink 2717 (time 1786693777.87 total 20.01 last 145.37)
 - /unlink 4120 (time 1786693787.87 total 30.01 last 140.30)
total: 4500 /unlink in 32.71 seconds: 137.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 638 (time 1786693806.05 total 10.01 last 63.74)
 - open/close 1409 (time 1786693816.05 total 20.01 last 77.10)
 - open/close 2162 (time 1786693826.06 total 30.02 last 75.20)
 - open/close 2960 (time 1786693836.07 total 40.03 last 79.74)
 - open/close 3667 (time 1786693846.08 total 50.04 last 70.63)
 - open/close 4327 (time 1786693856.09 total 60.05 last 65.90)
total: 4500 open/close in 62.58 seconds: 71.91 ops/second
 - /unlink 1418 (time 1786693873.70 total 10.00 last 141.77)
 - /unlink 2692 (time 1786693883.70 total 20.01 last 127.32)
 - /unlink 4174 (time 1786693893.71 total 30.01 last 148.17)
total: 4500 /unlink in 32.51 seconds: 138.44 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95338 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg215-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg215-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server
stop mds service on oleg215-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server
stop mds service on oleg215-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg215-server

unloading modules via unload_modules_local on: 'oleg215-server'
oleg215-server: oleg215-server.virtnet: executing unload_modules_local
oleg215-server: 
modules unloaded.