== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:07:40 (1787864860)
Stopping clients: oleg619-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg619-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg619-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg619-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg619-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg619-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg619-server
oleg619-server: oleg619-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg619-server'
oleg619-server: oleg619-server.virtnet: executing load_modules_local
oleg619-server: Loading modules from /home/green/git/lustre-release/lustre
oleg619-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg619-server: detected 4 online CPUs by sysfs
oleg619-server: MODOPTS_LIBCFS=
oleg619-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg619-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all
pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg619-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all
pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg619-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all
pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg619-client.virtnet:  -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 734 (time 1787865054.14 total 10.01 last 73.35)
 - open/close 1479 (time 1787865064.15 total 20.02 last 74.43)
 - open/close 2273 (time 1787865074.17 total 30.04 last 79.20)
 - open/close 3013 (time 1787865084.18 total 40.05 last 73.97)
 - open/close 3852 (time 1787865094.19 total 50.06 last 83.81)
total: 4500 open/close in 58.43 seconds: 77.02 ops/second
 - /unlink 1551 (time 1787865116.14 total 10.00 last 155.04)
 - /unlink 3070 (time 1787865126.14 total 20.01 last 151.84)
total: 4500 /unlink in 28.80 seconds: 156.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 808 (time 1787865148.98 total 10.01 last 80.74)
 - open/close 1580 (time 1787865158.98 total 20.01 last 77.16)
 - open/close 2398 (time 1787865168.99 total 30.02 last 81.74)
 - open/close 3174 (time 1787865178.99 total 40.02 last 77.57)
 - open/close 3991 (time 1787865189.00 total 50.03 last 81.65)
total: 4500 open/close in 57.18 seconds: 78.69 ops/second
 - /unlink 1295 (time 1787865210.84 total 10.02 last 129.29)
 - /unlink 2685 (time 1787865220.84 total 20.02 last 139.00)
 - /unlink 4114 (time 1787865230.85 total 30.03 last 142.73)
total: 4500 /unlink in 33.07 seconds: 136.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 760 (time 1787865248.34 total 10.01 last 75.93)
 - open/close 1584 (time 1787865258.35 total 20.02 last 82.35)
 - open/close 2413 (time 1787865268.35 total 30.02 last 82.87)
 - open/close 3236 (time 1787865278.35 total 40.03 last 82.25)
 - open/close 4053 (time 1787865288.36 total 50.03 last 81.70)
total: 4500 open/close in 55.27 seconds: 81.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1487 (time 1787865307.31 total 10.00 last 148.69)
 - /unlink 2930 (time 1787865317.32 total 20.00 last 144.24)
 - /unlink 4414 (time 1787865327.33 total 30.01 last 148.28)
total: 4500 /unlink in 30.75 seconds: 146.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 767 (time 1787865341.80 total 10.01 last 76.63)
 - open/close 1548 (time 1787865351.81 total 20.01 last 78.06)
 - open/close 2188 (time 1787865361.84 total 30.05 last 63.78)
 - open/close 2861 (time 1787865371.85 total 40.06 last 67.23)
 - open/close 3602 (time 1787865381.86 total 50.06 last 74.07)
 - open/close 4355 (time 1787865391.88 total 60.08 last 75.15)
total: 4500 open/close in 62.07 seconds: 72.50 ops/second
 - /unlink 1117 (time 1787865408.74 total 10.00 last 111.65)
 - /unlink 2443 (time 1787865418.74 total 20.01 last 132.58)
 - /unlink 3984 (time 1787865428.74 total 30.01 last 154.09)
total: 4500 /unlink in 33.50 seconds: 134.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 727 (time 1787865446.78 total 10.01 last 72.64)
 - open/close 1548 (time 1787865456.79 total 20.01 last 82.06)
 - open/close 2347 (time 1787865466.79 total 30.01 last 79.90)
 - open/close 3184 (time 1787865476.79 total 40.02 last 83.66)
 - open/close 3930 (time 1787865486.79 total 50.02 last 74.59)
total: 4500 open/close in 57.47 seconds: 78.30 ops/second
 - /unlink 1622 (time 1787865509.06 total 10.01 last 162.11)
 - /unlink 2970 (time 1787865519.06 total 20.01 last 134.77)
 - /unlink 4424 (time 1787865529.08 total 30.03 last 145.09)
total: 4500 /unlink in 30.95 seconds: 145.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 719 (time 1787865545.19 total 10.01 last 71.82)
 - open/close 1456 (time 1787865555.19 total 20.02 last 73.65)
 - open/close 2316 (time 1787865565.20 total 30.02 last 85.95)
 - open/close 3120 (time 1787865575.20 total 40.03 last 80.38)
 - open/close 3887 (time 1787865585.21 total 50.03 last 76.68)
total: 4500 open/close in 57.88 seconds: 77.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1445 (time 1787865606.80 total 10.00 last 144.43)
 - /unlink 2947 (time 1787865616.80 total 20.01 last 150.18)
 - /unlink 4490 (time 1787865626.81 total 30.01 last 154.19)
total: 4500 /unlink in 30.07 seconds: 149.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 762 (time 1787865641.19 total 10.02 last 76.02)
 - open/close 1500 (time 1787865651.19 total 20.02 last 73.79)
 - open/close 2149 (time 1787865661.19 total 30.03 last 64.89)
 - open/close 2811 (time 1787865671.20 total 40.03 last 66.15)
 - open/close 3525 (time 1787865681.20 total 50.04 last 71.38)
 - open/close 4216 (time 1787865691.23 total 60.06 last 68.92)
total: 4500 open/close in 63.91 seconds: 70.41 ops/second
 - /unlink 1382 (time 1787865710.57 total 10.01 last 138.03)
 - /unlink 2634 (time 1787865720.57 total 20.02 last 125.16)
 - /unlink 4088 (time 1787865730.57 total 30.02 last 145.39)
total: 4500 /unlink in 33.24 seconds: 135.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 784 (time 1787865748.93 total 10.00 last 78.38)
 - open/close 1501 (time 1787865758.94 total 20.01 last 71.64)
 - open/close 2190 (time 1787865768.94 total 30.01 last 68.89)
 - open/close 2853 (time 1787865778.95 total 40.03 last 66.20)
 - open/close 3442 (time 1787865788.95 total 50.03 last 58.90)
 - open/close 3994 (time 1787865798.96 total 60.04 last 55.15)
total: 4500 open/close in 68.35 seconds: 65.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1512 (time 1787865823.25 total 10.00 last 151.14)
 - /unlink 2787 (time 1787865833.25 total 20.00 last 127.50)
 - /unlink 4271 (time 1787865843.25 total 30.00 last 148.40)
total: 4500 /unlink in 31.84 seconds: 141.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 766 (time 1787865860.71 total 10.00 last 76.58)
 - open/close 1403 (time 1787865870.71 total 20.00 last 63.69)
 - open/close 2038 (time 1787865880.71 total 30.01 last 63.47)
 - open/close 2650 (time 1787865890.76 total 40.05 last 60.94)
 - open/close 3216 (time 1787865900.77 total 50.06 last 56.55)
 - open/close 3864 (time 1787865910.77 total 60.07 last 64.77)
total: 4500 open/close in 69.80 seconds: 64.47 ops/second
 - /unlink 1373 (time 1787865936.06 total 10.00 last 137.27)
 - /unlink 2677 (time 1787865946.07 total 20.01 last 130.35)
 - /unlink 4060 (time 1787865956.07 total 30.01 last 138.30)
total: 4500 /unlink in 32.94 seconds: 136.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 626 (time 1787865973.99 total 10.01 last 62.51)
 - open/close 1254 (time 1787865983.99 total 20.02 last 62.77)
 - open/close 1872 (time 1787865994.00 total 30.02 last 61.77)
 - open/close 2480 (time 1787866004.01 total 40.03 last 60.75)
 - open/close 3173 (time 1787866014.02 total 50.05 last 69.20)
 - open/close 3715 (time 1787866024.04 total 60.06 last 54.12)
 - open/close 4329 (time 1787866034.05 total 70.07 last 61.33)
total: 4500 open/close in 72.02 seconds: 62.48 ops/second
 - /unlink 1471 (time 1787866050.69 total 10.00 last 147.06)
 - /unlink 2812 (time 1787866060.69 total 20.00 last 134.10)
 - /unlink 4270 (time 1787866070.69 total 30.01 last 145.73)
total: 4500 /unlink in 31.76 seconds: 141.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 770 (time 1787866087.43 total 10.00 last 76.97)
 - open/close 1523 (time 1787866097.44 total 20.02 last 75.21)
 - open/close 2228 (time 1787866107.46 total 30.03 last 70.38)
 - open/close 2918 (time 1787866117.47 total 40.04 last 68.94)
 - open/close 3623 (time 1787866127.47 total 50.04 last 70.49)
 - open/close 4360 (time 1787866137.47 total 60.05 last 73.68)
total: 4500 open/close in 61.81 seconds: 72.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1357 (time 1787866153.79 total 10.01 last 135.58)
 - /unlink 2694 (time 1787866163.80 total 20.01 last 133.67)
 - /unlink 4161 (time 1787866173.80 total 30.01 last 146.66)
total: 4500 /unlink in 32.67 seconds: 137.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 682 (time 1787866191.05 total 10.00 last 68.18)
 - open/close 1413 (time 1787866201.06 total 20.01 last 73.05)
 - open/close 2110 (time 1787866211.06 total 30.01 last 69.70)
 - open/close 2922 (time 1787866221.06 total 40.01 last 81.19)
 - open/close 3672 (time 1787866231.06 total 50.02 last 74.94)
 - open/close 4452 (time 1787866241.08 total 60.03 last 77.90)
total: 4500 open/close in 60.75 seconds: 74.08 ops/second
 - /unlink 1392 (time 1787866256.41 total 10.00 last 139.15)
 - /unlink 2556 (time 1787866266.42 total 20.01 last 116.27)
 - /unlink 3700 (time 1787866276.42 total 30.02 last 114.39)
total: 4500 /unlink in 37.68 seconds: 119.42 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 705 (time 1787866299.47 total 10.00 last 70.47)
 - open/close 1466 (time 1787866309.48 total 20.01 last 76.04)
 - open/close 2229 (time 1787866319.49 total 30.02 last 76.25)
 - open/close 2960 (time 1787866329.49 total 40.02 last 73.05)
 - open/close 3634 (time 1787866339.50 total 50.03 last 67.34)
 - open/close 4365 (time 1787866349.50 total 60.03 last 73.10)
total: 4500 open/close in 61.92 seconds: 72.67 ops/second
 - /unlink 1462 (time 1787866366.01 total 10.01 last 146.09)
 - /unlink 2880 (time 1787866376.01 total 20.01 last 141.79)
total: 4500 /unlink in 29.96 seconds: 150.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 737 (time 1787866401.32 total 10.01 last 73.65)
 - open/close 1420 (time 1787866411.35 total 20.04 last 68.07)
 - open/close 2203 (time 1787866421.37 total 30.05 last 78.20)
 - open/close 2887 (time 1787866431.38 total 40.06 last 68.33)
 - open/close 3656 (time 1787866441.38 total 50.06 last 76.89)
 - open/close 4411 (time 1787866451.39 total 60.08 last 75.41)
total: 4500 open/close in 61.35 seconds: 73.35 ops/second
 - /unlink 1350 (time 1787866467.21 total 10.00 last 134.95)
 - /unlink 2748 (time 1787866477.22 total 20.01 last 139.68)
 - /unlink 3985 (time 1787866487.22 total 30.02 last 123.66)
total: 4500 /unlink in 34.72 seconds: 129.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 568 (time 1787866508.32 total 10.02 last 56.69)
 - open/close 1233 (time 1787866518.32 total 20.02 last 66.49)
 - open/close 2018 (time 1787866528.33 total 30.03 last 78.42)
 - open/close 2810 (time 1787866538.33 total 40.03 last 79.20)
 - open/close 3646 (time 1787866548.34 total 50.03 last 83.59)
total: 4500 open/close in 59.94 seconds: 75.08 ops/second
 - /unlink 1275 (time 1787866572.96 total 10.00 last 127.44)
 - /unlink 2464 (time 1787866582.96 total 20.01 last 118.85)
 - /unlink 3752 (time 1787866592.97 total 30.01 last 128.76)
total: 4500 /unlink in 36.66 seconds: 122.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 760 (time 1787866615.90 total 10.01 last 75.95)
 - open/close 1435 (time 1787866625.91 total 20.01 last 67.46)
 - open/close 2153 (time 1787866635.92 total 30.03 last 71.70)
 - open/close 2901 (time 1787866645.93 total 40.04 last 74.74)
 - open/close 3708 (time 1787866655.93 total 50.04 last 80.68)
 - open/close 4425 (time 1787866665.97 total 60.08 last 71.40)
total: 4500 open/close in 61.47 seconds: 73.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1331 (time 1787866682.17 total 10.01 last 132.94)
 - /unlink 2773 (time 1787866692.18 total 20.02 last 144.10)
 - /unlink 4162 (time 1787866702.18 total 30.02 last 138.84)
total: 4500 /unlink in 32.58 seconds: 138.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 739 (time 1787866719.52 total 10.01 last 73.86)
 - open/close 1491 (time 1787866729.52 total 20.01 last 75.19)
 - open/close 2212 (time 1787866739.53 total 30.02 last 72.02)
 - open/close 2980 (time 1787866749.53 total 40.02 last 76.79)
 - open/close 3795 (time 1787866759.53 total 50.02 last 81.48)
 - open/close 4452 (time 1787866769.56 total 60.04 last 65.55)
total: 4500 open/close in 60.71 seconds: 74.12 ops/second
 - /unlink 1487 (time 1787866784.92 total 10.01 last 148.56)
 - /unlink 2834 (time 1787866794.94 total 20.02 last 134.49)
 - /unlink 4276 (time 1787866804.94 total 30.03 last 144.17)
total: 4500 /unlink in 31.57 seconds: 142.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 670 (time 1787866821.55 total 10.00 last 66.99)
 - open/close 1305 (time 1787866831.56 total 20.01 last 63.45)
 - open/close 1934 (time 1787866841.58 total 30.02 last 62.82)
 - open/close 2535 (time 1787866851.59 total 40.04 last 60.01)
 - open/close 3257 (time 1787866861.59 total 50.04 last 72.18)
 - open/close 3951 (time 1787866871.60 total 60.05 last 69.35)
total: 4500 open/close in 67.59 seconds: 66.57 ops/second
 - /unlink 1309 (time 1787866893.35 total 10.00 last 130.86)
 - /unlink 2753 (time 1787866903.35 total 20.01 last 144.37)
 - /unlink 4120 (time 1787866913.36 total 30.01 last 136.59)
total: 4500 /unlink in 33.15 seconds: 135.74 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11400: 95317 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg619-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg619-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg619-server
stop mds service on oleg619-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg619-server
stop mds service on oleg619-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg619-server

unloading modules via unload_modules_local on: 'oleg619-server'
oleg619-server: oleg619-server.virtnet: executing unload_modules_local
oleg619-server: 
modules unloaded.