== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:28:03 (1782487683)
Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg416-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg416-server
oleg416-server: oleg416-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg416-server'
oleg416-server: oleg416-server.virtnet: executing load_modules_local
oleg416-server: Loading modules from /home/green/git/lustre-release/lustre
oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
oleg416-server: detected 4 online CPUs by sysfs
oleg416-server: MODOPTS_LIBCFS=
oleg416-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg416-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all
pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg416-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all
pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg416-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all
pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg416-client.virtnet:  -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 369 (time 1782487864.41 total 10.00 last 36.89)
 - open/close 841 (time 1782487874.42 total 20.02 last 47.12)
 - open/close 1403 (time 1782487884.43 total 30.03 last 56.14)
 - open/close 1968 (time 1782487894.45 total 40.04 last 56.41)
 - open/close 2649 (time 1782487904.45 total 50.05 last 68.06)
 - open/close 3317 (time 1782487914.47 total 60.07 last 66.66)
 - open/close 3931 (time 1782487924.48 total 70.08 last 61.36)
 - open/close 4499 (time 1782487934.48 total 80.08 last 56.80)
total: 4500 open/close in 80.08 seconds: 56.19 ops/second
 - /unlink 1066 (time 1782487948.93 total 10.00 last 106.55)
 - /unlink 2195 (time 1782487958.94 total 20.01 last 112.79)
 - /unlink 3294 (time 1782487968.95 total 30.03 last 109.76)
 - /unlink 4399 (time 1782487978.96 total 40.03 last 110.46)
total: 4500 /unlink in 40.82 seconds: 110.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 569 (time 1782487995.34 total 10.01 last 56.82)
 - open/close 1221 (time 1782488005.35 total 20.02 last 65.16)
 - open/close 1886 (time 1782488015.36 total 30.03 last 66.45)
 - open/close 2558 (time 1782488025.36 total 40.03 last 67.17)
 - open/close 3190 (time 1782488035.37 total 50.04 last 63.13)
 - open/close 3839 (time 1782488045.38 total 60.05 last 64.85)
 - open/close 4471 (time 1782488055.38 total 70.05 last 63.17)
total: 4500 open/close in 70.50 seconds: 63.83 ops/second
 - /unlink 1200 (time 1782488070.20 total 10.00 last 119.96)
 - /unlink 2387 (time 1782488080.20 total 20.00 last 118.68)
 - /unlink 3461 (time 1782488090.21 total 30.01 last 107.35)
total: 4500 /unlink in 38.70 seconds: 116.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 640 (time 1782488113.41 total 10.01 last 63.91)
 - open/close 1234 (time 1782488123.43 total 20.03 last 59.28)
 - open/close 1850 (time 1782488133.43 total 30.04 last 61.57)
 - open/close 2498 (time 1782488143.44 total 40.05 last 64.76)
 - open/close 3167 (time 1782488153.45 total 50.05 last 66.87)
 - open/close 3803 (time 1782488163.45 total 60.05 last 63.59)
 - open/close 4439 (time 1782488173.46 total 70.06 last 63.53)
total: 4500 open/close in 71.09 seconds: 63.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1203 (time 1782488189.53 total 10.00 last 120.27)
 - /unlink 2325 (time 1782488199.54 total 20.00 last 112.18)
 - /unlink 3382 (time 1782488209.55 total 30.01 last 105.58)
 - /unlink 4463 (time 1782488219.55 total 40.02 last 108.07)
total: 4500 /unlink in 40.68 seconds: 110.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 573 (time 1782488235.91 total 10.01 last 57.23)
 - open/close 1105 (time 1782488245.92 total 20.02 last 53.17)
 - open/close 1569 (time 1782488255.92 total 30.02 last 46.40)
 - open/close 2126 (time 1782488265.94 total 40.04 last 55.59)
 - open/close 2779 (time 1782488275.95 total 50.05 last 65.23)
 - open/close 3471 (time 1782488285.97 total 60.06 last 69.10)
 - open/close 4081 (time 1782488295.99 total 70.09 last 60.85)
total: 4500 open/close in 78.81 seconds: 57.10 ops/second
 - /unlink 1010 (time 1782488323.00 total 10.00 last 101.00)
 - /unlink 2086 (time 1782488333.01 total 20.01 last 107.55)
 - /unlink 2801 (time 1782488343.01 total 30.01 last 71.45)
 - /unlink 3507 (time 1782488353.02 total 40.02 last 70.53)
 - /unlink 4348 (time 1782488363.03 total 50.02 last 84.09)
total: 4500 /unlink in 52.06 seconds: 86.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 485 (time 1782488381.05 total 10.00 last 48.49)
 - open/close 1086 (time 1782488391.07 total 20.02 last 59.98)
 - open/close 1535 (time 1782488401.07 total 30.03 last 44.87)
 - open/close 2091 (time 1782488411.07 total 40.03 last 55.60)
 - open/close 2671 (time 1782488421.09 total 50.04 last 57.92)
 - open/close 3228 (time 1782488431.10 total 60.05 last 55.64)
 - open/close 3769 (time 1782488441.12 total 70.07 last 54.01)
 - open/close 4376 (time 1782488451.12 total 80.08 last 60.64)
total: 4500 open/close in 82.52 seconds: 54.53 ops/second
 - /unlink 1146 (time 1782488468.68 total 10.11 last 113.37)
 - /unlink 2149 (time 1782488478.69 total 20.12 last 100.17)
 - /unlink 3455 (time 1782488488.71 total 30.14 last 130.38)
total: 4500 /unlink in 37.87 seconds: 118.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 820 (time 1782488508.94 total 10.01 last 81.88)
 - open/close 1768 (time 1782488518.94 total 20.02 last 94.78)
 - open/close 2755 (time 1782488528.94 total 30.02 last 98.67)
 - open/close 3650 (time 1782488538.95 total 40.03 last 89.44)
total: 4500 open/close in 48.87 seconds: 92.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 2187 (time 1782488559.94 total 10.00 last 218.68)
 - /unlink 4401 (time 1782488569.94 total 20.00 last 221.37)
total: 4500 /unlink in 20.44 seconds: 220.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 1539 (time 1782488582.51 total 10.00 last 153.87)
 - open/close 4101 (time 1782488592.51 total 20.00 last 256.13)
total: 4500 open/close in 21.72 seconds: 207.20 ops/second
total: 4500 /unlink in 7.22 seconds: 623.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 3315 (time 1782488614.44 total 10.00 last 331.43)
total: 4500 open/close in 13.42 seconds: 335.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
total: 4500 /unlink in 7.17 seconds: 627.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 3330 (time 1782488637.76 total 10.00 last 332.99)
total: 4500 open/close in 13.17 seconds: 341.72 ops/second
total: 4500 /unlink in 7.71 seconds: 583.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 2636 (time 1782488661.59 total 10.00 last 263.59)
total: 4500 open/close in 16.33 seconds: 275.52 ops/second
total: 4500 /unlink in 8.17 seconds: 550.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 3238 (time 1782488689.27 total 10.00 last 323.77)
total: 4500 open/close in 13.66 seconds: 329.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
total: 4500 /unlink in 8.69 seconds: 517.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 2887 (time 1782488714.58 total 10.00 last 288.65)
total: 4500 open/close in 14.85 seconds: 303.05 ops/second
total: 4500 /unlink in 6.25 seconds: 719.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 3870 (time 1782488738.35 total 10.01 last 386.70)
total: 4500 open/close in 11.44 seconds: 393.39 ops/second
 - /unlink 2005 (time 1782488751.46 total 10.00 last 200.41)
 - /unlink 3955 (time 1782488761.46 total 20.01 last 194.98)
total: 4500 /unlink in 22.22 seconds: 202.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 879 (time 1782488776.65 total 10.01 last 87.84)
 - open/close 1850 (time 1782488786.65 total 20.01 last 97.06)
 - open/close 2899 (time 1782488796.67 total 30.03 last 104.75)
 - open/close 3725 (time 1782488806.67 total 40.03 last 82.56)
total: 4500 open/close in 49.64 seconds: 90.66 ops/second
 - /unlink 1368 (time 1782488830.12 total 10.00 last 136.74)
 - /unlink 2927 (time 1782488840.12 total 20.01 last 155.85)
total: 4500 /unlink in 28.69 seconds: 156.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 874 (time 1782488862.14 total 10.00 last 87.36)
 - open/close 1807 (time 1782488872.15 total 20.01 last 93.26)
 - open/close 2604 (time 1782488882.15 total 30.01 last 79.69)
 - open/close 3492 (time 1782488892.16 total 40.02 last 88.73)
total: 4500 open/close in 49.95 seconds: 90.09 ops/second
 - /unlink 1912 (time 1782488914.57 total 10.00 last 191.18)
 - /unlink 4166 (time 1782488924.57 total 20.00 last 225.37)
total: 4500 /unlink in 21.75 seconds: 206.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 855 (time 1782488939.26 total 10.00 last 85.47)
 - open/close 1838 (time 1782488949.26 total 20.01 last 98.26)
 - open/close 2825 (time 1782488959.27 total 30.01 last 98.67)
 - open/close 3820 (time 1782488969.27 total 40.01 last 99.47)
total: 4500 open/close in 47.39 seconds: 94.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1839 (time 1782488989.21 total 10.01 last 183.80)
 - /unlink 3675 (time 1782488999.21 total 20.01 last 183.56)
total: 4500 /unlink in 24.46 seconds: 183.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 921 (time 1782489016.40 total 10.01 last 92.04)
 - open/close 1849 (time 1782489026.41 total 20.02 last 92.69)
 - open/close 2878 (time 1782489036.41 total 30.02 last 102.89)
 - open/close 3953 (time 1782489046.41 total 40.02 last 107.48)
total: 4500 open/close in 46.29 seconds: 97.22 ops/second
 - /unlink 1831 (time 1782489065.28 total 10.00 last 183.07)
 - /unlink 4052 (time 1782489075.29 total 20.01 last 221.85)
total: 4500 /unlink in 21.87 seconds: 205.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 1524 (time 1782489089.02 total 10.01 last 152.31)
 - open/close 2225 (time 1782489099.03 total 20.02 last 69.99)
 - open/close 2744 (time 1782489109.04 total 30.03 last 51.87)
 - open/close 3187 (time 1782489119.06 total 40.05 last 44.21)
 - open/close 3458 (time 1782489129.09 total 50.08 last 27.01)
 - open/close 3773 (time 1782489139.15 total 60.14 last 31.33)
 - open/close 4154 (time 1782489149.18 total 70.16 last 38.00)
total: 4500 open/close in 79.84 seconds: 56.37 ops/second
 - /unlink 1326 (time 1782489174.93 total 10.00 last 132.59)
 - /unlink 2571 (time 1782489184.94 total 20.01 last 124.42)
 - /unlink 3442 (time 1782489194.95 total 30.02 last 86.99)
 - /unlink 4053 (time 1782489204.96 total 40.02 last 61.07)
total: 4500 /unlink in 43.81 seconds: 102.72 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11221: 94296 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg416-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg416-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server
stop mds service on oleg416-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server
stop mds service on oleg416-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server

unloading modules via unload_modules_local on: 'oleg416-server'
oleg416-server: oleg416-server.virtnet: executing unload_modules_local
oleg416-server: 
modules unloaded.