== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:08:21 (1790784501)
Stopping clients: oleg249-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg249-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg249-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg249-server
oleg249-server: oleg249-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg249-server'
oleg249-server: oleg249-server.virtnet: executing load_modules_local
oleg249-server: kmemleak disabled
oleg249-server: Loading modules from /home/green/git/lustre-release/lustre
oleg249-server: detected 4 online CPUs by sysfs
oleg249-server: MODOPTS_LIBCFS=
oleg249-server: Force libcfs to create 2 CPU partitions
oleg249-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg249-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all
pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg249-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all
pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg249-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all
pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg249-client.virtnet:  -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 676 (time 1790784730.39 total 10.00 last 67.57)
 - open/close 1372 (time 1790784740.41 total 20.02 last 69.46)
 - open/close 2079 (time 1790784750.42 total 30.04 last 70.62)
 - open/close 2794 (time 1790784760.43 total 40.04 last 71.49)
 - open/close 3579 (time 1790784770.43 total 50.05 last 78.44)
 - open/close 4353 (time 1790784780.45 total 60.06 last 77.27)
total: 4500 open/close in 61.87 seconds: 72.73 ops/second
 - /unlink 1317 (time 1790784797.85 total 10.00 last 131.68)
 - /unlink 2528 (time 1790784807.87 total 20.02 last 120.90)
 - /unlink 3788 (time 1790784817.87 total 30.02 last 125.95)
total: 4500 /unlink in 35.04 seconds: 128.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 635 (time 1790784837.97 total 10.01 last 63.41)
 - open/close 1345 (time 1790784847.97 total 20.02 last 70.99)
 - open/close 2069 (time 1790784857.98 total 30.03 last 72.32)
 - open/close 2848 (time 1790784867.98 total 40.03 last 77.88)
 - open/close 3573 (time 1790784877.99 total 50.04 last 72.44)
 - open/close 4276 (time 1790784888.00 total 60.04 last 70.28)
total: 4500 open/close in 63.18 seconds: 71.23 ops/second
 - /unlink 1404 (time 1790784906.28 total 10.00 last 140.34)
 - /unlink 2691 (time 1790784916.29 total 20.01 last 128.67)
 - /unlink 4113 (time 1790784926.29 total 30.01 last 142.14)
total: 4500 /unlink in 33.06 seconds: 136.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 685 (time 1790784944.98 total 10.01 last 68.45)
 - open/close 1351 (time 1790784955.00 total 20.02 last 66.49)
 - open/close 2155 (time 1790784965.01 total 30.03 last 80.31)
 - open/close 2914 (time 1790784975.03 total 40.05 last 75.76)
 - open/close 3656 (time 1790784985.03 total 50.06 last 74.15)
 - open/close 4289 (time 1790784995.05 total 60.07 last 63.23)
total: 4500 open/close in 63.64 seconds: 70.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1150 (time 1790785013.76 total 10.01 last 114.89)
 - /unlink 2262 (time 1790785023.76 total 20.01 last 111.17)
 - /unlink 3554 (time 1790785033.77 total 30.02 last 129.12)
total: 4500 /unlink in 37.92 seconds: 118.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 723 (time 1790785056.20 total 10.00 last 72.29)
 - open/close 1533 (time 1790785066.21 total 20.02 last 80.88)
 - open/close 2305 (time 1790785076.22 total 30.02 last 77.13)
 - open/close 3033 (time 1790785086.23 total 40.04 last 72.72)
 - open/close 3791 (time 1790785096.24 total 50.04 last 75.80)
 - open/close 4338 (time 1790785106.25 total 60.05 last 54.63)
total: 4500 open/close in 63.57 seconds: 70.79 ops/second
 - /unlink 1267 (time 1790785126.17 total 10.00 last 126.68)
 - /unlink 2515 (time 1790785136.17 total 20.01 last 124.75)
 - /unlink 3843 (time 1790785146.19 total 30.02 last 132.63)
total: 4500 /unlink in 36.06 seconds: 124.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 733 (time 1790785167.84 total 10.01 last 73.24)
 - open/close 1462 (time 1790785177.85 total 20.01 last 72.87)
 - open/close 2235 (time 1790785187.86 total 30.03 last 77.21)
 - open/close 2948 (time 1790785197.87 total 40.03 last 71.25)
 - open/close 3698 (time 1790785207.87 total 50.04 last 74.93)
 - open/close 4350 (time 1790785217.88 total 60.04 last 65.18)
total: 4500 open/close in 62.07 seconds: 72.50 ops/second
 - /unlink 1471 (time 1790785234.82 total 10.00 last 147.05)
 - /unlink 2747 (time 1790785244.82 total 20.00 last 127.60)
 - /unlink 4000 (time 1790785254.83 total 30.01 last 125.22)
total: 4500 /unlink in 34.09 seconds: 132.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 647 (time 1790785274.49 total 10.00 last 64.68)
 - open/close 1284 (time 1790785284.50 total 20.01 last 63.64)
 - open/close 1899 (time 1790785294.51 total 30.02 last 61.45)
 - open/close 2588 (time 1790785304.51 total 40.02 last 68.86)
 - open/close 3190 (time 1790785314.51 total 50.03 last 60.18)
 - open/close 3763 (time 1790785324.52 total 60.03 last 57.29)
 - open/close 4336 (time 1790785334.52 total 70.04 last 57.26)
total: 4500 open/close in 73.01 seconds: 61.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1044 (time 1790785352.56 total 10.00 last 104.35)
 - /unlink 2290 (time 1790785362.56 total 20.01 last 124.57)
 - /unlink 3622 (time 1790785372.57 total 30.02 last 133.09)
total: 4500 /unlink in 36.76 seconds: 122.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 732 (time 1790785395.24 total 10.01 last 73.16)
 - open/close 1424 (time 1790785405.26 total 20.03 last 69.06)
 - open/close 2114 (time 1790785415.27 total 30.03 last 68.95)
 - open/close 2681 (time 1790785425.27 total 40.04 last 56.69)
 - open/close 3307 (time 1790785435.27 total 50.04 last 62.59)
 - open/close 3949 (time 1790785445.29 total 60.06 last 64.06)
total: 4500 open/close in 69.46 seconds: 64.79 ops/second
 - /unlink 1248 (time 1790785470.92 total 10.00 last 124.80)
 - /unlink 2462 (time 1790785480.92 total 20.00 last 121.37)
 - /unlink 3951 (time 1790785490.92 total 30.00 last 148.87)
total: 4500 /unlink in 33.61 seconds: 133.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 633 (time 1790785509.95 total 10.03 last 63.09)
 - open/close 1187 (time 1790785519.97 total 20.06 last 55.26)
 - open/close 1731 (time 1790785529.98 total 30.06 last 54.37)
 - open/close 2223 (time 1790785539.99 total 40.07 last 49.14)
 - open/close 2724 (time 1790785549.99 total 50.08 last 50.09)
 - open/close 3367 (time 1790785559.99 total 60.08 last 64.27)
 - open/close 4134 (time 1790785570.00 total 70.09 last 76.63)
total: 4500 open/close in 75.84 seconds: 59.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1368 (time 1790785590.50 total 10.01 last 136.66)
 - /unlink 2718 (time 1790785600.50 total 20.01 last 134.98)
 - /unlink 3945 (time 1790785610.50 total 30.01 last 122.68)
total: 4500 /unlink in 33.96 seconds: 132.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 695 (time 1790785630.06 total 10.01 last 69.45)
 - open/close 1450 (time 1790785640.06 total 20.01 last 75.47)
 - open/close 2219 (time 1790785650.08 total 30.03 last 76.76)
 - open/close 2866 (time 1790785660.09 total 40.04 last 64.63)
 - open/close 3641 (time 1790785670.10 total 50.05 last 77.44)
 - open/close 4376 (time 1790785680.10 total 60.05 last 73.49)
total: 4500 open/close in 61.41 seconds: 73.27 ops/second
 - /unlink 1496 (time 1790785697.26 total 10.00 last 149.59)
 - /unlink 2585 (time 1790785707.26 total 20.00 last 108.86)
 - /unlink 3784 (time 1790785717.28 total 30.02 last 119.71)
total: 4500 /unlink in 37.36 seconds: 120.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 694 (time 1790785740.22 total 10.01 last 69.35)
 - open/close 1360 (time 1790785750.23 total 20.01 last 66.59)
 - open/close 1822 (time 1790785760.23 total 30.01 last 46.19)
 - open/close 2397 (time 1790785770.26 total 40.04 last 57.34)
 - open/close 2999 (time 1790785780.26 total 50.05 last 60.16)
 - open/close 3691 (time 1790785790.27 total 60.05 last 69.16)
 - open/close 4430 (time 1790785800.27 total 70.06 last 73.86)
total: 4500 open/close in 70.93 seconds: 63.44 ops/second
 - /unlink 1312 (time 1790785816.15 total 10.01 last 131.11)
 - /unlink 2419 (time 1790785826.15 total 20.01 last 110.68)
 - /unlink 3337 (time 1790785836.15 total 30.01 last 91.80)
 - /unlink 4351 (time 1790785846.16 total 40.01 last 101.35)
total: 4500 /unlink in 41.40 seconds: 108.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 584 (time 1790785863.33 total 10.01 last 58.34)
 - open/close 1291 (time 1790785873.34 total 20.02 last 70.61)
 - open/close 2013 (time 1790785883.35 total 30.03 last 72.16)
 - open/close 2647 (time 1790785893.35 total 40.03 last 63.39)
 - open/close 3382 (time 1790785903.35 total 50.03 last 73.47)
 - open/close 4053 (time 1790785913.36 total 60.04 last 67.09)
total: 4500 open/close in 66.20 seconds: 67.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1446 (time 1790785934.15 total 10.00 last 144.60)
 - /unlink 2715 (time 1790785944.17 total 20.02 last 126.65)
 - /unlink 4054 (time 1790785954.18 total 30.03 last 133.83)
total: 4500 /unlink in 33.57 seconds: 134.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 725 (time 1790785972.68 total 10.00 last 72.47)
 - open/close 1332 (time 1790785982.70 total 20.02 last 60.57)
 - open/close 1994 (time 1790785992.74 total 30.06 last 65.97)
 - open/close 2684 (time 1790786002.74 total 40.06 last 68.98)
 - open/close 3391 (time 1790786012.75 total 50.07 last 70.61)
 - open/close 4020 (time 1790786022.76 total 60.08 last 62.89)
total: 4500 open/close in 68.12 seconds: 66.06 ops/second
 - /unlink 1289 (time 1790786046.24 total 10.00 last 128.85)
 - /unlink 2445 (time 1790786056.24 total 20.00 last 115.60)
 - /unlink 3779 (time 1790786066.24 total 30.01 last 133.35)
total: 4500 /unlink in 35.51 seconds: 126.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 768 (time 1790786086.80 total 10.01 last 76.69)
 - open/close 1496 (time 1790786096.81 total 20.02 last 72.76)
 - open/close 2052 (time 1790786106.84 total 30.04 last 55.46)
 - open/close 2774 (time 1790786116.85 total 40.06 last 72.10)
 - open/close 3584 (time 1790786126.86 total 50.07 last 80.89)
 - open/close 4310 (time 1790786136.87 total 60.08 last 72.58)
total: 4500 open/close in 62.64 seconds: 71.84 ops/second
 - /unlink 1314 (time 1790786154.32 total 10.00 last 131.36)
 - /unlink 2742 (time 1790786164.33 total 20.01 last 142.75)
 - /unlink 4209 (time 1790786174.33 total 30.01 last 146.63)
total: 4500 /unlink in 32.40 seconds: 138.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 686 (time 1790786192.21 total 10.00 last 68.58)
 - open/close 1409 (time 1790786202.21 total 20.00 last 72.29)
 - open/close 2140 (time 1790786212.21 total 30.01 last 73.08)
 - open/close 2895 (time 1790786222.22 total 40.01 last 75.46)
 - open/close 3677 (time 1790786232.22 total 50.01 last 78.20)
 - open/close 4465 (time 1790786242.22 total 60.02 last 78.75)
total: 4500 open/close in 60.46 seconds: 74.42 ops/second
 - /unlink 1475 (time 1790786257.42 total 10.00 last 147.45)
 - /unlink 2846 (time 1790786267.42 total 20.01 last 137.03)
 - /unlink 4112 (time 1790786277.43 total 30.02 last 126.42)
total: 4500 /unlink in 33.08 seconds: 136.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 821 (time 1790786295.51 total 10.01 last 81.99)
 - open/close 1643 (time 1790786305.52 total 20.03 last 82.09)
 - open/close 2257 (time 1790786315.54 total 30.05 last 61.28)
 - open/close 2819 (time 1790786325.56 total 40.06 last 56.11)
 - open/close 3612 (time 1790786335.57 total 50.07 last 79.21)
 - open/close 4345 (time 1790786345.58 total 60.09 last 73.22)
total: 4500 open/close in 62.30 seconds: 72.23 ops/second
 - /unlink 1563 (time 1790786362.85 total 10.00 last 156.28)
 - /unlink 3017 (time 1790786372.86 total 20.00 last 145.37)
 - /unlink 4333 (time 1790786382.86 total 30.01 last 131.56)
total: 4500 /unlink in 31.60 seconds: 142.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 723 (time 1790786398.78 total 10.00 last 72.29)
 - open/close 1455 (time 1790786408.78 total 20.01 last 73.17)
 - open/close 2149 (time 1790786418.80 total 30.02 last 69.30)
 - open/close 2962 (time 1790786428.82 total 40.04 last 81.14)
 - open/close 3691 (time 1790786438.82 total 50.04 last 72.90)
 - open/close 4321 (time 1790786448.83 total 60.05 last 62.91)
total: 4500 open/close in 62.91 seconds: 71.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1106 (time 1790786467.88 total 10.02 last 110.36)
 - /unlink 2313 (time 1790786477.88 total 20.02 last 120.68)
 - /unlink 3673 (time 1790786487.89 total 30.03 last 135.91)
total: 4500 /unlink in 36.96 seconds: 121.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 573 (time 1790786511.43 total 10.01 last 57.22)
 - open/close 1223 (time 1790786521.43 total 20.02 last 64.97)
 - open/close 1928 (time 1790786531.43 total 30.02 last 70.49)
 - open/close 2628 (time 1790786541.44 total 40.03 last 69.95)
 - open/close 3391 (time 1790786551.44 total 50.03 last 76.26)
 - open/close 4188 (time 1790786561.45 total 60.04 last 79.66)
total: 4500 open/close in 64.69 seconds: 69.57 ops/second
 - /unlink 1303 (time 1790786580.81 total 10.01 last 130.23)
 - /unlink 2659 (time 1790786590.81 total 20.01 last 135.57)
 - /unlink 3906 (time 1790786600.82 total 30.01 last 124.66)
total: 4500 /unlink in 36.44 seconds: 123.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 625 (time 1790786623.67 total 10.01 last 62.46)
 - open/close 1356 (time 1790786633.68 total 20.01 last 73.04)
 - open/close 2054 (time 1790786643.68 total 30.02 last 69.76)
 - open/close 2697 (time 1790786653.68 total 40.02 last 64.29)
 - open/close 3394 (time 1790786663.69 total 50.02 last 69.67)
 - open/close 4111 (time 1790786673.70 total 60.04 last 71.59)
total: 4500 open/close in 65.44 seconds: 68.77 ops/second
 - /unlink 1218 (time 1790786693.80 total 10.01 last 121.72)
 - /unlink 2536 (time 1790786703.81 total 20.01 last 131.69)
 - /unlink 3896 (time 1790786713.81 total 30.02 last 135.99)
total: 4500 /unlink in 35.41 seconds: 127.09 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11358: 95565 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg249-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg249-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server
stop mds service on oleg249-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server
stop mds service on oleg249-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server
unloading modules via unload_modules_local on: 'oleg249-server'
oleg249-server: oleg249-server.virtnet: executing unload_modules_local
oleg249-server: kmemleak disabled
modules unloaded.