== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:20:45 (1788668445)
Stopping clients: oleg122-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg122-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg122-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg122-server
oleg122-server: oleg122-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing load_modules_local
oleg122-server: kmemleak disabled
oleg122-server: Loading modules from /home/green/git/lustre-release/lustre
oleg122-server: detected 4 online CPUs by sysfs
oleg122-server: MODOPTS_LIBCFS=
oleg122-server: Force libcfs to create 2 CPU partitions
oleg122-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg122-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg122-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg122-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all
pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg122-client.virtnet:  -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 757 (time 1788668662.45 total 10.00 last 75.66)
 - open/close 1676 (time 1788668672.46 total 20.02 last 91.80)
 - open/close 2404 (time 1788668682.47 total 30.03 last 72.69)
 - open/close 3109 (time 1788668692.48 total 40.03 last 70.49)
 - open/close 3837 (time 1788668702.49 total 50.05 last 72.70)
total: 4500 open/close in 58.53 seconds: 76.88 ops/second
 - /unlink 1375 (time 1788668725.43 total 10.01 last 137.42)
 - /unlink 2847 (time 1788668735.43 total 20.01 last 147.17)
 - /unlink 4253 (time 1788668745.43 total 30.01 last 140.60)
total: 4500 /unlink in 32.29 seconds: 139.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 752 (time 1788668762.83 total 10.01 last 75.16)
 - open/close 1531 (time 1788668772.85 total 20.03 last 77.73)
 - open/close 2257 (time 1788668782.88 total 30.06 last 72.40)
 - open/close 2993 (time 1788668792.89 total 40.06 last 73.56)
 - open/close 3673 (time 1788668802.89 total 50.07 last 67.94)
 - open/close 4423 (time 1788668812.90 total 60.08 last 74.95)
total: 4500 open/close in 61.25 seconds: 73.47 ops/second
 - /unlink 1216 (time 1788668829.91 total 10.01 last 121.51)
 - /unlink 2539 (time 1788668839.92 total 20.01 last 132.23)
 - /unlink 3753 (time 1788668849.92 total 30.02 last 121.32)
total: 4500 /unlink in 36.31 seconds: 123.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 752 (time 1788668871.32 total 10.00 last 75.17)
 - open/close 1512 (time 1788668881.33 total 20.02 last 75.88)
 - open/close 2266 (time 1788668891.34 total 30.02 last 75.38)
 - open/close 3032 (time 1788668901.36 total 40.04 last 76.43)
 - open/close 3706 (time 1788668911.38 total 50.07 last 67.23)
 - open/close 4230 (time 1788668921.39 total 60.08 last 52.35)
total: 4500 open/close in 65.03 seconds: 69.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1183 (time 1788668943.69 total 10.00 last 118.29)
 - /unlink 2204 (time 1788668953.70 total 20.01 last 102.00)
 - /unlink 3198 (time 1788668963.70 total 30.02 last 99.34)
total: 4500 /unlink in 39.82 seconds: 113.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 669 (time 1788668988.66 total 10.00 last 66.87)
 - open/close 1332 (time 1788668998.67 total 20.01 last 66.26)
 - open/close 2024 (time 1788669008.68 total 30.02 last 69.10)
 - open/close 2816 (time 1788669018.69 total 40.03 last 79.14)
 - open/close 3552 (time 1788669028.69 total 50.03 last 73.58)
 - open/close 4237 (time 1788669038.71 total 60.06 last 68.35)
total: 4500 open/close in 64.04 seconds: 70.27 ops/second
 - /unlink 1555 (time 1788669058.43 total 10.00 last 155.44)
 - /unlink 2808 (time 1788669068.43 total 20.00 last 125.30)
 - /unlink 4123 (time 1788669078.43 total 30.00 last 131.49)
total: 4500 /unlink in 32.83 seconds: 137.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 677 (time 1788669097.21 total 10.00 last 67.68)
 - open/close 1373 (time 1788669107.21 total 20.00 last 69.59)
 - open/close 2019 (time 1788669117.22 total 30.01 last 64.54)
 - open/close 2727 (time 1788669127.22 total 40.01 last 70.79)
 - open/close 3511 (time 1788669137.22 total 50.01 last 78.39)
 - open/close 4195 (time 1788669147.23 total 60.02 last 68.36)
total: 4500 open/close in 64.74 seconds: 69.50 ops/second
 - /unlink 1297 (time 1788669167.34 total 10.01 last 129.61)
 - /unlink 2718 (time 1788669177.34 total 20.01 last 142.06)
 - /unlink 4030 (time 1788669187.34 total 30.01 last 131.19)
total: 4500 /unlink in 33.84 seconds: 132.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 709 (time 1788669206.18 total 10.00 last 70.87)
 - open/close 1342 (time 1788669216.18 total 20.01 last 63.27)
 - open/close 2045 (time 1788669226.18 total 30.01 last 70.29)
 - open/close 2797 (time 1788669236.19 total 40.02 last 75.15)
 - open/close 3580 (time 1788669246.19 total 50.02 last 78.27)
 - open/close 4361 (time 1788669256.20 total 60.03 last 78.04)
total: 4500 open/close in 61.75 seconds: 72.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1365 (time 1788669272.68 total 10.01 last 136.40)
 - /unlink 2681 (time 1788669282.68 total 20.01 last 131.55)
 - /unlink 3860 (time 1788669292.68 total 30.01 last 117.89)
total: 4500 /unlink in 35.04 seconds: 128.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 650 (time 1788669314.19 total 10.04 last 64.73)
 - open/close 1442 (time 1788669324.21 total 20.06 last 79.09)
 - open/close 2233 (time 1788669334.21 total 30.06 last 79.09)
 - open/close 2814 (time 1788669344.22 total 40.07 last 58.02)
 - open/close 3445 (time 1788669354.23 total 50.07 last 63.07)
 - open/close 4092 (time 1788669364.24 total 60.09 last 64.63)
total: 4500 open/close in 66.51 seconds: 67.66 ops/second
 - /unlink 1333 (time 1788669387.25 total 10.01 last 133.23)
 - /unlink 2802 (time 1788669397.25 total 20.01 last 146.87)
 - /unlink 4024 (time 1788669407.26 total 30.01 last 122.16)
total: 4500 /unlink in 33.46 seconds: 134.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 797 (time 1788669426.38 total 10.01 last 79.61)
 - open/close 1600 (time 1788669436.41 total 20.03 last 80.12)
 - open/close 2344 (time 1788669446.41 total 30.04 last 74.39)
 - open/close 3180 (time 1788669456.42 total 40.05 last 83.51)
 - open/close 3994 (time 1788669466.43 total 50.06 last 81.30)
total: 4500 open/close in 56.38 seconds: 79.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1477 (time 1788669486.79 total 10.01 last 147.59)
 - /unlink 2719 (time 1788669496.79 total 20.01 last 124.15)
 - /unlink 4120 (time 1788669506.79 total 30.02 last 140.05)
total: 4500 /unlink in 32.60 seconds: 138.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 800 (time 1788669523.76 total 10.00 last 79.97)
 - open/close 1536 (time 1788669533.78 total 20.02 last 73.47)
 - open/close 2244 (time 1788669543.79 total 30.03 last 70.73)
 - open/close 2907 (time 1788669553.80 total 40.04 last 66.27)
 - open/close 3748 (time 1788669563.80 total 50.04 last 84.03)
 - open/close 4428 (time 1788669573.81 total 60.05 last 67.98)
total: 4500 open/close in 61.08 seconds: 73.67 ops/second
 - /unlink 1376 (time 1788669589.64 total 10.00 last 137.54)
 - /unlink 2870 (time 1788669599.64 total 20.00 last 149.40)
 - /unlink 4252 (time 1788669609.64 total 30.01 last 138.14)
total: 4500 /unlink in 32.11 seconds: 140.13 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 690 (time 1788669627.32 total 10.00 last 68.97)
 - open/close 1447 (time 1788669637.32 total 20.01 last 75.66)
 - open/close 2121 (time 1788669647.34 total 30.02 last 67.32)
 - open/close 2737 (time 1788669657.34 total 40.02 last 61.59)
 - open/close 3390 (time 1788669667.35 total 50.04 last 65.20)
 - open/close 4009 (time 1788669677.37 total 60.05 last 61.82)
total: 4500 open/close in 67.46 seconds: 66.70 ops/second
 - /unlink 1395 (time 1788669699.94 total 10.00 last 139.49)
 - /unlink 2689 (time 1788669709.94 total 20.00 last 129.37)
 - /unlink 3829 (time 1788669719.94 total 30.01 last 113.97)
total: 4500 /unlink in 34.66 seconds: 129.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 602 (time 1788669741.43 total 10.01 last 60.16)
 - open/close 1181 (time 1788669751.43 total 20.01 last 57.89)
 - open/close 1916 (time 1788669761.45 total 30.03 last 73.36)
 - open/close 2607 (time 1788669771.45 total 40.03 last 69.06)
 - open/close 3229 (time 1788669781.47 total 50.05 last 62.08)
 - open/close 3822 (time 1788669791.49 total 60.07 last 59.22)
 - open/close 4475 (time 1788669801.49 total 70.07 last 65.30)
total: 4500 open/close in 70.54 seconds: 63.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1384 (time 1788669817.90 total 10.00 last 138.40)
 - /unlink 2709 (time 1788669827.90 total 20.00 last 132.45)
 - /unlink 3986 (time 1788669837.90 total 30.00 last 127.69)
total: 4500 /unlink in 33.92 seconds: 132.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 825 (time 1788669858.19 total 10.01 last 82.45)
 - open/close 1640 (time 1788669868.20 total 20.02 last 81.40)
 - open/close 2386 (time 1788669878.21 total 30.03 last 74.50)
 - open/close 3161 (time 1788669888.22 total 40.04 last 77.44)
 - open/close 3957 (time 1788669898.22 total 50.04 last 79.59)
total: 4500 open/close in 58.07 seconds: 77.50 ops/second
 - /unlink 1284 (time 1788669921.82 total 10.01 last 128.29)
 - /unlink 2533 (time 1788669931.82 total 20.01 last 124.83)
 - /unlink 3918 (time 1788669941.83 total 30.02 last 138.45)
total: 4500 /unlink in 34.09 seconds: 132.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 676 (time 1788669961.54 total 10.01 last 67.55)
 - open/close 1401 (time 1788669971.55 total 20.02 last 72.43)
 - open/close 2158 (time 1788669981.56 total 30.03 last 75.63)
 - open/close 2859 (time 1788669991.56 total 40.03 last 70.08)
 - open/close 3581 (time 1788670001.56 total 50.03 last 72.18)
 - open/close 4300 (time 1788670011.57 total 60.04 last 71.83)
total: 4500 open/close in 62.49 seconds: 72.01 ops/second
 - /unlink 1010 (time 1788670028.18 total 10.00 last 100.98)
 - /unlink 1727 (time 1788670038.19 total 20.01 last 71.63)
 - /unlink 2749 (time 1788670048.19 total 30.01 last 102.19)
 - /unlink 4046 (time 1788670058.19 total 40.01 last 129.70)
total: 4500 /unlink in 43.45 seconds: 103.56 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 623 (time 1788670077.11 total 10.02 last 62.15)
 - open/close 1202 (time 1788670087.12 total 20.03 last 57.84)
 - open/close 1875 (time 1788670097.15 total 30.07 last 67.06)
 - open/close 2605 (time 1788670107.16 total 40.08 last 72.95)
 - open/close 3314 (time 1788670117.16 total 50.08 last 70.88)
 - open/close 3999 (time 1788670127.16 total 60.08 last 68.48)
total: 4500 open/close in 68.29 seconds: 65.90 ops/second
 - /unlink 1345 (time 1788670149.79 total 10.01 last 134.41)
 - /unlink 2705 (time 1788670159.79 total 20.01 last 135.96)
 - /unlink 4030 (time 1788670169.80 total 30.01 last 132.47)
total: 4500 /unlink in 33.78 seconds: 133.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 744 (time 1788670189.45 total 10.00 last 74.38)
 - open/close 1465 (time 1788670199.47 total 20.02 last 71.96)
 - open/close 2146 (time 1788670209.49 total 30.04 last 68.00)
 - open/close 2814 (time 1788670219.49 total 40.04 last 66.76)
 - open/close 3562 (time 1788670229.50 total 50.05 last 74.75)
 - open/close 4266 (time 1788670239.51 total 60.06 last 70.36)
total: 4500 open/close in 63.69 seconds: 70.65 ops/second
 - /unlink 1341 (time 1788670258.48 total 10.01 last 133.99)
 - /unlink 2585 (time 1788670268.48 total 20.01 last 124.39)
 - /unlink 3649 (time 1788670278.48 total 30.01 last 106.36)
total: 4500 /unlink in 36.42 seconds: 123.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 786 (time 1788670301.02 total 10.01 last 78.55)
 - open/close 1489 (time 1788670311.03 total 20.01 last 70.24)
 - open/close 2269 (time 1788670321.06 total 30.04 last 77.78)
 - open/close 2920 (time 1788670331.06 total 40.05 last 65.08)
 - open/close 3561 (time 1788670341.07 total 50.05 last 64.07)
 - open/close 4314 (time 1788670351.08 total 60.06 last 75.24)
total: 4500 open/close in 62.82 seconds: 71.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1356 (time 1788670369.28 total 10.01 last 135.49)
 - /unlink 2491 (time 1788670379.28 total 20.01 last 113.49)
 - /unlink 3949 (time 1788670389.31 total 30.04 last 145.42)
total: 4500 /unlink in 34.34 seconds: 131.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 687 (time 1788670409.25 total 10.00 last 68.67)
 - open/close 1441 (time 1788670419.26 total 20.02 last 75.30)
 - open/close 2264 (time 1788670429.28 total 30.04 last 82.13)
 - open/close 2962 (time 1788670439.29 total 40.05 last 69.73)
 - open/close 3637 (time 1788670449.31 total 50.06 last 67.40)
 - open/close 4275 (time 1788670459.34 total 60.09 last 63.62)
total: 4500 open/close in 63.20 seconds: 71.21 ops/second
 - /unlink 1496 (time 1788670478.03 total 10.00 last 149.58)
 - /unlink 2941 (time 1788670488.03 total 20.00 last 144.44)
 - /unlink 4251 (time 1788670498.04 total 30.01 last 130.91)
total: 4500 /unlink in 32.03 seconds: 140.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 798 (time 1788670515.45 total 10.01 last 79.73)
 - open/close 1474 (time 1788670525.46 total 20.02 last 67.53)
 - open/close 2181 (time 1788670535.46 total 30.03 last 70.65)
 - open/close 2935 (time 1788670545.47 total 40.03 last 75.35)
 - open/close 3679 (time 1788670555.48 total 50.04 last 74.37)
 - open/close 4350 (time 1788670565.53 total 60.09 last 66.75)
total: 4500 open/close in 62.27 seconds: 72.27 ops/second
 - /unlink 1538 (time 1788670582.26 total 10.00 last 153.78)
 - /unlink 2859 (time 1788670592.26 total 20.00 last 132.09)
 - /unlink 4169 (time 1788670602.27 total 30.01 last 130.87)
total: 4500 /unlink in 32.63 seconds: 137.91 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95253 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg122-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg122-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server
stop mds service on oleg122-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg122-server
unloading modules via unload_modules_local on: 'oleg122-server'
oleg122-server: oleg122-server.virtnet: executing unload_modules_local
oleg122-server: kmemleak disabled
modules unloaded.