== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:29:43 (1788442183)
Stopping clients: oleg657-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg657-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg657-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg657-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg657-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg657-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg657-server
oleg657-server: oleg657-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg657-server'
oleg657-server: oleg657-server.virtnet: executing load_modules_local
oleg657-server: kmemleak disabled
oleg657-server: Loading modules from /home/green/git/lustre-release/lustre
oleg657-server: detected 4 online CPUs by sysfs
oleg657-server: MODOPTS_LIBCFS=
oleg657-server: Force libcfs to create 2 CPU partitions
oleg657-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg657-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg657-server: oleg657-server.virtnet: executing set_default_debug -1 all
pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg657-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg657-server: oleg657-server.virtnet: executing set_default_debug -1 all
pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg657-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg657-server: oleg657-server.virtnet: executing set_default_debug -1 all
pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg657-client.virtnet:  -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 640 (time 1788442370.69 total 10.01 last 63.95)
 - open/close 1353 (time 1788442380.69 total 20.01 last 71.28)
 - open/close 2132 (time 1788442390.70 total 30.02 last 77.80)
 - open/close 2867 (time 1788442400.73 total 40.06 last 73.26)
 - open/close 3600 (time 1788442410.75 total 50.07 last 73.21)
 - open/close 4354 (time 1788442420.76 total 60.08 last 75.28)
total: 4500 open/close in 62.14 seconds: 72.42 ops/second
 - /unlink 1430 (time 1788442436.92 total 10.01 last 142.90)
 - /unlink 2912 (time 1788442446.93 total 20.02 last 148.02)
 - /unlink 4344 (time 1788442456.94 total 30.02 last 143.16)
total: 4500 /unlink in 31.34 seconds: 143.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 722 (time 1788442472.93 total 10.01 last 72.16)
 - open/close 1407 (time 1788442482.93 total 20.01 last 68.49)
 - open/close 2091 (time 1788442492.94 total 30.02 last 68.30)
 - open/close 2768 (time 1788442502.94 total 40.02 last 67.70)
 - open/close 3425 (time 1788442512.95 total 50.03 last 65.63)
 - open/close 4206 (time 1788442522.96 total 60.04 last 78.06)
total: 4500 open/close in 63.95 seconds: 70.37 ops/second
 - /unlink 1331 (time 1788442541.40 total 10.01 last 133.02)
 - /unlink 2649 (time 1788442551.40 total 20.01 last 131.79)
 - /unlink 4095 (time 1788442561.41 total 30.02 last 144.47)
total: 4500 /unlink in 32.96 seconds: 136.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 751 (time 1788442579.33 total 10.02 last 74.98)
 - open/close 1452 (time 1788442589.34 total 20.03 last 70.00)
 - open/close 2147 (time 1788442599.35 total 30.04 last 69.43)
 - open/close 2901 (time 1788442609.35 total 40.04 last 75.36)
 - open/close 3663 (time 1788442619.36 total 50.05 last 76.18)
 - open/close 4405 (time 1788442629.36 total 60.05 last 74.17)
total: 4500 open/close in 61.29 seconds: 73.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1411 (time 1788442645.31 total 10.00 last 141.09)
 - /unlink 2613 (time 1788442655.32 total 20.00 last 120.16)
 - /unlink 3950 (time 1788442665.32 total 30.01 last 133.64)
total: 4500 /unlink in 34.07 seconds: 132.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 753 (time 1788442684.47 total 10.01 last 75.22)
 - open/close 1499 (time 1788442694.48 total 20.02 last 74.54)
 - open/close 2164 (time 1788442704.51 total 30.06 last 66.25)
 - open/close 2917 (time 1788442714.52 total 40.06 last 75.26)
 - open/close 3613 (time 1788442724.53 total 50.07 last 69.56)
 - open/close 4234 (time 1788442734.53 total 60.08 last 62.06)
total: 4500 open/close in 64.14 seconds: 70.16 ops/second
 - /unlink 1385 (time 1788442753.71 total 10.01 last 138.41)
 - /unlink 2837 (time 1788442763.71 total 20.01 last 145.13)
 - /unlink 4209 (time 1788442773.72 total 30.01 last 137.16)
total: 4500 /unlink in 32.42 seconds: 138.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 764 (time 1788442791.08 total 10.00 last 76.38)
 - open/close 1474 (time 1788442801.09 total 20.01 last 70.92)
 - open/close 2253 (time 1788442811.09 total 30.02 last 77.89)
 - open/close 2962 (time 1788442821.10 total 40.03 last 70.83)
 - open/close 3703 (time 1788442831.10 total 50.03 last 74.09)
 - open/close 4428 (time 1788442841.11 total 60.04 last 72.42)
total: 4500 open/close in 61.29 seconds: 73.43 ops/second
 - /unlink 1335 (time 1788442857.26 total 10.00 last 133.44)
 - /unlink 2621 (time 1788442867.27 total 20.01 last 128.57)
 - /unlink 3912 (time 1788442877.27 total 30.01 last 129.03)
total: 4500 /unlink in 34.12 seconds: 131.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 675 (time 1788442897.08 total 10.00 last 67.48)
 - open/close 1372 (time 1788442907.10 total 20.03 last 69.54)
 - open/close 2012 (time 1788442917.10 total 30.03 last 63.98)
 - open/close 2718 (time 1788442927.11 total 40.04 last 70.54)
 - open/close 3414 (time 1788442937.11 total 50.04 last 69.60)
 - open/close 4173 (time 1788442947.11 total 60.04 last 75.89)
total: 4500 open/close in 65.07 seconds: 69.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1407 (time 1788442966.87 total 10.00 last 140.66)
 - /unlink 2599 (time 1788442976.87 total 20.01 last 119.17)
 - /unlink 3858 (time 1788442986.87 total 30.01 last 125.84)
total: 4500 /unlink in 35.65 seconds: 126.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 744 (time 1788443006.99 total 10.01 last 74.30)
 - open/close 1450 (time 1788443017.00 total 20.02 last 70.54)
 - open/close 2174 (time 1788443027.00 total 30.03 last 72.35)
 - open/close 2932 (time 1788443037.01 total 40.04 last 75.74)
 - open/close 3724 (time 1788443047.01 total 50.04 last 79.19)
 - open/close 4474 (time 1788443057.02 total 60.04 last 74.97)
total: 4500 open/close in 60.36 seconds: 74.55 ops/second
 - /unlink 1427 (time 1788443071.75 total 10.00 last 142.64)
 - /unlink 2720 (time 1788443081.75 total 20.01 last 129.29)
 - /unlink 4089 (time 1788443091.76 total 30.01 last 136.88)
total: 4500 /unlink in 34.13 seconds: 131.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 614 (time 1788443111.27 total 10.01 last 61.35)
 - open/close 1211 (time 1788443121.27 total 20.01 last 59.68)
 - open/close 1900 (time 1788443131.28 total 30.02 last 68.84)
 - open/close 2633 (time 1788443141.29 total 40.03 last 73.24)
 - open/close 3328 (time 1788443151.30 total 50.03 last 69.47)
 - open/close 4098 (time 1788443161.30 total 60.04 last 76.95)
total: 4500 open/close in 65.23 seconds: 68.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1373 (time 1788443180.99 total 10.00 last 137.28)
 - /unlink 2787 (time 1788443191.00 total 20.01 last 141.30)
 - /unlink 4188 (time 1788443201.00 total 30.01 last 140.10)
total: 4500 /unlink in 32.25 seconds: 139.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 766 (time 1788443217.66 total 10.01 last 76.55)
 - open/close 1499 (time 1788443227.66 total 20.01 last 73.27)
 - open/close 2226 (time 1788443237.67 total 30.02 last 72.64)
 - open/close 2961 (time 1788443247.68 total 40.02 last 73.47)
 - open/close 3763 (time 1788443257.70 total 50.05 last 80.03)
total: 4500 open/close in 59.34 seconds: 75.84 ops/second
 - /unlink 1370 (time 1788443281.35 total 10.00 last 136.97)
 - /unlink 2691 (time 1788443291.35 total 20.01 last 132.00)
 - /unlink 4075 (time 1788443301.35 total 30.01 last 138.38)
total: 4500 /unlink in 33.15 seconds: 135.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 699 (time 1788443318.88 total 10.01 last 69.82)
 - open/close 1432 (time 1788443328.89 total 20.02 last 73.23)
 - open/close 2175 (time 1788443338.90 total 30.03 last 74.23)
 - open/close 2860 (time 1788443348.91 total 40.04 last 68.43)
 - open/close 3505 (time 1788443358.91 total 50.05 last 64.47)
 - open/close 4280 (time 1788443368.93 total 60.06 last 77.38)
total: 4500 open/close in 63.25 seconds: 71.14 ops/second
 - /unlink 1472 (time 1788443386.60 total 10.00 last 147.19)
 - /unlink 2822 (time 1788443396.60 total 20.00 last 134.99)
 - /unlink 4082 (time 1788443406.60 total 30.00 last 125.96)
total: 4500 /unlink in 33.33 seconds: 135.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 787 (time 1788443424.58 total 10.01 last 78.66)
 - open/close 1560 (time 1788443434.59 total 20.01 last 77.28)
 - open/close 2302 (time 1788443444.59 total 30.01 last 74.18)
 - open/close 3039 (time 1788443454.60 total 40.02 last 73.64)
 - open/close 3797 (time 1788443464.60 total 50.02 last 75.77)
total: 4500 open/close in 59.54 seconds: 75.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1377 (time 1788443488.43 total 10.00 last 137.66)
 - /unlink 2808 (time 1788443498.43 total 20.01 last 143.03)
 - /unlink 4320 (time 1788443508.44 total 30.02 last 151.07)
total: 4500 /unlink in 31.54 seconds: 142.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 781 (time 1788443524.39 total 10.00 last 78.08)
 - open/close 1540 (time 1788443534.40 total 20.01 last 75.81)
 - open/close 2279 (time 1788443544.40 total 30.02 last 73.89)
 - open/close 2994 (time 1788443554.41 total 40.02 last 71.47)
 - open/close 3705 (time 1788443564.41 total 50.03 last 71.06)
 - open/close 4373 (time 1788443574.42 total 60.04 last 66.73)
total: 4500 open/close in 62.09 seconds: 72.48 ops/second
 - /unlink 1351 (time 1788443591.21 total 10.01 last 134.98)
 - /unlink 2742 (time 1788443601.23 total 20.02 last 138.89)
 - /unlink 4148 (time 1788443611.24 total 30.04 last 140.43)
total: 4500 /unlink in 32.91 seconds: 136.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 662 (time 1788443629.68 total 10.02 last 66.09)
 - open/close 1251 (time 1788443639.69 total 20.03 last 58.83)
 - open/close 2013 (time 1788443649.70 total 30.03 last 76.15)
 - open/close 2690 (time 1788443659.70 total 40.04 last 67.66)
 - open/close 3413 (time 1788443669.71 total 50.04 last 72.28)
 - open/close 3945 (time 1788443679.72 total 60.06 last 53.13)
total: 4500 open/close in 69.57 seconds: 64.68 ops/second
 - /unlink 1339 (time 1788443704.58 total 10.00 last 133.85)
 - /unlink 2668 (time 1788443714.59 total 20.01 last 132.77)
 - /unlink 4053 (time 1788443724.59 total 30.02 last 138.46)
total: 4500 /unlink in 33.40 seconds: 134.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 744 (time 1788443742.63 total 10.03 last 74.17)
 - open/close 1498 (time 1788443752.63 total 20.04 last 75.37)
 - open/close 2210 (time 1788443762.63 total 30.04 last 71.19)
 - open/close 3013 (time 1788443772.64 total 40.05 last 80.22)
 - open/close 3765 (time 1788443782.65 total 50.05 last 75.18)
total: 4500 open/close in 59.93 seconds: 75.09 ops/second
 - /unlink 1460 (time 1788443806.67 total 10.00 last 145.98)
 - /unlink 2776 (time 1788443816.67 total 20.01 last 131.56)
 - /unlink 4120 (time 1788443826.67 total 30.01 last 134.40)
total: 4500 /unlink in 32.72 seconds: 137.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 756 (time 1788443843.81 total 10.00 last 75.57)
 - open/close 1570 (time 1788443853.81 total 20.01 last 81.36)
 - open/close 2332 (time 1788443863.82 total 30.02 last 76.14)
 - open/close 3057 (time 1788443873.82 total 40.02 last 72.48)
 - open/close 3817 (time 1788443883.83 total 50.03 last 75.96)
total: 4500 open/close in 59.00 seconds: 76.27 ops/second
 - /unlink 1370 (time 1788443907.15 total 10.00 last 136.99)
 - /unlink 2777 (time 1788443917.15 total 20.00 last 140.70)
 - /unlink 4126 (time 1788443927.16 total 30.01 last 134.84)
total: 4500 /unlink in 32.71 seconds: 137.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 732 (time 1788443944.44 total 10.00 last 73.20)
 - open/close 1462 (time 1788443954.45 total 20.01 last 72.95)
 - open/close 2205 (time 1788443964.45 total 30.01 last 74.30)
 - open/close 2739 (time 1788443974.45 total 40.01 last 53.39)
 - open/close 3363 (time 1788443984.47 total 50.03 last 62.26)
 - open/close 3841 (time 1788443994.51 total 60.07 last 47.62)
 - open/close 4406 (time 1788444004.52 total 70.08 last 56.45)
total: 4500 open/close in 71.42 seconds: 63.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1494 (time 1788444020.11 total 10.00 last 149.37)
 - /unlink 2838 (time 1788444030.11 total 20.01 last 134.34)
 - /unlink 4110 (time 1788444040.12 total 30.02 last 127.08)
total: 4500 /unlink in 32.89 seconds: 136.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 741 (time 1788444059.46 total 10.00 last 74.09)
 - open/close 1455 (time 1788444069.46 total 20.00 last 71.39)
 - open/close 2188 (time 1788444079.47 total 30.01 last 73.24)
 - open/close 2913 (time 1788444089.48 total 40.02 last 72.46)
 - open/close 3595 (time 1788444099.50 total 50.04 last 68.05)
 - open/close 4215 (time 1788444109.51 total 60.05 last 61.94)
total: 4500 open/close in 65.73 seconds: 68.46 ops/second
 - /unlink 1379 (time 1788444130.67 total 10.00 last 137.89)
 - /unlink 2635 (time 1788444140.68 total 20.01 last 125.54)
 - /unlink 4084 (time 1788444150.68 total 30.01 last 144.83)
total: 4500 /unlink in 33.06 seconds: 136.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 807 (time 1788444168.08 total 10.01 last 80.59)
 - open/close 1447 (time 1788444178.08 total 20.01 last 63.99)
 - open/close 2167 (time 1788444188.09 total 30.02 last 71.96)
 - open/close 2875 (time 1788444198.10 total 40.03 last 70.75)
 - open/close 3544 (time 1788444208.11 total 50.04 last 66.81)
 - open/close 4287 (time 1788444218.12 total 60.05 last 74.22)
total: 4500 open/close in 62.49 seconds: 72.02 ops/second
 - /unlink 1255 (time 1788444235.03 total 10.00 last 125.45)
 - /unlink 2382 (time 1788444245.03 total 20.01 last 112.66)
 - /unlink 3669 (time 1788444255.04 total 30.02 last 128.55)
total: 4500 /unlink in 36.89 seconds: 122.00 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95160 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg657-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg657-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg657-server
stop mds service on oleg657-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg657-server
stop mds service on oleg657-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg657-server
unloading modules via unload_modules_local on: 'oleg657-server'
oleg657-server: oleg657-server.virtnet: executing unload_modules_local
oleg657-server: kmemleak disabled
modules unloaded.