== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:56:02 (1790967362)
Stopping clients: oleg301-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg301-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg301-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg301-server
oleg301-server: oleg301-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg301-server'
oleg301-server: oleg301-server.virtnet: executing load_modules_local
oleg301-server: kmemleak disabled
oleg301-server: Loading modules from /home/green/git/lustre-release/lustre
oleg301-server: detected 4 online CPUs by sysfs
oleg301-server: MODOPTS_LIBCFS=
oleg301-server: Force libcfs to create 2 CPU partitions
oleg301-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg301-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg301-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg301-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg301-client.virtnet:  -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 647 (time 1790967599.49 total 10.01 last 64.61)
 - open/close 1268 (time 1790967609.50 total 20.02 last 62.07)
 - open/close 1915 (time 1790967619.52 total 30.05 last 64.52)
 - open/close 2598 (time 1790967629.53 total 40.05 last 68.24)
 - open/close 3232 (time 1790967639.53 total 50.06 last 63.38)
 - open/close 3808 (time 1790967649.56 total 60.08 last 57.45)
 - open/close 4477 (time 1790967659.57 total 70.09 last 66.84)
total: 4500 open/close in 70.53 seconds: 63.80 ops/second
 - /unlink 1226 (time 1790967675.50 total 10.00 last 122.55)
 - /unlink 2466 (time 1790967685.51 total 20.01 last 123.89)
 - /unlink 3722 (time 1790967695.51 total 30.01 last 125.59)
total: 4500 /unlink in 36.27 seconds: 124.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 636 (time 1790967716.94 total 10.00 last 63.60)
 - open/close 1389 (time 1790967726.94 total 20.00 last 75.29)
 - open/close 2104 (time 1790967736.95 total 30.01 last 71.47)
 - open/close 2757 (time 1790967746.95 total 40.01 last 65.29)
 - open/close 3439 (time 1790967756.98 total 50.04 last 67.98)
 - open/close 4104 (time 1790967766.99 total 60.04 last 66.46)
total: 4500 open/close in 66.11 seconds: 68.07 ops/second
 - /unlink 1269 (time 1790967787.68 total 10.01 last 126.81)
 - /unlink 2474 (time 1790967797.69 total 20.01 last 120.46)
 - /unlink 3600 (time 1790967807.69 total 30.01 last 112.58)
total: 4500 /unlink in 36.59 seconds: 123.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 513 (time 1790967828.83 total 10.02 last 51.18)
 - open/close 943 (time 1790967838.83 total 20.02 last 42.99)
 - open/close 1356 (time 1790967848.87 total 30.07 last 41.13)
 - open/close 1810 (time 1790967858.87 total 40.07 last 45.38)
 - open/close 2417 (time 1790967868.88 total 50.08 last 60.65)
 - open/close 2990 (time 1790967878.89 total 60.08 last 57.27)
 - open/close 3593 (time 1790967888.89 total 70.08 last 60.30)
 - open/close 4052 (time 1790967898.89 total 80.08 last 45.90)
 - open/close 4493 (time 1790967908.90 total 90.10 last 44.04)
total: 4500 open/close in 90.27 seconds: 49.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 903 (time 1790967926.30 total 10.02 last 90.16)
 - /unlink 2208 (time 1790967936.30 total 20.02 last 130.44)
 - /unlink 3329 (time 1790967946.30 total 30.02 last 112.07)
total: 4500 /unlink in 39.19 seconds: 114.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 678 (time 1790967970.58 total 10.04 last 67.50)
 - open/close 1342 (time 1790967980.60 total 20.06 last 66.28)
 - open/close 2022 (time 1790967990.60 total 30.07 last 67.96)
 - open/close 2753 (time 1790968000.60 total 40.07 last 73.10)
 - open/close 3470 (time 1790968010.61 total 50.07 last 71.67)
 - open/close 4216 (time 1790968020.61 total 60.08 last 74.55)
total: 4500 open/close in 64.12 seconds: 70.18 ops/second
 - /unlink 1319 (time 1790968038.87 total 10.00 last 131.87)
 - /unlink 2559 (time 1790968048.88 total 20.00 last 123.97)
 - /unlink 3821 (time 1790968058.90 total 30.03 last 125.92)
total: 4500 /unlink in 35.98 seconds: 125.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 630 (time 1790968080.78 total 10.01 last 62.93)
 - open/close 1153 (time 1790968090.78 total 20.01 last 52.28)
 - open/close 1771 (time 1790968100.80 total 30.03 last 61.72)
 - open/close 2509 (time 1790968110.81 total 40.04 last 73.69)
 - open/close 3207 (time 1790968120.82 total 50.05 last 69.75)
 - open/close 3871 (time 1790968130.82 total 60.05 last 66.39)
total: 4500 open/close in 68.77 seconds: 65.43 ops/second
 - /unlink 1271 (time 1790968153.78 total 10.01 last 126.97)
 - /unlink 2552 (time 1790968163.79 total 20.02 last 127.97)
 - /unlink 3751 (time 1790968173.79 total 30.02 last 119.89)
total: 4500 /unlink in 36.80 seconds: 122.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 632 (time 1790968195.69 total 10.00 last 63.17)
 - open/close 1268 (time 1790968205.70 total 20.02 last 63.49)
 - open/close 1921 (time 1790968215.71 total 30.03 last 65.26)
 - open/close 2548 (time 1790968225.72 total 40.03 last 62.66)
 - open/close 3212 (time 1790968235.72 total 50.04 last 66.37)
 - open/close 3941 (time 1790968245.73 total 60.04 last 72.86)
total: 4500 open/close in 67.20 seconds: 66.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1121 (time 1790968267.78 total 10.00 last 112.09)
 - /unlink 2370 (time 1790968277.79 total 20.01 last 124.79)
 - /unlink 3717 (time 1790968287.79 total 30.01 last 134.63)
total: 4500 /unlink in 35.96 seconds: 125.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 706 (time 1790968309.35 total 10.01 last 70.53)
 - open/close 1448 (time 1790968319.40 total 20.06 last 73.83)
 - open/close 2112 (time 1790968329.40 total 30.07 last 66.37)
 - open/close 2695 (time 1790968339.41 total 40.07 last 58.26)
 - open/close 3292 (time 1790968349.42 total 50.08 last 59.64)
 - open/close 3944 (time 1790968359.44 total 60.10 last 65.06)
total: 4500 open/close in 68.76 seconds: 65.45 ops/second
 - /unlink 1114 (time 1790968383.62 total 10.01 last 111.31)
 - /unlink 2260 (time 1790968393.62 total 20.01 last 114.58)
 - /unlink 3398 (time 1790968403.63 total 30.02 last 113.74)
total: 4500 /unlink in 39.84 seconds: 112.94 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 623 (time 1790968429.06 total 10.01 last 62.22)
 - open/close 1312 (time 1790968439.07 total 20.02 last 68.86)
 - open/close 2001 (time 1790968449.08 total 30.03 last 68.79)
 - open/close 2649 (time 1790968459.09 total 40.04 last 64.75)
 - open/close 3283 (time 1790968469.09 total 50.04 last 63.39)
 - open/close 3912 (time 1790968479.10 total 60.05 last 62.86)
total: 4500 open/close in 69.55 seconds: 64.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1206 (time 1790968503.43 total 10.00 last 120.59)
 - /unlink 2447 (time 1790968513.43 total 20.00 last 124.10)
 - /unlink 3480 (time 1790968523.43 total 30.01 last 103.24)
total: 4500 /unlink in 39.29 seconds: 114.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 655 (time 1790968547.54 total 10.00 last 65.47)
 - open/close 1287 (time 1790968557.54 total 20.00 last 63.20)
 - open/close 1938 (time 1790968567.55 total 30.02 last 65.03)
 - open/close 2575 (time 1790968577.55 total 40.02 last 63.66)
 - open/close 3222 (time 1790968587.56 total 50.03 last 64.68)
 - open/close 3833 (time 1790968597.57 total 60.04 last 61.01)
total: 4500 open/close in 69.66 seconds: 64.60 ops/second
 - /unlink 1285 (time 1790968621.51 total 10.00 last 128.46)
 - /unlink 2547 (time 1790968631.51 total 20.00 last 126.20)
 - /unlink 3842 (time 1790968641.52 total 30.01 last 129.42)
total: 4500 /unlink in 35.20 seconds: 127.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 652 (time 1790968661.71 total 10.00 last 65.19)
 - open/close 1277 (time 1790968671.72 total 20.02 last 62.40)
 - open/close 1952 (time 1790968681.72 total 30.02 last 67.50)
 - open/close 2600 (time 1790968691.73 total 40.02 last 64.76)
 - open/close 3259 (time 1790968701.74 total 50.03 last 65.82)
 - open/close 3793 (time 1790968711.78 total 60.07 last 53.21)
 - open/close 4490 (time 1790968721.78 total 70.08 last 69.66)
total: 4500 open/close in 70.16 seconds: 64.14 ops/second
 - /unlink 1395 (time 1790968736.76 total 10.01 last 139.42)
 - /unlink 2807 (time 1790968746.77 total 20.02 last 141.04)
 - /unlink 4148 (time 1790968756.77 total 30.02 last 134.05)
total: 4500 /unlink in 32.75 seconds: 137.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 704 (time 1790968774.29 total 10.00 last 70.37)
 - open/close 1371 (time 1790968784.30 total 20.01 last 66.67)
 - open/close 2068 (time 1790968794.30 total 30.01 last 69.70)
 - open/close 2800 (time 1790968804.30 total 40.01 last 73.16)
 - open/close 3518 (time 1790968814.30 total 50.02 last 71.79)
 - open/close 4244 (time 1790968824.30 total 60.02 last 72.60)
total: 4500 open/close in 63.65 seconds: 70.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1366 (time 1790968841.62 total 10.00 last 136.55)
 - /unlink 2620 (time 1790968851.62 total 20.00 last 125.39)
 - /unlink 3895 (time 1790968861.63 total 30.01 last 127.37)
total: 4500 /unlink in 34.85 seconds: 129.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 642 (time 1790968881.01 total 10.00 last 64.17)
 - open/close 1216 (time 1790968891.02 total 20.01 last 57.34)
 - open/close 1972 (time 1790968901.03 total 30.03 last 75.51)
 - open/close 2722 (time 1790968911.05 total 40.05 last 74.81)
 - open/close 3391 (time 1790968921.05 total 50.05 last 66.90)
 - open/close 4074 (time 1790968931.06 total 60.06 last 68.25)
total: 4500 open/close in 66.65 seconds: 67.51 ops/second
 - /unlink 1327 (time 1790968951.86 total 10.00 last 132.70)
 - /unlink 2658 (time 1790968961.86 total 20.01 last 132.99)
 - /unlink 3909 (time 1790968971.87 total 30.01 last 125.03)
total: 4500 /unlink in 34.97 seconds: 128.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 601 (time 1790968992.94 total 10.02 last 60.01)
 - open/close 1270 (time 1790969002.95 total 20.03 last 66.83)
 - open/close 1900 (time 1790969012.96 total 30.04 last 62.90)
 - open/close 2650 (time 1790969022.97 total 40.05 last 74.92)
 - open/close 3320 (time 1790969032.98 total 50.05 last 66.99)
 - open/close 4006 (time 1790969042.98 total 60.06 last 68.56)
total: 4500 open/close in 67.87 seconds: 66.30 ops/second
 - /unlink 1276 (time 1790969065.81 total 10.00 last 127.58)
 - /unlink 2482 (time 1790969075.81 total 20.01 last 120.55)
 - /unlink 3667 (time 1790969085.82 total 30.01 last 118.45)
total: 4500 /unlink in 36.65 seconds: 122.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 684 (time 1790969107.02 total 10.01 last 68.36)
 - open/close 1431 (time 1790969117.02 total 20.01 last 74.69)
 - open/close 2142 (time 1790969127.04 total 30.02 last 70.97)
 - open/close 2829 (time 1790969137.06 total 40.05 last 68.52)
 - open/close 3524 (time 1790969147.07 total 50.06 last 69.45)
 - open/close 4264 (time 1790969157.08 total 60.06 last 73.95)
total: 4500 open/close in 63.63 seconds: 70.72 ops/second
 - /unlink 1432 (time 1790969175.47 total 10.00 last 143.19)
 - /unlink 2750 (time 1790969185.47 total 20.00 last 131.79)
 - /unlink 4011 (time 1790969195.48 total 30.01 last 125.98)
total: 4500 /unlink in 33.99 seconds: 132.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 675 (time 1790969215.19 total 10.00 last 67.49)
 - open/close 1309 (time 1790969225.20 total 20.01 last 63.35)
 - open/close 1945 (time 1790969235.20 total 30.01 last 63.60)
 - open/close 2585 (time 1790969245.20 total 40.01 last 63.97)
 - open/close 3263 (time 1790969255.21 total 50.02 last 67.77)
 - open/close 3963 (time 1790969265.21 total 60.03 last 69.95)
total: 4500 open/close in 67.81 seconds: 66.37 ops/second
 - /unlink 1406 (time 1790969287.16 total 10.01 last 140.41)
 - /unlink 2677 (time 1790969297.16 total 20.02 last 127.05)
 - /unlink 3957 (time 1790969307.18 total 30.04 last 127.77)
total: 4500 /unlink in 34.54 seconds: 130.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 648 (time 1790969326.18 total 10.02 last 64.69)
 - open/close 1275 (time 1790969336.19 total 20.02 last 62.68)
 - open/close 1910 (time 1790969346.19 total 30.03 last 63.46)
 - open/close 2622 (time 1790969356.20 total 40.04 last 71.13)
 - open/close 3304 (time 1790969366.20 total 50.04 last 68.20)
 - open/close 3989 (time 1790969376.21 total 60.05 last 68.44)
total: 4500 open/close in 67.12 seconds: 67.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1143 (time 1790969398.47 total 10.00 last 114.27)
 - /unlink 2239 (time 1790969408.47 total 20.01 last 109.54)
 - /unlink 3460 (time 1790969418.47 total 30.01 last 122.09)
total: 4500 /unlink in 39.17 seconds: 114.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 642 (time 1790969443.96 total 10.03 last 64.00)
 - open/close 1343 (time 1790969453.96 total 20.04 last 70.07)
 - open/close 2022 (time 1790969463.98 total 30.06 last 67.75)
 - open/close 2715 (time 1790969474.00 total 40.08 last 69.17)
 - open/close 3365 (time 1790969484.01 total 50.08 last 64.95)
 - open/close 4097 (time 1790969494.02 total 60.10 last 73.11)
total: 4500 open/close in 65.49 seconds: 68.71 ops/second
 - /unlink 1395 (time 1790969514.26 total 10.01 last 139.33)
 - /unlink 2616 (time 1790969524.26 total 20.01 last 122.07)
 - /unlink 3919 (time 1790969534.26 total 30.02 last 130.26)
total: 4500 /unlink in 34.26 seconds: 131.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 661 (time 1790969553.98 total 10.00 last 66.08)
 - open/close 1313 (time 1790969563.98 total 20.01 last 65.16)
 - open/close 1961 (time 1790969574.01 total 30.04 last 64.63)
 - open/close 2609 (time 1790969584.01 total 40.04 last 64.76)
 - open/close 3318 (time 1790969594.02 total 50.05 last 70.86)
 - open/close 3997 (time 1790969604.03 total 60.06 last 67.85)
total: 4500 open/close in 66.72 seconds: 67.45 ops/second
 - /unlink 1231 (time 1790969624.64 total 10.00 last 123.05)
 - /unlink 2436 (time 1790969634.64 total 20.01 last 120.47)
 - /unlink 3478 (time 1790969644.64 total 30.01 last 104.19)
total: 4500 /unlink in 38.25 seconds: 117.65 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95218 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg301-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg301-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server
stop mds service on oleg301-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server
stop mds service on oleg301-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server
unloading modules via unload_modules_local on: 'oleg301-server'
oleg301-server: oleg301-server.virtnet: executing unload_modules_local
oleg301-server: kmemleak disabled
modules unloaded.