== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:51:11 (1788843071)
Stopping clients: oleg446-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg446-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg446-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg446-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg446-server
oleg446-server: oleg446-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg446-server'
oleg446-server: oleg446-server.virtnet: executing load_modules_local
oleg446-server: kmemleak disabled
oleg446-server: Loading modules from /home/green/git/lustre-release/lustre
oleg446-server: detected 4 online CPUs by sysfs
oleg446-server: MODOPTS_LIBCFS=
oleg446-server: Force libcfs to create 2 CPU partitions
oleg446-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg446-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg446-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg446-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg446-client.virtnet:  -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 643 (time 1788843293.13 total 10.01 last 64.22)
 - open/close 1262 (time 1788843303.15 total 20.04 last 61.75)
 - open/close 1878 (time 1788843313.16 total 30.04 last 61.56)
 - open/close 2489 (time 1788843323.17 total 40.05 last 61.04)
 - open/close 3072 (time 1788843333.17 total 50.06 last 58.28)
 - open/close 3779 (time 1788843343.18 total 60.06 last 70.64)
 - open/close 4479 (time 1788843353.18 total 70.07 last 69.98)
total: 4500 open/close in 70.35 seconds: 63.96 ops/second
 - /unlink 1433 (time 1788843368.00 total 10.00 last 143.29)
 - /unlink 2704 (time 1788843378.01 total 20.00 last 127.06)
 - /unlink 3975 (time 1788843388.01 total 30.01 last 127.07)
total: 4500 /unlink in 34.17 seconds: 131.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 808 (time 1788843407.67 total 10.00 last 80.79)
 - open/close 1529 (time 1788843417.69 total 20.02 last 71.95)
 - open/close 2247 (time 1788843427.70 total 30.03 last 71.73)
 - open/close 3002 (time 1788843437.71 total 40.04 last 75.44)
 - open/close 3741 (time 1788843447.72 total 50.05 last 73.86)
 - open/close 4386 (time 1788843457.72 total 60.05 last 64.50)
total: 4500 open/close in 61.63 seconds: 73.02 ops/second
 - /unlink 1184 (time 1788843474.34 total 10.01 last 118.31)
 - /unlink 2387 (time 1788843484.35 total 20.02 last 120.16)
 - /unlink 3586 (time 1788843494.35 total 30.02 last 119.85)
total: 4500 /unlink in 37.99 seconds: 118.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 583 (time 1788843517.73 total 10.01 last 58.26)
 - open/close 1283 (time 1788843527.75 total 20.02 last 69.89)
 - open/close 1961 (time 1788843537.76 total 30.04 last 67.70)
 - open/close 2705 (time 1788843547.77 total 40.05 last 74.31)
 - open/close 3277 (time 1788843557.80 total 50.07 last 57.07)
 - open/close 3922 (time 1788843567.81 total 60.09 last 64.41)
 - open/close 4491 (time 1788843577.81 total 70.09 last 56.88)
total: 4500 open/close in 70.26 seconds: 64.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1196 (time 1788843593.45 total 10.00 last 119.57)
 - /unlink 2416 (time 1788843603.45 total 20.00 last 122.00)
 - /unlink 3716 (time 1788843613.45 total 30.01 last 129.94)
total: 4500 /unlink in 36.52 seconds: 123.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 565 (time 1788843635.66 total 10.00 last 56.49)
 - open/close 1072 (time 1788843645.67 total 20.01 last 50.66)
 - open/close 1756 (time 1788843655.69 total 30.03 last 68.27)
 - open/close 2374 (time 1788843665.69 total 40.03 last 61.76)
 - open/close 2989 (time 1788843675.71 total 50.05 last 61.41)
 - open/close 3569 (time 1788843685.73 total 60.07 last 57.89)
 - open/close 4099 (time 1788843695.73 total 70.08 last 52.97)
total: 4500 open/close in 77.27 seconds: 58.24 ops/second
 - /unlink 1121 (time 1788843718.28 total 10.02 last 111.88)
 - /unlink 2318 (time 1788843728.28 total 20.02 last 119.63)
 - /unlink 3623 (time 1788843738.29 total 30.03 last 130.44)
total: 4500 /unlink in 36.83 seconds: 122.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 702 (time 1788843760.80 total 10.01 last 70.15)
 - open/close 1304 (time 1788843770.80 total 20.01 last 60.18)
 - open/close 1987 (time 1788843780.82 total 30.03 last 68.19)
 - open/close 2656 (time 1788843790.84 total 40.05 last 66.76)
 - open/close 3420 (time 1788843800.85 total 50.05 last 76.35)
 - open/close 4123 (time 1788843810.85 total 60.06 last 70.29)
total: 4500 open/close in 65.58 seconds: 68.62 ops/second
 - /unlink 1121 (time 1788843831.41 total 10.00 last 112.08)
 - /unlink 2310 (time 1788843841.42 total 20.01 last 118.80)
 - /unlink 3498 (time 1788843851.44 total 30.03 last 118.51)
total: 4500 /unlink in 38.91 seconds: 115.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 577 (time 1788843875.60 total 10.00 last 57.70)
 - open/close 1223 (time 1788843885.60 total 20.01 last 64.56)
 - open/close 1952 (time 1788843895.61 total 30.01 last 72.85)
 - open/close 2548 (time 1788843905.62 total 40.02 last 59.56)
 - open/close 3100 (time 1788843915.63 total 50.03 last 55.15)
 - open/close 3681 (time 1788843925.63 total 60.03 last 58.08)
 - open/close 4208 (time 1788843935.63 total 70.04 last 52.69)
total: 4500 open/close in 75.31 seconds: 59.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1157 (time 1788843956.12 total 10.00 last 115.67)
 - /unlink 2503 (time 1788843966.13 total 20.01 last 134.46)
 - /unlink 3762 (time 1788843976.13 total 30.02 last 125.87)
total: 4500 /unlink in 36.57 seconds: 123.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 626 (time 1788843998.71 total 10.00 last 62.59)
 - open/close 1224 (time 1788844008.74 total 20.02 last 59.68)
 - open/close 1878 (time 1788844018.75 total 30.03 last 65.32)
 - open/close 2522 (time 1788844028.75 total 40.04 last 64.39)
 - open/close 3155 (time 1788844038.78 total 50.07 last 63.11)
 - open/close 3795 (time 1788844048.79 total 60.08 last 63.92)
 - open/close 4442 (time 1788844058.79 total 70.08 last 64.69)
total: 4500 open/close in 71.08 seconds: 63.31 ops/second
 - /unlink 1226 (time 1788844074.91 total 10.00 last 122.56)
 - /unlink 2360 (time 1788844084.93 total 20.02 last 113.24)
 - /unlink 3511 (time 1788844094.94 total 30.03 last 114.93)
 - /unlink 4497 (time 1788844104.95 total 40.04 last 98.53)
total: 4500 /unlink in 40.07 seconds: 112.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 584 (time 1788844121.07 total 10.01 last 58.34)
 - open/close 1217 (time 1788844131.08 total 20.02 last 63.22)
 - open/close 1810 (time 1788844141.08 total 30.02 last 59.29)
 - open/close 2461 (time 1788844151.08 total 40.03 last 65.07)
 - open/close 3145 (time 1788844161.08 total 50.03 last 68.39)
 - open/close 3857 (time 1788844171.10 total 60.05 last 71.06)
 - open/close 4477 (time 1788844181.12 total 70.07 last 61.88)
total: 4500 open/close in 70.47 seconds: 63.86 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1090 (time 1788844196.82 total 10.02 last 108.78)
 - /unlink 2260 (time 1788844206.84 total 20.04 last 116.80)
 - /unlink 3392 (time 1788844216.85 total 30.05 last 113.10)
total: 4500 /unlink in 38.65 seconds: 116.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 693 (time 1788844240.09 total 10.01 last 69.25)
 - open/close 1371 (time 1788844250.09 total 20.01 last 67.78)
 - open/close 1999 (time 1788844260.10 total 30.02 last 62.77)
 - open/close 2566 (time 1788844270.10 total 40.02 last 56.67)
 - open/close 3170 (time 1788844280.11 total 50.02 last 60.39)
 - open/close 3766 (time 1788844290.11 total 60.03 last 59.55)
 - open/close 4289 (time 1788844300.12 total 70.04 last 52.25)
total: 4500 open/close in 73.82 seconds: 60.96 ops/second
 - /unlink 1181 (time 1788844319.70 total 10.01 last 118.00)
 - /unlink 2362 (time 1788844329.71 total 20.02 last 118.01)
 - /unlink 3530 (time 1788844339.71 total 30.02 last 116.80)
total: 4500 /unlink in 37.87 seconds: 118.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 705 (time 1788844363.45 total 10.01 last 70.43)
 - open/close 1423 (time 1788844373.46 total 20.02 last 71.74)
 - open/close 2166 (time 1788844383.47 total 30.03 last 74.25)
 - open/close 2875 (time 1788844393.47 total 40.03 last 70.89)
 - open/close 3504 (time 1788844403.47 total 50.03 last 62.88)
 - open/close 4097 (time 1788844413.49 total 60.05 last 59.22)
total: 4500 open/close in 65.39 seconds: 68.82 ops/second
 - /unlink 1212 (time 1788844433.95 total 10.00 last 121.15)
 - /unlink 2353 (time 1788844443.95 total 20.00 last 114.09)
 - /unlink 3568 (time 1788844453.95 total 30.01 last 121.50)
total: 4500 /unlink in 39.46 seconds: 114.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 512 (time 1788844479.10 total 10.00 last 51.20)
 - open/close 1118 (time 1788844489.13 total 20.03 last 60.44)
 - open/close 1662 (time 1788844499.14 total 30.04 last 54.35)
 - open/close 2252 (time 1788844509.14 total 40.04 last 58.98)
 - open/close 2831 (time 1788844519.18 total 50.08 last 57.67)
 - open/close 3447 (time 1788844529.20 total 60.10 last 61.47)
 - open/close 3967 (time 1788844539.22 total 70.12 last 51.92)
total: 4500 open/close in 78.81 seconds: 57.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1278 (time 1788844562.46 total 10.01 last 127.74)
 - /unlink 2523 (time 1788844572.46 total 20.01 last 124.49)
 - /unlink 3828 (time 1788844582.47 total 30.01 last 130.39)
total: 4500 /unlink in 36.60 seconds: 122.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 609 (time 1788844604.51 total 10.01 last 60.85)
 - open/close 1300 (time 1788844614.54 total 20.03 last 68.92)
 - open/close 1967 (time 1788844624.55 total 30.05 last 66.61)
 - open/close 2600 (time 1788844634.56 total 40.06 last 63.24)
 - open/close 3194 (time 1788844644.57 total 50.07 last 59.35)
 - open/close 3750 (time 1788844654.59 total 60.09 last 55.48)
 - open/close 4213 (time 1788844664.60 total 70.10 last 46.24)
total: 4500 open/close in 75.27 seconds: 59.79 ops/second
 - /unlink 1202 (time 1788844685.75 total 10.03 last 119.89)
 - /unlink 2493 (time 1788844695.75 total 20.03 last 129.06)
 - /unlink 3552 (time 1788844705.76 total 30.04 last 105.84)
total: 4500 /unlink in 37.38 seconds: 120.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 481 (time 1788844729.48 total 10.00 last 48.09)
 - open/close 988 (time 1788844739.50 total 20.02 last 50.60)
 - open/close 1508 (time 1788844749.51 total 30.03 last 51.95)
 - open/close 2044 (time 1788844759.52 total 40.04 last 53.54)
 - open/close 2785 (time 1788844769.52 total 50.04 last 74.10)
 - open/close 3492 (time 1788844779.53 total 60.05 last 70.64)
 - open/close 4169 (time 1788844789.54 total 70.06 last 67.64)
total: 4500 open/close in 74.62 seconds: 60.31 ops/second
 - /unlink 1212 (time 1788844809.84 total 10.01 last 121.12)
 - /unlink 2321 (time 1788844819.84 total 20.01 last 110.87)
 - /unlink 3291 (time 1788844829.84 total 30.01 last 96.98)
 - /unlink 4156 (time 1788844839.85 total 40.01 last 86.47)
total: 4500 /unlink in 44.20 seconds: 101.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 456 (time 1788844859.91 total 10.01 last 45.57)
 - open/close 1025 (time 1788844869.91 total 20.01 last 56.88)
 - open/close 1733 (time 1788844879.93 total 30.03 last 70.66)
 - open/close 2397 (time 1788844889.95 total 40.05 last 66.29)
 - open/close 3138 (time 1788844899.97 total 50.07 last 73.96)
 - open/close 3824 (time 1788844909.97 total 60.07 last 68.55)
 - open/close 4473 (time 1788844919.98 total 70.08 last 64.87)
total: 4500 open/close in 70.56 seconds: 63.78 ops/second
 - /unlink 1266 (time 1788844935.14 total 10.00 last 126.59)
 - /unlink 2519 (time 1788844945.14 total 20.00 last 125.25)
 - /unlink 3688 (time 1788844955.15 total 30.01 last 116.87)
total: 4500 /unlink in 38.68 seconds: 116.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 556 (time 1788844979.86 total 10.00 last 55.60)
 - open/close 1192 (time 1788844989.87 total 20.01 last 63.54)
 - open/close 1919 (time 1788844999.87 total 30.02 last 72.65)
 - open/close 2642 (time 1788845009.89 total 40.03 last 72.20)
 - open/close 3393 (time 1788845019.89 total 50.03 last 75.08)
 - open/close 4159 (time 1788845029.92 total 60.06 last 76.37)
total: 4500 open/close in 65.17 seconds: 69.05 ops/second
 - /unlink 1349 (time 1788845049.95 total 10.00 last 134.88)
 - /unlink 2558 (time 1788845059.96 total 20.01 last 120.84)
 - /unlink 3815 (time 1788845069.97 total 30.02 last 125.55)
total: 4500 /unlink in 36.04 seconds: 124.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 664 (time 1788845091.36 total 10.00 last 66.39)
 - open/close 1265 (time 1788845101.37 total 20.01 last 60.07)
 - open/close 1855 (time 1788845111.37 total 30.01 last 58.97)
 - open/close 2466 (time 1788845121.38 total 40.02 last 61.02)
 - open/close 3128 (time 1788845131.39 total 50.03 last 66.16)
 - open/close 3854 (time 1788845141.39 total 60.03 last 72.60)
total: 4500 open/close in 68.57 seconds: 65.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1318 (time 1788845166.01 total 10.01 last 131.73)
 - /unlink 2466 (time 1788845176.03 total 20.03 last 114.57)
 - /unlink 3634 (time 1788845186.04 total 30.03 last 116.77)
total: 4500 /unlink in 36.77 seconds: 122.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 631 (time 1788845208.25 total 10.01 last 63.05)
 - open/close 1377 (time 1788845218.25 total 20.01 last 74.55)
 - open/close 2091 (time 1788845228.26 total 30.02 last 71.36)
 - open/close 2756 (time 1788845238.27 total 40.03 last 66.43)
 - open/close 3459 (time 1788845248.28 total 50.05 last 70.19)
 - open/close 4129 (time 1788845258.30 total 60.06 last 66.90)
total: 4500 open/close in 64.44 seconds: 69.84 ops/second
 - /unlink 1308 (time 1788845277.48 total 10.00 last 130.75)
 - /unlink 2494 (time 1788845287.48 total 20.01 last 118.56)
 - /unlink 3784 (time 1788845297.48 total 30.01 last 128.99)
total: 4500 /unlink in 36.16 seconds: 124.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 670 (time 1788845320.17 total 10.04 last 66.75)
 - open/close 1338 (time 1788845330.17 total 20.04 last 66.77)
 - open/close 1938 (time 1788845340.18 total 30.05 last 59.94)
 - open/close 2480 (time 1788845350.19 total 40.06 last 54.13)
 - open/close 3130 (time 1788845360.19 total 50.07 last 64.98)
 - open/close 3771 (time 1788845370.20 total 60.07 last 64.05)
 - open/close 4442 (time 1788845380.20 total 70.08 last 67.10)
total: 4500 open/close in 71.06 seconds: 63.32 ops/second
 - /unlink 1306 (time 1788845396.13 total 10.00 last 130.54)
 - /unlink 2557 (time 1788845406.13 total 20.01 last 125.02)
 - /unlink 3827 (time 1788845416.14 total 30.01 last 126.95)
total: 4500 /unlink in 35.50 seconds: 126.75 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95291 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg446-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg446-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server
stop mds service on oleg446-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server
stop mds service on oleg446-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg446-server
unloading modules via unload_modules_local on: 'oleg446-server'
oleg446-server: oleg446-server.virtnet: executing unload_modules_local
oleg446-server: kmemleak disabled
modules unloaded.