== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:33:08 (1789180388)
Stopping clients: oleg425-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg425-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg425-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg425-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg425-server
oleg425-server: oleg425-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg425-server'
oleg425-server: oleg425-server.virtnet: executing load_modules_local
oleg425-server: kmemleak disabled
oleg425-server: Loading modules from /home/green/git/lustre-release/lustre
oleg425-server: detected 4 online CPUs by sysfs
oleg425-server: MODOPTS_LIBCFS=
oleg425-server: Force libcfs to create 2 CPU partitions
oleg425-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg425-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg425-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg425-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg425-client.virtnet:  -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 678 (time 1789180611.00 total 10.02 last 67.66)
 - open/close 1291 (time 1789180621.02 total 20.04 last 61.16)
 - open/close 1951 (time 1789180631.03 total 30.05 last 65.97)
 - open/close 2613 (time 1789180641.04 total 40.06 last 66.10)
 - open/close 3252 (time 1789180651.05 total 50.07 last 63.85)
 - open/close 3831 (time 1789180661.07 total 60.09 last 57.78)
 - open/close 4417 (time 1789180671.07 total 70.09 last 58.59)
total: 4500 open/close in 71.43 seconds: 63.00 ops/second
 - /unlink 1172 (time 1789180687.82 total 10.01 last 117.11)
 - /unlink 2280 (time 1789180697.83 total 20.01 last 110.74)
 - /unlink 3440 (time 1789180707.84 total 30.02 last 115.93)
total: 4500 /unlink in 38.87 seconds: 115.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 648 (time 1789180732.52 total 10.03 last 64.59)
 - open/close 1221 (time 1789180742.54 total 20.05 last 57.23)
 - open/close 1840 (time 1789180752.55 total 30.06 last 61.83)
 - open/close 2492 (time 1789180762.55 total 40.06 last 65.19)
 - open/close 3165 (time 1789180772.56 total 50.07 last 67.22)
 - open/close 3838 (time 1789180782.56 total 60.07 last 67.29)
total: 4500 open/close in 70.05 seconds: 64.24 ops/second
 - /unlink 1336 (time 1789180807.93 total 10.01 last 133.45)
 - /unlink 2535 (time 1789180817.93 total 20.02 last 119.81)
 - /unlink 3609 (time 1789180827.93 total 30.02 last 107.40)
total: 4500 /unlink in 37.53 seconds: 119.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 775 (time 1789180850.82 total 10.01 last 77.42)
 - open/close 1484 (time 1789180860.83 total 20.02 last 70.84)
 - open/close 2160 (time 1789180870.87 total 30.05 last 67.38)
 - open/close 2837 (time 1789180880.89 total 40.07 last 67.55)
 - open/close 3495 (time 1789180890.89 total 50.08 last 65.78)
 - open/close 4165 (time 1789180900.90 total 60.08 last 66.95)
total: 4500 open/close in 64.61 seconds: 69.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1249 (time 1789180920.61 total 10.01 last 124.82)
 - /unlink 2345 (time 1789180930.62 total 20.01 last 109.56)
 - /unlink 3483 (time 1789180940.62 total 30.01 last 113.75)
total: 4500 /unlink in 39.18 seconds: 114.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 567 (time 1789180966.77 total 10.00 last 56.68)
 - open/close 1194 (time 1789180976.78 total 20.01 last 62.65)
 - open/close 1814 (time 1789180986.79 total 30.03 last 61.91)
 - open/close 2408 (time 1789180996.80 total 40.03 last 59.37)
 - open/close 2896 (time 1789181006.80 total 50.03 last 48.79)
 - open/close 3586 (time 1789181016.83 total 60.06 last 68.81)
 - open/close 4285 (time 1789181026.83 total 70.06 last 69.90)
total: 4500 open/close in 73.55 seconds: 61.18 ops/second
 - /unlink 1375 (time 1789181045.58 total 10.02 last 137.23)
 - /unlink 2641 (time 1789181055.58 total 20.03 last 126.51)
 - /unlink 3899 (time 1789181065.59 total 30.03 last 125.74)
total: 4500 /unlink in 34.22 seconds: 131.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 644 (time 1789181085.48 total 10.00 last 64.38)
 - open/close 1245 (time 1789181095.50 total 20.02 last 60.01)
 - open/close 1767 (time 1789181105.51 total 30.03 last 52.12)
 - open/close 2291 (time 1789181115.52 total 40.04 last 52.37)
 - open/close 2957 (time 1789181125.53 total 50.05 last 66.52)
 - open/close 3575 (time 1789181135.54 total 60.07 last 61.71)
 - open/close 4036 (time 1789181145.55 total 70.07 last 46.09)
total: 4500 open/close in 79.05 seconds: 56.93 ops/second
 - /unlink 941 (time 1789181171.64 total 10.00 last 94.10)
 - /unlink 2098 (time 1789181181.64 total 20.00 last 115.67)
 - /unlink 3278 (time 1789181191.65 total 30.01 last 117.92)
 - /unlink 4429 (time 1789181201.65 total 40.02 last 115.02)
total: 4500 /unlink in 40.82 seconds: 110.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 610 (time 1789181218.05 total 10.02 last 60.86)
 - open/close 1119 (time 1789181228.09 total 20.06 last 50.70)
 - open/close 1825 (time 1789181238.09 total 30.07 last 70.57)
 - open/close 2495 (time 1789181248.10 total 40.08 last 66.93)
 - open/close 3186 (time 1789181258.12 total 50.09 last 68.98)
 - open/close 3936 (time 1789181268.13 total 60.10 last 74.96)
total: 4500 open/close in 68.94 seconds: 65.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1091 (time 1789181291.94 total 10.00 last 109.06)
 - /unlink 2129 (time 1789181301.95 total 20.01 last 103.73)
 - /unlink 3216 (time 1789181311.95 total 30.02 last 108.65)
 - /unlink 4350 (time 1789181321.96 total 40.02 last 113.36)
total: 4500 /unlink in 41.43 seconds: 108.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 706 (time 1789181339.00 total 10.01 last 70.54)
 - open/close 1366 (time 1789181349.00 total 20.01 last 65.97)
 - open/close 1976 (time 1789181359.04 total 30.05 last 60.78)
 - open/close 2542 (time 1789181369.05 total 40.05 last 56.57)
 - open/close 3103 (time 1789181379.05 total 50.06 last 56.08)
 - open/close 3701 (time 1789181389.05 total 60.06 last 59.78)
 - open/close 4282 (time 1789181399.06 total 70.06 last 58.09)
total: 4500 open/close in 73.51 seconds: 61.22 ops/second
 - /unlink 1374 (time 1789181417.95 total 10.00 last 137.38)
 - /unlink 2534 (time 1789181427.96 total 20.01 last 115.95)
 - /unlink 3683 (time 1789181437.96 total 30.01 last 114.85)
total: 4500 /unlink in 36.00 seconds: 125.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 466 (time 1789181461.17 total 10.00 last 46.59)
 - open/close 1034 (time 1789181471.19 total 20.02 last 56.68)
 - open/close 1645 (time 1789181481.20 total 30.03 last 61.08)
 - open/close 2262 (time 1789181491.20 total 40.03 last 61.68)
 - open/close 2928 (time 1789181501.20 total 50.03 last 66.58)
 - open/close 3598 (time 1789181511.21 total 60.04 last 66.98)
 - open/close 4318 (time 1789181521.21 total 70.04 last 71.98)
total: 4500 open/close in 72.72 seconds: 61.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1154 (time 1789181539.30 total 10.00 last 115.35)
 - /unlink 2319 (time 1789181549.31 total 20.01 last 116.46)
 - /unlink 3453 (time 1789181559.31 total 30.01 last 113.35)
 - /unlink 4386 (time 1789181569.33 total 40.03 last 93.16)
total: 4500 /unlink in 41.24 seconds: 109.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 525 (time 1789181588.39 total 10.01 last 52.47)
 - open/close 1060 (time 1789181598.41 total 20.03 last 53.39)
 - open/close 1627 (time 1789181608.43 total 30.05 last 56.59)
 - open/close 2225 (time 1789181618.44 total 40.05 last 59.75)
 - open/close 2764 (time 1789181628.48 total 50.09 last 53.70)
 - open/close 3191 (time 1789181638.48 total 60.09 last 42.69)
 - open/close 3709 (time 1789181648.52 total 70.14 last 51.56)
 - open/close 4201 (time 1789181658.53 total 80.14 last 49.19)
total: 4500 open/close in 84.78 seconds: 53.08 ops/second
 - /unlink 1175 (time 1789181678.81 total 10.00 last 117.48)
 - /unlink 2368 (time 1789181688.81 total 20.01 last 119.26)
 - /unlink 3466 (time 1789181698.82 total 30.01 last 109.71)
 - /unlink 4431 (time 1789181708.82 total 40.02 last 96.47)
total: 4500 /unlink in 40.92 seconds: 109.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 595 (time 1789181726.03 total 10.03 last 59.34)
 - open/close 1192 (time 1789181736.05 total 20.05 last 59.59)
 - open/close 1867 (time 1789181746.06 total 30.05 last 67.46)
 - open/close 2527 (time 1789181756.06 total 40.06 last 65.98)
 - open/close 3168 (time 1789181766.07 total 50.07 last 64.04)
 - open/close 3861 (time 1789181776.08 total 60.07 last 69.26)
 - open/close 4481 (time 1789181786.09 total 70.08 last 61.92)
total: 4500 open/close in 70.46 seconds: 63.87 ops/second
 - /unlink 1200 (time 1789181801.17 total 10.00 last 119.95)
 - /unlink 2275 (time 1789181811.18 total 20.01 last 107.42)
 - /unlink 3236 (time 1789181821.18 total 30.01 last 96.10)
 - /unlink 4201 (time 1789181831.18 total 40.01 last 96.47)
total: 4500 /unlink in 43.14 seconds: 104.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 631 (time 1789181850.84 total 10.02 last 62.97)
 - open/close 1249 (time 1789181860.84 total 20.02 last 61.80)
 - open/close 1867 (time 1789181870.86 total 30.03 last 61.72)
 - open/close 2532 (time 1789181880.88 total 40.05 last 66.37)
 - open/close 3174 (time 1789181890.89 total 50.07 last 64.10)
 - open/close 3705 (time 1789181900.90 total 60.08 last 53.06)
 - open/close 4348 (time 1789181910.91 total 70.09 last 64.24)
total: 4500 open/close in 72.52 seconds: 62.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1222 (time 1789181928.42 total 10.00 last 122.15)
 - /unlink 2428 (time 1789181938.42 total 20.01 last 120.56)
 - /unlink 3580 (time 1789181948.43 total 30.01 last 115.14)
total: 4500 /unlink in 39.20 seconds: 114.79 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 653 (time 1789181973.00 total 10.00 last 65.29)
 - open/close 1398 (time 1789181983.02 total 20.02 last 74.36)
 - open/close 2151 (time 1789181993.02 total 30.03 last 75.25)
 - open/close 2873 (time 1789182003.03 total 40.04 last 72.12)
 - open/close 3555 (time 1789182013.04 total 50.04 last 68.18)
 - open/close 4261 (time 1789182023.04 total 60.04 last 70.59)
total: 4500 open/close in 63.56 seconds: 70.80 ops/second
 - /unlink 1355 (time 1789182041.17 total 10.00 last 135.45)
 - /unlink 2673 (time 1789182051.17 total 20.01 last 131.71)
 - /unlink 4018 (time 1789182061.17 total 30.01 last 134.48)
total: 4500 /unlink in 34.24 seconds: 131.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 706 (time 1789182081.41 total 10.01 last 70.53)
 - open/close 1374 (time 1789182091.43 total 20.03 last 66.67)
 - open/close 2052 (time 1789182101.43 total 30.03 last 67.77)
 - open/close 2653 (time 1789182111.44 total 40.04 last 60.09)
 - open/close 3337 (time 1789182121.45 total 50.05 last 68.32)
 - open/close 4060 (time 1789182131.45 total 60.05 last 72.25)
total: 4500 open/close in 66.35 seconds: 67.82 ops/second
 - /unlink 1449 (time 1789182152.13 total 10.01 last 144.82)
 - /unlink 2752 (time 1789182162.14 total 20.01 last 130.18)
 - /unlink 3948 (time 1789182172.14 total 30.02 last 119.55)
total: 4500 /unlink in 34.29 seconds: 131.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 685 (time 1789182191.38 total 10.00 last 68.47)
 - open/close 1390 (time 1789182201.39 total 20.01 last 70.47)
 - open/close 2055 (time 1789182211.39 total 30.01 last 66.46)
 - open/close 2701 (time 1789182221.41 total 40.03 last 64.51)
 - open/close 3393 (time 1789182231.42 total 50.04 last 69.11)
 - open/close 4077 (time 1789182241.46 total 60.08 last 68.13)
total: 4500 open/close in 67.15 seconds: 67.02 ops/second
 - /unlink 1204 (time 1789182264.01 total 10.00 last 120.38)
 - /unlink 2418 (time 1789182274.02 total 20.01 last 121.33)
 - /unlink 3387 (time 1789182284.02 total 30.01 last 96.88)
 - /unlink 4451 (time 1789182294.03 total 40.02 last 106.33)
total: 4500 /unlink in 40.58 seconds: 110.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 689 (time 1789182311.11 total 10.01 last 68.86)
 - open/close 1251 (time 1789182321.11 total 20.01 last 56.18)
 - open/close 1937 (time 1789182331.13 total 30.02 last 68.52)
 - open/close 2512 (time 1789182341.13 total 40.02 last 57.50)
 - open/close 3211 (time 1789182351.14 total 50.04 last 69.81)
 - open/close 3976 (time 1789182361.14 total 60.04 last 76.47)
total: 4500 open/close in 67.26 seconds: 66.90 ops/second
 - /unlink 1230 (time 1789182383.62 total 10.00 last 122.96)
 - /unlink 2412 (time 1789182393.63 total 20.01 last 118.12)
 - /unlink 3754 (time 1789182403.64 total 30.02 last 134.00)
total: 4500 /unlink in 36.18 seconds: 124.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 730 (time 1789182424.90 total 10.01 last 72.96)
 - open/close 1400 (time 1789182434.90 total 20.01 last 67.00)
 - open/close 2006 (time 1789182444.90 total 30.01 last 60.58)
 - open/close 2623 (time 1789182454.91 total 40.02 last 61.67)
 - open/close 3261 (time 1789182464.92 total 50.02 last 63.75)
 - open/close 3934 (time 1789182474.92 total 60.03 last 67.25)
total: 4500 open/close in 68.52 seconds: 65.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1185 (time 1789182498.89 total 10.00 last 118.45)
 - /unlink 2375 (time 1789182508.91 total 20.02 last 118.81)
 - /unlink 3689 (time 1789182518.91 total 30.02 last 131.38)
total: 4500 /unlink in 37.16 seconds: 121.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 515 (time 1789182541.96 total 10.02 last 51.41)
 - open/close 1077 (time 1789182551.97 total 20.03 last 56.14)
 - open/close 1668 (time 1789182561.97 total 30.03 last 59.09)
 - open/close 2307 (time 1789182571.99 total 40.05 last 63.76)
 - open/close 2998 (time 1789182582.00 total 50.06 last 69.05)
 - open/close 3679 (time 1789182592.00 total 60.06 last 68.08)
 - open/close 4277 (time 1789182602.00 total 70.06 last 59.79)
total: 4500 open/close in 73.87 seconds: 60.92 ops/second
 - /unlink 1178 (time 1789182621.01 total 10.00 last 117.78)
 - /unlink 2255 (time 1789182631.03 total 20.01 last 107.57)
 - /unlink 3275 (time 1789182641.03 total 30.02 last 101.92)
total: 4500 /unlink in 39.96 seconds: 112.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 600 (time 1789182667.41 total 10.01 last 59.96)
 - open/close 1245 (time 1789182677.42 total 20.02 last 64.44)
 - open/close 1836 (time 1789182687.42 total 30.02 last 59.06)
 - open/close 2471 (time 1789182697.45 total 40.04 last 63.36)
 - open/close 3253 (time 1789182707.45 total 50.05 last 78.18)
 - open/close 3934 (time 1789182717.47 total 60.07 last 67.93)
total: 4500 open/close in 67.67 seconds: 66.50 ops/second
 - /unlink 1277 (time 1789182740.64 total 10.01 last 127.61)
 - /unlink 2396 (time 1789182750.65 total 20.02 last 111.75)
 - /unlink 3239 (time 1789182760.65 total 30.02 last 84.29)
 - /unlink 4111 (time 1789182770.66 total 40.03 last 87.15)
total: 4500 /unlink in 44.45 seconds: 101.24 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11377: 95219 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg425-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg425-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server
stop mds service on oleg425-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server
stop mds service on oleg425-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg425-server
unloading modules via unload_modules_local on: 'oleg425-server'
oleg425-server: oleg425-server.virtnet: executing unload_modules_local
oleg425-server: kmemleak disabled
modules unloaded.