== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:59:57 (1789070397)
Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg220-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg220-server
oleg220-server: oleg220-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: kmemleak disabled
oleg220-server: Loading modules from /home/green/git/lustre-release/lustre
oleg220-server: detected 4 online CPUs by sysfs
oleg220-server: MODOPTS_LIBCFS=
oleg220-server: Force libcfs to create 2 CPU partitions
oleg220-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg220-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg220-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg220-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 697 (time 1789070613.78 total 10.01 last 69.62)
 - open/close 1391 (time 1789070623.79 total 20.02 last 69.35)
 - open/close 1984 (time 1789070633.80 total 30.03 last 59.25)
 - open/close 2591 (time 1789070643.83 total 40.06 last 60.50)
 - open/close 3199 (time 1789070653.84 total 50.06 last 60.77)
 - open/close 3808 (time 1789070663.84 total 60.07 last 60.85)
 - open/close 4423 (time 1789070673.85 total 70.07 last 61.50)
total: 4500 open/close in 70.92 seconds: 63.45 ops/second
 - /unlink 1226 (time 1789070689.79 total 10.00 last 122.59)
 - /unlink 2371 (time 1789070699.80 total 20.00 last 114.46)
 - /unlink 3542 (time 1789070709.80 total 30.01 last 117.08)
total: 4500 /unlink in 38.46 seconds: 117.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 461 (time 1789070734.62 total 10.03 last 45.98)
 - open/close 977 (time 1789070744.63 total 20.03 last 51.57)
 - open/close 1583 (time 1789070754.63 total 30.03 last 60.58)
 - open/close 2137 (time 1789070764.63 total 40.04 last 55.38)
 - open/close 2658 (time 1789070774.66 total 50.06 last 51.98)
 - open/close 3259 (time 1789070784.66 total 60.06 last 60.08)
 - open/close 3870 (time 1789070794.66 total 70.06 last 61.10)
 - open/close 4481 (time 1789070804.66 total 80.07 last 61.07)
total: 4500 open/close in 80.43 seconds: 55.95 ops/second
 - /unlink 1148 (time 1789070820.47 total 10.00 last 114.76)
 - /unlink 2432 (time 1789070830.48 total 20.01 last 128.35)
 - /unlink 3738 (time 1789070840.48 total 30.01 last 130.58)
total: 4500 /unlink in 36.72 seconds: 122.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 625 (time 1789070865.64 total 10.07 last 62.04)
 - open/close 1216 (time 1789070875.64 total 20.08 last 59.09)
 - open/close 1837 (time 1789070885.67 total 30.10 last 61.95)
 - open/close 2432 (time 1789070895.67 total 40.10 last 59.48)
 - open/close 2922 (time 1789070905.68 total 50.11 last 48.97)
 - open/close 3477 (time 1789070915.68 total 60.12 last 55.46)
 - open/close 4109 (time 1789070925.69 total 70.12 last 63.18)
total: 4500 open/close in 77.59 seconds: 58.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1181 (time 1789070948.86 total 10.02 last 117.87)
 - /unlink 2324 (time 1789070958.86 total 20.02 last 114.27)
 - /unlink 3332 (time 1789070968.87 total 30.03 last 100.72)
total: 4500 /unlink in 39.69 seconds: 113.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 626 (time 1789070993.52 total 10.00 last 62.59)
 - open/close 1300 (time 1789071003.53 total 20.01 last 67.34)
 - open/close 1893 (time 1789071013.56 total 30.05 last 59.10)
 - open/close 2514 (time 1789071023.58 total 40.06 last 61.99)
 - open/close 3108 (time 1789071033.59 total 50.07 last 59.37)
 - open/close 3709 (time 1789071043.59 total 60.07 last 60.08)
total: 4500 open/close in 69.98 seconds: 64.30 ops/second
 - /unlink 1280 (time 1789071068.76 total 10.00 last 127.95)
 - /unlink 2630 (time 1789071078.76 total 20.01 last 134.93)
 - /unlink 4030 (time 1789071088.77 total 30.01 last 139.96)
total: 4500 /unlink in 33.22 seconds: 135.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 672 (time 1789071107.25 total 10.01 last 67.12)
 - open/close 1281 (time 1789071117.26 total 20.02 last 60.84)
 - open/close 2028 (time 1789071127.27 total 30.03 last 74.62)
 - open/close 2666 (time 1789071137.29 total 40.05 last 63.70)
 - open/close 3278 (time 1789071147.30 total 50.06 last 61.15)
 - open/close 3916 (time 1789071157.30 total 60.06 last 63.79)
 - open/close 4475 (time 1789071167.30 total 70.06 last 55.90)
total: 4500 open/close in 70.47 seconds: 63.85 ops/second
 - /unlink 818 (time 1789071182.90 total 10.01 last 81.72)
 - /unlink 1606 (time 1789071192.92 total 20.03 last 78.64)
 - /unlink 2564 (time 1789071202.92 total 30.03 last 95.78)
 - /unlink 3600 (time 1789071212.92 total 40.03 last 103.58)
total: 4500 /unlink in 47.60 seconds: 94.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 516 (time 1789071236.83 total 10.02 last 51.47)
 - open/close 1014 (time 1789071246.85 total 20.04 last 49.74)
 - open/close 1686 (time 1789071256.85 total 30.04 last 67.16)
 - open/close 2273 (time 1789071266.85 total 40.05 last 58.68)
 - open/close 2746 (time 1789071276.87 total 50.06 last 47.22)
 - open/close 3290 (time 1789071286.88 total 60.07 last 54.35)
 - open/close 3937 (time 1789071296.89 total 70.08 last 64.64)
total: 4500 open/close in 78.39 seconds: 57.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1210 (time 1789071320.92 total 10.00 last 120.99)
 - /unlink 2283 (time 1789071330.93 total 20.01 last 107.24)
 - /unlink 3766 (time 1789071340.93 total 30.01 last 148.29)
total: 4500 /unlink in 36.40 seconds: 123.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 456 (time 1789071363.59 total 10.02 last 45.52)
 - open/close 883 (time 1789071373.59 total 20.02 last 42.67)
 - open/close 1516 (time 1789071383.60 total 30.03 last 63.28)
 - open/close 2080 (time 1789071393.60 total 40.03 last 56.37)
 - open/close 2695 (time 1789071403.60 total 50.03 last 61.50)
 - open/close 3533 (time 1789071413.60 total 60.03 last 83.79)
 - open/close 4193 (time 1789071423.60 total 70.03 last 65.99)
total: 4500 open/close in 75.45 seconds: 59.65 ops/second
 - /unlink 1163 (time 1789071444.84 total 10.01 last 116.22)
 - /unlink 2272 (time 1789071454.85 total 20.01 last 110.87)
 - /unlink 3519 (time 1789071464.85 total 30.01 last 124.67)
total: 4500 /unlink in 37.56 seconds: 119.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 626 (time 1789071488.76 total 10.01 last 62.55)
 - open/close 1213 (time 1789071498.78 total 20.02 last 58.62)
 - open/close 1768 (time 1789071508.78 total 30.02 last 55.50)
 - open/close 2340 (time 1789071518.78 total 40.02 last 57.20)
 - open/close 3009 (time 1789071528.79 total 50.03 last 66.86)
 - open/close 3630 (time 1789071538.79 total 60.03 last 62.08)
 - open/close 4195 (time 1789071548.79 total 70.04 last 56.47)
total: 4500 open/close in 75.57 seconds: 59.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1372 (time 1789071570.44 total 10.00 last 137.18)
 - /unlink 2828 (time 1789071580.47 total 20.03 last 145.18)
 - /unlink 4026 (time 1789071590.47 total 30.04 last 119.74)
total: 4500 /unlink in 34.00 seconds: 132.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 624 (time 1789071610.11 total 10.01 last 62.35)
 - open/close 1362 (time 1789071620.12 total 20.02 last 73.75)
 - open/close 2008 (time 1789071630.13 total 30.03 last 64.53)
 - open/close 2575 (time 1789071640.14 total 40.03 last 56.67)
 - open/close 3232 (time 1789071650.14 total 50.04 last 65.67)
 - open/close 3904 (time 1789071660.15 total 60.04 last 67.15)
 - open/close 4432 (time 1789071670.15 total 70.05 last 52.78)
total: 4500 open/close in 71.20 seconds: 63.20 ops/second
 - /unlink 928 (time 1789071687.64 total 10.01 last 92.75)
 - /unlink 1943 (time 1789071697.64 total 20.01 last 101.47)
 - /unlink 2870 (time 1789071707.65 total 30.01 last 92.67)
 - /unlink 3790 (time 1789071717.65 total 40.02 last 91.96)
total: 4500 /unlink in 49.84 seconds: 90.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 490 (time 1789071745.24 total 10.01 last 48.97)
 - open/close 1050 (time 1789071755.24 total 20.01 last 55.98)
 - open/close 1614 (time 1789071765.24 total 30.01 last 56.39)
 - open/close 2209 (time 1789071775.26 total 40.02 last 59.43)
 - open/close 2813 (time 1789071785.27 total 50.04 last 60.31)
 - open/close 3351 (time 1789071795.30 total 60.06 last 53.67)
 - open/close 3739 (time 1789071805.32 total 70.09 last 38.69)
 - open/close 4049 (time 1789071815.35 total 80.11 last 30.93)
 - open/close 4473 (time 1789071825.35 total 90.12 last 42.37)
total: 4500 open/close in 90.65 seconds: 49.64 ops/second
 - /unlink 1053 (time 1789071842.14 total 10.00 last 105.27)
 - /unlink 2024 (time 1789071852.14 total 20.00 last 97.08)
 - /unlink 2847 (time 1789071862.15 total 30.01 last 82.26)
 - /unlink 3794 (time 1789071872.15 total 40.01 last 94.67)
total: 4500 /unlink in 46.39 seconds: 97.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 596 (time 1789071895.65 total 10.02 last 59.48)
 - open/close 1197 (time 1789071905.66 total 20.03 last 60.05)
 - open/close 1719 (time 1789071915.66 total 30.03 last 52.18)
 - open/close 2360 (time 1789071925.67 total 40.04 last 64.03)
 - open/close 3047 (time 1789071935.68 total 50.05 last 68.64)
 - open/close 3723 (time 1789071945.69 total 60.06 last 67.55)
 - open/close 4293 (time 1789071955.71 total 70.08 last 56.87)
total: 4500 open/close in 73.21 seconds: 61.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1102 (time 1789071975.58 total 10.00 last 110.15)
 - /unlink 2231 (time 1789071985.60 total 20.02 last 112.75)
 - /unlink 3208 (time 1789071995.63 total 30.05 last 97.42)
 - /unlink 4242 (time 1789072005.63 total 40.05 last 103.36)
total: 4500 /unlink in 42.10 seconds: 106.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 569 (time 1789072024.14 total 10.01 last 56.86)
 - open/close 1244 (time 1789072034.16 total 20.02 last 67.41)
 - open/close 1905 (time 1789072044.17 total 30.03 last 66.03)
 - open/close 2754 (time 1789072054.17 total 40.04 last 84.85)
 - open/close 3396 (time 1789072064.18 total 50.05 last 64.14)
 - open/close 3975 (time 1789072074.20 total 60.06 last 57.82)
total: 4500 open/close in 69.07 seconds: 65.15 ops/second
 - /unlink 721 (time 1789072099.53 total 10.02 last 71.95)
 - /unlink 1487 (time 1789072109.55 total 20.04 last 76.45)
 - /unlink 2573 (time 1789072119.55 total 30.04 last 108.56)
 - /unlink 3561 (time 1789072129.56 total 40.05 last 98.75)
total: 4500 /unlink in 48.89 seconds: 92.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 381 (time 1789072155.78 total 10.05 last 37.91)
 - open/close 718 (time 1789072165.79 total 20.06 last 33.66)
 - open/close 1253 (time 1789072175.81 total 30.08 last 53.41)
 - open/close 1860 (time 1789072185.83 total 40.10 last 60.60)
 - open/close 2506 (time 1789072195.87 total 50.14 last 64.31)
 - open/close 3059 (time 1789072205.89 total 60.16 last 55.23)
 - open/close 3428 (time 1789072215.89 total 70.16 last 36.88)
 - open/close 4035 (time 1789072225.90 total 80.17 last 60.66)
total: 4500 open/close in 88.02 seconds: 51.13 ops/second
 - /unlink 1212 (time 1789072249.37 total 10.00 last 121.20)
 - /unlink 2255 (time 1789072259.37 total 20.01 last 104.25)
 - /unlink 3340 (time 1789072269.38 total 30.01 last 108.41)
total: 4500 /unlink in 39.53 seconds: 113.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 475 (time 1789072295.25 total 10.02 last 47.41)
 - open/close 909 (time 1789072305.26 total 20.03 last 43.37)
 - open/close 1462 (time 1789072315.26 total 30.03 last 55.26)
 - open/close 1974 (time 1789072325.27 total 40.04 last 51.16)
 - open/close 2550 (time 1789072335.29 total 50.05 last 57.52)
 - open/close 3149 (time 1789072345.29 total 60.06 last 59.85)
 - open/close 3663 (time 1789072355.30 total 70.07 last 51.38)
 - open/close 4252 (time 1789072365.31 total 80.07 last 58.86)
total: 4500 open/close in 84.43 seconds: 53.30 ops/second
 - /unlink 925 (time 1789072384.88 total 10.00 last 92.50)
 - /unlink 1915 (time 1789072394.89 total 20.01 last 98.94)
 - /unlink 3012 (time 1789072404.89 total 30.01 last 109.67)
 - /unlink 4272 (time 1789072414.89 total 40.01 last 125.96)
total: 4500 /unlink in 42.22 seconds: 106.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 631 (time 1789072433.49 total 10.01 last 63.05)
 - open/close 1245 (time 1789072443.51 total 20.03 last 61.28)
 - open/close 1809 (time 1789072453.53 total 30.04 last 56.33)
 - open/close 2430 (time 1789072463.54 total 40.05 last 62.03)
 - open/close 3109 (time 1789072473.55 total 50.07 last 67.80)
 - open/close 3763 (time 1789072483.55 total 60.07 last 65.39)
 - open/close 4380 (time 1789072493.56 total 70.08 last 61.65)
total: 4500 open/close in 71.90 seconds: 62.59 ops/second
 - /unlink 1114 (time 1789072510.70 total 10.00 last 111.38)
 - /unlink 2205 (time 1789072520.72 total 20.02 last 108.92)
 - /unlink 3091 (time 1789072530.72 total 30.02 last 88.56)
 - /unlink 4024 (time 1789072540.72 total 40.02 last 93.28)
total: 4500 /unlink in 46.80 seconds: 96.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 699 (time 1789072563.84 total 10.00 last 69.88)
 - open/close 1319 (time 1789072573.85 total 20.01 last 61.94)
 - open/close 1816 (time 1789072583.85 total 30.02 last 49.67)
 - open/close 2392 (time 1789072593.85 total 40.02 last 57.60)
 - open/close 3176 (time 1789072603.86 total 50.03 last 78.34)
 - open/close 3710 (time 1789072613.87 total 60.03 last 53.38)
 - open/close 4094 (time 1789072623.87 total 70.03 last 38.39)
 - open/close 4421 (time 1789072633.88 total 80.04 last 32.67)
total: 4500 open/close in 82.34 seconds: 54.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1035 (time 1789072654.45 total 10.02 last 103.33)
 - /unlink 2131 (time 1789072664.45 total 20.02 last 109.58)
 - /unlink 3172 (time 1789072674.45 total 30.02 last 104.07)
 - /unlink 4320 (time 1789072684.47 total 40.04 last 114.61)
total: 4500 /unlink in 41.72 seconds: 107.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 457 (time 1789072701.96 total 10.00 last 45.68)
 - open/close 900 (time 1789072711.96 total 20.01 last 44.30)
 - open/close 1512 (time 1789072721.97 total 30.02 last 61.12)
 - open/close 2160 (time 1789072731.98 total 40.03 last 64.76)
 - open/close 2787 (time 1789072741.99 total 50.03 last 62.65)
 - open/close 3389 (time 1789072751.99 total 60.04 last 60.19)
 - open/close 3961 (time 1789072762.00 total 70.05 last 57.14)
total: 4500 open/close in 79.91 seconds: 56.31 ops/second
 - /unlink 772 (time 1789072787.74 total 10.00 last 77.19)
 - /unlink 1743 (time 1789072797.75 total 20.01 last 97.06)
 - /unlink 2662 (time 1789072807.75 total 30.01 last 91.88)
 - /unlink 3431 (time 1789072817.78 total 40.04 last 76.67)
total: 4500 /unlink in 49.65 seconds: 90.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 428 (time 1789072844.98 total 10.02 last 42.72)
 - open/close 865 (time 1789072854.98 total 20.03 last 43.67)
 - open/close 1537 (time 1789072864.99 total 30.03 last 67.18)
 - open/close 2235 (time 1789072875.01 total 40.05 last 69.64)
 - open/close 2776 (time 1789072885.02 total 50.06 last 54.05)
 - open/close 3412 (time 1789072895.04 total 60.08 last 63.48)
 - open/close 3751 (time 1789072905.05 total 70.09 last 33.86)
 - open/close 4346 (time 1789072915.06 total 80.10 last 59.47)
total: 4500 open/close in 82.64 seconds: 54.45 ops/second
 - /unlink 1323 (time 1789072932.65 total 10.00 last 132.28)
 - /unlink 2373 (time 1789072942.65 total 20.00 last 104.99)
 - /unlink 3418 (time 1789072952.66 total 30.01 last 104.44)
total: 4500 /unlink in 39.74 seconds: 113.23 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95421 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server
stop mds service on oleg220-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server
stop mds service on oleg220-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server
unloading modules via unload_modules_local on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing unload_modules_local
oleg220-server: kmemleak disabled
modules unloaded.