== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:06:44 (1789301204)
Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg457-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg457-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg457-server
oleg457-server: oleg457-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg457-server'
oleg457-server: oleg457-server.virtnet: executing load_modules_local
oleg457-server: kmemleak disabled
oleg457-server: Loading modules from /home/green/git/lustre-release/lustre
oleg457-server: detected 4 online CPUs by sysfs
oleg457-server: MODOPTS_LIBCFS=
oleg457-server: Force libcfs to create 2 CPU partitions
oleg457-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg457-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all
pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg457-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all
pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg457-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all
pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg457-client.virtnet:  -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 630 (time 1789301442.37 total 10.01 last 62.96)
 - open/close 1204 (time 1789301452.38 total 20.02 last 57.32)
 - open/close 1805 (time 1789301462.39 total 30.04 last 60.02)
 - open/close 2479 (time 1789301472.41 total 40.05 last 67.32)
 - open/close 3126 (time 1789301482.41 total 50.05 last 64.70)
 - open/close 3727 (time 1789301492.42 total 60.06 last 60.01)
 - open/close 4371 (time 1789301502.43 total 70.07 last 64.33)
total: 4500 open/close in 72.02 seconds: 62.48 ops/second
 - /unlink 1068 (time 1789301519.65 total 10.01 last 106.74)
 - /unlink 1876 (time 1789301529.67 total 20.03 last 80.63)
 - /unlink 2820 (time 1789301539.68 total 30.03 last 94.36)
 - /unlink 3832 (time 1789301549.68 total 40.04 last 101.16)
total: 4500 /unlink in 45.51 seconds: 98.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 630 (time 1789301570.36 total 10.05 last 62.67)
 - open/close 1268 (time 1789301580.36 total 20.06 last 63.78)
 - open/close 1938 (time 1789301590.38 total 30.07 last 66.87)
 - open/close 2611 (time 1789301600.39 total 40.09 last 67.21)
 - open/close 3318 (time 1789301610.40 total 50.09 last 70.68)
 - open/close 3937 (time 1789301620.41 total 60.11 last 61.82)
total: 4500 open/close in 68.25 seconds: 65.94 ops/second
 - /unlink 1293 (time 1789301643.58 total 10.00 last 129.29)
 - /unlink 2616 (time 1789301653.59 total 20.01 last 132.24)
 - /unlink 3807 (time 1789301663.59 total 30.01 last 119.09)
total: 4500 /unlink in 36.62 seconds: 122.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 631 (time 1789301686.15 total 10.00 last 63.09)
 - open/close 1242 (time 1789301696.18 total 20.02 last 60.96)
 - open/close 1829 (time 1789301706.18 total 30.03 last 58.69)
 - open/close 2421 (time 1789301716.19 total 40.03 last 59.15)
 - open/close 3085 (time 1789301726.20 total 50.04 last 66.33)
 - open/close 3668 (time 1789301736.20 total 60.05 last 58.29)
 - open/close 4286 (time 1789301746.21 total 70.06 last 61.71)
total: 4500 open/close in 73.85 seconds: 60.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1057 (time 1789301765.36 total 10.01 last 105.63)
 - /unlink 2071 (time 1789301775.37 total 20.02 last 101.30)
 - /unlink 3034 (time 1789301785.37 total 30.02 last 96.28)
 - /unlink 4212 (time 1789301795.38 total 40.03 last 117.68)
total: 4500 /unlink in 42.39 seconds: 106.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 581 (time 1789301813.71 total 10.01 last 58.05)
 - open/close 1123 (time 1789301823.71 total 20.01 last 54.18)
 - open/close 1734 (time 1789301833.72 total 30.03 last 61.02)
 - open/close 2372 (time 1789301843.73 total 40.04 last 63.73)
 - open/close 3007 (time 1789301853.74 total 50.05 last 63.43)
 - open/close 3651 (time 1789301863.76 total 60.06 last 64.32)
 - open/close 4278 (time 1789301873.77 total 70.07 last 62.64)
total: 4500 open/close in 73.60 seconds: 61.14 ops/second
 - /unlink 1143 (time 1789301892.85 total 10.00 last 114.27)
 - /unlink 2305 (time 1789301902.85 total 20.00 last 116.17)
 - /unlink 3443 (time 1789301912.86 total 30.01 last 113.74)
total: 4500 /unlink in 38.57 seconds: 116.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 608 (time 1789301936.98 total 10.01 last 60.76)
 - open/close 1315 (time 1789301946.98 total 20.01 last 70.70)
 - open/close 1935 (time 1789301956.98 total 30.01 last 61.99)
 - open/close 2593 (time 1789301966.98 total 40.01 last 65.77)
 - open/close 3251 (time 1789301976.98 total 50.01 last 65.80)
 - open/close 3897 (time 1789301986.99 total 60.02 last 64.53)
 - open/close 4492 (time 1789301997.01 total 70.04 last 59.42)
total: 4500 open/close in 70.17 seconds: 64.13 ops/second
 - /unlink 1233 (time 1789302012.94 total 10.00 last 123.24)
 - /unlink 2399 (time 1789302022.95 total 20.01 last 116.53)
 - /unlink 3601 (time 1789302032.95 total 30.01 last 120.15)
total: 4500 /unlink in 37.48 seconds: 120.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 635 (time 1789302055.50 total 10.01 last 63.44)
 - open/close 1260 (time 1789302065.52 total 20.03 last 62.39)
 - open/close 1915 (time 1789302075.53 total 30.04 last 65.43)
 - open/close 2528 (time 1789302085.55 total 40.05 last 61.20)
 - open/close 3168 (time 1789302095.56 total 50.07 last 63.92)
 - open/close 3824 (time 1789302105.56 total 60.07 last 65.60)
 - open/close 4446 (time 1789302115.56 total 70.07 last 62.20)
total: 4500 open/close in 70.88 seconds: 63.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1145 (time 1789302131.67 total 10.01 last 114.42)
 - /unlink 2313 (time 1789302141.68 total 20.01 last 116.80)
 - /unlink 3559 (time 1789302151.68 total 30.01 last 124.56)
total: 4500 /unlink in 37.82 seconds: 118.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 689 (time 1789302174.03 total 10.00 last 68.88)
 - open/close 1373 (time 1789302184.04 total 20.02 last 68.31)
 - open/close 2027 (time 1789302194.06 total 30.03 last 65.31)
 - open/close 2731 (time 1789302204.06 total 40.04 last 70.35)
 - open/close 3380 (time 1789302214.07 total 50.04 last 64.84)
 - open/close 4018 (time 1789302224.10 total 60.07 last 63.63)
total: 4500 open/close in 69.27 seconds: 64.96 ops/second
 - /unlink 1179 (time 1789302249.34 total 10.00 last 117.88)
 - /unlink 2237 (time 1789302259.35 total 20.01 last 105.76)
 - /unlink 3459 (time 1789302269.35 total 30.01 last 122.12)
total: 4500 /unlink in 38.70 seconds: 116.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 617 (time 1789302293.64 total 10.01 last 61.65)
 - open/close 1265 (time 1789302303.64 total 20.01 last 64.78)
 - open/close 1927 (time 1789302313.66 total 30.02 last 66.11)
 - open/close 2549 (time 1789302323.66 total 40.03 last 62.17)
 - open/close 3116 (time 1789302333.67 total 50.04 last 56.63)
 - open/close 3778 (time 1789302343.67 total 60.04 last 66.20)
 - open/close 4384 (time 1789302353.71 total 70.08 last 60.39)
total: 4500 open/close in 71.89 seconds: 62.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1299 (time 1789302371.18 total 10.00 last 129.88)
 - /unlink 2449 (time 1789302381.19 total 20.01 last 114.94)
 - /unlink 3758 (time 1789302391.19 total 30.01 last 130.89)
total: 4500 /unlink in 36.60 seconds: 122.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 651 (time 1789302413.17 total 10.02 last 64.99)
 - open/close 1303 (time 1789302423.18 total 20.02 last 65.17)
 - open/close 1990 (time 1789302433.19 total 30.03 last 68.62)
 - open/close 2658 (time 1789302443.19 total 40.04 last 66.77)
 - open/close 3320 (time 1789302453.20 total 50.04 last 66.18)
 - open/close 4001 (time 1789302463.20 total 60.05 last 68.07)
total: 4500 open/close in 67.74 seconds: 66.43 ops/second
 - /unlink 1179 (time 1789302485.88 total 10.01 last 117.83)
 - /unlink 2313 (time 1789302495.89 total 20.01 last 113.31)
 - /unlink 3406 (time 1789302505.89 total 30.02 last 109.27)
total: 4500 /unlink in 39.83 seconds: 112.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 582 (time 1789302530.93 total 10.03 last 58.03)
 - open/close 1275 (time 1789302540.95 total 20.04 last 69.19)
 - open/close 1941 (time 1789302550.95 total 30.05 last 66.55)
 - open/close 2579 (time 1789302560.97 total 40.07 last 63.70)
 - open/close 3287 (time 1789302570.97 total 50.07 last 70.75)
 - open/close 3967 (time 1789302580.98 total 60.08 last 67.96)
total: 4500 open/close in 67.44 seconds: 66.73 ops/second
 - /unlink 1227 (time 1789302603.70 total 10.00 last 122.68)
 - /unlink 2376 (time 1789302613.70 total 20.00 last 114.89)
 - /unlink 3608 (time 1789302623.70 total 30.01 last 123.16)
total: 4500 /unlink in 38.52 seconds: 116.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 619 (time 1789302648.17 total 10.02 last 61.75)
 - open/close 1225 (time 1789302658.19 total 20.04 last 60.52)
 - open/close 1880 (time 1789302668.21 total 30.06 last 65.37)
 - open/close 2535 (time 1789302678.22 total 40.07 last 65.44)
 - open/close 3059 (time 1789302688.22 total 50.07 last 52.39)
 - open/close 3719 (time 1789302698.22 total 60.07 last 65.99)
 - open/close 4373 (time 1789302708.24 total 70.09 last 65.25)
total: 4500 open/close in 72.03 seconds: 62.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1387 (time 1789302724.67 total 10.00 last 138.70)
 - /unlink 2669 (time 1789302734.68 total 20.01 last 128.08)
 - /unlink 3787 (time 1789302744.69 total 30.02 last 111.69)
total: 4500 /unlink in 35.76 seconds: 125.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 658 (time 1789302765.22 total 10.01 last 65.76)
 - open/close 1231 (time 1789302775.23 total 20.01 last 57.26)
 - open/close 1861 (time 1789302785.24 total 30.02 last 62.96)
 - open/close 2552 (time 1789302795.24 total 40.03 last 69.06)
 - open/close 3235 (time 1789302805.26 total 50.04 last 68.21)
 - open/close 3814 (time 1789302815.26 total 60.04 last 57.90)
 - open/close 4480 (time 1789302825.26 total 70.04 last 66.57)
total: 4500 open/close in 70.35 seconds: 63.97 ops/second
 - /unlink 1042 (time 1789302840.73 total 10.00 last 104.15)
 - /unlink 2194 (time 1789302850.73 total 20.01 last 115.18)
 - /unlink 3324 (time 1789302860.76 total 30.03 last 112.68)
total: 4500 /unlink in 39.63 seconds: 113.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 688 (time 1789302884.36 total 10.02 last 68.65)
 - open/close 1350 (time 1789302894.37 total 20.03 last 66.17)
 - open/close 1997 (time 1789302904.40 total 30.06 last 64.47)
 - open/close 2566 (time 1789302914.42 total 40.07 last 56.82)
 - open/close 3122 (time 1789302924.43 total 50.08 last 55.56)
 - open/close 3731 (time 1789302934.43 total 60.08 last 60.90)
 - open/close 4355 (time 1789302944.46 total 70.11 last 62.21)
total: 4500 open/close in 72.48 seconds: 62.08 ops/second
 - /unlink 1190 (time 1789302961.66 total 10.00 last 118.94)
 - /unlink 2472 (time 1789302971.66 total 20.01 last 128.14)
 - /unlink 3793 (time 1789302981.67 total 30.02 last 132.02)
total: 4500 /unlink in 34.69 seconds: 129.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 685 (time 1789303002.10 total 10.01 last 68.44)
 - open/close 1385 (time 1789303012.11 total 20.02 last 69.93)
 - open/close 2094 (time 1789303022.11 total 30.02 last 70.87)
 - open/close 2750 (time 1789303032.11 total 40.03 last 65.57)
 - open/close 3450 (time 1789303042.12 total 50.03 last 69.98)
 - open/close 4044 (time 1789303052.13 total 60.04 last 59.32)
total: 4500 open/close in 67.99 seconds: 66.18 ops/second
 - /unlink 1143 (time 1789303075.27 total 10.01 last 114.18)
 - /unlink 2372 (time 1789303085.29 total 20.03 last 122.69)
 - /unlink 3671 (time 1789303095.30 total 30.03 last 129.81)
total: 4500 /unlink in 36.27 seconds: 124.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 625 (time 1789303116.00 total 10.02 last 62.39)
 - open/close 1198 (time 1789303126.01 total 20.03 last 57.20)
 - open/close 1735 (time 1789303136.02 total 30.04 last 53.68)
 - open/close 2282 (time 1789303146.03 total 40.05 last 54.66)
 - open/close 2851 (time 1789303156.04 total 50.06 last 56.84)
 - open/close 3456 (time 1789303166.04 total 60.06 last 60.49)
 - open/close 4081 (time 1789303176.08 total 70.10 last 62.23)
total: 4500 open/close in 76.70 seconds: 58.67 ops/second
 - /unlink 1248 (time 1789303197.60 total 10.00 last 124.80)
 - /unlink 2397 (time 1789303207.60 total 20.00 last 114.85)
 - /unlink 3715 (time 1789303217.61 total 30.01 last 131.75)
total: 4500 /unlink in 36.47 seconds: 123.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 634 (time 1789303239.35 total 10.02 last 63.29)
 - open/close 1259 (time 1789303249.35 total 20.02 last 62.50)
 - open/close 1913 (time 1789303259.35 total 30.02 last 65.38)
 - open/close 2607 (time 1789303269.37 total 40.04 last 69.28)
 - open/close 3261 (time 1789303279.38 total 50.04 last 65.37)
 - open/close 3960 (time 1789303289.39 total 60.05 last 69.82)
total: 4500 open/close in 68.36 seconds: 65.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1156 (time 1789303312.54 total 10.00 last 115.58)
 - /unlink 2411 (time 1789303322.54 total 20.00 last 125.49)
 - /unlink 3643 (time 1789303332.55 total 30.01 last 123.15)
total: 4500 /unlink in 37.62 seconds: 119.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 643 (time 1789303356.81 total 10.01 last 64.27)
 - open/close 1272 (time 1789303366.82 total 20.01 last 62.86)
 - open/close 1954 (time 1789303376.83 total 30.02 last 68.14)
 - open/close 2527 (time 1789303386.84 total 40.03 last 57.23)
 - open/close 3154 (time 1789303396.85 total 50.04 last 62.65)
 - open/close 3851 (time 1789303406.85 total 60.04 last 69.67)
total: 4500 open/close in 69.52 seconds: 64.73 ops/second
 - /unlink 1228 (time 1789303431.33 total 10.00 last 122.79)
 - /unlink 2252 (time 1789303441.33 total 20.01 last 102.34)
 - /unlink 3435 (time 1789303451.34 total 30.01 last 118.26)
total: 4500 /unlink in 37.90 seconds: 118.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 625 (time 1789303474.66 total 10.00 last 62.48)
 - open/close 1240 (time 1789303484.66 total 20.00 last 61.50)
 - open/close 1906 (time 1789303494.67 total 30.01 last 66.56)
 - open/close 2515 (time 1789303504.68 total 40.02 last 60.86)
 - open/close 3139 (time 1789303514.68 total 50.02 last 62.37)
 - open/close 3764 (time 1789303524.69 total 60.03 last 62.47)
 - open/close 4360 (time 1789303534.69 total 70.04 last 59.55)
total: 4500 open/close in 72.11 seconds: 62.40 ops/second
 - /unlink 1287 (time 1789303552.28 total 10.00 last 128.67)
 - /unlink 2477 (time 1789303562.28 total 20.00 last 118.98)
 - /unlink 3647 (time 1789303572.29 total 30.01 last 116.98)
total: 4500 /unlink in 37.23 seconds: 120.87 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95377 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg457-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg457-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server
stop mds service on oleg457-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server
stop mds service on oleg457-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg457-server
unloading modules via unload_modules_local on: 'oleg457-server'
oleg457-server: oleg457-server.virtnet: executing unload_modules_local
oleg457-server: kmemleak disabled
modules unloaded.