== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:37:29 (1788467849)
Stopping clients: oleg110-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg110-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg110-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg110-server
oleg110-server: oleg110-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg110-server'
oleg110-server: oleg110-server.virtnet: executing load_modules_local
oleg110-server: kmemleak disabled
oleg110-server: Loading modules from /home/green/git/lustre-release/lustre
oleg110-server: detected 4 online CPUs by sysfs
oleg110-server: MODOPTS_LIBCFS=
oleg110-server: Force libcfs to create 2 CPU partitions
oleg110-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg110-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg110-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg110-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg110-client.virtnet:  -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 693 (time 1788468049.33 total 10.01 last 69.24)
 - open/close 1394 (time 1788468059.40 total 20.08 last 69.57)
 - open/close 2077 (time 1788468069.40 total 30.09 last 68.30)
 - open/close 2685 (time 1788468079.41 total 40.09 last 60.77)
 - open/close 3367 (time 1788468089.43 total 50.11 last 68.07)
 - open/close 3844 (time 1788468099.43 total 60.11 last 47.69)
 - open/close 4485 (time 1788468109.44 total 70.12 last 64.07)
total: 4500 open/close in 70.32 seconds: 64.00 ops/second
 - /unlink 1269 (time 1788468123.67 total 10.00 last 126.89)
 - /unlink 2488 (time 1788468133.67 total 20.01 last 121.84)
 - /unlink 3853 (time 1788468143.67 total 30.01 last 136.49)
total: 4500 /unlink in 34.66 seconds: 129.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 621 (time 1788468163.22 total 10.05 last 61.77)
 - open/close 1348 (time 1788468173.22 total 20.06 last 72.66)
 - open/close 2016 (time 1788468183.24 total 30.07 last 66.70)
 - open/close 2734 (time 1788468193.25 total 40.08 last 71.74)
 - open/close 3437 (time 1788468203.27 total 50.10 last 70.14)
 - open/close 4147 (time 1788468213.32 total 60.16 last 70.63)
total: 4500 open/close in 65.65 seconds: 68.54 ops/second
 - /unlink 1379 (time 1788468234.04 total 10.00 last 137.85)
 - /unlink 2561 (time 1788468244.04 total 20.01 last 118.18)
 - /unlink 3944 (time 1788468254.04 total 30.01 last 138.29)
total: 4500 /unlink in 36.95 seconds: 121.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 425 (time 1788468277.95 total 10.02 last 42.41)
 - open/close 998 (time 1788468287.98 total 20.05 last 57.13)
 - open/close 1671 (time 1788468297.99 total 30.06 last 67.25)
 - open/close 2378 (time 1788468307.99 total 40.06 last 70.66)
 - open/close 3027 (time 1788468317.99 total 50.07 last 64.89)
 - open/close 3512 (time 1788468328.02 total 60.10 last 48.35)
 - open/close 3972 (time 1788468338.04 total 70.11 last 45.95)
total: 4500 open/close in 80.01 seconds: 56.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1309 (time 1788468363.12 total 10.00 last 130.87)
 - /unlink 2537 (time 1788468373.12 total 20.01 last 122.77)
 - /unlink 3912 (time 1788468383.13 total 30.01 last 137.45)
total: 4500 /unlink in 34.91 seconds: 128.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 658 (time 1788468402.99 total 10.00 last 65.78)
 - open/close 1362 (time 1788468413.00 total 20.01 last 70.36)
 - open/close 1979 (time 1788468423.01 total 30.02 last 61.63)
 - open/close 2554 (time 1788468433.02 total 40.03 last 57.45)
 - open/close 3348 (time 1788468443.06 total 50.07 last 79.07)
 - open/close 4078 (time 1788468453.06 total 60.07 last 72.99)
total: 4500 open/close in 65.06 seconds: 69.17 ops/second
 - /unlink 1216 (time 1788468473.41 total 10.00 last 121.58)
 - /unlink 2115 (time 1788468483.52 total 20.11 last 88.96)
 - /unlink 3114 (time 1788468493.52 total 30.11 last 99.89)
 - /unlink 4086 (time 1788468503.54 total 40.12 last 97.05)
total: 4500 /unlink in 43.88 seconds: 102.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 588 (time 1788468523.83 total 10.00 last 58.79)
 - open/close 1249 (time 1788468533.84 total 20.01 last 66.07)
 - open/close 1918 (time 1788468543.85 total 30.02 last 66.79)
 - open/close 2607 (time 1788468553.85 total 40.02 last 68.89)
 - open/close 3255 (time 1788468563.86 total 50.03 last 64.77)
 - open/close 3885 (time 1788468573.87 total 60.04 last 62.95)
total: 4500 open/close in 68.67 seconds: 65.53 ops/second
 - /unlink 1460 (time 1788468597.63 total 10.00 last 146.00)
 - /unlink 2899 (time 1788468607.63 total 20.00 last 143.87)
 - /unlink 4289 (time 1788468617.64 total 30.00 last 138.97)
total: 4500 /unlink in 31.40 seconds: 143.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 800 (time 1788468633.44 total 10.00 last 79.99)
 - open/close 1614 (time 1788468643.45 total 20.01 last 81.31)
 - open/close 2406 (time 1788468653.46 total 30.03 last 79.09)
 - open/close 3177 (time 1788468663.46 total 40.03 last 77.10)
 - open/close 3907 (time 1788468673.47 total 50.04 last 72.92)
total: 4500 open/close in 58.96 seconds: 76.32 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1475 (time 1788468697.13 total 10.00 last 147.44)
 - /unlink 2724 (time 1788468707.13 total 20.00 last 124.90)
 - /unlink 4217 (time 1788468717.15 total 30.02 last 149.03)
total: 4500 /unlink in 32.53 seconds: 138.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 692 (time 1788468734.61 total 10.01 last 69.17)
 - open/close 1578 (time 1788468744.63 total 20.02 last 88.44)
 - open/close 2278 (time 1788468754.64 total 30.03 last 69.93)
 - open/close 2913 (time 1788468764.65 total 40.04 last 63.45)
 - open/close 3566 (time 1788468774.65 total 50.04 last 65.29)
 - open/close 4319 (time 1788468784.67 total 60.06 last 75.14)
total: 4500 open/close in 63.32 seconds: 71.07 ops/second
 - /unlink 1522 (time 1788468802.58 total 10.00 last 152.18)
 - /unlink 2857 (time 1788468812.59 total 20.00 last 133.47)
 - /unlink 4156 (time 1788468822.59 total 30.01 last 129.83)
total: 4500 /unlink in 32.63 seconds: 137.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 765 (time 1788468840.35 total 10.00 last 76.49)
 - open/close 1531 (time 1788468850.36 total 20.01 last 76.51)
 - open/close 2298 (time 1788468860.36 total 30.01 last 76.69)
 - open/close 3013 (time 1788468870.36 total 40.02 last 71.46)
 - open/close 3610 (time 1788468880.38 total 50.03 last 59.63)
 - open/close 4238 (time 1788468890.39 total 60.05 last 62.69)
total: 4500 open/close in 63.64 seconds: 70.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1231 (time 1788468909.12 total 10.01 last 123.00)
 - /unlink 2278 (time 1788468919.14 total 20.03 last 104.48)
 - /unlink 3267 (time 1788468929.14 total 30.03 last 98.88)
 - /unlink 4185 (time 1788468939.15 total 40.04 last 91.73)
total: 4500 /unlink in 43.22 seconds: 104.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 712 (time 1788468958.06 total 10.01 last 71.16)
 - open/close 1396 (time 1788468968.07 total 20.02 last 68.31)
 - open/close 2159 (time 1788468978.10 total 30.05 last 76.07)
 - open/close 2882 (time 1788468988.14 total 40.08 last 72.05)
 - open/close 3446 (time 1788468998.14 total 50.09 last 56.36)
 - open/close 3866 (time 1788469008.16 total 60.10 last 41.95)
 - open/close 4346 (time 1788469018.16 total 70.11 last 47.97)
total: 4500 open/close in 71.94 seconds: 62.55 ops/second
 - /unlink 1066 (time 1788469035.38 total 10.02 last 106.35)
 - /unlink 2121 (time 1788469045.39 total 20.04 last 105.37)
 - /unlink 3218 (time 1788469055.39 total 30.04 last 109.69)
 - /unlink 4224 (time 1788469065.40 total 40.04 last 100.55)
total: 4500 /unlink in 43.29 seconds: 103.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 710 (time 1788469085.22 total 10.02 last 70.89)
 - open/close 1416 (time 1788469095.22 total 20.02 last 70.59)
 - open/close 2039 (time 1788469105.23 total 30.03 last 62.24)
 - open/close 2656 (time 1788469115.24 total 40.04 last 61.64)
 - open/close 3363 (time 1788469125.24 total 50.04 last 70.67)
 - open/close 4143 (time 1788469135.25 total 60.05 last 77.94)
total: 4500 open/close in 64.90 seconds: 69.33 ops/second
 - /unlink 1365 (time 1788469156.03 total 10.01 last 136.41)
 - /unlink 2277 (time 1788469166.03 total 20.01 last 91.13)
 - /unlink 3149 (time 1788469176.04 total 30.02 last 87.15)
 - /unlink 4082 (time 1788469186.05 total 40.04 last 93.16)
total: 4500 /unlink in 45.09 seconds: 99.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 675 (time 1788469206.72 total 10.01 last 67.44)
 - open/close 1438 (time 1788469216.72 total 20.01 last 76.28)
 - open/close 2084 (time 1788469226.73 total 30.02 last 64.53)
 - open/close 2598 (time 1788469236.74 total 40.03 last 51.36)
 - open/close 3088 (time 1788469246.75 total 50.04 last 48.95)
 - open/close 3629 (time 1788469256.75 total 60.04 last 54.08)
 - open/close 4213 (time 1788469266.76 total 70.05 last 58.37)
total: 4500 open/close in 74.46 seconds: 60.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1005 (time 1788469287.78 total 10.00 last 100.47)
 - /unlink 1952 (time 1788469297.79 total 20.01 last 94.63)
 - /unlink 2945 (time 1788469307.79 total 30.01 last 99.27)
 - /unlink 4428 (time 1788469317.79 total 40.02 last 148.24)
total: 4500 /unlink in 40.88 seconds: 110.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 609 (time 1788469334.55 total 10.01 last 60.86)
 - open/close 1274 (time 1788469344.56 total 20.01 last 66.48)
 - open/close 1929 (time 1788469354.61 total 30.07 last 65.13)
 - open/close 2681 (time 1788469364.63 total 40.08 last 75.07)
 - open/close 3421 (time 1788469374.64 total 50.09 last 73.90)
 - open/close 4219 (time 1788469384.65 total 60.10 last 79.77)
total: 4500 open/close in 63.55 seconds: 70.81 ops/second
 - /unlink 1216 (time 1788469403.41 total 10.00 last 121.55)
 - /unlink 2524 (time 1788469413.41 total 20.00 last 130.80)
 - /unlink 3815 (time 1788469423.41 total 30.01 last 129.04)
total: 4500 /unlink in 36.43 seconds: 123.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 836 (time 1788469445.93 total 10.01 last 83.54)
 - open/close 1651 (time 1788469455.93 total 20.01 last 81.45)
 - open/close 2409 (time 1788469465.94 total 30.02 last 75.76)
 - open/close 3071 (time 1788469475.94 total 40.02 last 66.15)
 - open/close 3696 (time 1788469485.98 total 50.06 last 62.30)
 - open/close 4281 (time 1788469495.98 total 60.06 last 58.46)
total: 4500 open/close in 62.88 seconds: 71.56 ops/second
 - /unlink 1081 (time 1788469514.55 total 10.00 last 108.08)
 - /unlink 2300 (time 1788469524.56 total 20.01 last 121.85)
 - /unlink 3617 (time 1788469534.57 total 30.01 last 131.59)
total: 4500 /unlink in 37.95 seconds: 118.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 770 (time 1788469558.38 total 10.01 last 76.92)
 - open/close 1480 (time 1788469568.39 total 20.01 last 70.98)
 - open/close 2229 (time 1788469578.39 total 30.01 last 74.89)
 - open/close 2812 (time 1788469588.40 total 40.03 last 58.24)
 - open/close 3536 (time 1788469598.40 total 50.03 last 72.37)
 - open/close 4202 (time 1788469608.42 total 60.05 last 66.50)
total: 4500 open/close in 67.12 seconds: 67.04 ops/second
 - /unlink 1055 (time 1788469631.39 total 10.01 last 105.42)
 - /unlink 2104 (time 1788469641.39 total 20.01 last 104.88)
 - /unlink 3090 (time 1788469651.40 total 30.01 last 98.56)
 - /unlink 3739 (time 1788469661.44 total 40.06 last 64.61)
 - /unlink 4408 (time 1788469671.44 total 50.06 last 66.89)
total: 4500 /unlink in 51.50 seconds: 87.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 711 (time 1788469690.16 total 10.00 last 71.08)
 - open/close 1382 (time 1788469700.17 total 20.01 last 67.08)
 - open/close 1973 (time 1788469710.17 total 30.01 last 59.06)
 - open/close 2505 (time 1788469720.18 total 40.02 last 53.16)
 - open/close 3143 (time 1788469730.19 total 50.03 last 63.76)
 - open/close 3786 (time 1788469740.19 total 60.03 last 64.29)
total: 4500 open/close in 69.80 seconds: 64.47 ops/second
 - /unlink 1414 (time 1788469764.11 total 10.00 last 141.37)
 - /unlink 2797 (time 1788469774.11 total 20.01 last 138.25)
 - /unlink 4201 (time 1788469784.11 total 30.01 last 140.39)
total: 4500 /unlink in 31.67 seconds: 142.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 737 (time 1788469799.42 total 10.00 last 73.68)
 - open/close 1393 (time 1788469809.42 total 20.01 last 65.56)
 - open/close 2100 (time 1788469819.43 total 30.01 last 70.66)
 - open/close 2890 (time 1788469829.44 total 40.02 last 78.94)
 - open/close 3610 (time 1788469839.46 total 50.04 last 71.86)
 - open/close 4359 (time 1788469849.46 total 60.04 last 74.90)
total: 4500 open/close in 62.09 seconds: 72.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1138 (time 1788469866.95 total 10.01 last 113.72)
 - /unlink 2230 (time 1788469876.96 total 20.02 last 109.09)
 - /unlink 3249 (time 1788469886.96 total 30.02 last 101.90)
total: 4500 /unlink in 38.93 seconds: 115.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 586 (time 1788469912.48 total 10.02 last 58.48)
 - open/close 1254 (time 1788469922.48 total 20.02 last 66.80)
 - open/close 2002 (time 1788469932.48 total 30.03 last 74.74)
 - open/close 2677 (time 1788469942.50 total 40.04 last 67.41)
 - open/close 3259 (time 1788469952.50 total 50.05 last 58.17)
 - open/close 3840 (time 1788469962.52 total 60.06 last 58.02)
 - open/close 4334 (time 1788469972.52 total 70.07 last 49.37)
total: 4500 open/close in 73.99 seconds: 60.82 ops/second
 - /unlink 1145 (time 1788469993.05 total 10.00 last 114.48)
 - /unlink 2146 (time 1788470003.05 total 20.01 last 100.06)
 - /unlink 3192 (time 1788470013.06 total 30.01 last 104.56)
 - /unlink 4049 (time 1788470023.06 total 40.01 last 85.67)
total: 4500 /unlink in 44.69 seconds: 100.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 501 (time 1788470044.98 total 10.01 last 50.04)
 - open/close 1100 (time 1788470054.98 total 20.02 last 59.88)
 - open/close 1619 (time 1788470064.98 total 30.02 last 51.89)
 - open/close 2230 (time 1788470074.99 total 40.02 last 61.09)
 - open/close 2824 (time 1788470085.00 total 50.04 last 59.30)
 - open/close 3330 (time 1788470095.03 total 60.06 last 50.46)
 - open/close 3832 (time 1788470105.05 total 70.08 last 50.12)
 - open/close 4468 (time 1788470115.05 total 80.08 last 63.58)
total: 4500 open/close in 80.46 seconds: 55.93 ops/second
 - /unlink 1001 (time 1788470130.50 total 10.00 last 100.08)
 - /unlink 2106 (time 1788470140.50 total 20.00 last 110.49)
 - /unlink 2769 (time 1788470151.27 total 30.78 last 61.53)
 - /unlink 3598 (time 1788470161.28 total 40.79 last 82.82)
total: 4500 /unlink in 48.32 seconds: 93.12 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95103 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg110-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server
stop mds service on oleg110-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server
stop mds service on oleg110-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server
unloading modules via unload_modules_local on: 'oleg110-server'
oleg110-server: oleg110-server.virtnet: executing unload_modules_local
oleg110-server: kmemleak disabled
modules unloaded.