== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:47:26 (1789638446)
Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg339-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server
oleg339-server: oleg339-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg339-server'
oleg339-server: oleg339-server.virtnet: executing load_modules_local
oleg339-server: kmemleak disabled
oleg339-server: Loading modules from /home/green/git/lustre-release/lustre
oleg339-server: detected 4 online CPUs by sysfs
oleg339-server: MODOPTS_LIBCFS=
oleg339-server: Force libcfs to create 2 CPU partitions
oleg339-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg339-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg339-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg339-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg339-client.virtnet:  -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 658 (time 1789638650.09 total 10.00 last 65.80)
 - open/close 1314 (time 1789638660.11 total 20.01 last 65.51)
 - open/close 2021 (time 1789638670.12 total 30.03 last 70.58)
 - open/close 2677 (time 1789638680.14 total 40.04 last 65.52)
 - open/close 3383 (time 1789638690.14 total 50.04 last 70.60)
 - open/close 4122 (time 1789638700.15 total 60.05 last 73.83)
total: 4500 open/close in 65.79 seconds: 68.40 ops/second
 - /unlink 1279 (time 1789638720.28 total 10.01 last 127.77)
 - /unlink 2674 (time 1789638730.28 total 20.01 last 139.49)
 - /unlink 3976 (time 1789638740.30 total 30.03 last 129.89)
total: 4500 /unlink in 34.48 seconds: 130.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 654 (time 1789638759.60 total 10.01 last 65.35)
 - open/close 1282 (time 1789638769.60 total 20.01 last 62.80)
 - open/close 1962 (time 1789638779.61 total 30.02 last 67.91)
 - open/close 2608 (time 1789638789.63 total 40.04 last 64.49)
 - open/close 3203 (time 1789638799.63 total 50.04 last 59.49)
 - open/close 3875 (time 1789638809.64 total 60.05 last 67.17)
total: 4500 open/close in 69.68 seconds: 64.58 ops/second
 - /unlink 1336 (time 1789638833.46 total 10.01 last 133.52)
 - /unlink 2520 (time 1789638843.47 total 20.01 last 118.34)
 - /unlink 3692 (time 1789638853.48 total 30.03 last 117.02)
total: 4500 /unlink in 36.37 seconds: 123.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 705 (time 1789638874.43 total 10.01 last 70.39)
 - open/close 1402 (time 1789638884.43 total 20.02 last 69.66)
 - open/close 2023 (time 1789638894.44 total 30.03 last 62.04)
 - open/close 2551 (time 1789638904.47 total 40.06 last 52.67)
 - open/close 3119 (time 1789638914.49 total 50.08 last 56.68)
 - open/close 3740 (time 1789638924.50 total 60.09 last 62.02)
 - open/close 4415 (time 1789638934.52 total 70.11 last 67.39)
total: 4500 open/close in 71.48 seconds: 62.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1130 (time 1789638951.20 total 10.00 last 112.99)
 - /unlink 1838 (time 1789638961.20 total 20.01 last 70.75)
 - /unlink 2521 (time 1789638971.22 total 30.02 last 68.19)
 - /unlink 3336 (time 1789638981.22 total 40.03 last 81.48)
 - /unlink 4485 (time 1789638991.22 total 50.03 last 114.87)
total: 4500 /unlink in 50.12 seconds: 89.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 618 (time 1789639005.79 total 10.02 last 61.65)
 - open/close 1216 (time 1789639015.80 total 20.03 last 59.74)
 - open/close 1885 (time 1789639025.80 total 30.04 last 66.86)
 - open/close 2543 (time 1789639035.83 total 40.06 last 65.66)
 - open/close 3252 (time 1789639045.83 total 50.07 last 70.85)
 - open/close 3966 (time 1789639055.84 total 60.08 last 71.32)
total: 4500 open/close in 67.67 seconds: 66.50 ops/second
 - /unlink 1443 (time 1789639077.11 total 10.00 last 144.27)
 - /unlink 2878 (time 1789639087.11 total 20.01 last 143.43)
total: 4500 /unlink in 29.25 seconds: 153.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 932 (time 1789639109.24 total 10.01 last 93.10)
 - open/close 1782 (time 1789639119.25 total 20.02 last 84.89)
 - open/close 2558 (time 1789639129.26 total 30.03 last 77.52)
 - open/close 3297 (time 1789639139.27 total 40.04 last 73.89)
 - open/close 4138 (time 1789639149.27 total 50.04 last 84.07)
total: 4500 open/close in 54.71 seconds: 82.26 ops/second
 - /unlink 1408 (time 1789639167.36 total 10.00 last 140.80)
 - /unlink 3000 (time 1789639177.36 total 20.00 last 159.14)
total: 4500 /unlink in 29.26 seconds: 153.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 791 (time 1789639200.31 total 10.01 last 79.05)
 - open/close 1594 (time 1789639210.31 total 20.01 last 80.29)
 - open/close 2315 (time 1789639220.32 total 30.01 last 72.06)
 - open/close 3069 (time 1789639230.33 total 40.03 last 75.30)
 - open/close 3839 (time 1789639240.35 total 50.04 last 76.87)
total: 4500 open/close in 59.50 seconds: 75.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1367 (time 1789639263.34 total 10.00 last 136.66)
 - /unlink 2875 (time 1789639273.35 total 20.01 last 150.73)
 - /unlink 4340 (time 1789639283.36 total 30.01 last 146.41)
total: 4500 /unlink in 31.02 seconds: 145.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 660 (time 1789639297.69 total 10.01 last 65.90)
 - open/close 1040 (time 1789639307.82 total 20.15 last 37.49)
 - open/close 1529 (time 1789639318.10 total 30.42 last 47.61)
 - open/close 1893 (time 1789639328.11 total 40.43 last 36.36)
 - open/close 2215 (time 1789639338.12 total 50.44 last 32.17)
 - open/close 2666 (time 1789639348.12 total 60.44 last 45.10)
 - open/close 3228 (time 1789639358.15 total 70.48 last 56.01)
 - open/close 3847 (time 1789639368.16 total 80.48 last 61.87)
total: 4500 open/close in 89.21 seconds: 50.44 ops/second
 - /unlink 1329 (time 1789639391.47 total 10.00 last 132.87)
 - /unlink 2651 (time 1789639401.47 total 20.01 last 132.09)
 - /unlink 3984 (time 1789639411.48 total 30.01 last 133.26)
total: 4500 /unlink in 34.72 seconds: 129.60 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 556 (time 1789639432.37 total 10.03 last 55.45)
 - open/close 1100 (time 1789639442.37 total 20.03 last 54.36)
 - open/close 1740 (time 1789639452.40 total 30.07 last 63.81)
 - open/close 2353 (time 1789639462.41 total 40.07 last 61.26)
 - open/close 2894 (time 1789639472.41 total 50.08 last 54.08)
 - open/close 3339 (time 1789639482.43 total 60.09 last 44.43)
 - open/close 3794 (time 1789639492.44 total 70.10 last 45.46)
 - open/close 4324 (time 1789639502.45 total 80.11 last 52.94)
total: 4500 open/close in 82.62 seconds: 54.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1251 (time 1789639519.86 total 10.00 last 125.09)
 - /unlink 2313 (time 1789639529.87 total 20.00 last 106.16)
 - /unlink 3527 (time 1789639539.88 total 30.01 last 121.28)
total: 4500 /unlink in 37.68 seconds: 119.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 658 (time 1789639562.64 total 10.01 last 65.71)
 - open/close 1323 (time 1789639572.65 total 20.03 last 66.41)
 - open/close 1958 (time 1789639582.66 total 30.03 last 63.46)
 - open/close 2548 (time 1789639592.66 total 40.04 last 58.97)
 - open/close 3236 (time 1789639602.69 total 50.06 last 68.64)
 - open/close 3950 (time 1789639612.69 total 60.07 last 71.36)
total: 4500 open/close in 69.88 seconds: 64.39 ops/second
 - /unlink 1153 (time 1789639639.01 total 10.00 last 115.29)
 - /unlink 2265 (time 1789639649.02 total 20.00 last 111.16)
 - /unlink 3203 (time 1789639659.02 total 30.01 last 93.77)
 - /unlink 4341 (time 1789639669.02 total 40.01 last 113.79)
total: 4500 /unlink in 42.22 seconds: 106.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 560 (time 1789639688.81 total 10.02 last 55.87)
 - open/close 1195 (time 1789639698.84 total 20.05 last 63.31)
 - open/close 1747 (time 1789639708.84 total 30.05 last 55.20)
 - open/close 2374 (time 1789639718.86 total 40.07 last 62.59)
 - open/close 3012 (time 1789639728.87 total 50.08 last 63.76)
 - open/close 3697 (time 1789639738.88 total 60.09 last 68.41)
 - open/close 4328 (time 1789639748.88 total 70.09 last 63.10)
total: 4500 open/close in 72.59 seconds: 61.99 ops/second
 - /unlink 1295 (time 1789639765.64 total 10.00 last 129.45)
 - /unlink 2539 (time 1789639775.65 total 20.01 last 124.27)
 - /unlink 3743 (time 1789639785.65 total 30.02 last 120.36)
total: 4500 /unlink in 36.74 seconds: 122.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 660 (time 1789639807.60 total 10.02 last 65.89)
 - open/close 1310 (time 1789639817.62 total 20.03 last 64.90)
 - open/close 2028 (time 1789639827.63 total 30.04 last 71.71)
 - open/close 2651 (time 1789639837.63 total 40.05 last 62.30)
 - open/close 3346 (time 1789639847.63 total 50.05 last 69.50)
 - open/close 4017 (time 1789639857.64 total 60.05 last 67.06)
total: 4500 open/close in 67.34 seconds: 66.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1135 (time 1789639880.29 total 10.02 last 113.31)
 - /unlink 2303 (time 1789639890.30 total 20.03 last 116.70)
 - /unlink 3541 (time 1789639900.30 total 30.03 last 123.71)
total: 4500 /unlink in 38.04 seconds: 118.29 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 654 (time 1789639924.29 total 10.01 last 65.32)
 - open/close 1321 (time 1789639934.29 total 20.02 last 66.68)
 - open/close 1974 (time 1789639944.30 total 30.02 last 65.26)
 - open/close 2652 (time 1789639954.30 total 40.03 last 67.76)
 - open/close 3353 (time 1789639964.32 total 50.04 last 69.98)
 - open/close 4001 (time 1789639974.32 total 60.04 last 64.80)
total: 4500 open/close in 66.97 seconds: 67.19 ops/second
 - /unlink 1207 (time 1789639995.49 total 10.01 last 120.63)
 - /unlink 2479 (time 1789640005.50 total 20.02 last 127.07)
 - /unlink 3614 (time 1789640015.50 total 30.02 last 113.44)
total: 4500 /unlink in 36.40 seconds: 123.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 588 (time 1789640035.81 total 10.01 last 58.72)
 - open/close 1134 (time 1789640045.84 total 20.04 last 54.46)
 - open/close 1753 (time 1789640055.85 total 30.05 last 61.84)
 - open/close 2324 (time 1789640065.85 total 40.06 last 57.05)
 - open/close 2933 (time 1789640075.87 total 50.07 last 60.81)
 - open/close 3596 (time 1789640085.87 total 60.08 last 66.28)
 - open/close 4272 (time 1789640095.90 total 70.10 last 67.41)
total: 4500 open/close in 73.38 seconds: 61.33 ops/second
 - /unlink 1378 (time 1789640113.39 total 10.00 last 137.76)
 - /unlink 2573 (time 1789640123.40 total 20.01 last 119.45)
 - /unlink 3535 (time 1789640133.40 total 30.01 last 96.14)
total: 4500 /unlink in 38.90 seconds: 115.69 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 747 (time 1789640157.47 total 10.02 last 74.58)
 - open/close 1405 (time 1789640167.48 total 20.03 last 65.73)
 - open/close 2069 (time 1789640177.48 total 30.03 last 66.37)
 - open/close 2764 (time 1789640187.49 total 40.04 last 69.41)
 - open/close 3469 (time 1789640197.52 total 50.07 last 70.30)
 - open/close 4105 (time 1789640207.54 total 60.09 last 63.52)
total: 4500 open/close in 65.40 seconds: 68.80 ops/second
 - /unlink 1259 (time 1789640227.66 total 10.00 last 125.87)
 - /unlink 2424 (time 1789640237.67 total 20.01 last 116.43)
 - /unlink 3746 (time 1789640247.67 total 30.01 last 132.15)
total: 4500 /unlink in 35.67 seconds: 126.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 709 (time 1789640268.11 total 10.01 last 70.86)
 - open/close 1437 (time 1789640278.11 total 20.01 last 72.78)
 - open/close 2134 (time 1789640288.12 total 30.02 last 69.63)
 - open/close 2775 (time 1789640298.13 total 40.03 last 64.05)
 - open/close 3472 (time 1789640308.15 total 50.05 last 69.56)
 - open/close 4110 (time 1789640318.15 total 60.05 last 63.77)
total: 4500 open/close in 66.72 seconds: 67.45 ops/second
 - /unlink 1160 (time 1789640340.03 total 10.00 last 115.99)
 - /unlink 2388 (time 1789640350.03 total 20.01 last 122.74)
 - /unlink 3563 (time 1789640360.04 total 30.01 last 117.45)
total: 4500 /unlink in 37.31 seconds: 120.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 654 (time 1789640382.50 total 10.01 last 65.35)
 - open/close 1322 (time 1789640392.50 total 20.01 last 66.79)
 - open/close 2009 (time 1789640402.51 total 30.02 last 68.65)
 - open/close 2681 (time 1789640412.51 total 40.02 last 67.16)
 - open/close 3342 (time 1789640422.52 total 50.03 last 66.07)
 - open/close 4014 (time 1789640432.52 total 60.03 last 67.16)
total: 4500 open/close in 67.07 seconds: 67.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1324 (time 1789640453.54 total 10.00 last 132.36)
 - /unlink 2608 (time 1789640463.55 total 20.01 last 128.25)
 - /unlink 3881 (time 1789640473.57 total 30.03 last 127.05)
total: 4500 /unlink in 35.28 seconds: 127.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 662 (time 1789640493.41 total 10.00 last 66.19)
 - open/close 1280 (time 1789640503.43 total 20.02 last 61.71)
 - open/close 2044 (time 1789640513.43 total 30.02 last 76.37)
 - open/close 2715 (time 1789640523.43 total 40.02 last 67.09)
 - open/close 3363 (time 1789640533.44 total 50.03 last 64.72)
 - open/close 4037 (time 1789640543.45 total 60.03 last 67.39)
total: 4500 open/close in 66.76 seconds: 67.41 ops/second
 - /unlink 1356 (time 1789640564.76 total 10.00 last 135.59)
 - /unlink 2648 (time 1789640574.77 total 20.01 last 129.05)
 - /unlink 3870 (time 1789640584.77 total 30.01 last 122.20)
total: 4500 /unlink in 35.56 seconds: 126.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 643 (time 1789640606.22 total 10.01 last 64.23)
 - open/close 1372 (time 1789640616.23 total 20.02 last 72.81)
 - open/close 2044 (time 1789640626.24 total 30.03 last 67.17)
 - open/close 2754 (time 1789640636.26 total 40.05 last 70.86)
 - open/close 3446 (time 1789640646.26 total 50.05 last 69.20)
 - open/close 4076 (time 1789640656.26 total 60.05 last 62.97)
total: 4500 open/close in 66.02 seconds: 68.16 ops/second
 - /unlink 1327 (time 1789640677.15 total 10.01 last 132.61)
 - /unlink 2570 (time 1789640687.15 total 20.01 last 124.26)
 - /unlink 3772 (time 1789640697.16 total 30.02 last 120.10)
total: 4500 /unlink in 36.18 seconds: 124.37 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95024 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg339-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg339-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server
stop mds service on oleg339-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server
stop mds service on oleg339-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server
unloading modules via unload_modules_local on: 'oleg339-server'
oleg339-server: oleg339-server.virtnet: executing unload_modules_local
oleg339-server: kmemleak disabled
modules unloaded.