== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:37:30 (1781242650)
Stopping clients: oleg110-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg110-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg110-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg110-server
oleg110-server: oleg110-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg110-server'
oleg110-server: oleg110-server.virtnet: executing load_modules_local
oleg110-server: Loading modules from /home/green/git/lustre-release/lustre
oleg110-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg110-server: detected 4 online CPUs by sysfs
oleg110-server: MODOPTS_LIBCFS=
oleg110-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg110-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg110-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg110-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg110-client.virtnet:  -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 847 (time 1781242840.91 total 10.00 last 84.67)
 - open/close 1623 (time 1781242850.91 total 20.01 last 77.58)
 - open/close 2389 (time 1781242860.92 total 30.02 last 76.51)
 - open/close 3122 (time 1781242870.96 total 40.05 last 73.07)
 - open/close 3873 (time 1781242880.97 total 50.07 last 74.99)
total: 4500 open/close in 58.05 seconds: 77.52 ops/second
 - /unlink 1301 (time 1781242903.30 total 10.00 last 130.09)
 - /unlink 2647 (time 1781242913.31 total 20.01 last 134.41)
 - /unlink 4049 (time 1781242923.32 total 30.02 last 140.16)
total: 4500 /unlink in 33.52 seconds: 134.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 682 (time 1781242942.18 total 10.00 last 68.17)
 - open/close 1391 (time 1781242952.19 total 20.01 last 70.82)
 - open/close 2099 (time 1781242962.20 total 30.03 last 70.69)
 - open/close 2729 (time 1781242972.20 total 40.03 last 62.99)
 - open/close 3471 (time 1781242982.21 total 50.04 last 74.17)
 - open/close 4240 (time 1781242992.23 total 60.06 last 76.73)
total: 4500 open/close in 63.96 seconds: 70.36 ops/second
 - /unlink 1339 (time 1781243010.21 total 10.00 last 133.89)
 - /unlink 2784 (time 1781243020.22 total 20.01 last 144.35)
 - /unlink 4120 (time 1781243030.22 total 30.01 last 133.58)
total: 4500 /unlink in 32.63 seconds: 137.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 707 (time 1781243048.64 total 10.00 last 70.69)
 - open/close 1314 (time 1781243058.64 total 20.01 last 60.65)
 - open/close 2076 (time 1781243068.65 total 30.01 last 76.17)
 - open/close 2845 (time 1781243078.65 total 40.01 last 76.89)
 - open/close 3603 (time 1781243088.65 total 50.02 last 75.79)
 - open/close 4366 (time 1781243098.66 total 60.02 last 76.25)
total: 4500 open/close in 61.63 seconds: 73.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1255 (time 1781243115.11 total 10.00 last 125.50)
 - /unlink 2550 (time 1781243125.11 total 20.01 last 129.39)
 - /unlink 3855 (time 1781243135.12 total 30.02 last 130.39)
total: 4500 /unlink in 35.60 seconds: 126.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 662 (time 1781243156.14 total 10.01 last 66.11)
 - open/close 1318 (time 1781243166.15 total 20.02 last 65.55)
 - open/close 2056 (time 1781243176.15 total 30.02 last 73.80)
 - open/close 2767 (time 1781243186.16 total 40.03 last 71.05)
 - open/close 3461 (time 1781243196.17 total 50.04 last 69.35)
 - open/close 4175 (time 1781243206.17 total 60.04 last 71.36)
total: 4500 open/close in 63.77 seconds: 70.56 ops/second
 - /unlink 1339 (time 1781243224.84 total 10.00 last 133.88)
 - /unlink 2614 (time 1781243234.84 total 20.00 last 127.50)
 - /unlink 3909 (time 1781243244.85 total 30.01 last 129.44)
total: 4500 /unlink in 33.79 seconds: 133.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 757 (time 1781243263.34 total 10.01 last 75.63)
 - open/close 1571 (time 1781243273.34 total 20.01 last 81.36)
 - open/close 2430 (time 1781243283.36 total 30.03 last 85.79)
 - open/close 3250 (time 1781243293.36 total 40.03 last 81.94)
 - open/close 4067 (time 1781243303.38 total 50.05 last 81.54)
total: 4500 open/close in 55.60 seconds: 80.93 ops/second
 - /unlink 1498 (time 1781243322.89 total 10.00 last 149.76)
 - /unlink 2959 (time 1781243332.90 total 20.01 last 145.96)
 - /unlink 4499 (time 1781243342.90 total 30.02 last 153.94)
total: 4500 /unlink in 30.02 seconds: 149.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 798 (time 1781243356.85 total 10.00 last 79.78)
 - open/close 1595 (time 1781243366.85 total 20.00 last 79.69)
 - open/close 2438 (time 1781243376.85 total 30.01 last 84.27)
 - open/close 3194 (time 1781243386.88 total 40.03 last 75.41)
 - open/close 4031 (time 1781243396.88 total 50.04 last 83.66)
total: 4500 open/close in 55.94 seconds: 80.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1339 (time 1781243416.77 total 10.01 last 133.76)
 - /unlink 2720 (time 1781243426.78 total 20.02 last 138.03)
 - /unlink 4069 (time 1781243436.78 total 30.02 last 134.88)
total: 4500 /unlink in 33.01 seconds: 136.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 828 (time 1781243454.08 total 10.00 last 82.76)
 - open/close 1591 (time 1781243464.08 total 20.01 last 76.27)
 - open/close 2410 (time 1781243474.09 total 30.01 last 81.85)
 - open/close 3198 (time 1781243484.09 total 40.02 last 78.74)
 - open/close 4113 (time 1781243494.10 total 50.03 last 91.42)
total: 4500 open/close in 55.67 seconds: 80.83 ops/second
 - /unlink 1388 (time 1781243513.89 total 10.01 last 138.65)
 - /unlink 2723 (time 1781243523.89 total 20.02 last 133.42)
 - /unlink 4085 (time 1781243533.90 total 30.02 last 136.15)
total: 4500 /unlink in 33.00 seconds: 136.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 801 (time 1781243551.16 total 10.01 last 80.04)
 - open/close 1697 (time 1781243561.17 total 20.01 last 89.54)
 - open/close 2556 (time 1781243571.18 total 30.03 last 85.78)
 - open/close 3388 (time 1781243581.18 total 40.03 last 83.18)
 - open/close 4167 (time 1781243591.20 total 50.04 last 77.79)
total: 4500 open/close in 54.34 seconds: 82.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1450 (time 1781243609.26 total 10.01 last 144.92)
 - /unlink 2836 (time 1781243619.26 total 20.01 last 138.55)
 - /unlink 4347 (time 1781243629.27 total 30.02 last 150.99)
total: 4500 /unlink in 31.12 seconds: 144.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 798 (time 1781243644.65 total 10.01 last 79.73)
 - open/close 1661 (time 1781243654.66 total 20.02 last 86.23)
 - open/close 2557 (time 1781243664.67 total 30.02 last 89.53)
 - open/close 3433 (time 1781243674.68 total 40.03 last 87.52)
 - open/close 4318 (time 1781243684.68 total 50.04 last 88.46)
total: 4500 open/close in 52.36 seconds: 85.94 ops/second
 - /unlink 1522 (time 1781243700.53 total 10.00 last 152.18)
 - /unlink 3004 (time 1781243710.53 total 20.01 last 148.12)
 - /unlink 4058 (time 1781243720.53 total 30.01 last 105.40)
total: 4500 /unlink in 33.90 seconds: 132.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 424 (time 1781243743.49 total 10.02 last 42.32)
 - open/close 813 (time 1781243753.49 total 20.02 last 38.88)
 - open/close 1393 (time 1781243763.50 total 30.03 last 57.96)
 - open/close 2051 (time 1781243773.50 total 40.03 last 65.80)
 - open/close 2646 (time 1781243783.51 total 50.04 last 59.43)
 - open/close 3260 (time 1781243793.51 total 60.04 last 61.38)
 - open/close 3923 (time 1781243803.53 total 70.06 last 66.21)
total: 4500 open/close in 77.29 seconds: 58.23 ops/second
 - /unlink 1219 (time 1781243826.29 total 10.00 last 121.90)
 - /unlink 2671 (time 1781243836.31 total 20.02 last 144.95)
 - /unlink 4033 (time 1781243846.31 total 30.02 last 136.15)
total: 4500 /unlink in 33.65 seconds: 133.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 634 (time 1781243865.45 total 10.00 last 63.38)
 - open/close 1392 (time 1781243875.45 total 20.01 last 75.78)
 - open/close 2125 (time 1781243885.47 total 30.02 last 73.20)
 - open/close 2835 (time 1781243895.47 total 40.02 last 70.97)
 - open/close 3549 (time 1781243905.48 total 50.03 last 71.36)
 - open/close 4323 (time 1781243915.48 total 60.04 last 77.37)
total: 4500 open/close in 62.42 seconds: 72.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1193 (time 1781243933.35 total 10.00 last 119.25)
 - /unlink 2347 (time 1781243943.36 total 20.02 last 115.27)
 - /unlink 3486 (time 1781243953.37 total 30.02 last 113.86)
total: 4500 /unlink in 38.21 seconds: 117.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 736 (time 1781243977.43 total 10.01 last 73.55)
 - open/close 1493 (time 1781243987.43 total 20.01 last 75.67)
 - open/close 2166 (time 1781243997.44 total 30.02 last 67.25)
 - open/close 2789 (time 1781244007.44 total 40.02 last 62.29)
 - open/close 3369 (time 1781244017.46 total 50.04 last 57.89)
 - open/close 4079 (time 1781244027.47 total 60.05 last 70.93)
total: 4500 open/close in 65.76 seconds: 68.43 ops/second
 - /unlink 1072 (time 1781244049.82 total 10.00 last 107.17)
 - /unlink 2208 (time 1781244059.82 total 20.00 last 113.59)
 - /unlink 3459 (time 1781244069.83 total 30.01 last 125.08)
total: 4500 /unlink in 39.12 seconds: 115.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 685 (time 1781244094.65 total 10.01 last 68.44)
 - open/close 1384 (time 1781244104.66 total 20.02 last 69.82)
 - open/close 2075 (time 1781244114.67 total 30.02 last 69.07)
 - open/close 2846 (time 1781244124.67 total 40.03 last 77.06)
 - open/close 3647 (time 1781244134.68 total 50.04 last 80.05)
 - open/close 4368 (time 1781244144.69 total 60.04 last 72.05)
total: 4500 open/close in 61.94 seconds: 72.65 ops/second
 - /unlink 1317 (time 1781244161.54 total 10.00 last 131.68)
 - /unlink 2667 (time 1781244171.55 total 20.00 last 134.95)
 - /unlink 3781 (time 1781244181.55 total 30.00 last 111.40)
total: 4500 /unlink in 37.54 seconds: 119.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 762 (time 1781244204.92 total 10.00 last 76.18)
 - open/close 1482 (time 1781244214.93 total 20.01 last 71.95)
 - open/close 2252 (time 1781244224.94 total 30.02 last 76.90)
 - open/close 2880 (time 1781244234.95 total 40.03 last 62.74)
 - open/close 3550 (time 1781244244.97 total 50.05 last 66.89)
 - open/close 4276 (time 1781244254.97 total 60.05 last 72.56)
total: 4500 open/close in 63.60 seconds: 70.76 ops/second
 - /unlink 1195 (time 1781244274.12 total 10.00 last 119.45)
 - /unlink 2547 (time 1781244284.13 total 20.01 last 135.06)
 - /unlink 3938 (time 1781244294.14 total 30.02 last 139.03)
total: 4500 /unlink in 34.84 seconds: 129.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 818 (time 1781244316.13 total 10.00 last 81.76)
 - open/close 1538 (time 1781244326.14 total 20.01 last 71.94)
 - open/close 2345 (time 1781244336.14 total 30.01 last 80.69)
 - open/close 3144 (time 1781244346.15 total 40.02 last 79.86)
 - open/close 3964 (time 1781244356.16 total 50.03 last 81.89)
total: 4500 open/close in 57.48 seconds: 78.28 ops/second
 - /unlink 1007 (time 1781244378.25 total 10.01 last 100.65)
 - /unlink 2127 (time 1781244388.25 total 20.01 last 111.97)
 - /unlink 3581 (time 1781244398.26 total 30.01 last 145.31)
total: 4500 /unlink in 38.29 seconds: 117.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 639 (time 1781244422.37 total 10.01 last 63.82)
 - open/close 1287 (time 1781244432.37 total 20.01 last 64.79)
 - open/close 1914 (time 1781244442.37 total 30.02 last 62.69)
 - open/close 2536 (time 1781244452.38 total 40.03 last 62.13)
 - open/close 3075 (time 1781244462.40 total 50.05 last 53.80)
 - open/close 3748 (time 1781244472.41 total 60.05 last 67.25)
 - open/close 4444 (time 1781244482.42 total 70.07 last 69.50)
total: 4500 open/close in 71.14 seconds: 63.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1346 (time 1781244499.42 total 10.00 last 134.57)
 - /unlink 2791 (time 1781244509.44 total 20.02 last 144.21)
 - /unlink 3877 (time 1781244519.44 total 30.02 last 108.59)
total: 4500 /unlink in 35.55 seconds: 126.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 694 (time 1781244539.86 total 10.01 last 69.35)
 - open/close 1467 (time 1781244549.88 total 20.03 last 77.10)
 - open/close 2141 (time 1781244559.89 total 30.04 last 67.36)
 - open/close 2796 (time 1781244569.90 total 40.05 last 65.42)
 - open/close 3398 (time 1781244579.91 total 50.06 last 60.16)
 - open/close 4039 (time 1781244589.91 total 60.07 last 64.06)
total: 4500 open/close in 67.13 seconds: 67.03 ops/second
 - /unlink 1361 (time 1781244612.37 total 10.00 last 136.09)
 - /unlink 2631 (time 1781244622.38 total 20.00 last 126.99)
 - /unlink 3821 (time 1781244632.38 total 30.01 last 118.94)
total: 4500 /unlink in 35.82 seconds: 125.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 697 (time 1781244655.02 total 10.01 last 69.63)
 - open/close 1366 (time 1781244665.03 total 20.02 last 66.83)
 - open/close 2056 (time 1781244675.03 total 30.02 last 68.99)
 - open/close 2664 (time 1781244685.03 total 40.02 last 60.79)
 - open/close 3182 (time 1781244695.05 total 50.03 last 51.74)
 - open/close 3786 (time 1781244705.05 total 60.04 last 60.36)
 - open/close 4393 (time 1781244715.09 total 70.08 last 60.48)
total: 4500 open/close in 71.64 seconds: 62.81 ops/second
 - /unlink 1265 (time 1781244732.43 total 10.00 last 126.45)
 - /unlink 2565 (time 1781244742.43 total 20.00 last 130.00)
 - /unlink 3999 (time 1781244752.44 total 30.01 last 143.34)
total: 4500 /unlink in 34.00 seconds: 132.37 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11195: 94047 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg110-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server
stop mds service on oleg110-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server
stop mds service on oleg110-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server

unloading modules via unload_modules_local on: 'oleg110-server'
oleg110-server: oleg110-server.virtnet: executing unload_modules_local
oleg110-server: 
modules unloaded.