== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:47:02 (1788443222)
Stopping clients: oleg341-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg341-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg341-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg341-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg341-server
oleg341-server: oleg341-server.virtnet: executing set_hostid
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg341-server'
oleg341-server: oleg341-server.virtnet: executing load_modules_local
oleg341-server: kmemleak disabled
oleg341-server: Loading modules from /home/green/git/lustre-release/lustre
oleg341-server: detected 4 online CPUs by sysfs
oleg341-server: MODOPTS_LIBCFS=
oleg341-server: Force libcfs to create 2 CPU partitions
oleg341-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg341-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all
pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg341-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all
pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg341-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all
pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg341-client.virtnet:  -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 775 (time 1788443410.00 total 10.02 last 77.37)
 - open/close 1552 (time 1788443420.00 total 20.02 last 77.70)
 - open/close 2394 (time 1788443430.02 total 30.04 last 84.04)
 - open/close 3177 (time 1788443440.03 total 40.05 last 78.22)
 - open/close 4028 (time 1788443450.03 total 50.05 last 85.07)
total: 4500 open/close in 55.55 seconds: 81.01 ops/second
 - /unlink 1550 (time 1788443469.02 total 10.01 last 154.79)
 - /unlink 3189 (time 1788443479.03 total 20.02 last 163.80)
total: 4500 /unlink in 28.15 seconds: 159.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 894 (time 1788443501.99 total 10.00 last 89.40)
 - open/close 1730 (time 1788443512.01 total 20.02 last 83.44)
 - open/close 2549 (time 1788443522.03 total 30.04 last 81.76)
 - open/close 3397 (time 1788443532.03 total 40.04 last 84.75)
 - open/close 4066 (time 1788443542.04 total 50.05 last 66.86)
total: 4500 open/close in 57.76 seconds: 77.91 ops/second
 - /unlink 1356 (time 1788443565.26 total 10.00 last 135.59)
 - /unlink 2797 (time 1788443575.26 total 20.01 last 143.98)
 - /unlink 3992 (time 1788443585.27 total 30.01 last 119.47)
total: 4500 /unlink in 35.93 seconds: 125.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 850 (time 1788443606.98 total 10.01 last 84.93)
 - open/close 1664 (time 1788443616.99 total 20.02 last 81.32)
 - open/close 2503 (time 1788443627.00 total 30.02 last 83.85)
 - open/close 3301 (time 1788443637.01 total 40.04 last 79.68)
 - open/close 4076 (time 1788443647.02 total 50.05 last 77.45)
total: 4500 open/close in 55.97 seconds: 80.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1435 (time 1788443667.86 total 10.01 last 143.32)
 - /unlink 2573 (time 1788443677.87 total 20.03 last 113.63)
 - /unlink 3844 (time 1788443687.87 total 30.03 last 127.09)
total: 4500 /unlink in 34.55 seconds: 130.26 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 694 (time 1788443708.36 total 10.00 last 69.39)
 - open/close 1284 (time 1788443718.38 total 20.02 last 58.91)
 - open/close 1933 (time 1788443728.39 total 30.03 last 64.82)
 - open/close 2686 (time 1788443738.40 total 40.03 last 75.25)
 - open/close 3422 (time 1788443748.40 total 50.04 last 73.55)
 - open/close 4221 (time 1788443758.41 total 60.04 last 79.89)
total: 4500 open/close in 64.26 seconds: 70.03 ops/second
 - /unlink 1221 (time 1788443779.82 total 10.00 last 122.08)
 - /unlink 2811 (time 1788443789.82 total 20.00 last 158.98)
 - /unlink 4349 (time 1788443799.82 total 30.01 last 153.75)
total: 4500 /unlink in 31.20 seconds: 144.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 860 (time 1788443815.10 total 10.00 last 85.96)
 - open/close 1646 (time 1788443825.11 total 20.02 last 78.50)
 - open/close 2372 (time 1788443835.12 total 30.02 last 72.57)
 - open/close 3150 (time 1788443845.12 total 40.03 last 77.78)
 - open/close 3955 (time 1788443855.13 total 50.03 last 80.43)
total: 4500 open/close in 56.77 seconds: 79.26 ops/second
 - /unlink 1491 (time 1788443876.01 total 10.00 last 149.10)
 - /unlink 2949 (time 1788443886.02 total 20.01 last 145.69)
total: 4500 /unlink in 29.91 seconds: 150.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 845 (time 1788443910.68 total 10.00 last 84.49)
 - open/close 1583 (time 1788443920.68 total 20.00 last 73.79)
 - open/close 2382 (time 1788443930.68 total 30.01 last 79.88)
 - open/close 3120 (time 1788443940.69 total 40.01 last 73.74)
 - open/close 3870 (time 1788443950.70 total 50.02 last 74.91)
total: 4500 open/close in 59.50 seconds: 75.63 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1364 (time 1788443976.21 total 10.00 last 136.38)
 - /unlink 2760 (time 1788443986.21 total 20.00 last 139.58)
 - /unlink 4232 (time 1788443996.21 total 30.00 last 147.18)
total: 4500 /unlink in 31.90 seconds: 141.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 718 (time 1788444012.94 total 10.01 last 71.75)
 - open/close 1301 (time 1788444022.94 total 20.01 last 58.26)
 - open/close 1786 (time 1788444032.95 total 30.02 last 48.46)
 - open/close 2287 (time 1788444042.96 total 40.03 last 50.04)
 - open/close 3034 (time 1788444052.97 total 50.04 last 74.67)
 - open/close 3810 (time 1788444062.98 total 60.05 last 77.51)
total: 4500 open/close in 68.22 seconds: 65.97 ops/second
 - /unlink 1414 (time 1788444085.76 total 10.00 last 141.35)
 - /unlink 2819 (time 1788444095.76 total 20.01 last 140.47)
 - /unlink 3964 (time 1788444105.78 total 30.02 last 114.31)
total: 4500 /unlink in 33.90 seconds: 132.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 605 (time 1788444124.78 total 10.00 last 60.49)
 - open/close 1163 (time 1788444134.78 total 20.00 last 55.80)
 - open/close 1895 (time 1788444144.79 total 30.01 last 73.14)
 - open/close 2568 (time 1788444154.80 total 40.02 last 67.22)
 - open/close 3200 (time 1788444164.82 total 50.04 last 63.08)
 - open/close 3896 (time 1788444174.83 total 60.05 last 69.51)
total: 4500 open/close in 68.62 seconds: 65.58 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1318 (time 1788444199.03 total 10.00 last 131.76)
 - /unlink 2697 (time 1788444209.04 total 20.01 last 137.86)
 - /unlink 3981 (time 1788444219.04 total 30.01 last 128.40)
total: 4500 /unlink in 33.61 seconds: 133.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 832 (time 1788444237.62 total 10.01 last 83.13)
 - open/close 1578 (time 1788444247.64 total 20.02 last 74.48)
 - open/close 2445 (time 1788444257.64 total 30.03 last 86.67)
 - open/close 3215 (time 1788444267.65 total 40.03 last 76.96)
 - open/close 4055 (time 1788444277.65 total 50.04 last 83.99)
total: 4500 open/close in 56.05 seconds: 80.28 ops/second
 - /unlink 1342 (time 1788444298.36 total 10.01 last 134.12)
 - /unlink 2543 (time 1788444308.36 total 20.01 last 120.00)
 - /unlink 3824 (time 1788444318.36 total 30.01 last 128.09)
total: 4500 /unlink in 34.71 seconds: 129.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 814 (time 1788444336.49 total 10.00 last 81.39)
 - open/close 1534 (time 1788444346.51 total 20.02 last 71.85)
 - open/close 2341 (time 1788444356.53 total 30.04 last 80.54)
 - open/close 3171 (time 1788444366.53 total 40.05 last 82.94)
 - open/close 3896 (time 1788444376.54 total 50.06 last 72.44)
total: 4500 open/close in 57.90 seconds: 77.72 ops/second
 - /unlink 1445 (time 1788444399.64 total 10.00 last 144.45)
 - /unlink 2477 (time 1788444409.65 total 20.01 last 103.17)
 - /unlink 3864 (time 1788444419.65 total 30.01 last 138.62)
total: 4500 /unlink in 34.03 seconds: 132.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 817 (time 1788444438.86 total 10.01 last 81.65)
 - open/close 1600 (time 1788444448.86 total 20.01 last 78.27)
 - open/close 2369 (time 1788444458.88 total 30.02 last 76.81)
 - open/close 3095 (time 1788444468.89 total 40.03 last 72.51)
 - open/close 3785 (time 1788444478.92 total 50.06 last 68.81)
total: 4500 open/close in 58.62 seconds: 76.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1333 (time 1788444502.20 total 10.00 last 133.30)
 - /unlink 2795 (time 1788444512.21 total 20.00 last 146.19)
 - /unlink 4269 (time 1788444522.21 total 30.00 last 147.36)
total: 4500 /unlink in 31.28 seconds: 143.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 815 (time 1788444537.70 total 10.01 last 81.41)
 - open/close 1595 (time 1788444547.70 total 20.01 last 77.99)
 - open/close 2437 (time 1788444557.71 total 30.02 last 84.12)
 - open/close 3173 (time 1788444567.72 total 40.03 last 73.57)
 - open/close 3843 (time 1788444577.73 total 50.04 last 66.91)
 - open/close 4423 (time 1788444587.73 total 60.04 last 57.98)
total: 4500 open/close in 61.72 seconds: 72.91 ops/second
 - /unlink 1458 (time 1788444604.30 total 10.00 last 145.77)
 - /unlink 2833 (time 1788444614.32 total 20.02 last 137.30)
 - /unlink 4288 (time 1788444624.32 total 30.02 last 145.45)
total: 4500 /unlink in 31.80 seconds: 141.50 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 766 (time 1788444640.87 total 10.01 last 76.54)
 - open/close 1513 (time 1788444650.88 total 20.02 last 74.58)
 - open/close 2179 (time 1788444660.88 total 30.02 last 66.59)
 - open/close 2972 (time 1788444670.89 total 40.03 last 79.27)
 - open/close 3787 (time 1788444680.89 total 50.03 last 81.46)
total: 4500 open/close in 58.37 seconds: 77.09 ops/second
 - /unlink 1554 (time 1788444703.31 total 10.00 last 155.36)
 - /unlink 3081 (time 1788444713.32 total 20.01 last 152.53)
total: 4500 /unlink in 29.37 seconds: 153.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 778 (time 1788444737.60 total 10.02 last 77.67)
 - open/close 1430 (time 1788444747.63 total 20.04 last 65.02)
 - open/close 2099 (time 1788444757.64 total 30.05 last 66.86)
 - open/close 2753 (time 1788444767.64 total 40.05 last 65.39)
 - open/close 3465 (time 1788444777.64 total 50.06 last 71.17)
 - open/close 4278 (time 1788444787.67 total 60.08 last 81.09)
total: 4500 open/close in 62.76 seconds: 71.70 ops/second
 - /unlink 1397 (time 1788444804.65 total 10.00 last 139.70)
 - /unlink 2883 (time 1788444814.66 total 20.00 last 148.56)
 - /unlink 4385 (time 1788444824.66 total 30.01 last 150.12)
total: 4500 /unlink in 31.11 seconds: 144.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 823 (time 1788444840.44 total 10.01 last 82.24)
 - open/close 1682 (time 1788444850.45 total 20.01 last 85.86)
 - open/close 2500 (time 1788444860.45 total 30.02 last 81.78)
 - open/close 3378 (time 1788444870.46 total 40.03 last 87.70)
 - open/close 4126 (time 1788444880.47 total 50.03 last 74.75)
total: 4500 open/close in 55.65 seconds: 80.87 ops/second
 - /unlink 1580 (time 1788444900.87 total 10.00 last 157.96)
 - /unlink 3078 (time 1788444910.88 total 20.00 last 149.77)
 - /unlink 4352 (time 1788444920.88 total 30.01 last 127.39)
total: 4500 /unlink in 31.02 seconds: 145.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 688 (time 1788444937.62 total 10.00 last 68.78)
 - open/close 1367 (time 1788444947.64 total 20.02 last 67.78)
 - open/close 2165 (time 1788444957.64 total 30.02 last 79.78)
 - open/close 2871 (time 1788444967.65 total 40.03 last 70.55)
 - open/close 3603 (time 1788444977.65 total 50.04 last 73.16)
 - open/close 4371 (time 1788444987.67 total 60.05 last 76.67)
total: 4500 open/close in 61.80 seconds: 72.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1627 (time 1788445003.32 total 10.00 last 162.70)
 - /unlink 3271 (time 1788445013.32 total 20.00 last 164.35)
total: 4500 /unlink in 27.95 seconds: 160.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 728 (time 1788445035.73 total 10.02 last 72.65)
 - open/close 1232 (time 1788445045.78 total 20.07 last 50.13)
 - open/close 1890 (time 1788445055.79 total 30.08 last 65.75)
 - open/close 2582 (time 1788445065.81 total 40.10 last 69.04)
 - open/close 3353 (time 1788445075.81 total 50.11 last 77.08)
 - open/close 4223 (time 1788445085.82 total 60.12 last 86.93)
total: 4500 open/close in 63.65 seconds: 70.70 ops/second
 - /unlink 1580 (time 1788445103.44 total 10.00 last 157.97)
 - /unlink 2859 (time 1788445113.44 total 20.01 last 127.84)
 - /unlink 4176 (time 1788445123.45 total 30.01 last 131.60)
total: 4500 /unlink in 32.46 seconds: 138.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 795 (time 1788445140.73 total 10.01 last 79.40)
 - open/close 1563 (time 1788445150.73 total 20.02 last 76.74)
 - open/close 2224 (time 1788445160.74 total 30.03 last 66.03)
 - open/close 2863 (time 1788445170.75 total 40.03 last 63.87)
 - open/close 3528 (time 1788445180.78 total 50.07 last 66.30)
 - open/close 4152 (time 1788445190.78 total 60.07 last 62.36)
total: 4500 open/close in 65.12 seconds: 69.10 ops/second
 - /unlink 1392 (time 1788445210.16 total 10.00 last 139.16)
 - /unlink 2793 (time 1788445220.17 total 20.01 last 140.05)
 - /unlink 4322 (time 1788445230.17 total 30.01 last 152.86)
total: 4500 /unlink in 31.46 seconds: 143.05 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95170 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg341-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg341-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server
stop mds service on oleg341-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server
stop mds service on oleg341-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg341-server
unloading modules via unload_modules_local on: 'oleg341-server'
oleg341-server: oleg341-server.virtnet: executing unload_modules_local
oleg341-server: kmemleak disabled
modules unloaded.