== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:03:39 (1787583819)
Stopping clients: oleg107-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg107-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg107-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg107-server
oleg107-server: oleg107-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg107-server'
oleg107-server: oleg107-server.virtnet: executing load_modules_local
oleg107-server: Loading modules from /home/green/git/lustre-release/lustre
oleg107-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg107-server: detected 4 online CPUs by sysfs
oleg107-server: MODOPTS_LIBCFS=
oleg107-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg107-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all
pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg107-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all
pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg107-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all
pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg107-client.virtnet:  -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 882 (time 1787584025.96 total 10.01 last 88.15)
 - open/close 1584 (time 1787584035.97 total 20.01 last 70.14)
 - open/close 2513 (time 1787584045.97 total 30.02 last 92.89)
 - open/close 3403 (time 1787584055.97 total 40.02 last 88.99)
 - open/close 4132 (time 1787584065.98 total 50.03 last 72.82)
total: 4500 open/close in 53.63 seconds: 83.90 ops/second
 - /unlink 1514 (time 1787584083.91 total 10.00 last 151.36)
 - /unlink 2982 (time 1787584093.91 total 20.00 last 146.79)
total: 4500 /unlink in 29.40 seconds: 153.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 836 (time 1787584118.82 total 10.01 last 83.52)
 - open/close 1615 (time 1787584128.83 total 20.02 last 77.82)
 - open/close 2556 (time 1787584138.84 total 30.03 last 93.99)
 - open/close 3554 (time 1787584148.84 total 40.03 last 99.79)
 - open/close 4436 (time 1787584158.84 total 50.03 last 88.19)
total: 4500 open/close in 50.78 seconds: 88.62 ops/second
 - /unlink 1272 (time 1787584175.26 total 10.01 last 127.11)
 - /unlink 2567 (time 1787584185.26 total 20.01 last 129.50)
 - /unlink 4259 (time 1787584195.26 total 30.01 last 169.16)
total: 4500 /unlink in 31.57 seconds: 142.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 889 (time 1787584212.35 total 10.00 last 88.87)
 - open/close 1698 (time 1787584222.36 total 20.00 last 80.89)
 - open/close 2581 (time 1787584232.36 total 30.01 last 88.24)
 - open/close 3457 (time 1787584242.36 total 40.01 last 87.59)
 - open/close 4268 (time 1787584252.37 total 50.02 last 81.05)
total: 4500 open/close in 53.35 seconds: 84.35 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1341 (time 1787584270.58 total 10.01 last 134.00)
 - /unlink 2231 (time 1787584280.58 total 20.01 last 88.95)
 - /unlink 3489 (time 1787584290.60 total 30.03 last 125.60)
 - /unlink 4196 (time 1787584300.61 total 40.04 last 70.62)
total: 4500 /unlink in 44.03 seconds: 102.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 633 (time 1787584320.92 total 10.01 last 63.26)
 - open/close 1130 (time 1787584330.93 total 20.01 last 49.67)
 - open/close 1709 (time 1787584340.94 total 30.03 last 57.82)
 - open/close 2269 (time 1787584350.96 total 40.04 last 55.90)
 - open/close 3096 (time 1787584360.96 total 50.05 last 82.67)
 - open/close 3946 (time 1787584370.97 total 60.05 last 84.96)
total: 4500 open/close in 67.12 seconds: 67.05 ops/second
 - /unlink 1745 (time 1787584392.12 total 10.00 last 174.47)
 - /unlink 3257 (time 1787584402.12 total 20.00 last 151.17)
total: 4500 /unlink in 29.22 seconds: 153.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 1034 (time 1787584425.73 total 10.00 last 103.35)
 - open/close 1941 (time 1787584435.76 total 20.03 last 90.49)
 - open/close 2787 (time 1787584445.77 total 30.05 last 84.45)
 - open/close 3635 (time 1787584455.78 total 40.05 last 84.79)
 - open/close 4488 (time 1787584465.80 total 50.07 last 85.13)
total: 4500 open/close in 50.21 seconds: 89.63 ops/second
 - /unlink 1636 (time 1787584480.42 total 10.01 last 163.51)
 - /unlink 3273 (time 1787584490.42 total 20.01 last 163.67)
total: 4500 /unlink in 27.76 seconds: 162.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 892 (time 1787584512.77 total 10.01 last 89.12)
 - open/close 1646 (time 1787584522.78 total 20.01 last 75.36)
 - open/close 2417 (time 1787584532.78 total 30.02 last 77.04)
 - open/close 3260 (time 1787584542.78 total 40.02 last 84.30)
 - open/close 3830 (time 1787584552.79 total 50.03 last 56.98)
total: 4500 open/close in 58.21 seconds: 77.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1694 (time 1787584574.83 total 10.00 last 169.36)
 - /unlink 3411 (time 1787584584.83 total 20.00 last 171.68)
total: 4500 /unlink in 26.64 seconds: 168.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 855 (time 1787584605.81 total 10.00 last 85.47)
 - open/close 1744 (time 1787584615.81 total 20.00 last 88.89)
 - open/close 2622 (time 1787584625.82 total 30.01 last 87.74)
 - open/close 3498 (time 1787584635.82 total 40.02 last 87.56)
 - open/close 4415 (time 1787584645.83 total 50.02 last 91.70)
total: 4500 open/close in 50.92 seconds: 88.37 ops/second
 - /unlink 1617 (time 1787584661.09 total 10.00 last 161.62)
 - /unlink 3138 (time 1787584671.10 total 20.01 last 151.96)
total: 4500 /unlink in 29.02 seconds: 155.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 910 (time 1787584694.60 total 10.00 last 90.99)
 - open/close 1692 (time 1787584704.60 total 20.01 last 78.17)
 - open/close 2441 (time 1787584714.61 total 30.01 last 74.85)
 - open/close 3275 (time 1787584724.62 total 40.02 last 83.37)
 - open/close 4132 (time 1787584734.62 total 50.02 last 85.64)
total: 4500 open/close in 54.00 seconds: 83.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1434 (time 1787584752.92 total 10.01 last 143.29)
 - /unlink 2551 (time 1787584762.92 total 20.01 last 111.65)
 - /unlink 3927 (time 1787584772.93 total 30.01 last 137.57)
total: 4500 /unlink in 34.54 seconds: 130.27 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 819 (time 1787584792.85 total 10.00 last 81.90)
 - open/close 1627 (time 1787584802.88 total 20.03 last 80.59)
 - open/close 2406 (time 1787584812.88 total 30.03 last 77.89)
 - open/close 3110 (time 1787584822.89 total 40.04 last 70.34)
 - open/close 3874 (time 1787584832.89 total 50.04 last 76.39)
total: 4500 open/close in 58.19 seconds: 77.34 ops/second
 - /unlink 1247 (time 1787584855.86 total 10.00 last 124.67)
 - /unlink 2573 (time 1787584865.86 total 20.00 last 132.60)
 - /unlink 3880 (time 1787584875.87 total 30.01 last 130.65)
total: 4500 /unlink in 33.99 seconds: 132.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 820 (time 1787584896.03 total 10.00 last 81.96)
 - open/close 1589 (time 1787584906.03 total 20.00 last 76.90)
 - open/close 2514 (time 1787584916.04 total 30.01 last 92.47)
 - open/close 3349 (time 1787584926.04 total 40.01 last 83.48)
 - open/close 4224 (time 1787584936.05 total 50.02 last 87.42)
total: 4500 open/close in 53.21 seconds: 84.57 ops/second
 - /unlink 1285 (time 1787584954.43 total 10.02 last 128.29)
 - /unlink 2794 (time 1787584964.43 total 20.02 last 150.79)
 - /unlink 4220 (time 1787584974.44 total 30.03 last 142.56)
total: 4500 /unlink in 31.99 seconds: 140.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 884 (time 1787584990.66 total 10.01 last 88.27)
 - open/close 1731 (time 1787585000.67 total 20.02 last 84.61)
 - open/close 2647 (time 1787585010.67 total 30.03 last 91.56)
 - open/close 3522 (time 1787585020.68 total 40.03 last 87.45)
 - open/close 4402 (time 1787585030.68 total 50.04 last 87.97)
total: 4500 open/close in 51.29 seconds: 87.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1597 (time 1787585047.41 total 10.03 last 159.25)
 - /unlink 3279 (time 1787585057.42 total 20.03 last 168.16)
total: 4500 /unlink in 26.27 seconds: 171.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 888 (time 1787585078.19 total 10.00 last 88.80)
 - open/close 1741 (time 1787585088.20 total 20.01 last 85.21)
 - open/close 2656 (time 1787585098.20 total 30.02 last 91.44)
 - open/close 3469 (time 1787585108.21 total 40.02 last 81.24)
 - open/close 4260 (time 1787585118.22 total 50.03 last 79.03)
total: 4500 open/close in 53.74 seconds: 83.74 ops/second
 - /unlink 1117 (time 1787585137.84 total 10.01 last 111.58)
 - /unlink 2671 (time 1787585147.85 total 20.02 last 155.28)
 - /unlink 4221 (time 1787585157.85 total 30.02 last 154.97)
total: 4500 /unlink in 32.17 seconds: 139.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 990 (time 1787585175.58 total 10.00 last 98.99)
 - open/close 1874 (time 1787585185.58 total 20.01 last 88.36)
 - open/close 2492 (time 1787585195.59 total 30.01 last 61.75)
 - open/close 3128 (time 1787585205.59 total 40.02 last 63.58)
 - open/close 3718 (time 1787585215.60 total 50.02 last 58.98)
total: 4500 open/close in 59.20 seconds: 76.01 ops/second
 - /unlink 1490 (time 1787585240.22 total 10.00 last 148.93)
 - /unlink 2998 (time 1787585250.22 total 20.01 last 150.79)
 - /unlink 4466 (time 1787585260.23 total 30.01 last 146.76)
total: 4500 /unlink in 30.24 seconds: 148.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 718 (time 1787585275.01 total 10.00 last 71.80)
 - open/close 1526 (time 1787585285.01 total 20.00 last 80.79)
 - open/close 2115 (time 1787585295.02 total 30.02 last 58.79)
 - open/close 2700 (time 1787585305.06 total 40.05 last 58.30)
 - open/close 3348 (time 1787585315.07 total 50.06 last 64.74)
 - open/close 4176 (time 1787585325.09 total 60.09 last 82.60)
total: 4500 open/close in 65.05 seconds: 69.18 ops/second
 - /unlink 1495 (time 1787585345.22 total 10.00 last 149.46)
 - /unlink 3062 (time 1787585355.23 total 20.01 last 156.65)
total: 4500 /unlink in 29.32 seconds: 153.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 869 (time 1787585378.75 total 10.01 last 86.80)
 - open/close 1838 (time 1787585388.75 total 20.01 last 96.89)
 - open/close 2760 (time 1787585398.76 total 30.02 last 92.12)
 - open/close 3666 (time 1787585408.77 total 40.03 last 90.49)
total: 4500 open/close in 49.03 seconds: 91.79 ops/second
 - /unlink 1588 (time 1787585432.17 total 10.00 last 158.74)
 - /unlink 2937 (time 1787585442.18 total 20.01 last 134.87)
 - /unlink 4426 (time 1787585452.18 total 30.01 last 148.88)
total: 4500 /unlink in 30.60 seconds: 147.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 948 (time 1787585467.19 total 10.01 last 94.75)
 - open/close 1651 (time 1787585477.20 total 20.01 last 70.29)
 - open/close 2453 (time 1787585487.21 total 30.02 last 80.09)
 - open/close 3207 (time 1787585497.21 total 40.02 last 75.37)
 - open/close 3742 (time 1787585507.23 total 50.05 last 53.39)
 - open/close 4487 (time 1787585517.24 total 60.05 last 74.44)
total: 4500 open/close in 60.23 seconds: 74.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1471 (time 1787585532.16 total 10.04 last 146.49)
 - /unlink 2616 (time 1787585542.16 total 20.04 last 114.46)
 - /unlink 4085 (time 1787585552.16 total 30.05 last 146.87)
total: 4500 /unlink in 33.56 seconds: 134.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 606 (time 1787585571.29 total 10.03 last 60.45)
 - open/close 1205 (time 1787585581.30 total 20.03 last 59.86)
 - open/close 2023 (time 1787585591.30 total 30.03 last 81.79)
 - open/close 2827 (time 1787585601.30 total 40.03 last 80.39)
 - open/close 3625 (time 1787585611.31 total 50.04 last 79.74)
 - open/close 4290 (time 1787585621.31 total 60.04 last 66.49)
total: 4500 open/close in 62.15 seconds: 72.41 ops/second
 - /unlink 1980 (time 1787585637.78 total 10.00 last 197.95)
 - /unlink 3436 (time 1787585647.78 total 20.00 last 145.60)
total: 4500 /unlink in 26.41 seconds: 170.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 962 (time 1787585668.89 total 10.00 last 96.19)
 - open/close 1946 (time 1787585678.89 total 20.00 last 98.39)
 - open/close 2773 (time 1787585688.93 total 30.04 last 82.35)
 - open/close 3449 (time 1787585698.96 total 40.06 last 67.46)
 - open/close 4067 (time 1787585708.96 total 50.07 last 61.79)
total: 4500 open/close in 58.41 seconds: 77.04 ops/second
 - /unlink 1578 (time 1787585733.08 total 10.00 last 157.75)
 - /unlink 2987 (time 1787585743.09 total 20.01 last 140.87)
 - /unlink 4477 (time 1787585753.10 total 30.02 last 148.85)
total: 4500 /unlink in 30.36 seconds: 148.21 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95309 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg107-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg107-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server
stop mds service on oleg107-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server
stop mds service on oleg107-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server

unloading modules via unload_modules_local on: 'oleg107-server'
oleg107-server: oleg107-server.virtnet: executing unload_modules_local
oleg107-server: 
modules unloaded.