== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:11:27 (1781241087)
Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg116-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg116-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg116-server
oleg116-server: oleg116-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg116-server'
oleg116-server: oleg116-server.virtnet: executing load_modules_local
oleg116-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg116-server: Loading modules from /home/green/git/lustre-release/lustre
oleg116-server: detected 4 online CPUs by sysfs
oleg116-server: MODOPTS_LIBCFS=
oleg116-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg116-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all
pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg116-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all
pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg116-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all
pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg116-client.virtnet:  -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 639 (time 1781241288.74 total 10.01 last 63.82)
 - open/close 1394 (time 1781241298.75 total 20.02 last 75.42)
 - open/close 2088 (time 1781241308.76 total 30.04 last 69.32)
 - open/close 2845 (time 1781241318.77 total 40.05 last 75.60)
 - open/close 3706 (time 1781241328.77 total 50.05 last 86.08)
 - open/close 4451 (time 1781241338.78 total 60.06 last 74.46)
total: 4500 open/close in 60.76 seconds: 74.06 ops/second
 - /unlink 1250 (time 1781241354.06 total 10.00 last 124.97)
 - /unlink 2536 (time 1781241364.06 total 20.01 last 128.53)
 - /unlink 3893 (time 1781241374.06 total 30.01 last 135.69)
total: 4500 /unlink in 34.25 seconds: 131.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 657 (time 1781241393.49 total 10.02 last 65.58)
 - open/close 1255 (time 1781241403.49 total 20.02 last 59.77)
 - open/close 2007 (time 1781241413.50 total 30.03 last 75.18)
 - open/close 2787 (time 1781241423.52 total 40.05 last 77.82)
 - open/close 3547 (time 1781241433.53 total 50.06 last 75.95)
 - open/close 4226 (time 1781241443.54 total 60.07 last 67.83)
total: 4500 open/close in 63.54 seconds: 70.82 ops/second
 - /unlink 1297 (time 1781241461.67 total 10.00 last 129.65)
 - /unlink 2407 (time 1781241471.68 total 20.01 last 110.95)
 - /unlink 3715 (time 1781241481.68 total 30.02 last 130.69)
total: 4500 /unlink in 34.71 seconds: 129.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 747 (time 1781241501.93 total 10.01 last 74.63)
 - open/close 1413 (time 1781241511.95 total 20.03 last 66.48)
 - open/close 2027 (time 1781241521.97 total 30.04 last 61.29)
 - open/close 2735 (time 1781241531.97 total 40.05 last 70.79)
 - open/close 3453 (time 1781241542.01 total 50.08 last 71.54)
 - open/close 4113 (time 1781241552.02 total 60.09 last 65.93)
total: 4500 open/close in 65.14 seconds: 69.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1286 (time 1781241571.13 total 10.00 last 128.56)
 - /unlink 2611 (time 1781241581.15 total 20.02 last 132.23)
 - /unlink 3954 (time 1781241591.15 total 30.03 last 134.27)
total: 4500 /unlink in 34.78 seconds: 129.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 677 (time 1781241609.86 total 10.00 last 67.69)
 - open/close 1296 (time 1781241619.87 total 20.01 last 61.84)
 - open/close 1933 (time 1781241629.87 total 30.01 last 63.69)
 - open/close 2622 (time 1781241639.93 total 40.07 last 68.49)
 - open/close 3313 (time 1781241649.94 total 50.08 last 69.01)
 - open/close 3975 (time 1781241659.95 total 60.09 last 66.14)
total: 4500 open/close in 67.90 seconds: 66.27 ops/second
 - /unlink 1282 (time 1781241682.40 total 10.02 last 127.97)
 - /unlink 2508 (time 1781241692.40 total 20.02 last 122.58)
 - /unlink 3975 (time 1781241702.40 total 30.02 last 146.69)
total: 4500 /unlink in 33.73 seconds: 133.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 737 (time 1781241721.75 total 10.01 last 73.63)
 - open/close 1471 (time 1781241731.75 total 20.02 last 73.35)
 - open/close 2244 (time 1781241741.75 total 30.02 last 77.29)
 - open/close 2971 (time 1781241751.76 total 40.02 last 72.69)
 - open/close 3768 (time 1781241761.76 total 50.03 last 79.65)
 - open/close 4419 (time 1781241771.78 total 60.04 last 64.99)
total: 4500 open/close in 61.07 seconds: 73.69 ops/second
 - /unlink 1388 (time 1781241787.40 total 10.00 last 138.76)
 - /unlink 2581 (time 1781241797.42 total 20.02 last 119.09)
 - /unlink 3873 (time 1781241807.42 total 30.02 last 129.17)
total: 4500 /unlink in 33.82 seconds: 133.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 697 (time 1781241826.80 total 10.00 last 69.68)
 - open/close 1463 (time 1781241836.82 total 20.03 last 76.42)
 - open/close 2229 (time 1781241846.82 total 30.03 last 76.59)
 - open/close 2892 (time 1781241856.85 total 40.06 last 66.12)
 - open/close 3631 (time 1781241866.86 total 50.06 last 73.84)
 - open/close 4319 (time 1781241876.86 total 60.06 last 68.79)
total: 4500 open/close in 62.90 seconds: 71.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1412 (time 1781241894.27 total 10.00 last 141.18)
 - /unlink 2738 (time 1781241904.27 total 20.00 last 132.56)
 - /unlink 4055 (time 1781241914.27 total 30.01 last 131.68)
total: 4500 /unlink in 33.45 seconds: 134.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 558 (time 1781241934.84 total 10.00 last 55.79)
 - open/close 1189 (time 1781241944.85 total 20.01 last 63.05)
 - open/close 1864 (time 1781241954.86 total 30.02 last 67.46)
 - open/close 2520 (time 1781241964.86 total 40.02 last 65.56)
 - open/close 3220 (time 1781241974.87 total 50.03 last 69.96)
 - open/close 3951 (time 1781241984.88 total 60.04 last 73.02)
total: 4500 open/close in 67.52 seconds: 66.65 ops/second
 - /unlink 1383 (time 1781242007.53 total 10.00 last 138.24)
 - /unlink 2539 (time 1781242017.53 total 20.01 last 115.59)
 - /unlink 3696 (time 1781242027.54 total 30.01 last 115.61)
total: 4500 /unlink in 36.61 seconds: 122.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 641 (time 1781242049.21 total 10.01 last 64.07)
 - open/close 1352 (time 1781242059.22 total 20.01 last 71.04)
 - open/close 2066 (time 1781242069.25 total 30.04 last 71.22)
 - open/close 2783 (time 1781242079.25 total 40.04 last 71.68)
 - open/close 3499 (time 1781242089.25 total 50.04 last 71.58)
 - open/close 4303 (time 1781242099.26 total 60.05 last 80.35)
total: 4500 open/close in 62.97 seconds: 71.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1301 (time 1781242116.90 total 10.00 last 130.05)
 - /unlink 2616 (time 1781242126.90 total 20.00 last 131.49)
 - /unlink 3806 (time 1781242136.90 total 30.01 last 118.91)
total: 4500 /unlink in 35.72 seconds: 125.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 743 (time 1781242156.81 total 10.01 last 74.24)
 - open/close 1496 (time 1781242166.81 total 20.01 last 75.27)
 - open/close 2182 (time 1781242176.82 total 30.02 last 68.57)
 - open/close 2911 (time 1781242186.84 total 40.04 last 72.75)
 - open/close 3705 (time 1781242196.84 total 50.04 last 79.39)
 - open/close 4424 (time 1781242206.84 total 60.04 last 71.89)
total: 4500 open/close in 60.93 seconds: 73.85 ops/second
 - /unlink 1344 (time 1781242222.18 total 10.01 last 134.30)
 - /unlink 2540 (time 1781242232.18 total 20.01 last 119.56)
 - /unlink 3813 (time 1781242242.18 total 30.01 last 127.29)
total: 4500 /unlink in 35.45 seconds: 126.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 733 (time 1781242262.80 total 10.01 last 73.26)
 - open/close 1358 (time 1781242272.81 total 20.02 last 62.43)
 - open/close 2080 (time 1781242282.82 total 30.02 last 72.16)
 - open/close 2827 (time 1781242292.82 total 40.03 last 74.66)
 - open/close 3456 (time 1781242302.82 total 50.03 last 62.89)
 - open/close 3999 (time 1781242312.83 total 60.04 last 54.26)
total: 4500 open/close in 67.68 seconds: 66.49 ops/second
 - /unlink 1220 (time 1781242335.76 total 10.00 last 121.99)
 - /unlink 2439 (time 1781242345.76 total 20.00 last 121.89)
 - /unlink 3752 (time 1781242355.76 total 30.00 last 131.26)
total: 4500 /unlink in 35.89 seconds: 125.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 715 (time 1781242376.63 total 10.00 last 71.48)
 - open/close 1430 (time 1781242386.64 total 20.01 last 71.46)
 - open/close 2095 (time 1781242396.65 total 30.02 last 66.46)
 - open/close 2771 (time 1781242406.65 total 40.02 last 67.60)
 - open/close 3507 (time 1781242416.65 total 50.02 last 73.58)
 - open/close 4127 (time 1781242426.65 total 60.02 last 62.00)
total: 4500 open/close in 64.83 seconds: 69.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1383 (time 1781242445.98 total 10.00 last 138.27)
 - /unlink 2655 (time 1781242455.99 total 20.01 last 127.12)
 - /unlink 3968 (time 1781242465.99 total 30.01 last 131.23)
total: 4500 /unlink in 34.75 seconds: 129.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 660 (time 1781242486.50 total 10.00 last 65.99)
 - open/close 1304 (time 1781242496.53 total 20.03 last 64.23)
 - open/close 1931 (time 1781242506.55 total 30.05 last 62.56)
 - open/close 2625 (time 1781242516.55 total 40.05 last 69.37)
 - open/close 3221 (time 1781242526.57 total 50.08 last 59.47)
 - open/close 3835 (time 1781242536.59 total 60.09 last 61.32)
total: 4500 open/close in 69.07 seconds: 65.15 ops/second
 - /unlink 1182 (time 1781242560.24 total 10.00 last 118.16)
 - /unlink 2446 (time 1781242570.24 total 20.00 last 126.39)
 - /unlink 3967 (time 1781242580.24 total 30.01 last 152.08)
total: 4500 /unlink in 34.35 seconds: 130.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 657 (time 1781242599.49 total 10.00 last 65.68)
 - open/close 1406 (time 1781242609.50 total 20.01 last 74.85)
 - open/close 2030 (time 1781242619.50 total 30.02 last 62.36)
 - open/close 2652 (time 1781242629.51 total 40.02 last 62.15)
 - open/close 3119 (time 1781242639.52 total 50.03 last 46.66)
 - open/close 3728 (time 1781242649.53 total 60.05 last 60.82)
 - open/close 4346 (time 1781242659.54 total 70.05 last 61.75)
total: 4500 open/close in 71.94 seconds: 62.55 ops/second
 - /unlink 1278 (time 1781242676.39 total 10.01 last 127.65)
 - /unlink 2410 (time 1781242686.39 total 20.02 last 113.15)
 - /unlink 3644 (time 1781242696.39 total 30.02 last 123.39)
total: 4500 /unlink in 37.68 seconds: 119.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 786 (time 1781242720.34 total 10.01 last 78.54)
 - open/close 1449 (time 1781242730.37 total 20.03 last 66.13)
 - open/close 2162 (time 1781242740.37 total 30.04 last 71.24)
 - open/close 2885 (time 1781242750.37 total 40.04 last 72.29)
 - open/close 3632 (time 1781242760.38 total 50.05 last 74.64)
 - open/close 4316 (time 1781242770.40 total 60.07 last 68.28)
total: 4500 open/close in 62.79 seconds: 71.66 ops/second
 - /unlink 1358 (time 1781242787.52 total 10.00 last 135.76)
 - /unlink 2658 (time 1781242797.52 total 20.01 last 129.95)
 - /unlink 4012 (time 1781242807.53 total 30.01 last 135.33)
total: 4500 /unlink in 33.56 seconds: 134.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 742 (time 1781242826.55 total 10.01 last 74.13)
 - open/close 1483 (time 1781242836.55 total 20.02 last 74.05)
 - open/close 2281 (time 1781242846.56 total 30.02 last 79.78)
 - open/close 3008 (time 1781242856.58 total 40.04 last 72.56)
 - open/close 3728 (time 1781242866.58 total 50.04 last 71.98)
 - open/close 4442 (time 1781242876.58 total 60.04 last 71.36)
total: 4500 open/close in 60.73 seconds: 74.10 ops/second
 - /unlink 1470 (time 1781242891.61 total 10.01 last 146.87)
 - /unlink 2875 (time 1781242901.62 total 20.01 last 140.45)
 - /unlink 4309 (time 1781242911.62 total 30.02 last 143.32)
total: 4500 /unlink in 31.66 seconds: 142.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 719 (time 1781242928.45 total 10.01 last 71.86)
 - open/close 1458 (time 1781242938.47 total 20.02 last 73.80)
 - open/close 2279 (time 1781242948.47 total 30.03 last 82.04)
 - open/close 3086 (time 1781242958.50 total 40.05 last 80.48)
 - open/close 3858 (time 1781242968.50 total 50.05 last 77.20)
total: 4500 open/close in 59.18 seconds: 76.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1399 (time 1781242991.62 total 10.01 last 139.75)
 - /unlink 2854 (time 1781243001.62 total 20.01 last 145.47)
 - /unlink 4296 (time 1781243011.63 total 30.02 last 144.13)
total: 4500 /unlink in 31.73 seconds: 141.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 688 (time 1781243028.97 total 10.00 last 68.80)
 - open/close 1337 (time 1781243038.99 total 20.01 last 64.83)
 - open/close 1985 (time 1781243048.99 total 30.01 last 64.79)
 - open/close 2752 (time 1781243058.99 total 40.01 last 76.70)
 - open/close 3525 (time 1781243069.00 total 50.02 last 77.22)
 - open/close 4157 (time 1781243079.01 total 60.03 last 63.13)
total: 4500 open/close in 64.88 seconds: 69.36 ops/second
 - /unlink 1216 (time 1781243098.91 total 10.00 last 121.59)
 - /unlink 2574 (time 1781243108.92 total 20.01 last 135.73)
 - /unlink 3885 (time 1781243118.92 total 30.01 last 131.05)
total: 4500 /unlink in 34.82 seconds: 129.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 806 (time 1781243138.80 total 10.01 last 80.53)
 - open/close 1456 (time 1781243148.81 total 20.01 last 64.99)
 - open/close 2151 (time 1781243158.82 total 30.02 last 69.42)
 - open/close 2882 (time 1781243168.82 total 40.02 last 73.07)
 - open/close 3615 (time 1781243178.85 total 50.06 last 73.07)
 - open/close 4291 (time 1781243188.85 total 60.06 last 67.59)
total: 4500 open/close in 62.72 seconds: 71.75 ops/second
 - /unlink 1308 (time 1781243207.02 total 10.00 last 130.77)
 - /unlink 2573 (time 1781243217.04 total 20.01 last 126.35)
 - /unlink 3952 (time 1781243227.05 total 30.03 last 137.69)
total: 4500 /unlink in 34.21 seconds: 131.54 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11195: 94073 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg116-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg116-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server
stop mds service on oleg116-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server
stop mds service on oleg116-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg116-server

unloading modules via unload_modules_local on: 'oleg116-server'
oleg116-server: oleg116-server.virtnet: executing unload_modules_local
oleg116-server: 
modules unloaded.