== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:30:30 (1787383830)
Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg202-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server
oleg202-server: oleg202-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing load_modules_local
oleg202-server: Loading modules from /home/green/git/lustre-release/lustre
oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg202-server: detected 4 online CPUs by sysfs
oleg202-server: MODOPTS_LIBCFS=
oleg202-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg202-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg202-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg202-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all
pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg202-client.virtnet:  -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H fnv_1a_64  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 810 (time 1787384047.06 total 10.01 last 80.95)
 - open/close 1364 (time 1787384057.07 total 20.01 last 55.36)
 - open/close 2032 (time 1787384067.08 total 30.02 last 66.75)
 - open/close 2768 (time 1787384077.09 total 40.03 last 73.52)
 - open/close 3516 (time 1787384087.09 total 50.03 last 74.80)
 - open/close 4163 (time 1787384097.10 total 60.04 last 64.65)
total: 4500 open/close in 66.10 seconds: 68.08 ops/second
 - /unlink 1191 (time 1787384118.28 total 10.02 last 118.87)
 - /unlink 2441 (time 1787384128.29 total 20.03 last 124.91)
 - /unlink 3687 (time 1787384138.29 total 30.03 last 124.52)
total: 4500 /unlink in 36.92 seconds: 121.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 754 (time 1787384159.86 total 10.00 last 75.40)
 - open/close 1412 (time 1787384169.86 total 20.00 last 65.79)
 - open/close 2049 (time 1787384179.88 total 30.02 last 63.61)
 - open/close 2715 (time 1787384189.89 total 40.03 last 66.52)
 - open/close 3531 (time 1787384199.91 total 50.04 last 81.46)
 - open/close 4267 (time 1787384209.91 total 60.05 last 73.54)
total: 4500 open/close in 63.18 seconds: 71.23 ops/second
 - /unlink 1167 (time 1787384228.52 total 10.00 last 116.69)
 - /unlink 2194 (time 1787384238.52 total 20.00 last 102.70)
 - /unlink 3419 (time 1787384248.52 total 30.01 last 122.44)
total: 4500 /unlink in 37.83 seconds: 118.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 665 (time 1787384271.46 total 10.00 last 66.49)
 - open/close 1361 (time 1787384281.46 total 20.01 last 69.55)
 - open/close 2146 (time 1787384291.47 total 30.01 last 78.46)
 - open/close 2731 (time 1787384301.48 total 40.03 last 58.43)
 - open/close 3554 (time 1787384311.49 total 50.03 last 82.27)
 - open/close 4259 (time 1787384321.50 total 60.04 last 70.43)
total: 4500 open/close in 64.29 seconds: 70.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1234 (time 1787384341.53 total 10.01 last 123.28)
 - /unlink 2466 (time 1787384351.53 total 20.01 last 123.19)
 - /unlink 3766 (time 1787384361.53 total 30.01 last 129.96)
total: 4500 /unlink in 35.59 seconds: 126.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 718 (time 1787384381.39 total 10.03 last 71.62)
 - open/close 1390 (time 1787384391.40 total 20.04 last 67.11)
 - open/close 2196 (time 1787384401.40 total 30.04 last 80.59)
 - open/close 3021 (time 1787384411.41 total 40.05 last 82.44)
 - open/close 3789 (time 1787384421.43 total 50.07 last 76.66)
 - open/close 4468 (time 1787384431.43 total 60.07 last 67.87)
total: 4500 open/close in 60.73 seconds: 74.10 ops/second
 - /unlink 1045 (time 1787384447.67 total 10.00 last 104.45)
 - /unlink 2302 (time 1787384457.67 total 20.01 last 125.63)
 - /unlink 3614 (time 1787384467.68 total 30.02 last 131.13)
total: 4500 /unlink in 37.63 seconds: 119.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 530 (time 1787384492.63 total 10.00 last 52.99)
 - open/close 1128 (time 1787384502.64 total 20.02 last 59.71)
 - open/close 1728 (time 1787384512.65 total 30.03 last 59.93)
 - open/close 2357 (time 1787384522.65 total 40.03 last 62.89)
 - open/close 3100 (time 1787384532.66 total 50.04 last 74.24)
 - open/close 3825 (time 1787384542.67 total 60.04 last 72.48)
 - open/close 4454 (time 1787384552.68 total 70.05 last 62.84)
total: 4500 open/close in 70.90 seconds: 63.47 ops/second
 - /unlink 1216 (time 1787384568.07 total 10.00 last 121.58)
 - /unlink 2483 (time 1787384578.07 total 20.01 last 126.65)
 - /unlink 3804 (time 1787384588.08 total 30.01 last 132.01)
total: 4500 /unlink in 34.96 seconds: 128.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 606 (time 1787384608.82 total 10.03 last 60.44)
 - open/close 1077 (time 1787384618.86 total 20.07 last 46.89)
 - open/close 1576 (time 1787384628.87 total 30.08 last 49.86)
 - open/close 2103 (time 1787384638.87 total 40.08 last 52.67)
 - open/close 2752 (time 1787384648.88 total 50.09 last 64.83)
 - open/close 3436 (time 1787384658.89 total 60.10 last 68.34)
 - open/close 4167 (time 1787384668.90 total 70.11 last 73.03)
total: 4500 open/close in 74.70 seconds: 60.24 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1225 (time 1787384688.72 total 10.00 last 122.48)
 - /unlink 2571 (time 1787384698.72 total 20.00 last 134.60)
 - /unlink 3814 (time 1787384708.73 total 30.01 last 124.25)
total: 4500 /unlink in 35.84 seconds: 125.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 576 (time 1787384729.92 total 10.01 last 57.57)
 - open/close 1378 (time 1787384739.94 total 20.03 last 80.04)
 - open/close 2023 (time 1787384749.95 total 30.03 last 64.49)
 - open/close 2685 (time 1787384759.97 total 40.06 last 66.01)
 - open/close 3293 (time 1787384769.99 total 50.07 last 60.73)
 - open/close 3911 (time 1787384779.99 total 60.08 last 61.75)
total: 4500 open/close in 69.07 seconds: 65.15 ops/second
 - /unlink 1245 (time 1787384804.32 total 10.01 last 124.42)
 - /unlink 2445 (time 1787384814.32 total 20.01 last 119.95)
 - /unlink 3649 (time 1787384824.33 total 30.02 last 120.32)
total: 4500 /unlink in 36.34 seconds: 123.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 701 (time 1787384846.02 total 10.00 last 70.09)
 - open/close 1281 (time 1787384856.03 total 20.02 last 57.91)
 - open/close 1785 (time 1787384866.05 total 30.03 last 50.33)
 - open/close 2203 (time 1787384876.06 total 40.04 last 41.76)
 - open/close 2932 (time 1787384886.10 total 50.08 last 72.63)
 - open/close 3549 (time 1787384896.10 total 60.08 last 61.67)
 - open/close 3963 (time 1787384906.11 total 70.09 last 41.36)
total: 4500 open/close in 79.67 seconds: 56.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1095 (time 1787384931.57 total 10.01 last 109.44)
 - /unlink 2053 (time 1787384941.58 total 20.01 last 95.74)
 - /unlink 3403 (time 1787384951.58 total 30.02 last 134.94)
total: 4500 /unlink in 38.11 seconds: 118.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 730 (time 1787384974.89 total 10.01 last 72.95)
 - open/close 1335 (time 1787384984.90 total 20.02 last 60.43)
 - open/close 2040 (time 1787384994.90 total 30.02 last 70.48)
 - open/close 2701 (time 1787385004.91 total 40.03 last 66.05)
 - open/close 3367 (time 1787385014.91 total 50.03 last 66.59)
 - open/close 4145 (time 1787385024.91 total 60.03 last 77.79)
total: 4500 open/close in 64.73 seconds: 69.52 ops/second
 - /unlink 1326 (time 1787385044.20 total 10.00 last 132.55)
 - /unlink 2696 (time 1787385054.20 total 20.01 last 136.95)
 - /unlink 4274 (time 1787385064.20 total 30.01 last 157.77)
total: 4500 /unlink in 31.64 seconds: 142.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 723 (time 1787385079.93 total 10.01 last 72.23)
 - open/close 1423 (time 1787385089.94 total 20.02 last 69.93)
 - open/close 2201 (time 1787385099.94 total 30.02 last 77.79)
 - open/close 2972 (time 1787385109.97 total 40.05 last 76.88)
 - open/close 3636 (time 1787385119.97 total 50.06 last 66.35)
 - open/close 4334 (time 1787385129.98 total 60.07 last 69.73)
total: 4500 open/close in 62.18 seconds: 72.37 ops/second
 - /unlink 1358 (time 1787385147.17 total 10.00 last 135.78)
 - /unlink 2638 (time 1787385157.17 total 20.00 last 128.00)
 - /unlink 4054 (time 1787385167.17 total 30.01 last 141.53)
total: 4500 /unlink in 33.10 seconds: 135.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 703 (time 1787385186.10 total 10.03 last 70.09)
 - open/close 1421 (time 1787385196.11 total 20.04 last 71.74)
 - open/close 2053 (time 1787385206.13 total 30.06 last 63.07)
 - open/close 2764 (time 1787385216.13 total 40.06 last 71.09)
 - open/close 3356 (time 1787385226.13 total 50.06 last 59.19)
 - open/close 4045 (time 1787385236.14 total 60.07 last 68.84)
total: 4500 open/close in 65.67 seconds: 68.52 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1437 (time 1787385256.66 total 10.00 last 143.64)
 - /unlink 2958 (time 1787385266.66 total 20.01 last 152.06)
 - /unlink 4284 (time 1787385276.66 total 30.01 last 132.60)
total: 4500 /unlink in 31.85 seconds: 141.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 594 (time 1787385294.42 total 10.01 last 59.37)
 - open/close 1239 (time 1787385304.44 total 20.03 last 64.37)
 - open/close 1848 (time 1787385314.45 total 30.03 last 60.86)
 - open/close 2477 (time 1787385324.45 total 40.04 last 62.89)
 - open/close 3239 (time 1787385334.45 total 50.04 last 76.16)
 - open/close 3985 (time 1787385344.46 total 60.05 last 74.56)
total: 4500 open/close in 66.15 seconds: 68.03 ops/second
 - /unlink 1308 (time 1787385365.52 total 10.01 last 130.70)
 - /unlink 2619 (time 1787385375.52 total 20.01 last 131.10)
 - /unlink 4085 (time 1787385385.52 total 30.01 last 146.52)
total: 4500 /unlink in 33.13 seconds: 135.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 752 (time 1787385403.46 total 10.00 last 75.18)
 - open/close 1442 (time 1787385413.46 total 20.01 last 68.95)
 - open/close 2229 (time 1787385423.47 total 30.01 last 78.68)
 - open/close 2966 (time 1787385433.47 total 40.02 last 73.67)
 - open/close 3712 (time 1787385443.47 total 50.02 last 74.59)
 - open/close 4465 (time 1787385453.47 total 60.02 last 75.29)
total: 4500 open/close in 60.71 seconds: 74.13 ops/second
 - /unlink 1305 (time 1787385468.22 total 10.01 last 130.41)
 - /unlink 2655 (time 1787385478.23 total 20.02 last 134.88)
 - /unlink 3993 (time 1787385488.23 total 30.02 last 133.78)
total: 4500 /unlink in 33.88 seconds: 132.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 707 (time 1787385507.43 total 10.02 last 70.55)
 - open/close 1462 (time 1787385517.44 total 20.03 last 75.44)
 - open/close 2253 (time 1787385527.44 total 30.04 last 79.05)
 - open/close 2926 (time 1787385537.45 total 40.04 last 67.27)
 - open/close 3605 (time 1787385547.46 total 50.05 last 67.83)
 - open/close 4283 (time 1787385557.46 total 60.05 last 67.79)
total: 4500 open/close in 64.39 seconds: 69.88 ops/second
 - /unlink 1350 (time 1787385576.96 total 10.00 last 134.95)
 - /unlink 2615 (time 1787385586.96 total 20.01 last 126.48)
 - /unlink 3940 (time 1787385596.97 total 30.01 last 132.45)
total: 4500 /unlink in 34.35 seconds: 131.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 850 (time 1787385616.06 total 10.00 last 84.98)
 - open/close 1629 (time 1787385626.07 total 20.01 last 77.85)
 - open/close 2302 (time 1787385636.07 total 30.01 last 67.27)
 - open/close 3051 (time 1787385646.08 total 40.02 last 74.85)
 - open/close 3775 (time 1787385656.10 total 50.04 last 72.27)
total: 4500 open/close in 59.97 seconds: 75.04 ops/second
 - /unlink 1310 (time 1787385680.24 total 10.00 last 130.96)
 - /unlink 2435 (time 1787385690.24 total 20.01 last 112.43)
 - /unlink 3654 (time 1787385700.27 total 30.04 last 121.54)
total: 4500 /unlink in 36.13 seconds: 124.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 663 (time 1787385721.35 total 10.00 last 66.30)
 - open/close 1311 (time 1787385731.37 total 20.01 last 64.72)
 - open/close 1923 (time 1787385741.39 total 30.04 last 61.05)
 - open/close 2583 (time 1787385751.40 total 40.04 last 65.97)
 - open/close 3322 (time 1787385761.40 total 50.05 last 73.86)
 - open/close 3923 (time 1787385771.43 total 60.08 last 59.91)
 - open/close 4436 (time 1787385781.44 total 70.09 last 51.27)
total: 4500 open/close in 71.18 seconds: 63.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1169 (time 1787385798.25 total 10.00 last 116.85)
 - /unlink 2352 (time 1787385808.25 total 20.01 last 118.24)
 - /unlink 3682 (time 1787385818.25 total 30.01 last 133.00)
total: 4500 /unlink in 39.08 seconds: 115.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 602 (time 1787385843.71 total 10.01 last 60.16)
 - open/close 1212 (time 1787385853.72 total 20.02 last 60.94)
 - open/close 1785 (time 1787385863.73 total 30.02 last 57.25)
 - open/close 2487 (time 1787385873.74 total 40.03 last 70.13)
 - open/close 3161 (time 1787385883.75 total 50.04 last 67.38)
 - open/close 3939 (time 1787385893.75 total 60.04 last 77.80)
total: 4500 open/close in 68.14 seconds: 66.04 ops/second
 - /unlink 1386 (time 1787385916.36 total 10.00 last 138.56)
 - /unlink 2667 (time 1787385926.38 total 20.02 last 127.84)
 - /unlink 4194 (time 1787385936.39 total 30.04 last 152.52)
total: 4500 /unlink in 32.54 seconds: 138.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 663 (time 1787385954.57 total 10.00 last 66.27)
 - open/close 1242 (time 1787385964.57 total 20.01 last 57.89)
 - open/close 1986 (time 1787385974.59 total 30.02 last 74.31)
 - open/close 2483 (time 1787385984.59 total 40.02 last 49.69)
 - open/close 3013 (time 1787385994.61 total 50.04 last 52.89)
 - open/close 3604 (time 1787386004.62 total 60.05 last 59.02)
 - open/close 4337 (time 1787386014.63 total 70.06 last 73.22)
total: 4500 open/close in 71.86 seconds: 62.62 ops/second
 - /unlink 1278 (time 1787386031.71 total 10.00 last 127.77)
 - /unlink 2556 (time 1787386041.73 total 20.02 last 127.54)
 - /unlink 3891 (time 1787386051.73 total 30.03 last 133.47)
total: 4500 /unlink in 34.82 seconds: 129.24 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94891 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user 'cl1': success
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user 'cl1': success

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg202-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg202-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server
stop mds service on oleg202-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server

unloading modules via unload_modules_local on: 'oleg202-server'
oleg202-server: oleg202-server.virtnet: executing unload_modules_local
oleg202-server: 
modules unloaded.