== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:21:21 (1769077281)
Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg351-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg351-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg351-server
oleg351-server: oleg351-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg351-server'
oleg351-server: oleg351-server.virtnet: executing load_modules_local
oleg351-server: Loading modules from /home/green/git/lustre-release/lustre
oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg351-server: detected 4 online CPUs by sysfs
oleg351-server: MODOPTS_LIBCFS=
oleg351-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg351-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg351-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg351-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg351-client.virtnet:  -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 536 (time 1769077486.86 total 10.00 last 53.57)
 - open/close 1080 (time 1769077496.88 total 20.02 last 54.30)
 - open/close 1796 (time 1769077506.88 total 30.03 last 71.54)
 - open/close 2489 (time 1769077516.89 total 40.04 last 69.27)
 - open/close 3244 (time 1769077526.90 total 50.05 last 75.42)
 - open/close 4036 (time 1769077536.90 total 60.05 last 79.16)
total: 4500 open/close in 65.92 seconds: 68.27 ops/second
 - /unlink 1431 (time 1769077557.97 total 10.00 last 143.05)
 - /unlink 2889 (time 1769077567.97 total 20.01 last 145.75)
 - /unlink 4221 (time 1769077577.98 total 30.01 last 133.18)
total: 4500 /unlink in 32.13 seconds: 140.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 734 (time 1769077595.01 total 10.01 last 73.29)
 - open/close 1397 (time 1769077605.01 total 20.02 last 66.27)
 - open/close 2127 (time 1769077615.02 total 30.03 last 72.91)
 - open/close 2830 (time 1769077625.02 total 40.03 last 70.29)
 - open/close 3883 (time 1769077635.03 total 50.03 last 105.29)
total: 4500 open/close in 59.71 seconds: 75.37 ops/second
 - /unlink 1207 (time 1769077660.39 total 10.01 last 120.62)
 - /unlink 2255 (time 1769077670.39 total 20.01 last 104.78)
 - /unlink 3378 (time 1769077680.39 total 30.01 last 112.25)
total: 4500 /unlink in 39.32 seconds: 114.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 494 (time 1769077704.42 total 10.02 last 49.32)
 - open/close 1148 (time 1769077714.43 total 20.03 last 65.31)
 - open/close 1835 (time 1769077724.44 total 30.04 last 68.62)
 - open/close 2555 (time 1769077734.47 total 40.07 last 71.82)
 - open/close 3150 (time 1769077744.47 total 50.08 last 59.46)
 - open/close 3638 (time 1769077754.48 total 60.08 last 48.76)
 - open/close 4236 (time 1769077764.48 total 70.08 last 59.79)
total: 4500 open/close in 74.93 seconds: 60.05 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1407 (time 1769077784.06 total 10.01 last 140.57)
 - /unlink 2694 (time 1769077794.07 total 20.02 last 128.63)
 - /unlink 4213 (time 1769077804.07 total 30.02 last 151.86)
total: 4500 /unlink in 33.04 seconds: 136.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 780 (time 1769077823.62 total 10.00 last 77.99)
 - open/close 1492 (time 1769077833.62 total 20.00 last 71.17)
 - open/close 2215 (time 1769077843.62 total 30.01 last 72.27)
 - open/close 3001 (time 1769077853.63 total 40.02 last 78.52)
 - open/close 3707 (time 1769077863.64 total 50.02 last 70.58)
 - open/close 4370 (time 1769077873.64 total 60.02 last 66.29)
total: 4500 open/close in 61.65 seconds: 72.99 ops/second
 - /unlink 1399 (time 1769077890.05 total 10.01 last 139.78)
 - /unlink 2692 (time 1769077900.06 total 20.01 last 129.26)
 - /unlink 4030 (time 1769077910.06 total 30.01 last 133.77)
total: 4500 /unlink in 33.76 seconds: 133.31 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 748 (time 1769077928.74 total 10.00 last 74.77)
 - open/close 1504 (time 1769077938.75 total 20.01 last 75.53)
 - open/close 2254 (time 1769077948.75 total 30.02 last 74.95)
 - open/close 3049 (time 1769077958.77 total 40.03 last 79.38)
 - open/close 3861 (time 1769077968.77 total 50.04 last 81.18)
total: 4500 open/close in 57.71 seconds: 77.98 ops/second
 - /unlink 1328 (time 1769077991.23 total 10.00 last 132.79)
 - /unlink 2609 (time 1769078001.24 total 20.01 last 128.04)
 - /unlink 3722 (time 1769078011.24 total 30.01 last 111.21)
total: 4500 /unlink in 36.70 seconds: 122.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 650 (time 1769078033.26 total 10.00 last 64.98)
 - open/close 1325 (time 1769078043.26 total 20.00 last 67.49)
 - open/close 2010 (time 1769078053.26 total 30.01 last 68.49)
 - open/close 2770 (time 1769078063.29 total 40.03 last 75.78)
 - open/close 3439 (time 1769078073.29 total 50.04 last 66.89)
 - open/close 4060 (time 1769078083.30 total 60.05 last 62.02)
total: 4500 open/close in 66.12 seconds: 68.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1435 (time 1769078103.97 total 10.00 last 143.47)
 - /unlink 2731 (time 1769078113.98 total 20.01 last 129.51)
 - /unlink 3978 (time 1769078123.98 total 30.01 last 124.66)
total: 4500 /unlink in 34.15 seconds: 131.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 751 (time 1769078143.71 total 10.01 last 75.03)
 - open/close 1440 (time 1769078153.73 total 20.02 last 68.82)
 - open/close 1904 (time 1769078163.73 total 30.03 last 46.37)
 - open/close 2280 (time 1769078173.75 total 40.04 last 37.54)
 - open/close 2579 (time 1769078183.75 total 50.05 last 29.88)
 - open/close 3046 (time 1769078193.76 total 60.05 last 46.68)
 - open/close 3697 (time 1769078203.77 total 70.06 last 65.01)
 - open/close 4393 (time 1769078213.78 total 80.07 last 69.53)
total: 4500 open/close in 81.94 seconds: 54.92 ops/second
 - /unlink 1302 (time 1769078232.53 total 10.07 last 129.29)
 - /unlink 2531 (time 1769078242.58 total 20.12 last 122.28)
 - /unlink 3648 (time 1769078252.59 total 30.13 last 111.63)
total: 4500 /unlink in 35.92 seconds: 125.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 717 (time 1769078273.20 total 10.01 last 71.62)
 - open/close 1302 (time 1769078283.21 total 20.02 last 58.43)
 - open/close 1847 (time 1769078293.25 total 30.07 last 54.26)
 - open/close 2526 (time 1769078303.25 total 40.07 last 67.89)
 - open/close 3184 (time 1769078313.26 total 50.07 last 65.76)
 - open/close 3823 (time 1769078323.26 total 60.08 last 63.89)
total: 4500 open/close in 69.69 seconds: 64.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1216 (time 1769078347.82 total 10.00 last 121.59)
 - /unlink 2636 (time 1769078357.82 total 20.00 last 141.99)
 - /unlink 4174 (time 1769078367.82 total 30.00 last 153.79)
total: 4500 /unlink in 32.26 seconds: 139.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 815 (time 1769078385.78 total 10.00 last 81.50)
 - open/close 1722 (time 1769078395.79 total 20.00 last 90.68)
 - open/close 2447 (time 1769078405.79 total 30.01 last 72.47)
 - open/close 3159 (time 1769078415.81 total 40.02 last 71.09)
 - open/close 3808 (time 1769078425.82 total 50.03 last 64.85)
total: 4500 open/close in 59.33 seconds: 75.84 ops/second
 - /unlink 1251 (time 1769078449.79 total 10.00 last 125.05)
 - /unlink 2429 (time 1769078459.79 total 20.01 last 117.78)
 - /unlink 3613 (time 1769078469.81 total 30.02 last 118.26)
total: 4500 /unlink in 36.37 seconds: 123.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 631 (time 1769078492.56 total 10.02 last 62.99)
 - open/close 1326 (time 1769078502.57 total 20.03 last 69.42)
 - open/close 2130 (time 1769078512.57 total 30.03 last 80.40)
 - open/close 2738 (time 1769078522.58 total 40.04 last 60.72)
 - open/close 3436 (time 1769078532.59 total 50.05 last 69.74)
 - open/close 4248 (time 1769078542.60 total 60.06 last 81.16)
total: 4500 open/close in 63.42 seconds: 70.95 ops/second
 - /unlink 1644 (time 1769078561.45 total 10.00 last 164.33)
 - /unlink 3369 (time 1769078571.45 total 20.01 last 172.48)
total: 4500 /unlink in 26.70 seconds: 168.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 694 (time 1769078593.03 total 10.01 last 69.31)
 - open/close 1379 (time 1769078603.04 total 20.02 last 68.42)
 - open/close 2120 (time 1769078613.06 total 30.04 last 73.96)
 - open/close 2675 (time 1769078623.06 total 40.05 last 55.47)
 - open/close 3378 (time 1769078633.06 total 50.05 last 70.28)
 - open/close 4019 (time 1769078643.07 total 60.06 last 64.04)
total: 4500 open/close in 66.66 seconds: 67.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1431 (time 1769078664.65 total 10.01 last 143.01)
 - /unlink 2658 (time 1769078674.66 total 20.01 last 122.63)
 - /unlink 4200 (time 1769078684.66 total 30.01 last 154.17)
total: 4500 /unlink in 31.43 seconds: 143.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 624 (time 1769078701.66 total 10.01 last 62.31)
 - open/close 1356 (time 1769078711.70 total 20.05 last 72.96)
 - open/close 1963 (time 1769078721.70 total 30.05 last 60.67)
 - open/close 2794 (time 1769078731.71 total 40.06 last 83.06)
 - open/close 3487 (time 1769078741.72 total 50.07 last 69.20)
 - open/close 4301 (time 1769078751.73 total 60.08 last 81.37)
total: 4500 open/close in 62.26 seconds: 72.27 ops/second
 - /unlink 1412 (time 1769078768.17 total 10.00 last 141.18)
 - /unlink 2933 (time 1769078778.17 total 20.00 last 152.07)
 - /unlink 4152 (time 1769078788.18 total 30.01 last 121.85)
total: 4500 /unlink in 32.44 seconds: 138.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 780 (time 1769078805.73 total 10.00 last 77.97)
 - open/close 1480 (time 1769078815.75 total 20.03 last 69.84)
 - open/close 2227 (time 1769078825.76 total 30.03 last 74.65)
 - open/close 2925 (time 1769078835.77 total 40.04 last 69.73)
 - open/close 3612 (time 1769078845.77 total 50.05 last 68.69)
 - open/close 4390 (time 1769078855.78 total 60.06 last 77.72)
total: 4500 open/close in 61.29 seconds: 73.42 ops/second
 - /unlink 1323 (time 1769078872.59 total 10.00 last 132.30)
 - /unlink 2362 (time 1769078882.60 total 20.01 last 103.76)
 - /unlink 3145 (time 1769078892.61 total 30.02 last 78.29)
 - /unlink 4285 (time 1769078902.61 total 40.02 last 113.91)
total: 4500 /unlink in 42.16 seconds: 106.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 676 (time 1769078919.52 total 10.01 last 67.54)
 - open/close 1426 (time 1769078929.54 total 20.03 last 74.84)
 - open/close 2071 (time 1769078939.54 total 30.03 last 64.49)
 - open/close 2591 (time 1769078949.55 total 40.04 last 51.95)
 - open/close 3322 (time 1769078959.57 total 50.06 last 72.97)
 - open/close 4015 (time 1769078969.58 total 60.07 last 69.23)
total: 4500 open/close in 65.94 seconds: 68.25 ops/second
 - /unlink 1344 (time 1769078990.81 total 10.00 last 134.39)
 - /unlink 2491 (time 1769079000.82 total 20.01 last 114.63)
 - /unlink 3700 (time 1769079010.82 total 30.01 last 120.86)
total: 4500 /unlink in 36.73 seconds: 122.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 776 (time 1769079031.97 total 10.00 last 77.58)
 - open/close 1501 (time 1769079041.98 total 20.01 last 72.43)
 - open/close 2276 (time 1769079051.99 total 30.02 last 77.46)
 - open/close 2912 (time 1769079061.99 total 40.02 last 63.56)
 - open/close 3485 (time 1769079072.00 total 50.03 last 57.28)
 - open/close 4156 (time 1769079082.01 total 60.04 last 66.99)
total: 4500 open/close in 64.50 seconds: 69.77 ops/second
 - /unlink 1106 (time 1769079101.55 total 10.01 last 110.54)
 - /unlink 1893 (time 1769079111.56 total 20.01 last 78.62)
 - /unlink 2631 (time 1769079121.62 total 30.07 last 73.38)
 - /unlink 3657 (time 1769079131.63 total 40.09 last 102.45)
total: 4500 /unlink in 46.80 seconds: 96.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 846 (time 1769079154.31 total 10.01 last 84.52)
 - open/close 1579 (time 1769079164.32 total 20.02 last 73.23)
 - open/close 2292 (time 1769079174.32 total 30.02 last 71.29)
 - open/close 3098 (time 1769079184.32 total 40.02 last 80.59)
 - open/close 3862 (time 1769079194.32 total 50.02 last 76.38)
total: 4500 open/close in 58.39 seconds: 77.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1230 (time 1769079217.17 total 10.00 last 122.99)
 - /unlink 2531 (time 1769079227.17 total 20.00 last 130.09)
 - /unlink 4031 (time 1769079237.17 total 30.00 last 149.96)
total: 4500 /unlink in 33.36 seconds: 134.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 738 (time 1769079255.26 total 10.01 last 73.76)
 - open/close 1534 (time 1769079265.28 total 20.02 last 79.48)
 - open/close 2337 (time 1769079275.28 total 30.03 last 80.25)
 - open/close 3107 (time 1769079285.29 total 40.03 last 76.95)
 - open/close 3854 (time 1769079295.29 total 50.04 last 74.67)
total: 4500 open/close in 57.22 seconds: 78.64 ops/second
 - /unlink 1592 (time 1769079315.88 total 10.01 last 159.10)
 - /unlink 3035 (time 1769079325.88 total 20.01 last 144.26)
total: 4500 /unlink in 29.13 seconds: 154.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 854 (time 1769079349.54 total 10.00 last 85.38)
 - open/close 1613 (time 1769079359.56 total 20.02 last 75.78)
 - open/close 2405 (time 1769079369.56 total 30.02 last 79.16)
 - open/close 3218 (time 1769079379.57 total 40.03 last 81.28)
 - open/close 4157 (time 1769079389.57 total 50.03 last 93.84)
total: 4500 open/close in 53.66 seconds: 83.86 ops/second
 - /unlink 1610 (time 1769079407.24 total 10.01 last 160.83)
 - /unlink 3210 (time 1769079417.24 total 20.01 last 159.99)
total: 4500 /unlink in 28.10 seconds: 160.17 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11101: 94142 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg351-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg351-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server
stop mds service on oleg351-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server
stop mds service on oleg351-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg351-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg351-server'
oleg351-server: oleg351-server.virtnet: executing unload_modules_local
oleg351-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg351-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.