== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:59:28 (1787857168)
Stopping clients: oleg208-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg208-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg208-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg208-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg208-server
oleg208-server: oleg208-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg208-server'
oleg208-server: oleg208-server.virtnet: executing load_modules_local
oleg208-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg208-server: Loading modules from /home/green/git/lustre-release/lustre
oleg208-server: detected 4 online CPUs by sysfs
oleg208-server: MODOPTS_LIBCFS=
oleg208-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg208-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all
pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg208-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all
pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg208-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all
pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg208-client.virtnet:  -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 790 (time 1787857386.54 total 10.01 last 78.94)
 - open/close 1490 (time 1787857396.55 total 20.02 last 69.89)
 - open/close 2062 (time 1787857406.55 total 30.02 last 57.20)
 - open/close 2704 (time 1787857416.56 total 40.03 last 64.14)
 - open/close 3523 (time 1787857426.56 total 50.03 last 81.90)
 - open/close 4331 (time 1787857436.57 total 60.04 last 80.78)
total: 4500 open/close in 62.57 seconds: 71.91 ops/second
 - /unlink 1511 (time 1787857453.98 total 10.00 last 151.08)
 - /unlink 2832 (time 1787857463.99 total 20.01 last 131.99)
 - /unlink 4166 (time 1787857474.00 total 30.01 last 133.34)
total: 4500 /unlink in 33.07 seconds: 136.08 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 755 (time 1787857492.07 total 10.00 last 75.47)
 - open/close 1525 (time 1787857502.07 total 20.01 last 76.97)
 - open/close 2399 (time 1787857512.09 total 30.03 last 87.24)
 - open/close 3215 (time 1787857522.09 total 40.03 last 81.58)
 - open/close 3955 (time 1787857532.10 total 50.04 last 73.95)
total: 4500 open/close in 57.64 seconds: 78.06 ops/second
 - /unlink 1297 (time 1787857554.44 total 10.00 last 129.65)
 - /unlink 2686 (time 1787857564.44 total 20.01 last 138.87)
 - /unlink 4227 (time 1787857574.45 total 30.02 last 153.93)
total: 4500 /unlink in 32.45 seconds: 138.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 515 (time 1787857593.81 total 10.02 last 51.41)
 - open/close 1126 (time 1787857603.82 total 20.02 last 61.07)
 - open/close 1773 (time 1787857613.84 total 30.04 last 64.56)
 - open/close 2535 (time 1787857623.84 total 40.05 last 76.17)
 - open/close 3227 (time 1787857633.85 total 50.05 last 69.18)
 - open/close 3920 (time 1787857643.85 total 60.05 last 69.29)
total: 4500 open/close in 68.83 seconds: 65.38 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1247 (time 1787857667.13 total 10.00 last 124.67)
 - /unlink 2459 (time 1787857677.13 total 20.00 last 121.20)
 - /unlink 3881 (time 1787857687.13 total 30.00 last 142.19)
total: 4500 /unlink in 34.75 seconds: 129.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 546 (time 1787857707.71 total 10.07 last 54.20)
 - open/close 1105 (time 1787857717.72 total 20.09 last 55.82)
 - open/close 1728 (time 1787857727.72 total 30.09 last 62.29)
 - open/close 2635 (time 1787857737.73 total 40.10 last 90.59)
 - open/close 3325 (time 1787857747.74 total 50.11 last 68.93)
 - open/close 3995 (time 1787857757.75 total 60.12 last 66.94)
total: 4500 open/close in 67.66 seconds: 66.51 ops/second
 - /unlink 1272 (time 1787857780.21 total 10.01 last 127.11)
 - /unlink 2537 (time 1787857790.21 total 20.01 last 126.50)
 - /unlink 3929 (time 1787857800.21 total 30.01 last 139.19)
total: 4500 /unlink in 34.50 seconds: 130.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 708 (time 1787857819.45 total 10.01 last 70.74)
 - open/close 1402 (time 1787857829.46 total 20.02 last 69.33)
 - open/close 2171 (time 1787857839.48 total 30.04 last 76.77)
 - open/close 2826 (time 1787857849.48 total 40.04 last 65.47)
 - open/close 3474 (time 1787857859.49 total 50.05 last 64.75)
 - open/close 4278 (time 1787857869.50 total 60.06 last 80.34)
total: 4500 open/close in 62.76 seconds: 71.71 ops/second
 - /unlink 1408 (time 1787857887.34 total 10.00 last 140.78)
 - /unlink 2743 (time 1787857897.35 total 20.01 last 133.36)
 - /unlink 3894 (time 1787857907.36 total 30.02 last 115.04)
total: 4500 /unlink in 36.05 seconds: 124.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 666 (time 1787857928.07 total 10.02 last 66.49)
 - open/close 1360 (time 1787857938.07 total 20.02 last 69.38)
 - open/close 2062 (time 1787857948.10 total 30.05 last 70.02)
 - open/close 2567 (time 1787857958.12 total 40.06 last 50.42)
 - open/close 3123 (time 1787857968.12 total 50.07 last 55.57)
 - open/close 3640 (time 1787857978.13 total 60.07 last 51.67)
 - open/close 4289 (time 1787857988.14 total 70.09 last 64.82)
total: 4500 open/close in 73.51 seconds: 61.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1030 (time 1787858008.74 total 10.01 last 102.92)
 - /unlink 2235 (time 1787858018.75 total 20.01 last 120.46)
 - /unlink 3181 (time 1787858028.75 total 30.02 last 94.54)
 - /unlink 4468 (time 1787858038.75 total 40.02 last 128.70)
total: 4500 /unlink in 40.44 seconds: 111.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 746 (time 1787858054.48 total 10.01 last 74.54)
 - open/close 1305 (time 1787858064.49 total 20.02 last 55.86)
 - open/close 1885 (time 1787858074.49 total 30.02 last 57.97)
 - open/close 2505 (time 1787858084.50 total 40.02 last 61.99)
 - open/close 3172 (time 1787858094.50 total 50.03 last 66.67)
 - open/close 3828 (time 1787858104.53 total 60.06 last 65.40)
total: 4500 open/close in 69.43 seconds: 64.81 ops/second
 - /unlink 1265 (time 1787858129.69 total 10.00 last 126.47)
 - /unlink 2476 (time 1787858139.69 total 20.00 last 121.08)
 - /unlink 3822 (time 1787858149.69 total 30.01 last 134.54)
total: 4500 /unlink in 36.37 seconds: 123.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 747 (time 1787858172.16 total 10.00 last 74.67)
 - open/close 1456 (time 1787858182.16 total 20.01 last 70.88)
 - open/close 2232 (time 1787858192.17 total 30.02 last 77.53)
 - open/close 2884 (time 1787858202.18 total 40.02 last 65.17)
 - open/close 3551 (time 1787858212.18 total 50.02 last 66.69)
 - open/close 4153 (time 1787858222.19 total 60.03 last 60.14)
total: 4500 open/close in 65.38 seconds: 68.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1493 (time 1787858242.38 total 10.00 last 149.25)
 - /unlink 2736 (time 1787858252.38 total 20.00 last 124.29)
 - /unlink 4113 (time 1787858262.39 total 30.00 last 137.69)
total: 4500 /unlink in 33.05 seconds: 136.14 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 724 (time 1787858281.30 total 10.01 last 72.33)
 - open/close 1356 (time 1787858291.31 total 20.02 last 63.11)
 - open/close 2061 (time 1787858301.33 total 30.04 last 70.36)
 - open/close 2712 (time 1787858311.35 total 40.06 last 64.98)
 - open/close 3427 (time 1787858321.36 total 50.07 last 71.46)
 - open/close 4073 (time 1787858331.36 total 60.07 last 64.60)
total: 4500 open/close in 67.02 seconds: 67.14 ops/second
 - /unlink 1208 (time 1787858353.81 total 10.00 last 120.76)
 - /unlink 2567 (time 1787858363.81 total 20.01 last 135.87)
 - /unlink 3940 (time 1787858373.82 total 30.01 last 137.24)
total: 4500 /unlink in 34.03 seconds: 132.22 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 581 (time 1787858393.89 total 10.02 last 57.96)
 - open/close 1251 (time 1787858403.90 total 20.04 last 66.87)
 - open/close 1953 (time 1787858413.91 total 30.05 last 70.17)
 - open/close 2537 (time 1787858423.92 total 40.06 last 58.34)
 - open/close 3106 (time 1787858433.92 total 50.06 last 56.88)
 - open/close 3825 (time 1787858443.94 total 60.08 last 71.77)
total: 4500 open/close in 69.40 seconds: 64.84 ops/second
 - /unlink 1284 (time 1787858469.26 total 10.00 last 128.40)
 - /unlink 2502 (time 1787858479.26 total 20.01 last 121.73)
 - /unlink 3765 (time 1787858489.27 total 30.01 last 126.21)
total: 4500 /unlink in 36.88 seconds: 122.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 597 (time 1787858512.35 total 10.03 last 59.53)
 - open/close 1237 (time 1787858522.36 total 20.03 last 64.00)
 - open/close 1905 (time 1787858532.36 total 30.03 last 66.78)
 - open/close 2620 (time 1787858542.37 total 40.04 last 71.45)
 - open/close 3240 (time 1787858552.37 total 50.04 last 61.99)
 - open/close 3976 (time 1787858562.38 total 60.06 last 73.49)
total: 4500 open/close in 68.23 seconds: 65.96 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1257 (time 1787858585.77 total 10.00 last 125.65)
 - /unlink 2599 (time 1787858595.78 total 20.01 last 134.16)
 - /unlink 3842 (time 1787858605.78 total 30.01 last 124.22)
total: 4500 /unlink in 35.75 seconds: 125.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 606 (time 1787858627.27 total 10.03 last 60.42)
 - open/close 1215 (time 1787858637.28 total 20.04 last 60.82)
 - open/close 1909 (time 1787858647.29 total 30.05 last 69.36)
 - open/close 2527 (time 1787858657.29 total 40.05 last 61.80)
 - open/close 3236 (time 1787858667.30 total 50.06 last 70.84)
 - open/close 3891 (time 1787858677.30 total 60.06 last 65.49)
total: 4500 open/close in 68.46 seconds: 65.73 ops/second
 - /unlink 1306 (time 1787858701.36 total 10.01 last 130.52)
 - /unlink 2623 (time 1787858711.37 total 20.01 last 131.60)
 - /unlink 3765 (time 1787858721.37 total 30.01 last 114.19)
total: 4500 /unlink in 35.77 seconds: 125.82 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 588 (time 1787858742.59 total 10.01 last 58.74)
 - open/close 1083 (time 1787858752.60 total 20.02 last 49.46)
 - open/close 1661 (time 1787858762.61 total 30.03 last 57.75)
 - open/close 2286 (time 1787858772.62 total 40.04 last 62.42)
 - open/close 2858 (time 1787858782.63 total 50.05 last 57.14)
 - open/close 3519 (time 1787858792.63 total 60.05 last 66.09)
 - open/close 4154 (time 1787858802.65 total 70.06 last 63.43)
total: 4500 open/close in 75.60 seconds: 59.53 ops/second
 - /unlink 1131 (time 1787858824.08 total 10.00 last 113.09)
 - /unlink 2270 (time 1787858834.09 total 20.01 last 113.86)
 - /unlink 3479 (time 1787858844.09 total 30.01 last 120.89)
total: 4500 /unlink in 38.91 seconds: 115.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 603 (time 1787858868.39 total 10.00 last 60.30)
 - open/close 1053 (time 1787858878.42 total 20.03 last 44.89)
 - open/close 1624 (time 1787858888.43 total 30.04 last 57.05)
 - open/close 2390 (time 1787858898.44 total 40.05 last 76.49)
 - open/close 3168 (time 1787858908.45 total 50.06 last 77.76)
 - open/close 3880 (time 1787858918.46 total 60.07 last 71.11)
total: 4500 open/close in 69.84 seconds: 64.43 ops/second
 - /unlink 1192 (time 1787858944.35 total 10.00 last 119.15)
 - /unlink 2243 (time 1787858954.36 total 20.01 last 105.05)
 - /unlink 3382 (time 1787858964.36 total 30.01 last 113.89)
total: 4500 /unlink in 39.93 seconds: 112.71 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 669 (time 1787858989.87 total 10.01 last 66.85)
 - open/close 1351 (time 1787858999.88 total 20.01 last 68.16)
 - open/close 2060 (time 1787859009.88 total 30.01 last 70.90)
 - open/close 2807 (time 1787859019.88 total 40.02 last 74.64)
 - open/close 3516 (time 1787859029.90 total 50.04 last 70.79)
 - open/close 4140 (time 1787859039.91 total 60.04 last 62.36)
total: 4500 open/close in 66.51 seconds: 67.66 ops/second
 - /unlink 1197 (time 1787859062.58 total 10.01 last 119.63)
 - /unlink 2454 (time 1787859072.58 total 20.01 last 125.67)
 - /unlink 3890 (time 1787859082.58 total 30.01 last 143.60)
total: 4500 /unlink in 34.95 seconds: 128.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 615 (time 1787859103.30 total 10.00 last 61.48)
 - open/close 1165 (time 1787859113.39 total 20.09 last 54.51)
 - open/close 1591 (time 1787859123.40 total 30.10 last 42.55)
 - open/close 2177 (time 1787859133.42 total 40.13 last 58.48)
 - open/close 2769 (time 1787859143.42 total 50.13 last 59.19)
 - open/close 3351 (time 1787859153.44 total 60.14 last 58.14)
 - open/close 4167 (time 1787859163.44 total 70.14 last 81.59)
total: 4500 open/close in 74.77 seconds: 60.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1223 (time 1787859181.92 total 10.01 last 122.19)
 - /unlink 2552 (time 1787859191.92 total 20.01 last 132.90)
 - /unlink 3766 (time 1787859201.93 total 30.01 last 121.38)
total: 4500 /unlink in 36.10 seconds: 124.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 612 (time 1787859224.19 total 10.01 last 61.11)
 - open/close 1151 (time 1787859234.20 total 20.02 last 53.85)
 - open/close 1572 (time 1787859244.21 total 30.04 last 42.05)
 - open/close 2119 (time 1787859254.21 total 40.04 last 54.69)
 - open/close 2709 (time 1787859264.22 total 50.05 last 58.93)
 - open/close 3343 (time 1787859274.25 total 60.08 last 63.23)
 - open/close 4022 (time 1787859284.26 total 70.09 last 67.86)
total: 4500 open/close in 77.99 seconds: 57.70 ops/second
 - /unlink 1157 (time 1787859306.85 total 10.00 last 115.67)
 - /unlink 2163 (time 1787859316.85 total 20.00 last 100.58)
 - /unlink 3328 (time 1787859326.86 total 30.01 last 116.48)
total: 4500 /unlink in 39.20 seconds: 114.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 639 (time 1787859351.61 total 10.01 last 63.81)
 - open/close 1226 (time 1787859361.62 total 20.02 last 58.64)
 - open/close 1925 (time 1787859371.63 total 30.03 last 69.84)
 - open/close 2581 (time 1787859381.63 total 40.04 last 65.59)
 - open/close 3148 (time 1787859391.64 total 50.04 last 56.66)
 - open/close 3779 (time 1787859401.68 total 60.08 last 62.85)
 - open/close 4375 (time 1787859411.69 total 70.10 last 59.50)
total: 4500 open/close in 71.74 seconds: 62.73 ops/second
 - /unlink 1387 (time 1787859428.77 total 10.00 last 138.68)
 - /unlink 2573 (time 1787859438.78 total 20.01 last 118.52)
 - /unlink 3796 (time 1787859448.78 total 30.01 last 122.27)
total: 4500 /unlink in 35.79 seconds: 125.74 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11369: 95369 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg208-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg208-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server
stop mds service on oleg208-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server
stop mds service on oleg208-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg208-server

unloading modules via unload_modules_local on: 'oleg208-server'
oleg208-server: oleg208-server.virtnet: executing unload_modules_local
oleg208-server: 
modules unloaded.