== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:43:33 (1786952613)
Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg424-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg424-server
oleg424-server: oleg424-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg424-server'
oleg424-server: oleg424-server.virtnet: executing load_modules_local
oleg424-server: Loading modules from /home/green/git/lustre-release/lustre
oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg424-server: detected 4 online CPUs by sysfs
oleg424-server: MODOPTS_LIBCFS=
oleg424-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg424-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg424-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg424-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg424-client.virtnet:  -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 635 (time 1786952829.79 total 10.02 last 63.40)
 - open/close 1271 (time 1786952839.80 total 20.02 last 63.56)
 - open/close 1953 (time 1786952849.81 total 30.03 last 68.14)
 - open/close 2628 (time 1786952859.81 total 40.03 last 67.49)
 - open/close 3389 (time 1786952869.82 total 50.05 last 76.00)
 - open/close 4022 (time 1786952879.83 total 60.05 last 63.27)
total: 4500 open/close in 68.41 seconds: 65.78 ops/second
 - /unlink 1268 (time 1786952903.00 total 10.00 last 126.75)
 - /unlink 2541 (time 1786952913.00 total 20.01 last 127.27)
 - /unlink 3774 (time 1786952923.01 total 30.01 last 123.21)
total: 4500 /unlink in 35.89 seconds: 125.37 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 681 (time 1786952942.94 total 10.00 last 68.08)
 - open/close 1376 (time 1786952952.96 total 20.01 last 69.42)
 - open/close 2086 (time 1786952962.97 total 30.03 last 70.92)
 - open/close 2790 (time 1786952972.98 total 40.04 last 70.33)
 - open/close 3425 (time 1786952982.98 total 50.04 last 63.45)
 - open/close 4012 (time 1786952993.02 total 60.08 last 58.51)
total: 4500 open/close in 68.49 seconds: 65.70 ops/second
 - /unlink 1184 (time 1786953016.79 total 10.00 last 118.37)
 - /unlink 2492 (time 1786953026.80 total 20.01 last 130.69)
 - /unlink 3540 (time 1786953036.80 total 30.01 last 104.78)
total: 4500 /unlink in 39.39 seconds: 114.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 595 (time 1786953062.42 total 10.01 last 59.46)
 - open/close 1219 (time 1786953072.43 total 20.02 last 62.33)
 - open/close 1825 (time 1786953082.44 total 30.02 last 60.58)
 - open/close 2412 (time 1786953092.45 total 40.03 last 58.66)
 - open/close 3028 (time 1786953102.45 total 50.03 last 61.58)
 - open/close 3636 (time 1786953112.45 total 60.03 last 60.78)
 - open/close 4260 (time 1786953122.45 total 70.04 last 62.39)
total: 4500 open/close in 73.97 seconds: 60.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1049 (time 1786953141.52 total 10.00 last 104.88)
 - /unlink 2019 (time 1786953151.53 total 20.01 last 96.96)
 - /unlink 3138 (time 1786953161.53 total 30.01 last 111.84)
 - /unlink 4221 (time 1786953171.53 total 40.02 last 108.26)
total: 4500 /unlink in 42.35 seconds: 106.25 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 530 (time 1786953190.13 total 10.02 last 52.89)
 - open/close 1126 (time 1786953200.13 total 20.02 last 59.58)
 - open/close 1785 (time 1786953210.13 total 30.03 last 65.88)
 - open/close 2408 (time 1786953220.14 total 40.04 last 62.25)
 - open/close 3116 (time 1786953230.15 total 50.05 last 70.72)
 - open/close 3756 (time 1786953240.16 total 60.06 last 63.95)
 - open/close 4351 (time 1786953250.17 total 70.07 last 59.44)
total: 4500 open/close in 72.05 seconds: 62.46 ops/second
 - /unlink 1080 (time 1786953267.21 total 10.00 last 108.00)
 - /unlink 2217 (time 1786953277.22 total 20.00 last 113.65)
 - /unlink 3481 (time 1786953287.22 total 30.01 last 126.40)
total: 4500 /unlink in 39.97 seconds: 112.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 603 (time 1786953312.81 total 10.01 last 60.26)
 - open/close 1209 (time 1786953322.81 total 20.02 last 60.54)
 - open/close 1855 (time 1786953332.82 total 30.02 last 64.55)
 - open/close 2401 (time 1786953342.82 total 40.02 last 54.59)
 - open/close 2954 (time 1786953352.83 total 50.03 last 55.26)
 - open/close 3492 (time 1786953362.84 total 60.04 last 53.76)
 - open/close 4092 (time 1786953372.86 total 70.06 last 59.87)
total: 4500 open/close in 75.93 seconds: 59.27 ops/second
 - /unlink 1117 (time 1786953393.44 total 10.00 last 111.69)
 - /unlink 2250 (time 1786953403.45 total 20.00 last 113.27)
 - /unlink 3218 (time 1786953413.46 total 30.01 last 96.69)
 - /unlink 4071 (time 1786953423.46 total 40.02 last 85.28)
total: 4500 /unlink in 44.90 seconds: 100.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 588 (time 1786953444.67 total 10.04 last 58.56)
 - open/close 1065 (time 1786953454.69 total 20.06 last 47.62)
 - open/close 1588 (time 1786953464.70 total 30.06 last 52.26)
 - open/close 2209 (time 1786953474.71 total 40.07 last 62.05)
 - open/close 2719 (time 1786953484.72 total 50.09 last 50.91)
 - open/close 3224 (time 1786953494.73 total 60.09 last 50.48)
 - open/close 3707 (time 1786953504.74 total 70.10 last 48.26)
 - open/close 4219 (time 1786953514.75 total 80.11 last 51.15)
total: 4500 open/close in 85.06 seconds: 52.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1020 (time 1786953535.58 total 10.00 last 102.00)
 - /unlink 2049 (time 1786953545.58 total 20.00 last 102.88)
 - /unlink 3098 (time 1786953555.58 total 30.01 last 104.83)
 - /unlink 4233 (time 1786953565.59 total 40.01 last 113.47)
total: 4500 /unlink in 43.35 seconds: 103.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 523 (time 1786953585.82 total 10.02 last 52.21)
 - open/close 1058 (time 1786953595.83 total 20.03 last 53.44)
 - open/close 1706 (time 1786953605.83 total 30.03 last 64.76)
 - open/close 2236 (time 1786953615.84 total 40.05 last 52.93)
 - open/close 2732 (time 1786953625.84 total 50.05 last 49.60)
 - open/close 3267 (time 1786953635.85 total 60.06 last 53.45)
 - open/close 3848 (time 1786953645.86 total 70.06 last 58.06)
 - open/close 4464 (time 1786953655.89 total 80.09 last 61.44)
total: 4500 open/close in 80.59 seconds: 55.84 ops/second
 - /unlink 1042 (time 1786953671.15 total 10.00 last 104.18)
 - /unlink 2118 (time 1786953681.16 total 20.02 last 107.42)
 - /unlink 3192 (time 1786953691.17 total 30.03 last 107.32)
 - /unlink 4356 (time 1786953701.19 total 40.05 last 116.14)
total: 4500 /unlink in 41.26 seconds: 109.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 620 (time 1786953717.25 total 10.03 last 61.84)
 - open/close 1168 (time 1786953727.25 total 20.03 last 54.80)
 - open/close 1826 (time 1786953737.25 total 30.03 last 65.78)
 - open/close 2385 (time 1786953747.27 total 40.05 last 55.79)
 - open/close 2826 (time 1786953757.30 total 50.08 last 43.99)
 - open/close 3246 (time 1786953767.31 total 60.09 last 41.94)
 - open/close 3813 (time 1786953777.32 total 70.10 last 56.64)
 - open/close 4385 (time 1786953787.33 total 80.11 last 57.14)
total: 4500 open/close in 81.85 seconds: 54.98 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1074 (time 1786953804.40 total 10.01 last 107.27)
 - /unlink 2061 (time 1786953814.43 total 20.04 last 98.44)
 - /unlink 2988 (time 1786953824.45 total 30.06 last 92.52)
 - /unlink 3952 (time 1786953834.45 total 40.06 last 96.39)
total: 4500 /unlink in 44.82 seconds: 100.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 586 (time 1786953855.48 total 10.00 last 58.59)
 - open/close 1163 (time 1786953865.50 total 20.02 last 57.61)
 - open/close 1759 (time 1786953875.50 total 30.02 last 59.59)
 - open/close 2374 (time 1786953885.52 total 40.03 last 61.41)
 - open/close 2873 (time 1786953895.53 total 50.05 last 49.80)
 - open/close 3448 (time 1786953905.55 total 60.06 last 57.43)
 - open/close 3984 (time 1786953915.59 total 70.11 last 53.36)
total: 4500 open/close in 79.42 seconds: 56.66 ops/second
 - /unlink 1227 (time 1786953940.14 total 10.00 last 122.68)
 - /unlink 2237 (time 1786953950.15 total 20.00 last 100.99)
 - /unlink 3292 (time 1786953960.15 total 30.01 last 105.48)
 - /unlink 4247 (time 1786953970.16 total 40.02 last 95.39)
total: 4500 /unlink in 42.61 seconds: 105.61 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 544 (time 1786953988.92 total 10.01 last 54.35)
 - open/close 1094 (time 1786953998.94 total 20.03 last 54.86)
 - open/close 1581 (time 1786954008.96 total 30.05 last 48.62)
 - open/close 2146 (time 1786954018.96 total 40.05 last 56.48)
 - open/close 2628 (time 1786954028.96 total 50.06 last 48.19)
 - open/close 3172 (time 1786954038.99 total 60.08 last 54.28)
 - open/close 3776 (time 1786954049.00 total 70.09 last 60.33)
 - open/close 4368 (time 1786954059.00 total 80.09 last 59.19)
total: 4500 open/close in 81.92 seconds: 54.93 ops/second
 - /unlink 895 (time 1786954076.23 total 10.01 last 89.43)
 - /unlink 1795 (time 1786954086.24 total 20.01 last 89.96)
 - /unlink 2788 (time 1786954096.25 total 30.03 last 99.14)
 - /unlink 3674 (time 1786954106.26 total 40.04 last 88.52)
 - /unlink 4460 (time 1786954116.27 total 50.04 last 78.54)
total: 4500 /unlink in 50.61 seconds: 88.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 558 (time 1786954132.53 total 10.02 last 55.66)
 - open/close 1078 (time 1786954142.53 total 20.03 last 51.98)
 - open/close 1667 (time 1786954152.54 total 30.04 last 58.86)
 - open/close 2276 (time 1786954162.55 total 40.05 last 60.81)
 - open/close 2904 (time 1786954172.56 total 50.06 last 62.77)
 - open/close 3579 (time 1786954182.57 total 60.06 last 67.44)
 - open/close 4046 (time 1786954192.58 total 70.07 last 46.66)
total: 4500 open/close in 78.48 seconds: 57.34 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1146 (time 1786954216.68 total 10.00 last 114.58)
 - /unlink 2149 (time 1786954226.68 total 20.00 last 100.29)
 - /unlink 3286 (time 1786954236.68 total 30.00 last 113.69)
 - /unlink 4328 (time 1786954246.69 total 40.01 last 104.16)
total: 4500 /unlink in 42.34 seconds: 106.28 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 570 (time 1786954265.03 total 10.01 last 56.95)
 - open/close 1140 (time 1786954275.04 total 20.02 last 56.94)
 - open/close 1742 (time 1786954285.05 total 30.03 last 60.11)
 - open/close 2309 (time 1786954295.07 total 40.05 last 56.61)
 - open/close 2983 (time 1786954305.07 total 50.06 last 67.36)
 - open/close 3579 (time 1786954315.08 total 60.06 last 59.57)
 - open/close 4246 (time 1786954325.09 total 70.07 last 66.63)
total: 4500 open/close in 74.73 seconds: 60.22 ops/second
 - /unlink 1179 (time 1786954345.25 total 10.01 last 117.83)
 - /unlink 2156 (time 1786954355.25 total 20.01 last 97.68)
 - /unlink 3316 (time 1786954365.27 total 30.02 last 115.83)
 - /unlink 4315 (time 1786954375.28 total 40.03 last 99.82)
total: 4500 /unlink in 41.65 seconds: 108.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 576 (time 1786954392.51 total 10.01 last 57.57)
 - open/close 1200 (time 1786954402.52 total 20.01 last 62.38)
 - open/close 1768 (time 1786954412.52 total 30.02 last 56.76)
 - open/close 2321 (time 1786954422.55 total 40.04 last 55.17)
 - open/close 2870 (time 1786954432.55 total 50.05 last 54.86)
 - open/close 3408 (time 1786954442.56 total 60.06 last 53.74)
 - open/close 3963 (time 1786954452.57 total 70.06 last 55.48)
total: 4500 open/close in 78.21 seconds: 57.53 ops/second
 - /unlink 1007 (time 1786954476.86 total 10.01 last 100.57)
 - /unlink 1974 (time 1786954486.87 total 20.02 last 96.59)
 - /unlink 2915 (time 1786954496.87 total 30.02 last 94.09)
 - /unlink 4055 (time 1786954506.87 total 40.02 last 114.00)
total: 4500 /unlink in 44.55 seconds: 101.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 624 (time 1786954526.49 total 10.01 last 62.32)
 - open/close 1250 (time 1786954536.49 total 20.01 last 62.60)
 - open/close 1864 (time 1786954546.50 total 30.02 last 61.34)
 - open/close 2374 (time 1786954556.50 total 40.03 last 50.97)
 - open/close 2897 (time 1786954566.51 total 50.04 last 52.25)
 - open/close 3497 (time 1786954576.52 total 60.04 last 59.99)
 - open/close 4165 (time 1786954586.53 total 70.05 last 66.72)
total: 4500 open/close in 75.62 seconds: 59.51 ops/second
 - /unlink 1197 (time 1786954607.52 total 10.00 last 119.68)
 - /unlink 2271 (time 1786954617.53 total 20.00 last 107.37)
 - /unlink 3566 (time 1786954627.53 total 30.01 last 129.45)
total: 4500 /unlink in 37.52 seconds: 119.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 588 (time 1786954650.30 total 10.01 last 58.76)
 - open/close 1211 (time 1786954660.31 total 20.02 last 62.24)
 - open/close 1807 (time 1786954670.32 total 30.02 last 59.57)
 - open/close 2392 (time 1786954680.34 total 40.04 last 58.40)
 - open/close 3082 (time 1786954690.35 total 50.05 last 68.93)
 - open/close 3815 (time 1786954700.36 total 60.06 last 73.22)
 - open/close 4451 (time 1786954710.36 total 70.06 last 63.60)
total: 4500 open/close in 70.69 seconds: 63.66 ops/second
 - /unlink 1438 (time 1786954726.23 total 10.00 last 143.79)
 - /unlink 2648 (time 1786954736.23 total 20.01 last 120.94)
 - /unlink 3820 (time 1786954746.24 total 30.01 last 117.14)
total: 4500 /unlink in 36.28 seconds: 124.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 610 (time 1786954767.45 total 10.00 last 60.98)
 - open/close 1244 (time 1786954777.46 total 20.01 last 63.36)
 - open/close 1930 (time 1786954787.48 total 30.03 last 68.49)
 - open/close 2621 (time 1786954797.48 total 40.03 last 69.08)
 - open/close 3157 (time 1786954807.50 total 50.05 last 53.50)
 - open/close 3777 (time 1786954817.50 total 60.05 last 61.96)
 - open/close 4447 (time 1786954827.51 total 70.06 last 66.94)
total: 4500 open/close in 71.09 seconds: 63.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1300 (time 1786954844.41 total 10.01 last 129.93)
 - /unlink 2394 (time 1786954854.42 total 20.01 last 109.35)
 - /unlink 3454 (time 1786954864.42 total 30.01 last 106.00)
total: 4500 /unlink in 39.85 seconds: 112.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 547 (time 1786954889.67 total 10.01 last 54.65)
 - open/close 1194 (time 1786954899.68 total 20.02 last 64.64)
 - open/close 1837 (time 1786954909.69 total 30.03 last 64.25)
 - open/close 2448 (time 1786954919.69 total 40.03 last 61.06)
 - open/close 3159 (time 1786954929.71 total 50.05 last 71.02)
 - open/close 3769 (time 1786954939.71 total 60.05 last 61.00)
 - open/close 4260 (time 1786954949.72 total 70.06 last 49.02)
total: 4500 open/close in 73.75 seconds: 61.02 ops/second
 - /unlink 1101 (time 1786954969.22 total 10.00 last 110.07)
 - /unlink 2190 (time 1786954979.22 total 20.01 last 108.85)
 - /unlink 3155 (time 1786954989.22 total 30.01 last 96.48)
 - /unlink 4317 (time 1786954999.23 total 40.01 last 116.19)
total: 4500 /unlink in 41.71 seconds: 107.88 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 638 (time 1786955016.08 total 10.01 last 63.76)
 - open/close 1306 (time 1786955026.09 total 20.02 last 66.71)
 - open/close 1907 (time 1786955036.09 total 30.02 last 60.10)
 - open/close 2512 (time 1786955046.10 total 40.03 last 60.47)
 - open/close 3153 (time 1786955056.11 total 50.03 last 64.04)
 - open/close 3788 (time 1786955066.12 total 60.05 last 63.40)
 - open/close 4433 (time 1786955076.13 total 70.06 last 64.45)
total: 4500 open/close in 71.10 seconds: 63.29 ops/second
 - /unlink 949 (time 1786955092.59 total 10.01 last 94.82)
 - /unlink 2028 (time 1786955102.59 total 20.01 last 107.85)
 - /unlink 2898 (time 1786955112.62 total 30.04 last 86.75)
 - /unlink 3992 (time 1786955122.64 total 40.06 last 109.22)
total: 4500 /unlink in 44.74 seconds: 100.57 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95370 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg424-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg424-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server
stop mds service on oleg424-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server
stop mds service on oleg424-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server

unloading modules via unload_modules_local on: 'oleg424-server'
oleg424-server: oleg424-server.virtnet: executing unload_modules_local
oleg424-server: 
modules unloaded.