== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:17:32 (1778620652)
Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg134-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg134-server
oleg134-server: oleg134-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg134-server'
oleg134-server: oleg134-server.virtnet: executing load_modules_local
oleg134-server: Loading modules from /home/green/git/lustre-release/lustre
oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg134-server: detected 4 online CPUs by sysfs
oleg134-server: MODOPTS_LIBCFS=
oleg134-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg134-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all
pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg134-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all
pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg134-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all
pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg134-client.virtnet:  -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 610 (time 1778620873.54 total 10.00 last 60.99)
 - open/close 1151 (time 1778620883.55 total 20.00 last 54.08)
 - open/close 1713 (time 1778620893.56 total 30.01 last 56.16)
 - open/close 2275 (time 1778620903.58 total 40.04 last 56.06)
 - open/close 2863 (time 1778620913.58 total 50.04 last 58.79)
 - open/close 3394 (time 1778620923.62 total 60.08 last 52.89)
 - open/close 3846 (time 1778620933.63 total 70.08 last 45.18)
 - open/close 4401 (time 1778620943.63 total 80.09 last 55.46)
total: 4500 open/close in 81.99 seconds: 54.89 ops/second
 - /unlink 1091 (time 1778620962.12 total 10.00 last 109.06)
 - /unlink 2199 (time 1778620972.12 total 20.00 last 110.80)
 - /unlink 3305 (time 1778620982.13 total 30.01 last 110.53)
 - /unlink 4385 (time 1778620992.13 total 40.01 last 107.97)
total: 4500 /unlink in 41.07 seconds: 109.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 614 (time 1778621009.29 total 10.01 last 61.33)
 - open/close 1229 (time 1778621019.30 total 20.02 last 61.43)
 - open/close 1887 (time 1778621029.32 total 30.04 last 65.70)
 - open/close 2569 (time 1778621039.34 total 40.06 last 68.06)
 - open/close 3254 (time 1778621049.34 total 50.06 last 68.47)
 - open/close 3918 (time 1778621059.34 total 60.06 last 66.38)
total: 4500 open/close in 68.35 seconds: 65.84 ops/second
 - /unlink 1388 (time 1778621082.07 total 10.01 last 138.72)
 - /unlink 2502 (time 1778621092.08 total 20.01 last 111.34)
 - /unlink 3730 (time 1778621102.08 total 30.01 last 122.76)
total: 4500 /unlink in 35.11 seconds: 128.17 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 704 (time 1778621122.35 total 10.00 last 70.39)
 - open/close 1392 (time 1778621132.35 total 20.00 last 68.79)
 - open/close 2055 (time 1778621142.36 total 30.01 last 66.22)
 - open/close 2831 (time 1778621152.37 total 40.02 last 77.53)
 - open/close 3497 (time 1778621162.39 total 50.04 last 66.46)
 - open/close 4188 (time 1778621172.40 total 60.05 last 69.09)
total: 4500 open/close in 64.28 seconds: 70.01 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1228 (time 1778621191.91 total 10.00 last 122.79)
 - /unlink 2438 (time 1778621201.91 total 20.01 last 120.94)
 - /unlink 3540 (time 1778621211.91 total 30.01 last 110.18)
total: 4500 /unlink in 37.75 seconds: 119.21 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 699 (time 1778621234.73 total 10.01 last 69.84)
 - open/close 1383 (time 1778621244.73 total 20.01 last 68.39)
 - open/close 2041 (time 1778621254.74 total 30.02 last 65.74)
 - open/close 2646 (time 1778621264.76 total 40.04 last 60.36)
 - open/close 3354 (time 1778621274.79 total 50.07 last 70.62)
 - open/close 3977 (time 1778621284.79 total 60.07 last 62.30)
total: 4500 open/close in 69.26 seconds: 64.97 ops/second
 - /unlink 1163 (time 1778621309.29 total 10.01 last 116.19)
 - /unlink 2370 (time 1778621319.30 total 20.01 last 120.67)
 - /unlink 3553 (time 1778621329.31 total 30.02 last 118.19)
total: 4500 /unlink in 36.67 seconds: 122.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 694 (time 1778621350.87 total 10.00 last 69.39)
 - open/close 1368 (time 1778621360.91 total 20.03 last 67.19)
 - open/close 2120 (time 1778621370.91 total 30.04 last 75.16)
 - open/close 2761 (time 1778621380.92 total 40.04 last 64.06)
 - open/close 3428 (time 1778621390.92 total 50.05 last 66.67)
 - open/close 4069 (time 1778621400.92 total 60.05 last 64.09)
total: 4500 open/close in 65.69 seconds: 68.51 ops/second
 - /unlink 1209 (time 1778621421.04 total 10.01 last 120.82)
 - /unlink 2367 (time 1778621431.04 total 20.01 last 115.77)
 - /unlink 3660 (time 1778621441.04 total 30.01 last 129.29)
total: 4500 /unlink in 37.98 seconds: 118.48 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 632 (time 1778621464.87 total 10.01 last 63.14)
 - open/close 1271 (time 1778621474.88 total 20.02 last 63.83)
 - open/close 1958 (time 1778621484.88 total 30.02 last 68.70)
 - open/close 2682 (time 1778621494.88 total 40.02 last 72.39)
 - open/close 3373 (time 1778621504.90 total 50.04 last 68.97)
 - open/close 4039 (time 1778621514.91 total 60.04 last 66.57)
total: 4500 open/close in 66.39 seconds: 67.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1281 (time 1778621536.51 total 10.00 last 128.08)
 - /unlink 2535 (time 1778621546.53 total 20.01 last 125.25)
 - /unlink 3691 (time 1778621556.53 total 30.01 last 115.58)
total: 4500 /unlink in 36.73 seconds: 122.51 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 675 (time 1778621578.36 total 10.01 last 67.46)
 - open/close 1428 (time 1778621588.37 total 20.01 last 75.24)
 - open/close 2071 (time 1778621598.37 total 30.02 last 64.27)
 - open/close 2690 (time 1778621608.38 total 40.03 last 61.85)
 - open/close 3286 (time 1778621618.38 total 50.03 last 59.57)
 - open/close 3971 (time 1778621628.38 total 60.03 last 68.50)
total: 4500 open/close in 67.82 seconds: 66.35 ops/second
 - /unlink 1260 (time 1778621650.87 total 10.00 last 125.98)
 - /unlink 2303 (time 1778621660.87 total 20.01 last 104.26)
 - /unlink 3321 (time 1778621670.88 total 30.02 last 101.69)
total: 4500 /unlink in 39.81 seconds: 113.03 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 762 (time 1778621695.21 total 10.01 last 76.16)
 - open/close 1467 (time 1778621705.22 total 20.02 last 70.42)
 - open/close 2084 (time 1778621715.22 total 30.02 last 61.65)
 - open/close 2818 (time 1778621725.23 total 40.03 last 73.36)
 - open/close 3604 (time 1778621735.24 total 50.04 last 78.55)
 - open/close 4249 (time 1778621745.25 total 60.05 last 64.44)
total: 4500 open/close in 64.19 seconds: 70.11 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1354 (time 1778621764.52 total 10.00 last 135.34)
 - /unlink 2667 (time 1778621774.53 total 20.01 last 131.19)
 - /unlink 3768 (time 1778621784.53 total 30.01 last 110.08)
total: 4500 /unlink in 35.48 seconds: 126.84 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 661 (time 1778621804.68 total 10.02 last 66.00)
 - open/close 1420 (time 1778621814.69 total 20.03 last 75.81)
 - open/close 2172 (time 1778621824.70 total 30.04 last 75.13)
 - open/close 2828 (time 1778621834.70 total 40.04 last 65.59)
 - open/close 3543 (time 1778621844.71 total 50.05 last 71.45)
 - open/close 4238 (time 1778621854.71 total 60.05 last 69.50)
total: 4500 open/close in 64.88 seconds: 69.36 ops/second
 - /unlink 1373 (time 1778621874.84 total 10.00 last 137.25)
 - /unlink 2593 (time 1778621884.85 total 20.01 last 121.97)
 - /unlink 3740 (time 1778621894.85 total 30.01 last 114.63)
total: 4500 /unlink in 35.66 seconds: 126.19 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 718 (time 1778621915.25 total 10.01 last 71.75)
 - open/close 1281 (time 1778621925.27 total 20.02 last 56.22)
 - open/close 1810 (time 1778621935.27 total 30.03 last 52.86)
 - open/close 2500 (time 1778621945.28 total 40.04 last 68.94)
 - open/close 3215 (time 1778621955.28 total 50.04 last 71.49)
 - open/close 3803 (time 1778621965.29 total 60.05 last 58.75)
 - open/close 4279 (time 1778621975.30 total 70.06 last 47.56)
total: 4500 open/close in 73.86 seconds: 60.92 ops/second
 - /unlink 929 (time 1778621995.40 total 10.00 last 92.89)
 - /unlink 1942 (time 1778622005.42 total 20.01 last 101.19)
 - /unlink 3306 (time 1778622015.42 total 30.01 last 136.37)
 - /unlink 4371 (time 1778622025.42 total 40.02 last 106.49)
total: 4500 /unlink in 41.27 seconds: 109.04 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 488 (time 1778622043.08 total 10.01 last 48.77)
 - open/close 1185 (time 1778622053.08 total 20.01 last 69.70)
 - open/close 1785 (time 1778622063.10 total 30.02 last 59.89)
 - open/close 2397 (time 1778622073.12 total 40.04 last 61.11)
 - open/close 3033 (time 1778622083.12 total 50.04 last 63.58)
 - open/close 3735 (time 1778622093.13 total 60.05 last 70.15)
 - open/close 4403 (time 1778622103.14 total 70.06 last 66.73)
total: 4500 open/close in 71.26 seconds: 63.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1176 (time 1778622119.91 total 10.01 last 117.44)
 - /unlink 2499 (time 1778622129.91 total 20.02 last 132.26)
 - /unlink 3755 (time 1778622139.92 total 30.02 last 125.50)
total: 4500 /unlink in 36.01 seconds: 124.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 758 (time 1778622160.46 total 10.01 last 75.69)
 - open/close 1452 (time 1778622170.46 total 20.02 last 69.35)
 - open/close 2138 (time 1778622180.47 total 30.02 last 68.59)
 - open/close 2996 (time 1778622190.47 total 40.03 last 85.75)
 - open/close 3657 (time 1778622200.48 total 50.04 last 66.04)
 - open/close 4220 (time 1778622210.50 total 60.06 last 56.17)
total: 4500 open/close in 64.05 seconds: 70.26 ops/second
 - /unlink 1186 (time 1778622230.30 total 10.01 last 118.52)
 - /unlink 2254 (time 1778622240.30 total 20.01 last 106.80)
 - /unlink 3595 (time 1778622250.30 total 30.01 last 134.09)
total: 4500 /unlink in 37.86 seconds: 118.87 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 757 (time 1778622274.16 total 10.00 last 75.69)
 - open/close 1474 (time 1778622284.16 total 20.00 last 71.69)
 - open/close 2041 (time 1778622294.18 total 30.02 last 56.61)
 - open/close 2828 (time 1778622304.19 total 40.03 last 78.60)
 - open/close 3679 (time 1778622314.20 total 50.04 last 85.05)
 - open/close 4429 (time 1778622324.20 total 60.04 last 75.00)
total: 4500 open/close in 61.02 seconds: 73.75 ops/second
 - /unlink 1106 (time 1778622341.13 total 10.00 last 110.58)
 - /unlink 2324 (time 1778622351.13 total 20.01 last 121.70)
 - /unlink 3497 (time 1778622361.14 total 30.02 last 117.25)
total: 4500 /unlink in 36.97 seconds: 121.72 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 748 (time 1778622383.14 total 10.01 last 74.75)
 - open/close 1384 (time 1778622393.14 total 20.01 last 63.56)
 - open/close 2019 (time 1778622403.14 total 30.01 last 63.50)
 - open/close 2696 (time 1778622413.15 total 40.02 last 67.68)
 - open/close 3392 (time 1778622423.15 total 50.02 last 69.56)
 - open/close 4058 (time 1778622433.16 total 60.03 last 66.55)
total: 4500 open/close in 65.05 seconds: 69.18 ops/second
 - /unlink 1342 (time 1778622452.91 total 10.01 last 134.08)
 - /unlink 2296 (time 1778622462.91 total 20.01 last 95.40)
 - /unlink 3015 (time 1778622472.92 total 30.02 last 71.85)
 - /unlink 4093 (time 1778622482.92 total 40.02 last 107.75)
total: 4500 /unlink in 42.98 seconds: 104.70 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 702 (time 1778622502.16 total 10.01 last 70.13)
 - open/close 1350 (time 1778622512.16 total 20.02 last 64.77)
 - open/close 1981 (time 1778622522.16 total 30.02 last 63.09)
 - open/close 2692 (time 1778622532.18 total 40.03 last 71.01)
 - open/close 3272 (time 1778622542.18 total 50.03 last 57.97)
 - open/close 3808 (time 1778622552.19 total 60.04 last 53.57)
 - open/close 4319 (time 1778622562.19 total 70.04 last 51.10)
total: 4500 open/close in 74.01 seconds: 60.81 ops/second
 - /unlink 1075 (time 1778622581.95 total 10.01 last 107.39)
 - /unlink 2120 (time 1778622591.96 total 20.02 last 104.38)
 - /unlink 3416 (time 1778622601.97 total 30.03 last 129.51)
total: 4500 /unlink in 37.78 seconds: 119.10 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 594 (time 1778622625.94 total 10.01 last 59.34)
 - open/close 1145 (time 1778622635.95 total 20.01 last 55.08)
 - open/close 1874 (time 1778622645.95 total 30.01 last 72.89)
 - open/close 2568 (time 1778622655.95 total 40.02 last 69.36)
 - open/close 3388 (time 1778622665.96 total 50.02 last 81.95)
 - open/close 4132 (time 1778622675.96 total 60.03 last 74.38)
total: 4500 open/close in 65.04 seconds: 69.18 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1519 (time 1778622696.42 total 10.00 last 151.87)
 - /unlink 2666 (time 1778622706.42 total 20.00 last 114.68)
 - /unlink 3743 (time 1778622716.42 total 30.01 last 107.67)
total: 4500 /unlink in 37.10 seconds: 121.30 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 537 (time 1778622739.60 total 10.00 last 53.69)
 - open/close 1095 (time 1778622749.61 total 20.01 last 55.75)
 - open/close 1704 (time 1778622759.63 total 30.03 last 60.78)
 - open/close 2326 (time 1778622769.64 total 40.04 last 62.13)
 - open/close 2959 (time 1778622779.64 total 50.04 last 63.29)
 - open/close 3540 (time 1778622789.68 total 60.08 last 57.89)
 - open/close 4093 (time 1778622799.68 total 70.08 last 55.30)
total: 4500 open/close in 78.52 seconds: 57.31 ops/second
 - /unlink 1249 (time 1778622823.57 total 10.00 last 124.88)
 - /unlink 2483 (time 1778622833.58 total 20.01 last 123.30)
 - /unlink 3742 (time 1778622843.58 total 30.01 last 125.84)
total: 4500 /unlink in 37.12 seconds: 121.23 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 625 (time 1778622866.17 total 10.02 last 62.38)
 - open/close 1339 (time 1778622876.17 total 20.02 last 71.36)
 - open/close 2076 (time 1778622886.18 total 30.03 last 73.66)
 - open/close 2670 (time 1778622896.19 total 40.04 last 59.31)
 - open/close 3189 (time 1778622906.21 total 50.06 last 51.80)
 - open/close 3813 (time 1778622916.21 total 60.07 last 62.39)
 - open/close 4451 (time 1778622926.21 total 70.07 last 63.80)
total: 4500 open/close in 70.65 seconds: 63.69 ops/second
 - /unlink 1116 (time 1778622942.77 total 10.01 last 111.50)
 - /unlink 2298 (time 1778622952.77 total 20.01 last 118.19)
 - /unlink 3396 (time 1778622962.78 total 30.02 last 109.70)
 - /unlink 4458 (time 1778622972.79 total 40.03 last 106.09)
total: 4500 /unlink in 40.61 seconds: 110.81 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 94350 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg134-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg134-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server
stop mds service on oleg134-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server
stop mds service on oleg134-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server

unloading modules via unload_modules_local on: 'oleg134-server'
oleg134-server: oleg134-server.virtnet: executing unload_modules_local
oleg134-server: 
modules unloaded.