== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:19:07 (1789132747)
Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg133-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg133-server
oleg133-server: oleg133-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg133-server'
oleg133-server: oleg133-server.virtnet: executing load_modules_local
oleg133-server: Loading modules from /home/green/git/lustre-release/lustre
oleg133-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy
oleg133-server: detected 4 online CPUs by sysfs
oleg133-server: MODOPTS_LIBCFS=
oleg133-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg133-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all
pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg133-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all
pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg133-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all
pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg133-client.virtnet:  -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 596 (time 1789132986.05 total 10.00 last 59.58)
 - open/close 1235 (time 1789132996.05 total 20.01 last 63.87)
 - open/close 1937 (time 1789133006.07 total 30.02 last 70.11)
 - open/close 2577 (time 1789133016.07 total 40.02 last 63.98)
 - open/close 3231 (time 1789133026.07 total 50.02 last 65.39)
 - open/close 3947 (time 1789133036.07 total 60.03 last 71.58)
total: 4500 open/close in 67.03 seconds: 67.14 ops/second
 - /unlink 1370 (time 1789133058.03 total 10.00 last 136.98)
 - /unlink 2570 (time 1789133068.03 total 20.01 last 119.95)
 - /unlink 3670 (time 1789133078.03 total 30.01 last 109.97)
total: 4500 /unlink in 36.60 seconds: 122.95 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 661 (time 1789133100.68 total 10.01 last 66.06)
 - open/close 1218 (time 1789133110.69 total 20.01 last 55.65)
 - open/close 1888 (time 1789133120.71 total 30.03 last 66.89)
 - open/close 2629 (time 1789133130.72 total 40.04 last 74.05)
 - open/close 3111 (time 1789133140.72 total 50.04 last 48.17)
 - open/close 3608 (time 1789133150.72 total 60.05 last 49.69)
 - open/close 4165 (time 1789133160.73 total 70.05 last 55.65)
total: 4500 open/close in 76.17 seconds: 59.08 ops/second
 - /unlink 1283 (time 1789133184.36 total 10.00 last 128.27)
 - /unlink 2323 (time 1789133194.36 total 20.00 last 103.99)
 - /unlink 3401 (time 1789133204.36 total 30.01 last 107.74)
 - /unlink 4360 (time 1789133214.38 total 40.03 last 95.75)
total: 4500 /unlink in 42.09 seconds: 106.91 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 669 (time 1789133233.50 total 10.01 last 66.84)
 - open/close 1410 (time 1789133243.51 total 20.02 last 74.05)
 - open/close 2113 (time 1789133253.51 total 30.02 last 70.29)
 - open/close 2647 (time 1789133263.52 total 40.03 last 53.32)
 - open/close 3236 (time 1789133273.53 total 50.04 last 58.87)
 - open/close 3822 (time 1789133283.56 total 60.07 last 58.42)
total: 4500 open/close in 69.47 seconds: 64.77 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1430 (time 1789133307.48 total 10.01 last 142.88)
 - /unlink 2761 (time 1789133317.49 total 20.02 last 132.91)
 - /unlink 4156 (time 1789133327.50 total 30.03 last 139.44)
total: 4500 /unlink in 32.73 seconds: 137.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 886 (time 1789133344.85 total 10.00 last 88.58)
 - open/close 1712 (time 1789133354.86 total 20.01 last 82.56)
 - open/close 2539 (time 1789133364.87 total 30.02 last 82.56)
 - open/close 3254 (time 1789133374.88 total 40.03 last 71.44)
 - open/close 3978 (time 1789133384.88 total 50.03 last 72.39)
total: 4500 open/close in 57.19 seconds: 78.68 ops/second
 - /unlink 1435 (time 1789133407.11 total 10.02 last 143.26)
 - /unlink 3007 (time 1789133417.12 total 20.02 last 157.18)
 - /unlink 4260 (time 1789133427.12 total 30.03 last 125.21)
total: 4500 /unlink in 31.81 seconds: 141.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 741 (time 1789133444.45 total 10.01 last 74.00)
 - open/close 1520 (time 1789133454.46 total 20.03 last 77.81)
 - open/close 2167 (time 1789133464.47 total 30.03 last 64.64)
 - open/close 2662 (time 1789133474.47 total 40.04 last 49.47)
 - open/close 3309 (time 1789133484.48 total 50.04 last 64.69)
 - open/close 4079 (time 1789133494.48 total 60.05 last 76.97)
total: 4500 open/close in 65.70 seconds: 68.49 ops/second
 - /unlink 1788 (time 1789133515.54 total 10.00 last 178.77)
 - /unlink 3334 (time 1789133525.54 total 20.01 last 154.52)
total: 4500 /unlink in 29.61 seconds: 151.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 692 (time 1789133549.61 total 10.01 last 69.16)
 - open/close 1375 (time 1789133559.62 total 20.01 last 68.27)
 - open/close 2012 (time 1789133569.63 total 30.02 last 63.64)
 - open/close 2589 (time 1789133579.63 total 40.02 last 57.69)
 - open/close 3008 (time 1789133589.66 total 50.05 last 41.77)
 - open/close 3349 (time 1789133599.68 total 60.07 last 34.04)
 - open/close 3836 (time 1789133609.69 total 70.08 last 48.64)
 - open/close 4184 (time 1789133619.70 total 80.10 last 34.75)
total: 4500 open/close in 88.28 seconds: 50.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 944 (time 1789133644.03 total 10.01 last 94.33)
 - /unlink 1774 (time 1789133654.03 total 20.01 last 82.99)
 - /unlink 2751 (time 1789133664.04 total 30.01 last 97.67)
 - /unlink 4004 (time 1789133674.04 total 40.02 last 125.26)
total: 4500 /unlink in 44.15 seconds: 101.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 758 (time 1789133693.28 total 10.01 last 75.75)
 - open/close 1460 (time 1789133703.28 total 20.01 last 70.19)
 - open/close 2104 (time 1789133713.28 total 30.01 last 64.38)
 - open/close 2676 (time 1789133723.29 total 40.02 last 57.14)
 - open/close 3384 (time 1789133733.30 total 50.03 last 70.73)
 - open/close 4102 (time 1789133743.30 total 60.03 last 71.79)
total: 4500 open/close in 67.55 seconds: 66.62 ops/second
 - /unlink 1196 (time 1789133766.74 total 10.01 last 119.50)
 - /unlink 2466 (time 1789133776.76 total 20.02 last 126.83)
 - /unlink 3670 (time 1789133786.76 total 30.02 last 120.37)
total: 4500 /unlink in 36.29 seconds: 123.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 639 (time 1789133807.48 total 10.01 last 63.84)
 - open/close 1326 (time 1789133817.48 total 20.02 last 68.65)
 - open/close 2023 (time 1789133827.49 total 30.02 last 69.66)
 - open/close 2781 (time 1789133837.50 total 40.03 last 75.75)
 - open/close 3505 (time 1789133847.50 total 50.03 last 72.37)
 - open/close 4260 (time 1789133857.50 total 60.04 last 75.48)
total: 4500 open/close in 63.47 seconds: 70.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1084 (time 1789133877.37 total 10.02 last 108.23)
 - /unlink 1855 (time 1789133887.39 total 20.03 last 76.96)
 - /unlink 2906 (time 1789133897.39 total 30.03 last 105.10)
 - /unlink 4137 (time 1789133907.39 total 40.04 last 123.06)
total: 4500 /unlink in 42.59 seconds: 105.66 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 603 (time 1789133926.81 total 10.01 last 60.26)
 - open/close 1151 (time 1789133936.81 total 20.01 last 54.75)
 - open/close 1757 (time 1789133946.83 total 30.03 last 60.53)
 - open/close 2277 (time 1789133956.84 total 40.04 last 51.93)
 - open/close 2952 (time 1789133966.84 total 50.04 last 67.49)
 - open/close 3626 (time 1789133976.86 total 60.06 last 67.29)
 - open/close 4188 (time 1789133986.87 total 70.07 last 56.14)
total: 4500 open/close in 74.64 seconds: 60.29 ops/second
 - /unlink 1009 (time 1789134006.65 total 10.02 last 100.69)
 - /unlink 1864 (time 1789134016.66 total 20.02 last 85.50)
 - /unlink 2843 (time 1789134026.66 total 30.02 last 97.88)
 - /unlink 4080 (time 1789134036.67 total 40.04 last 123.48)
total: 4500 /unlink in 44.81 seconds: 100.41 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 585 (time 1789134058.73 total 10.01 last 58.42)
 - open/close 1017 (time 1789134068.73 total 20.02 last 43.17)
 - open/close 1574 (time 1789134078.74 total 30.02 last 55.69)
 - open/close 2232 (time 1789134088.74 total 40.03 last 65.77)
 - open/close 2912 (time 1789134098.77 total 50.05 last 67.83)
 - open/close 3502 (time 1789134108.77 total 60.06 last 58.97)
 - open/close 4189 (time 1789134118.78 total 70.06 last 68.66)
total: 4500 open/close in 73.81 seconds: 60.96 ops/second
 - /unlink 1443 (time 1789134136.82 total 10.01 last 144.22)
 - /unlink 2654 (time 1789134146.82 total 20.01 last 121.09)
 - /unlink 4020 (time 1789134156.82 total 30.01 last 136.56)
total: 4500 /unlink in 36.12 seconds: 124.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 505 (time 1789134180.24 total 10.01 last 50.44)
 - open/close 1084 (time 1789134190.26 total 20.03 last 57.79)
 - open/close 1617 (time 1789134200.28 total 30.04 last 53.23)
 - open/close 2300 (time 1789134210.29 total 40.06 last 68.22)
 - open/close 3050 (time 1789134220.29 total 50.06 last 74.95)
 - open/close 3859 (time 1789134230.30 total 60.07 last 80.81)
total: 4500 open/close in 68.75 seconds: 65.46 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1242 (time 1789134254.10 total 10.00 last 124.20)
 - /unlink 2556 (time 1789134264.10 total 20.00 last 131.35)
 - /unlink 3728 (time 1789134274.11 total 30.01 last 117.10)
total: 4500 /unlink in 35.88 seconds: 125.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 767 (time 1789134294.68 total 10.00 last 76.70)
 - open/close 1414 (time 1789134304.69 total 20.01 last 64.65)
 - open/close 1900 (time 1789134314.71 total 30.03 last 48.49)
 - open/close 2418 (time 1789134324.72 total 40.04 last 51.76)
 - open/close 2966 (time 1789134334.72 total 50.04 last 54.79)
 - open/close 3493 (time 1789134344.74 total 60.06 last 52.62)
 - open/close 4020 (time 1789134354.75 total 70.07 last 52.64)
total: 4500 open/close in 79.59 seconds: 56.54 ops/second
 - /unlink 1028 (time 1789134379.96 total 10.00 last 102.78)
 - /unlink 2159 (time 1789134389.97 total 20.01 last 113.00)
 - /unlink 3288 (time 1789134399.97 total 30.01 last 112.89)
 - /unlink 4404 (time 1789134409.97 total 40.02 last 111.53)
total: 4500 /unlink in 41.00 seconds: 109.76 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 472 (time 1789134428.87 total 10.00 last 47.19)
 - open/close 1022 (time 1789134438.88 total 20.01 last 54.97)
 - open/close 1587 (time 1789134448.90 total 30.03 last 56.40)
 - open/close 2410 (time 1789134458.90 total 40.03 last 82.26)
 - open/close 3066 (time 1789134468.90 total 50.03 last 65.59)
 - open/close 3831 (time 1789134478.91 total 60.04 last 76.41)
total: 4500 open/close in 68.09 seconds: 66.09 ops/second
 - /unlink 1316 (time 1789134502.39 total 10.00 last 131.54)
 - /unlink 2404 (time 1789134512.39 total 20.00 last 108.80)
 - /unlink 3561 (time 1789134522.40 total 30.01 last 115.64)
total: 4500 /unlink in 38.91 seconds: 115.65 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 655 (time 1789134547.62 total 10.01 last 65.41)
 - open/close 1131 (time 1789134557.62 total 20.02 last 47.57)
 - open/close 1635 (time 1789134567.62 total 30.02 last 50.40)
 - open/close 2425 (time 1789134577.63 total 40.03 last 78.94)
 - open/close 3116 (time 1789134587.65 total 50.05 last 68.95)
 - open/close 3732 (time 1789134597.67 total 60.06 last 61.52)
 - open/close 4416 (time 1789134607.67 total 70.07 last 68.38)
total: 4500 open/close in 71.30 seconds: 63.11 ops/second
 - /unlink 1226 (time 1789134624.24 total 10.01 last 122.51)
 - /unlink 2624 (time 1789134634.24 total 20.01 last 139.73)
 - /unlink 3799 (time 1789134644.24 total 30.02 last 117.47)
total: 4500 /unlink in 37.13 seconds: 121.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 712 (time 1789134668.39 total 10.01 last 71.14)
 - open/close 1353 (time 1789134678.42 total 20.05 last 63.86)
 - open/close 1934 (time 1789134688.42 total 30.05 last 58.10)
 - open/close 2718 (time 1789134698.43 total 40.05 last 78.35)
 - open/close 3431 (time 1789134708.43 total 50.05 last 71.28)
 - open/close 4191 (time 1789134718.44 total 60.06 last 75.93)
total: 4500 open/close in 64.15 seconds: 70.15 ops/second
 - /unlink 1246 (time 1789134737.57 total 10.02 last 124.37)
 - /unlink 2434 (time 1789134747.58 total 20.02 last 118.74)
 - /unlink 3795 (time 1789134757.58 total 30.03 last 136.07)
total: 4500 /unlink in 34.98 seconds: 128.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 503 (time 1789134779.72 total 10.00 last 50.28)
 - open/close 1086 (time 1789134789.73 total 20.02 last 58.22)
 - open/close 1794 (time 1789134799.75 total 30.03 last 70.70)
 - open/close 2474 (time 1789134809.75 total 40.04 last 67.97)
 - open/close 3038 (time 1789134819.76 total 50.04 last 56.39)
 - open/close 3794 (time 1789134829.76 total 60.05 last 75.55)
 - open/close 4463 (time 1789134839.77 total 70.05 last 66.85)
total: 4500 open/close in 70.78 seconds: 63.57 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1423 (time 1789134857.13 total 10.00 last 142.24)
 - /unlink 2903 (time 1789134867.14 total 20.01 last 147.94)
 - /unlink 4242 (time 1789134877.14 total 30.01 last 133.90)
total: 4500 /unlink in 31.71 seconds: 141.90 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 692 (time 1789134894.49 total 10.00 last 69.18)
 - open/close 1350 (time 1789134904.50 total 20.01 last 65.73)
 - open/close 1818 (time 1789134914.52 total 30.03 last 46.73)
 - open/close 2576 (time 1789134924.53 total 40.04 last 75.73)
 - open/close 3093 (time 1789134934.54 total 50.05 last 51.65)
 - open/close 3681 (time 1789134944.56 total 60.08 last 58.63)
 - open/close 4083 (time 1789134954.59 total 70.10 last 40.09)
total: 4500 open/close in 79.75 seconds: 56.43 ops/second
 - /unlink 1067 (time 1789134981.34 total 10.01 last 106.63)
 - /unlink 2028 (time 1789134991.35 total 20.02 last 95.99)
 - /unlink 3229 (time 1789135001.36 total 30.03 last 119.98)
 - /unlink 4370 (time 1789135011.37 total 40.03 last 114.06)
total: 4500 /unlink in 41.24 seconds: 109.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 549 (time 1789135028.46 total 10.02 last 54.81)
 - open/close 1156 (time 1789135038.46 total 20.02 last 60.69)
 - open/close 1736 (time 1789135048.47 total 30.02 last 57.99)
 - open/close 2382 (time 1789135058.47 total 40.03 last 64.55)
 - open/close 3150 (time 1789135068.48 total 50.03 last 76.78)
 - open/close 3821 (time 1789135078.49 total 60.04 last 67.02)
 - open/close 4459 (time 1789135088.50 total 70.06 last 63.72)
total: 4500 open/close in 70.78 seconds: 63.58 ops/second
 - /unlink 984 (time 1789135105.46 total 10.01 last 98.33)
 - /unlink 2122 (time 1789135115.48 total 20.03 last 113.57)
 - /unlink 3230 (time 1789135125.48 total 30.03 last 110.76)
 - /unlink 4352 (time 1789135135.49 total 40.04 last 112.13)
total: 4500 /unlink in 41.32 seconds: 108.91 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95387 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg133-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg133-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server
stop mds service on oleg133-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server
stop mds service on oleg133-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server

unloading modules via unload_modules_local on: 'oleg133-server'
oleg133-server: oleg133-server.virtnet: executing unload_modules_local
oleg133-server: 
modules unloaded.