== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:23:56 (1783517036) start mds service on oleg346-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg346-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg346-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50096 529 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1671 (time 1783517086.29 total 10.01 last 167.00) - open/close/unlink 3420 (time 1783517096.30 total 20.02 last 174.73) - open/close/unlink 5329 (time 1783517106.31 total 30.02 last 190.76) - open/close/unlink 7071 (time 1783517116.31 total 40.03 last 174.13) - open/close/unlink 8411 (time 1783517126.32 total 50.04 last 133.90) - open/close/unlink 10000 (time 1783517135.71 total 59.42 last 169.23) - open/close/unlink 11850 (time 1783517145.72 total 69.43 last 184.90) - open/close/unlink 13761 (time 1783517155.72 total 79.43 last 191.05) - open/close/unlink 15191 (time 1783517165.73 total 89.44 last 142.90) - open/close/unlink 16877 (time 1783517175.73 total 99.44 last 168.59) - open/close/unlink 18814 (time 1783517185.73 total 109.44 last 193.64) - open/close/unlink 20000 (time 1783517191.97 total 115.69 last 189.94) - open/close/unlink 21690 (time 1783517201.97 total 125.69 last 169.00) - open/close/unlink 23095 (time 1783517211.98 total 135.69 last 140.47) - open/close/unlink 24046 (time 1783517221.98 total 145.69 last 95.08) - open/close/unlink 24934 (time 1783517231.98 total 155.70 last 88.74) - open/close/unlink 25820 (time 1783517241.99 total 165.70 last 88.59) - open/close/unlink 26814 (time 1783517251.99 total 175.70 last 99.40) - open/close/unlink 27836 (time 1783517261.99 total 185.70 last 102.20) - open/close/unlink 29207 (time 1783517271.99 total 195.70 last 137.06) - open/close/unlink 30000 (time 1783517277.70 total 201.41 last 138.95) - open/close/unlink 31300 (time 1783517287.70 total 211.41 last 130.00) - open/close/unlink 32795 (time 1783517297.70 total 221.41 last 149.46) - open/close/unlink 34651 (time 1783517307.70 total 231.41 last 185.58) - open/close/unlink 36503 (time 1783517317.72 total 241.44 last 184.74) - open/close/unlink 38340 (time 1783517327.73 total 251.44 last 183.67) - open/close/unlink 40000 (time 1783517335.79 total 259.51 last 205.77) - open/close/unlink 41993 (time 1783517345.80 total 269.51 last 199.23) - open/close/unlink 43702 (time 1783517355.85 total 279.57 last 169.95) - open/close/unlink 45762 (time 1783517365.86 total 289.57 last 205.90) - open/close/unlink 47767 (time 1783517375.86 total 299.58 last 200.42) - open/close/unlink 49880 (time 1783517385.86 total 309.58 last 211.28) - open/close/unlink 52068 (time 1783517395.87 total 319.58 last 218.71) - open/close/unlink 54107 (time 1783517405.87 total 329.59 last 203.80) - open/close/unlink 56196 (time 1783517415.87 total 339.59 last 208.88) - open/close/unlink 58204 (time 1783517425.87 total 349.59 last 200.78) - open/close/unlink 60000 (time 1783517434.33 total 358.04 last 212.44) - open/close/unlink 62092 (time 1783517444.33 total 368.04 last 209.19) - open/close/unlink 64247 (time 1783517454.33 total 378.04 last 215.48) total: 64768 open/close/unlink in 380.44 seconds: 170.24 ops/second - open/close/unlink 1662 (time 1783517086.18 total 10.01 last 166.00) - open/close/unlink 3417 (time 1783517096.18 total 20.02 last 175.42) - open/close/unlink 5312 (time 1783517106.19 total 30.02 last 189.38) - open/close/unlink 7050 (time 1783517116.19 total 40.02 last 173.78) - open/close/unlink 8384 (time 1783517126.19 total 50.02 last 133.40) - open/close/unlink 10000 (time 1783517135.78 total 59.62 last 168.47) - open/close/unlink 11831 (time 1783517145.79 total 69.62 last 183.05) - open/close/unlink 13739 (time 1783517155.79 total 79.62 last 190.78) - open/close/unlink 15157 (time 1783517165.79 total 89.62 last 141.74) - open/close/unlink 16837 (time 1783517175.79 total 99.63 last 167.97) - open/close/unlink 18797 (time 1783517185.80 total 109.63 last 195.96) - open/close/unlink 20000 (time 1783517192.14 total 115.97 last 189.60) - open/close/unlink 21698 (time 1783517202.14 total 125.97 last 169.78) - open/close/unlink 23104 (time 1783517212.14 total 135.97 last 140.59) - open/close/unlink 24043 (time 1783517222.15 total 145.98 last 93.82) - open/close/unlink 24928 (time 1783517232.15 total 155.99 last 88.48) - open/close/unlink 25825 (time 1783517242.16 total 165.99 last 89.66) - open/close/unlink 26819 (time 1783517252.16 total 175.99 last 99.39) - open/close/unlink 27858 (time 1783517262.16 total 186.00 last 103.84) - open/close/unlink 29228 (time 1783517272.17 total 196.00 last 136.92) - open/close/unlink 30000 (time 1783517277.70 total 201.53 last 139.56) - open/close/unlink 31297 (time 1783517287.70 total 211.54 last 129.68) - open/close/unlink 32787 (time 1783517297.71 total 221.54 last 148.94) - open/close/unlink 34631 (time 1783517307.71 total 231.54 last 184.35) - open/close/unlink 36452 (time 1783517317.71 total 241.55 last 182.06) - open/close/unlink 38302 (time 1783517327.72 total 251.55 last 184.94) - open/close/unlink 40000 (time 1783517335.97 total 259.80 last 205.80) - open/close/unlink 41974 (time 1783517345.97 total 269.80 last 197.34) - open/close/unlink 43655 (time 1783517355.97 total 279.80 last 168.09) - open/close/unlink 45714 (time 1783517365.97 total 289.81 last 205.86) - open/close/unlink 47716 (time 1783517375.97 total 299.81 last 200.18) - open/close/unlink 49842 (time 1783517385.98 total 309.81 last 212.52) - open/close/unlink 52046 (time 1783517395.98 total 319.81 last 220.29) - open/close/unlink 54063 (time 1783517405.99 total 329.82 last 201.61) - open/close/unlink 56178 (time 1783517415.99 total 339.82 last 211.47) - open/close/unlink 58174 (time 1783517425.99 total 349.82 last 199.57) - open/close/unlink 60000 (time 1783517434.67 total 358.51 last 210.28) - open/close/unlink 62123 (time 1783517444.68 total 368.51 last 212.26) - open/close/unlink 64291 (time 1783517454.68 total 378.51 last 216.79) total: 64768 open/close/unlink in 381.20 seconds: 169.91 ops/second - open/close/unlink 1659 (time 1783517086.29 total 10.00 last 165.89) - open/close/unlink 3410 (time 1783517096.29 total 20.00 last 175.08) - open/close/unlink 5293 (time 1783517106.30 total 30.01 last 188.09) - open/close/unlink 7050 (time 1783517116.31 total 40.03 last 175.47) - open/close/unlink 8391 (time 1783517126.32 total 50.03 last 134.00) - open/close/unlink 10000 (time 1783517135.81 total 59.52 last 169.54) - open/close/unlink 11812 (time 1783517145.81 total 69.53 last 181.14) - open/close/unlink 13739 (time 1783517155.82 total 79.53 last 192.59) - open/close/unlink 15152 (time 1783517165.82 total 89.53 last 141.29) - open/close/unlink 16831 (time 1783517175.82 total 99.53 last 167.88) - open/close/unlink 18729 (time 1783517185.82 total 109.54 last 189.73) - open/close/unlink 20000 (time 1783517192.66 total 116.38 last 185.87) - open/close/unlink 21703 (time 1783517202.66 total 126.38 last 170.29) - open/close/unlink 23093 (time 1783517212.67 total 136.38 last 138.91) - open/close/unlink 24023 (time 1783517222.67 total 146.39 last 92.96) - open/close/unlink 24904 (time 1783517232.68 total 156.40 last 88.03) - open/close/unlink 25796 (time 1783517242.69 total 166.41 last 89.09) - open/close/unlink 26808 (time 1783517252.70 total 176.42 last 101.12) - open/close/unlink 27825 (time 1783517262.70 total 186.42 last 101.67) - open/close/unlink 29219 (time 1783517272.71 total 196.42 last 139.37) - open/close/unlink 30000 (time 1783517278.30 total 202.02 last 139.51) - open/close/unlink 31309 (time 1783517288.32 total 212.03 last 130.76) - open/close/unlink 32818 (time 1783517298.32 total 222.03 last 150.86) - open/close/unlink 34690 (time 1783517308.32 total 232.04 last 187.13) - open/close/unlink 36519 (time 1783517318.32 total 242.04 last 182.86) - open/close/unlink 38401 (time 1783517328.32 total 252.04 last 188.19) - open/close/unlink 40000 (time 1783517336.07 total 259.78 last 206.46) - open/close/unlink 42001 (time 1783517346.07 total 269.79 last 200.02) - open/close/unlink 43683 (time 1783517356.08 total 279.79 last 168.16) - open/close/unlink 45766 (time 1783517366.08 total 289.79 last 208.25) - open/close/unlink 47752 (time 1783517376.08 total 299.79 last 198.57) - open/close/unlink 49886 (time 1783517386.08 total 309.79 last 213.40) - open/close/unlink 52087 (time 1783517396.08 total 319.80 last 220.05) - open/close/unlink 54082 (time 1783517406.08 total 329.80 last 199.44) - open/close/unlink 56194 (time 1783517416.09 total 339.80 last 211.13) - open/close/unlink 58185 (time 1783517426.09 total 349.80 last 199.08) - open/close/unlink 60000 (time 1783517434.69 total 358.40 last 211.09) - open/close/unlink 62108 (time 1783517444.69 total 368.40 last 210.77) - open/close/unlink 64271 (time 1783517454.69 total 378.41 last 216.22) total: 64768 open/close/unlink in 381.23 seconds: 169.89 ops/second - open/close/unlink 1653 (time 1783517086.35 total 10.02 last 165.01) - open/close/unlink 3393 (time 1783517096.35 total 20.02 last 173.92) - open/close/unlink 5294 (time 1783517106.35 total 30.03 last 190.05) - open/close/unlink 7029 (time 1783517116.36 total 40.03 last 173.40) - open/close/unlink 8370 (time 1783517126.36 total 50.03 last 134.10) - open/close/unlink 10000 (time 1783517135.99 total 59.67 last 169.17) - open/close/unlink 11828 (time 1783517146.00 total 69.67 last 182.78) - open/close/unlink 13754 (time 1783517156.00 total 79.67 last 192.50) - open/close/unlink 15160 (time 1783517166.01 total 89.68 last 140.52) - open/close/unlink 16845 (time 1783517176.01 total 99.68 last 168.41) - open/close/unlink 18781 (time 1783517186.01 total 109.68 last 193.58) - open/close/unlink 20000 (time 1783517192.68 total 116.35 last 182.81) - open/close/unlink 21710 (time 1783517202.68 total 126.36 last 170.95) - open/close/unlink 23092 (time 1783517212.68 total 136.36 last 138.19) - open/close/unlink 24017 (time 1783517222.69 total 146.36 last 92.46) - open/close/unlink 24898 (time 1783517232.69 total 156.37 last 88.07) - open/close/unlink 25791 (time 1783517242.69 total 166.37 last 89.29) - open/close/unlink 26796 (time 1783517252.71 total 176.38 last 100.37) - open/close/unlink 27820 (time 1783517262.72 total 186.39 last 102.30) - open/close/unlink 29206 (time 1783517272.72 total 196.39 last 138.59) - open/close/unlink 30000 (time 1783517278.34 total 202.02 last 141.13) - open/close/unlink 31302 (time 1783517288.35 total 212.02 last 130.12) - open/close/unlink 32808 (time 1783517298.35 total 222.02 last 150.55) - open/close/unlink 34677 (time 1783517308.36 total 232.03 last 186.86) - open/close/unlink 36487 (time 1783517318.36 total 242.03 last 180.97) - open/close/unlink 38363 (time 1783517328.36 total 252.03 last 187.53) - open/close/unlink 40000 (time 1783517336.22 total 259.89 last 208.28) - open/close/unlink 41959 (time 1783517346.22 total 269.89 last 195.89) - open/close/unlink 43664 (time 1783517356.23 total 279.90 last 170.38) - open/close/unlink 45734 (time 1783517366.23 total 289.90 last 206.98) - open/close/unlink 47696 (time 1783517376.23 total 299.90 last 196.14) - open/close/unlink 49836 (time 1783517386.23 total 309.90 last 213.99) - open/close/unlink 52051 (time 1783517396.23 total 319.91 last 221.46) - open/close/unlink 54047 (time 1783517406.24 total 329.91 last 199.56) - open/close/unlink 56168 (time 1783517416.24 total 339.91 last 212.03) - open/close/unlink 58160 (time 1783517426.25 total 349.92 last 199.06) - open/close/unlink 60000 (time 1783517435.03 total 358.70 last 209.47) - open/close/unlink 62113 (time 1783517445.03 total 368.70 last 211.24) - open/close/unlink 64321 (time 1783517455.03 total 378.71 last 220.77) total: 64768 open/close/unlink in 381.55 seconds: 169.75 ops/second - open/close/unlink 1682 (time 1783517086.39 total 10.01 last 168.10) - open/close/unlink 3415 (time 1783517096.40 total 20.01 last 173.20) - open/close/unlink 5301 (time 1783517106.41 total 30.02 last 188.40) - open/close/unlink 7034 (time 1783517116.41 total 40.02 last 173.28) - open/close/unlink 8376 (time 1783517126.41 total 50.03 last 134.16) - open/close/unlink 10000 (time 1783517135.96 total 59.57 last 170.22) - open/close/unlink 11834 (time 1783517145.96 total 69.57 last 183.34) - open/close/unlink 13753 (time 1783517155.96 total 79.57 last 191.86) - open/close/unlink 15168 (time 1783517165.97 total 89.58 last 141.38) - open/close/unlink 16849 (time 1783517175.97 total 99.58 last 168.06) - open/close/unlink 18775 (time 1783517185.97 total 109.58 last 192.59) - open/close/unlink 20000 (time 1783517192.60 total 116.21 last 184.74) - open/close/unlink 21700 (time 1783517202.60 total 126.22 last 169.97) - open/close/unlink 23093 (time 1783517212.61 total 136.22 last 139.21) - open/close/unlink 24028 (time 1783517222.62 total 146.23 last 93.46) - open/close/unlink 24909 (time 1783517232.62 total 156.23 last 88.07) - open/close/unlink 25797 (time 1783517242.63 total 166.25 last 88.66) - open/close/unlink 26796 (time 1783517252.64 total 176.25 last 99.87) - open/close/unlink 27815 (time 1783517262.65 total 186.26 last 101.77) - open/close/unlink 29196 (time 1783517272.67 total 196.28 last 137.91) - open/close/unlink 30000 (time 1783517278.42 total 202.04 last 139.63) - open/close/unlink 31298 (time 1783517288.43 total 212.04 last 129.78) - open/close/unlink 32807 (time 1783517298.43 total 222.04 last 150.89) - open/close/unlink 34672 (time 1783517308.43 total 232.04 last 186.38) - open/close/unlink 36519 (time 1783517318.44 total 242.05 last 184.64) - open/close/unlink 38396 (time 1783517328.44 total 252.05 last 187.65) - open/close/unlink 40000 (time 1783517336.18 total 259.79 last 207.15) - open/close/unlink 41984 (time 1783517346.18 total 269.80 last 198.34) - open/close/unlink 43688 (time 1783517356.19 total 279.80 last 170.35) - open/close/unlink 45742 (time 1783517366.19 total 289.80 last 205.33) - open/close/unlink 47756 (time 1783517376.19 total 299.81 last 201.32) - open/close/unlink 49890 (time 1783517386.20 total 309.81 last 213.34) - open/close/unlink 52075 (time 1783517396.20 total 319.81 last 218.45) - open/close/unlink 54071 (time 1783517406.20 total 329.81 last 199.57) - open/close/unlink 56182 (time 1783517416.20 total 339.82 last 211.05) - open/close/unlink 58184 (time 1783517426.21 total 349.82 last 200.06) - open/close/unlink 60000 (time 1783517435.01 total 358.62 last 206.45) - open/close/unlink 62123 (time 1783517445.01 total 368.62 last 212.30) - open/close/unlink 64296 (time 1783517455.01 total 378.62 last 217.22) total: 64768 open/close/unlink in 381.69 seconds: 169.69 ops/second Waiting for MDT destroys to complete oleg346-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg346-server unloading modules via unload_modules_local on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2