== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:45:43 (1781660743) start mds service on oleg357-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1399 (time 1781660771.19 total 10.00 last 139.84) - open/close/unlink 2880 (time 1781660781.20 total 20.02 last 147.90) - open/close/unlink 4331 (time 1781660791.20 total 30.02 last 145.04) - open/close/unlink 5808 (time 1781660801.21 total 40.02 last 147.67) - open/close/unlink 7343 (time 1781660811.21 total 50.02 last 153.49) - open/close/unlink 8741 (time 1781660821.21 total 60.03 last 139.76) - open/close/unlink 10000 (time 1781660828.53 total 67.35 last 171.88) - open/close/unlink 11466 (time 1781660838.54 total 77.36 last 146.48) - open/close/unlink 12937 (time 1781660848.54 total 87.36 last 147.08) - open/close/unlink 14401 (time 1781660858.54 total 97.36 last 146.40) - open/close/unlink 15958 (time 1781660868.55 total 107.36 last 155.68) - open/close/unlink 17223 (time 1781660878.55 total 117.37 last 126.43) - open/close/unlink 18435 (time 1781660888.56 total 127.38 last 121.12) - open/close/unlink 19754 (time 1781660898.56 total 137.38 last 131.88) - open/close/unlink 21188 (time 1781660908.57 total 147.39 last 143.28) - open/close/unlink 22501 (time 1781660918.57 total 157.39 last 131.23) - open/close/unlink 23856 (time 1781660928.58 total 167.40 last 135.39) - open/close/unlink 24889 (time 1781660938.59 total 177.40 last 103.24) - open/close/unlink 26041 (time 1781660948.59 total 187.41 last 115.13) - open/close/unlink 27406 (time 1781660958.59 total 197.41 last 136.50) - open/close/unlink 28806 (time 1781660968.60 total 207.42 last 139.92) - open/close/unlink 30000 (time 1781660977.66 total 216.48 last 131.72) - open/close/unlink 31196 (time 1781660987.66 total 226.48 last 119.58) - open/close/unlink 32430 (time 1781660997.67 total 236.48 last 123.39) - open/close/unlink 33637 (time 1781661007.67 total 246.49 last 120.64) - open/close/unlink 34948 (time 1781661017.67 total 256.49 last 131.05) - open/close/unlink 36188 (time 1781661027.68 total 266.50 last 123.92) - open/close/unlink 37518 (time 1781661037.68 total 276.50 last 132.99) - open/close/unlink 38842 (time 1781661047.69 total 286.51 last 132.31) - open/close/unlink 40000 (time 1781661055.09 total 293.91 last 156.47) - open/close/unlink 41209 (time 1781661065.11 total 303.93 last 120.66) - open/close/unlink 42605 (time 1781661075.12 total 313.94 last 139.47) - open/close/unlink 43778 (time 1781661085.12 total 323.94 last 117.23) - open/close/unlink 44986 (time 1781661095.13 total 333.95 last 120.72) - open/close/unlink 46151 (time 1781661105.14 total 343.95 last 116.43) - open/close/unlink 47328 (time 1781661115.14 total 353.96 last 117.64) - open/close/unlink 48787 (time 1781661125.14 total 363.96 last 145.88) - open/close/unlink 50000 (time 1781661134.62 total 373.44 last 127.99) - open/close/unlink 51198 (time 1781661144.62 total 383.44 last 119.79) - open/close/unlink 52285 (time 1781661154.62 total 393.44 last 108.70) - open/close/unlink 53456 (time 1781661164.63 total 403.45 last 117.03) - open/close/unlink 54580 (time 1781661174.63 total 413.45 last 112.34) - open/close/unlink 55723 (time 1781661184.64 total 423.46 last 114.23) - open/close/unlink 56881 (time 1781661194.64 total 433.46 last 115.79) - open/close/unlink 57978 (time 1781661204.65 total 443.47 last 109.59) - open/close/unlink 59186 (time 1781661214.70 total 453.52 last 120.20) - open/close/unlink 60000 (time 1781661221.54 total 460.36 last 119.03) - open/close/unlink 61234 (time 1781661231.54 total 470.36 last 123.38) - open/close/unlink 62569 (time 1781661241.54 total 480.36 last 133.47) - open/close/unlink 63933 (time 1781661251.54 total 490.36 last 136.40) total: 64768 open/close/unlink in 496.11 seconds: 130.55 ops/second - open/close/unlink 1400 (time 1781660771.23 total 10.00 last 139.95) - open/close/unlink 2869 (time 1781660781.23 total 20.01 last 146.85) - open/close/unlink 4304 (time 1781660791.23 total 30.01 last 143.44) - open/close/unlink 5766 (time 1781660801.23 total 40.01 last 146.18) - open/close/unlink 7308 (time 1781660811.24 total 50.02 last 154.11) - open/close/unlink 8716 (time 1781660821.24 total 60.02 last 140.79) - open/close/unlink 10000 (time 1781660828.74 total 67.52 last 171.25) - open/close/unlink 11465 (time 1781660838.74 total 77.52 last 146.47) - open/close/unlink 12928 (time 1781660848.74 total 87.52 last 146.27) - open/close/unlink 14407 (time 1781660858.75 total 97.52 last 147.85) - open/close/unlink 15983 (time 1781660868.75 total 107.53 last 157.57) - open/close/unlink 17223 (time 1781660878.75 total 117.53 last 123.93) - open/close/unlink 18428 (time 1781660888.75 total 127.53 last 120.49) - open/close/unlink 19766 (time 1781660898.76 total 137.54 last 133.76) - open/close/unlink 21217 (time 1781660908.76 total 147.54 last 145.10) - open/close/unlink 22528 (time 1781660918.76 total 157.54 last 131.09) - open/close/unlink 23886 (time 1781660928.76 total 167.54 last 135.72) - open/close/unlink 24922 (time 1781660938.77 total 177.55 last 103.54) - open/close/unlink 26067 (time 1781660948.77 total 187.55 last 114.48) - open/close/unlink 27457 (time 1781660958.78 total 197.55 last 138.95) - open/close/unlink 28840 (time 1781660968.78 total 207.56 last 138.19) - open/close/unlink 30000 (time 1781660977.57 total 216.35 last 132.05) - open/close/unlink 31186 (time 1781660987.58 total 226.35 last 118.51) - open/close/unlink 32416 (time 1781660997.58 total 236.36 last 122.98) - open/close/unlink 33607 (time 1781661007.58 total 246.36 last 119.10) - open/close/unlink 34945 (time 1781661017.58 total 256.36 last 133.73) - open/close/unlink 36186 (time 1781661027.58 total 266.36 last 124.10) - open/close/unlink 37516 (time 1781661037.59 total 276.37 last 132.93) - open/close/unlink 38862 (time 1781661047.59 total 286.37 last 134.55) - open/close/unlink 40000 (time 1781661054.66 total 293.43 last 161.13) - open/close/unlink 41203 (time 1781661064.66 total 303.44 last 120.28) - open/close/unlink 42560 (time 1781661074.66 total 313.44 last 135.68) - open/close/unlink 43736 (time 1781661084.66 total 323.44 last 117.53) - open/close/unlink 44929 (time 1781661094.66 total 333.44 last 119.29) - open/close/unlink 46089 (time 1781661104.67 total 343.44 last 115.99) - open/close/unlink 47280 (time 1781661114.67 total 353.45 last 119.00) - open/close/unlink 48689 (time 1781661124.68 total 363.46 last 140.86) - open/close/unlink 50000 (time 1781661134.65 total 373.43 last 131.48) - open/close/unlink 51177 (time 1781661144.65 total 383.43 last 117.68) - open/close/unlink 52257 (time 1781661154.65 total 393.43 last 107.95) - open/close/unlink 53437 (time 1781661164.66 total 403.44 last 117.94) - open/close/unlink 54560 (time 1781661174.66 total 413.44 last 112.29) - open/close/unlink 55708 (time 1781661184.67 total 423.45 last 114.72) - open/close/unlink 56867 (time 1781661194.67 total 433.45 last 115.83) - open/close/unlink 57961 (time 1781661204.68 total 443.46 last 109.33) - open/close/unlink 59159 (time 1781661214.70 total 453.48 last 119.55) - open/close/unlink 60000 (time 1781661221.80 total 460.57 last 118.54) - open/close/unlink 61224 (time 1781661231.80 total 470.58 last 122.38) - open/close/unlink 62575 (time 1781661241.80 total 480.58 last 135.05) - open/close/unlink 63942 (time 1781661251.80 total 490.58 last 136.67) total: 64768 open/close/unlink in 496.48 seconds: 130.45 ops/second - open/close/unlink 1409 (time 1781660771.24 total 10.00 last 140.89) - open/close/unlink 2885 (time 1781660781.24 total 20.00 last 147.58) - open/close/unlink 4330 (time 1781660791.25 total 30.00 last 144.47) - open/close/unlink 5802 (time 1781660801.25 total 40.01 last 147.14) - open/close/unlink 7342 (time 1781660811.25 total 50.01 last 153.99) - open/close/unlink 8747 (time 1781660821.25 total 60.01 last 140.45) - open/close/unlink 10000 (time 1781660828.53 total 67.29 last 172.25) - open/close/unlink 11469 (time 1781660838.53 total 77.29 last 146.90) - open/close/unlink 12957 (time 1781660848.53 total 87.29 last 148.77) - open/close/unlink 14420 (time 1781660858.54 total 97.29 last 146.23) - open/close/unlink 15978 (time 1781660868.54 total 107.30 last 155.75) - open/close/unlink 17249 (time 1781660878.54 total 117.30 last 127.03) - open/close/unlink 18449 (time 1781660888.55 total 127.31 last 119.92) - open/close/unlink 19784 (time 1781660898.55 total 137.31 last 133.49) - open/close/unlink 21220 (time 1781660908.55 total 147.31 last 143.57) - open/close/unlink 22541 (time 1781660918.56 total 157.32 last 131.99) - open/close/unlink 23891 (time 1781660928.58 total 167.34 last 134.72) - open/close/unlink 24931 (time 1781660938.59 total 177.35 last 103.95) - open/close/unlink 26091 (time 1781660948.59 total 187.35 last 115.98) - open/close/unlink 27455 (time 1781660958.59 total 197.35 last 136.32) - open/close/unlink 28858 (time 1781660968.61 total 207.36 last 140.13) - open/close/unlink 30000 (time 1781660977.43 total 216.19 last 129.46) - open/close/unlink 31199 (time 1781660987.43 total 226.19 last 119.89) - open/close/unlink 32442 (time 1781660997.43 total 236.19 last 124.28) - open/close/unlink 33632 (time 1781661007.43 total 246.19 last 118.98) - open/close/unlink 34958 (time 1781661017.44 total 256.20 last 132.53) - open/close/unlink 36194 (time 1781661027.44 total 266.20 last 123.54) - open/close/unlink 37534 (time 1781661037.44 total 276.20 last 133.96) - open/close/unlink 38877 (time 1781661047.45 total 286.21 last 134.25) - open/close/unlink 40000 (time 1781661054.53 total 293.28 last 158.67) - open/close/unlink 41204 (time 1781661064.53 total 303.29 last 120.32) - open/close/unlink 42559 (time 1781661074.54 total 313.30 last 135.44) - open/close/unlink 43732 (time 1781661084.54 total 323.30 last 117.24) - open/close/unlink 44938 (time 1781661094.55 total 333.30 last 120.57) - open/close/unlink 46097 (time 1781661104.55 total 343.31 last 115.84) - open/close/unlink 47285 (time 1781661114.56 total 353.31 last 118.73) - open/close/unlink 48704 (time 1781661124.56 total 363.32 last 141.88) - open/close/unlink 50000 (time 1781661134.48 total 373.24 last 130.66) - open/close/unlink 51175 (time 1781661144.48 total 383.24 last 117.45) - open/close/unlink 52256 (time 1781661154.48 total 393.24 last 108.09) - open/close/unlink 53434 (time 1781661164.48 total 403.24 last 117.78) - open/close/unlink 54555 (time 1781661174.49 total 413.25 last 112.03) - open/close/unlink 55691 (time 1781661184.50 total 423.25 last 113.54) - open/close/unlink 56847 (time 1781661194.50 total 433.26 last 115.56) - open/close/unlink 57955 (time 1781661204.50 total 443.26 last 110.74) - open/close/unlink 59166 (time 1781661214.53 total 453.29 last 120.81) - open/close/unlink 60000 (time 1781661221.73 total 460.49 last 115.87) - open/close/unlink 61228 (time 1781661231.73 total 470.49 last 122.71) - open/close/unlink 62561 (time 1781661241.74 total 480.50 last 133.26) - open/close/unlink 63917 (time 1781661251.74 total 490.50 last 135.59) total: 64768 open/close/unlink in 496.69 seconds: 130.40 ops/second - open/close/unlink 1397 (time 1781660771.18 total 10.00 last 139.66) - open/close/unlink 2872 (time 1781660781.18 total 20.00 last 147.47) - open/close/unlink 4308 (time 1781660791.20 total 30.01 last 143.45) - open/close/unlink 5761 (time 1781660801.20 total 40.02 last 145.23) - open/close/unlink 7296 (time 1781660811.21 total 50.03 last 153.40) - open/close/unlink 8710 (time 1781660821.21 total 60.03 last 141.37) - open/close/unlink 10000 (time 1781660828.75 total 67.57 last 171.13) - open/close/unlink 11465 (time 1781660838.75 total 77.57 last 146.44) - open/close/unlink 12908 (time 1781660848.75 total 87.57 last 144.27) - open/close/unlink 14379 (time 1781660858.76 total 97.58 last 147.03) - open/close/unlink 15929 (time 1781660868.76 total 107.58 last 154.96) - open/close/unlink 17176 (time 1781660878.76 total 117.58 last 124.68) - open/close/unlink 18392 (time 1781660888.76 total 127.58 last 121.60) - open/close/unlink 19724 (time 1781660898.77 total 137.59 last 133.13) - open/close/unlink 21163 (time 1781660908.77 total 147.59 last 143.82) - open/close/unlink 22475 (time 1781660918.77 total 157.59 last 131.19) - open/close/unlink 23837 (time 1781660928.78 total 167.60 last 136.11) - open/close/unlink 24878 (time 1781660938.78 total 177.60 last 104.09) - open/close/unlink 26022 (time 1781660948.78 total 187.60 last 114.38) - open/close/unlink 27418 (time 1781660958.79 total 197.60 last 139.57) - open/close/unlink 28791 (time 1781660968.79 total 207.61 last 137.23) - open/close/unlink 30000 (time 1781660977.92 total 216.74 last 132.37) - open/close/unlink 31190 (time 1781660987.93 total 226.75 last 118.96) - open/close/unlink 32420 (time 1781660997.93 total 236.75 last 122.98) - open/close/unlink 33625 (time 1781661007.93 total 246.75 last 120.49) - open/close/unlink 34925 (time 1781661017.93 total 256.75 last 129.95) - open/close/unlink 36156 (time 1781661027.94 total 266.76 last 123.05) - open/close/unlink 37496 (time 1781661037.94 total 276.76 last 133.98) - open/close/unlink 38836 (time 1781661047.94 total 286.76 last 133.96) - open/close/unlink 40000 (time 1781661055.53 total 294.35 last 153.41) - open/close/unlink 41202 (time 1781661065.53 total 304.35 last 120.13) - open/close/unlink 42594 (time 1781661075.54 total 314.36 last 139.09) - open/close/unlink 43754 (time 1781661085.55 total 324.37 last 115.87) - open/close/unlink 44961 (time 1781661095.56 total 334.38 last 120.62) - open/close/unlink 46132 (time 1781661105.56 total 344.38 last 117.09) - open/close/unlink 47315 (time 1781661115.57 total 354.39 last 118.23) - open/close/unlink 48779 (time 1781661125.57 total 364.39 last 146.37) - open/close/unlink 50000 (time 1781661135.10 total 373.92 last 128.11) - open/close/unlink 51201 (time 1781661145.10 total 383.92 last 120.07) - open/close/unlink 52271 (time 1781661155.11 total 393.93 last 106.94) - open/close/unlink 53449 (time 1781661165.12 total 403.94 last 117.71) - open/close/unlink 54586 (time 1781661175.12 total 413.94 last 113.63) - open/close/unlink 55717 (time 1781661185.12 total 423.94 last 113.08) - open/close/unlink 56872 (time 1781661195.13 total 433.95 last 115.45) - open/close/unlink 57975 (time 1781661205.13 total 443.95 last 110.26) - open/close/unlink 59137 (time 1781661215.14 total 453.96 last 116.09) - open/close/unlink 60000 (time 1781661222.17 total 460.99 last 122.71) - open/close/unlink 61240 (time 1781661232.18 total 471.00 last 123.96) - open/close/unlink 62592 (time 1781661242.19 total 481.00 last 135.08) - open/close/unlink 63945 (time 1781661252.19 total 491.01 last 135.27) total: 64768 open/close/unlink in 497.61 seconds: 130.16 ops/second - open/close/unlink 1403 (time 1781660771.22 total 10.00 last 140.29) - open/close/unlink 2869 (time 1781660781.23 total 20.01 last 146.53) - open/close/unlink 4325 (time 1781660791.23 total 30.01 last 145.54) - open/close/unlink 5790 (time 1781660801.24 total 40.01 last 146.44) - open/close/unlink 7333 (time 1781660811.24 total 50.02 last 154.24) - open/close/unlink 8745 (time 1781660821.24 total 60.02 last 141.16) - open/close/unlink 10000 (time 1781660828.61 total 67.39 last 170.31) - open/close/unlink 11459 (time 1781660838.62 total 77.39 last 145.87) - open/close/unlink 12918 (time 1781660848.62 total 87.40 last 145.78) - open/close/unlink 14395 (time 1781660858.62 total 97.40 last 147.69) - open/close/unlink 15958 (time 1781660868.63 total 107.40 last 156.25) - open/close/unlink 17210 (time 1781660878.63 total 117.41 last 125.16) - open/close/unlink 18417 (time 1781660888.63 total 127.41 last 120.66) - open/close/unlink 19749 (time 1781660898.64 total 137.41 last 133.15) - open/close/unlink 21189 (time 1781660908.64 total 147.42 last 143.96) - open/close/unlink 22494 (time 1781660918.64 total 157.42 last 130.48) - open/close/unlink 23845 (time 1781660928.64 total 167.42 last 135.06) - open/close/unlink 24879 (time 1781660938.65 total 177.42 last 103.37) - open/close/unlink 26049 (time 1781660948.65 total 187.43 last 116.96) - open/close/unlink 27420 (time 1781660958.65 total 197.43 last 137.06) - open/close/unlink 28815 (time 1781660968.67 total 207.45 last 139.23) - open/close/unlink 30000 (time 1781660977.79 total 216.56 last 130.02) - open/close/unlink 31191 (time 1781660987.79 total 226.57 last 119.07) - open/close/unlink 32432 (time 1781660997.79 total 236.57 last 124.06) - open/close/unlink 33639 (time 1781661007.79 total 246.57 last 120.68) - open/close/unlink 34934 (time 1781661017.80 total 256.58 last 129.43) - open/close/unlink 36189 (time 1781661027.81 total 266.58 last 125.44) - open/close/unlink 37533 (time 1781661037.81 total 276.59 last 134.33) - open/close/unlink 38854 (time 1781661047.82 total 286.59 last 132.02) - open/close/unlink 40000 (time 1781661055.11 total 293.88 last 157.21) - open/close/unlink 41197 (time 1781661065.11 total 303.89 last 119.66) - open/close/unlink 42573 (time 1781661075.11 total 313.89 last 137.59) - open/close/unlink 43727 (time 1781661085.12 total 323.89 last 115.31) - open/close/unlink 44922 (time 1781661095.12 total 333.90 last 119.45) - open/close/unlink 46089 (time 1781661105.12 total 343.90 last 116.66) - open/close/unlink 47247 (time 1781661115.13 total 353.90 last 115.79) - open/close/unlink 48701 (time 1781661125.13 total 363.90 last 145.37) - open/close/unlink 49996 (time 1781661135.13 total 373.91 last 129.45) - open/close/unlink 51176 (time 1781661145.13 total 383.91 last 117.99) - open/close/unlink 52251 (time 1781661155.14 total 393.91 last 107.45) - open/close/unlink 53431 (time 1781661165.14 total 403.91 last 117.99) - open/close/unlink 54562 (time 1781661175.14 total 413.92 last 113.05) - open/close/unlink 55699 (time 1781661185.15 total 423.92 last 113.65) - open/close/unlink 56855 (time 1781661195.15 total 433.93 last 115.55) - open/close/unlink 57956 (time 1781661205.16 total 443.93 last 110.05) - open/close/unlink 59130 (time 1781661215.16 total 453.93 last 117.37) - open/close/unlink 60000 (time 1781661222.36 total 461.14 last 120.78) - open/close/unlink 61235 (time 1781661232.36 total 471.14 last 123.48) - open/close/unlink 62588 (time 1781661242.37 total 481.14 last 135.25) - open/close/unlink 63938 (time 1781661252.37 total 491.15 last 134.90) total: 64768 open/close/unlink in 497.96 seconds: 130.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg357-server: cannot open 'lustre-ost2': no such pool pdsh@oleg357-client: oleg357-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local oleg357-server: modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2