== conf-sanity test 69: replace an OST with the same index ========================================================== 07:15:41 (1788866141) start mds service on oleg420-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: kmemleak disabled oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg420-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg420-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg420-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 561 (time 1788866236.60 total 10.01 last 56.05) - open/close 1176 (time 1788866246.60 total 20.01 last 61.47) - open/close 1749 (time 1788866256.61 total 30.02 last 57.28) - open/close 2259 (time 1788866266.61 total 40.03 last 50.95) - open/close 2732 (time 1788866276.62 total 50.03 last 47.27) - open/close 3268 (time 1788866286.64 total 60.05 last 53.51) - open/close 3858 (time 1788866296.65 total 70.06 last 58.93) - open/close 4461 (time 1788866306.66 total 80.07 last 60.26) - open/close 5048 (time 1788866316.66 total 90.07 last 58.69) - open/close 5761 (time 1788866326.67 total 100.08 last 71.23) - open/close 6343 (time 1788866336.67 total 110.08 last 58.19) - open/close 6943 (time 1788866346.69 total 120.11 last 59.85) - open/close 7610 (time 1788866356.71 total 130.13 last 66.57) - open/close 8246 (time 1788866366.71 total 140.13 last 63.60) - open/close 8875 (time 1788866376.73 total 150.15 last 62.77) - open/close 9517 (time 1788866386.74 total 160.15 last 64.16) total: 10000 open/close in 167.70 seconds: 59.63 ops/second - unlinked 0 (time 1788866398 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 587 (time 1788866515.38 total 10.00 last 58.69) - open/close 1204 (time 1788866525.39 total 20.01 last 61.65) - open/close 1746 (time 1788866535.43 total 30.05 last 53.99) - open/close 2200 (time 1788866545.45 total 40.07 last 45.29) - open/close 2647 (time 1788866555.46 total 50.08 last 44.67) - open/close 3128 (time 1788866565.46 total 60.08 last 48.07) - open/close 3652 (time 1788866575.46 total 70.09 last 52.39) - open/close 4205 (time 1788866585.47 total 80.09 last 55.28) - open/close 4784 (time 1788866595.48 total 90.10 last 57.84) - open/close 5445 (time 1788866605.49 total 100.11 last 66.04) - open/close 6012 (time 1788866615.50 total 110.12 last 56.64) - open/close 6613 (time 1788866625.52 total 120.14 last 59.99) - open/close 7166 (time 1788866635.53 total 130.15 last 55.24) - open/close 7754 (time 1788866645.55 total 140.17 last 58.68) - open/close 8270 (time 1788866655.55 total 150.17 last 51.59) - open/close 8854 (time 1788866665.55 total 160.17 last 58.38) - open/close 9437 (time 1788866675.57 total 170.19 last 58.21) total: 10000 open/close in 180.18 seconds: 55.50 ops/second - unlinked 0 (time 1788866691 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 621 (time 1788866803.91 total 10.01 last 62.04) - open/close 1278 (time 1788866813.91 total 20.01 last 65.70) - open/close 1807 (time 1788866823.91 total 30.01 last 52.87) - open/close 2266 (time 1788866833.92 total 40.02 last 45.86) - open/close 2803 (time 1788866843.95 total 50.05 last 53.58) - open/close 3356 (time 1788866853.95 total 60.05 last 55.28) - open/close 3919 (time 1788866863.96 total 70.06 last 56.26) - open/close 4484 (time 1788866873.96 total 80.06 last 56.48) - open/close 5182 (time 1788866883.97 total 90.08 last 69.69) - open/close 5827 (time 1788866893.98 total 100.08 last 64.46) - open/close 6454 (time 1788866903.99 total 110.09 last 62.63) - open/close 7083 (time 1788866914.03 total 120.13 last 62.69) - open/close 7742 (time 1788866924.03 total 130.13 last 65.89) - open/close 8383 (time 1788866934.03 total 140.14 last 64.06) - open/close 9063 (time 1788866944.06 total 150.16 last 67.86) - open/close 9720 (time 1788866954.06 total 160.17 last 65.64) total: 10000 open/close in 165.25 seconds: 60.51 ops/second - unlinked 0 (time 1788866963 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 625 (time 1788867082.83 total 10.01 last 62.42) - open/close 1244 (time 1788867092.84 total 20.01 last 61.89) - open/close 1951 (time 1788867102.85 total 30.03 last 70.60) - open/close 2595 (time 1788867112.87 total 40.05 last 64.27) - open/close 3141 (time 1788867122.87 total 50.05 last 54.59) - open/close 3715 (time 1788867132.89 total 60.06 last 57.33) - open/close 4307 (time 1788867142.89 total 70.06 last 59.20) - open/close 4942 (time 1788867152.89 total 80.07 last 63.47) - open/close 5546 (time 1788867162.91 total 90.09 last 60.26) - open/close 6191 (time 1788867172.94 total 100.12 last 64.29) - open/close 6841 (time 1788867182.98 total 110.16 last 64.76) - open/close 7496 (time 1788867192.99 total 120.17 last 65.43) - open/close 8098 (time 1788867203.00 total 130.18 last 60.15) - open/close 8682 (time 1788867213.00 total 140.18 last 58.40) - open/close 9272 (time 1788867223.02 total 150.20 last 58.89) - open/close 9788 (time 1788867233.02 total 160.20 last 51.59) total: 10000 open/close in 163.96 seconds: 60.99 ops/second - unlinked 0 (time 1788867242 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 622 (time 1788867358.75 total 10.02 last 62.09) - open/close 1166 (time 1788867368.76 total 20.03 last 54.34) - open/close 1775 (time 1788867378.78 total 30.04 last 60.83) - open/close 2372 (time 1788867388.78 total 40.05 last 59.67) - open/close 3030 (time 1788867398.78 total 50.05 last 65.78) - open/close 3700 (time 1788867408.79 total 60.05 last 66.98) - open/close 4331 (time 1788867418.80 total 70.07 last 63.00) - open/close 5024 (time 1788867428.80 total 80.07 last 69.28) - open/close 5656 (time 1788867438.83 total 90.10 last 63.01) - open/close 6295 (time 1788867448.84 total 100.10 last 63.89) - open/close 6961 (time 1788867458.84 total 110.11 last 66.54) - open/close 7669 (time 1788867468.85 total 120.12 last 70.75) - open/close 8285 (time 1788867478.86 total 130.13 last 61.54) - open/close 8925 (time 1788867488.86 total 140.13 last 63.99) - open/close 9539 (time 1788867498.87 total 150.13 last 61.38) total: 10000 open/close in 156.97 seconds: 63.71 ops/second - unlinked 0 (time 1788867509 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 609 (time 1788867624.78 total 10.00 last 60.90) - open/close 1252 (time 1788867634.78 total 20.01 last 64.26) - open/close 1897 (time 1788867644.79 total 30.01 last 64.46) - open/close 2546 (time 1788867654.82 total 40.04 last 64.72) - open/close 3106 (time 1788867664.82 total 50.04 last 55.98) - open/close 3666 (time 1788867674.83 total 60.05 last 55.93) - open/close 4265 (time 1788867684.84 total 70.06 last 59.88) - open/close 4869 (time 1788867694.84 total 80.07 last 60.35) - open/close 5450 (time 1788867704.85 total 90.08 last 58.04) - open/close 6078 (time 1788867714.87 total 100.10 last 62.67) - open/close 6637 (time 1788867724.88 total 110.10 last 55.89) - open/close 7282 (time 1788867734.88 total 120.11 last 64.45) - open/close 7947 (time 1788867744.89 total 130.11 last 66.44) - open/close 8559 (time 1788867754.90 total 140.12 last 61.17) - open/close 9167 (time 1788867764.90 total 150.13 last 60.77) - open/close 9800 (time 1788867774.91 total 160.13 last 63.27) total: 10000 open/close in 163.60 seconds: 61.12 ops/second - unlinked 0 (time 1788867783 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 560 (time 1788867893.74 total 10.01 last 55.92) - open/close 1180 (time 1788867903.74 total 20.02 last 61.96) - open/close 1815 (time 1788867913.75 total 30.03 last 63.46) - open/close 2440 (time 1788867923.76 total 40.03 last 62.46) - open/close 3013 (time 1788867933.77 total 50.05 last 57.23) - open/close 3675 (time 1788867943.78 total 60.06 last 66.14) - open/close 4348 (time 1788867953.78 total 70.06 last 67.26) - open/close 5036 (time 1788867963.80 total 80.08 last 68.66) - open/close 5758 (time 1788867973.81 total 90.09 last 72.15) - open/close 6360 (time 1788867983.82 total 100.10 last 60.14) - open/close 7055 (time 1788867993.83 total 110.11 last 69.45) - open/close 7661 (time 1788868003.84 total 120.12 last 60.52) - open/close 8227 (time 1788868013.85 total 130.13 last 56.56) - open/close 8865 (time 1788868023.85 total 140.13 last 63.79) - open/close 9510 (time 1788868033.88 total 150.15 last 64.33) total: 10000 open/close in 157.99 seconds: 63.29 ops/second - unlinked 0 (time 1788868046 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 626 (time 1788868155.08 total 10.00 last 62.59) - open/close 1219 (time 1788868165.09 total 20.01 last 59.26) - open/close 1868 (time 1788868175.09 total 30.01 last 64.87) - open/close 2503 (time 1788868185.11 total 40.03 last 63.39) - open/close 3084 (time 1788868195.13 total 50.05 last 57.99) - open/close 3686 (time 1788868205.13 total 60.05 last 60.19) - open/close 4234 (time 1788868215.16 total 70.08 last 54.65) - open/close 4876 (time 1788868225.17 total 80.09 last 64.15) - open/close 5490 (time 1788868235.18 total 90.10 last 61.33) - open/close 6108 (time 1788868245.19 total 100.11 last 61.75) - open/close 6709 (time 1788868255.20 total 110.12 last 60.06) - open/close 7326 (time 1788868265.20 total 120.12 last 61.69) - open/close 7934 (time 1788868275.21 total 130.13 last 60.72) - open/close 8534 (time 1788868285.22 total 140.14 last 59.93) - open/close 9156 (time 1788868295.23 total 150.15 last 62.16) - open/close 9788 (time 1788868305.24 total 160.16 last 63.15) total: 10000 open/close in 163.82 seconds: 61.04 ops/second - unlinked 0 (time 1788868313 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 702 (time 1788868420.61 total 10.00 last 70.18) - open/close 1400 (time 1788868430.61 total 20.01 last 69.76) - open/close 2087 (time 1788868440.62 total 30.02 last 68.63) - open/close 2761 (time 1788868450.63 total 40.02 last 67.37) - open/close 3472 (time 1788868460.63 total 50.02 last 71.08) - open/close 4131 (time 1788868470.64 total 60.03 last 65.86) - open/close 4727 (time 1788868480.64 total 70.03 last 59.60) - open/close 5355 (time 1788868490.64 total 80.03 last 62.78) - open/close 5965 (time 1788868500.65 total 90.04 last 60.96) - open/close 6637 (time 1788868510.66 total 100.05 last 67.13) - open/close 7304 (time 1788868520.67 total 110.06 last 66.61) - open/close 7971 (time 1788868530.68 total 120.08 last 66.61) - open/close 8637 (time 1788868540.69 total 130.09 last 66.53) - open/close 9334 (time 1788868550.72 total 140.12 last 69.49) total: 10000 open/close in 149.81 seconds: 66.75 ops/second - unlinked 0 (time 1788868565 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second