== conf-sanity test 69: replace an OST with the same index ========================================================== 20:56:00 (1782608160) start mds service on oleg153-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing load_modules_local oleg153-server: Loading modules from /home/green/git/lustre-release/lustre oleg153-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg153-server: detected 4 online CPUs by sysfs oleg153-server: MODOPTS_LIBCFS= oleg153-server: Force libcfs to create 2 CPU partitions oleg153-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg153-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg153-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg153-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49535 inodes available. Want 99872. rc=0 - open/close 895 (time 1782608278.28 total 10.00 last 89.49) - open/close 1710 (time 1782608288.30 total 20.03 last 81.29) - open/close 2431 (time 1782608298.32 total 30.05 last 71.96) - open/close 3233 (time 1782608308.33 total 40.05 last 80.19) - open/close 3955 (time 1782608318.33 total 50.05 last 72.17) - open/close 4774 (time 1782608328.33 total 60.05 last 81.89) - open/close 5689 (time 1782608338.33 total 70.06 last 91.48) - open/close 6642 (time 1782608348.34 total 80.06 last 95.24) - open/close 7466 (time 1782608358.35 total 90.07 last 82.34) - open/close 8195 (time 1782608368.36 total 100.08 last 72.83) - open/close 8878 (time 1782608378.36 total 110.08 last 68.25) - open/close 9575 (time 1782608388.37 total 120.09 last 69.64) total: 10000 open/close in 125.85 seconds: 79.46 ops/second - unlinked 0 (time 1782608399 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 857 (time 1782608492.27 total 10.00 last 85.68) - open/close 1713 (time 1782608502.28 total 20.01 last 85.58) - open/close 2584 (time 1782608512.29 total 30.02 last 86.98) - open/close 3344 (time 1782608522.30 total 40.03 last 75.92) - open/close 4164 (time 1782608532.31 total 50.04 last 81.93) - open/close 4871 (time 1782608542.31 total 60.04 last 70.68) - open/close 5482 (time 1782608552.31 total 70.04 last 61.09) - open/close 6139 (time 1782608562.32 total 80.05 last 65.65) - open/close 6956 (time 1782608572.33 total 90.06 last 81.64) - open/close 7785 (time 1782608582.33 total 100.06 last 82.90) - open/close 8672 (time 1782608592.33 total 110.06 last 88.68) - open/close 9506 (time 1782608602.35 total 120.08 last 83.22) total: 10000 open/close in 125.40 seconds: 79.75 ops/second - unlinked 0 (time 1782608612 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 824 (time 1782608693.89 total 10.00 last 82.37) - open/close 1669 (time 1782608703.89 total 20.01 last 84.46) - open/close 2465 (time 1782608713.90 total 30.02 last 79.49) - open/close 3171 (time 1782608723.91 total 40.03 last 70.54) - open/close 3867 (time 1782608733.93 total 50.05 last 69.49) - open/close 4632 (time 1782608743.93 total 60.05 last 76.44) - open/close 5420 (time 1782608753.95 total 70.07 last 78.71) - open/close 6116 (time 1782608763.96 total 80.08 last 69.53) - open/close 6891 (time 1782608773.96 total 90.08 last 77.44) - open/close 7674 (time 1782608783.97 total 100.09 last 78.29) - open/close 8419 (time 1782608793.97 total 110.09 last 74.46) - open/close 9133 (time 1782608803.99 total 120.10 last 71.30) total: 10000 open/close in 130.01 seconds: 76.92 ops/second - unlinked 0 (time 1782608817 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 656 (time 1782608902.11 total 10.03 last 65.40) - open/close 1416 (time 1782608912.12 total 20.04 last 75.94) - open/close 2198 (time 1782608922.12 total 30.04 last 78.20) - open/close 2980 (time 1782608932.12 total 40.04 last 78.19) - open/close 3694 (time 1782608942.12 total 50.04 last 71.39) - open/close 4550 (time 1782608952.13 total 60.05 last 85.55) - open/close 5416 (time 1782608962.15 total 70.07 last 86.42) - open/close 6149 (time 1782608972.17 total 80.09 last 73.14) - open/close 6826 (time 1782608982.17 total 90.09 last 67.68) - open/close 7494 (time 1782608992.18 total 100.10 last 66.76) - open/close 8163 (time 1782609002.19 total 110.11 last 66.84) - open/close 8822 (time 1782609012.19 total 120.11 last 65.89) - open/close 9730 (time 1782609022.20 total 130.12 last 90.70) total: 10000 open/close in 133.36 seconds: 74.99 ops/second - unlinked 0 (time 1782609029 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 716 (time 1782609114.20 total 10.01 last 71.53) - open/close 1429 (time 1782609124.22 total 20.03 last 71.17) - open/close 2249 (time 1782609134.23 total 30.04 last 81.89) - open/close 3220 (time 1782609144.23 total 40.04 last 97.09) - open/close 4194 (time 1782609154.23 total 50.05 last 97.37) - open/close 5095 (time 1782609164.24 total 60.05 last 90.05) - open/close 5984 (time 1782609174.24 total 70.05 last 88.89) - open/close 6876 (time 1782609184.25 total 80.06 last 89.10) - open/close 7735 (time 1782609194.25 total 90.06 last 85.89) - open/close 8464 (time 1782609204.27 total 100.08 last 72.79) - open/close 9334 (time 1782609214.27 total 110.08 last 86.95) total: 10000 open/close in 117.89 seconds: 84.82 ops/second - unlinked 0 (time 1782609227 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 611 (time 1782609316.21 total 10.00 last 61.07) - open/close 1356 (time 1782609326.21 total 20.01 last 74.47) - open/close 1986 (time 1782609336.22 total 30.02 last 62.94) - open/close 2715 (time 1782609346.23 total 40.02 last 72.87) - open/close 3430 (time 1782609356.25 total 50.04 last 71.37) - open/close 4360 (time 1782609366.26 total 60.05 last 92.89) - open/close 5161 (time 1782609376.26 total 70.05 last 80.09) - open/close 6074 (time 1782609386.26 total 80.06 last 91.29) - open/close 6928 (time 1782609396.28 total 90.07 last 85.25) - open/close 7798 (time 1782609406.28 total 100.08 last 86.96) - open/close 8671 (time 1782609416.29 total 110.08 last 87.28) - open/close 9345 (time 1782609426.29 total 120.09 last 67.34) total: 10000 open/close in 128.66 seconds: 77.72 ops/second - unlinked 0 (time 1782609439 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 715 (time 1782609532.41 total 10.01 last 71.45) - open/close 1462 (time 1782609542.41 total 20.01 last 74.67) - open/close 2036 (time 1782609552.41 total 30.01 last 57.39) - open/close 2735 (time 1782609562.42 total 40.02 last 69.85) - open/close 3587 (time 1782609572.42 total 50.02 last 85.18) - open/close 4413 (time 1782609582.43 total 60.03 last 82.55) - open/close 5266 (time 1782609592.43 total 70.04 last 85.25) - open/close 6024 (time 1782609602.45 total 80.05 last 75.69) - open/close 6736 (time 1782609612.46 total 90.06 last 71.15) - open/close 7496 (time 1782609622.46 total 100.06 last 75.95) - open/close 8270 (time 1782609632.46 total 110.06 last 77.39) - open/close 9226 (time 1782609642.46 total 120.07 last 95.60) - open/close 9985 (time 1782609652.47 total 130.08 last 75.82) total: 10000 open/close in 130.27 seconds: 76.76 ops/second - unlinked 0 (time 1782609659 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 877 (time 1782609749.20 total 10.00 last 87.68) - open/close 1714 (time 1782609759.22 total 20.01 last 83.60) - open/close 2421 (time 1782609769.22 total 30.02 last 70.67) - open/close 3348 (time 1782609779.23 total 40.03 last 92.62) - open/close 4304 (time 1782609789.23 total 50.03 last 95.55) - open/close 5201 (time 1782609799.26 total 60.06 last 89.45) - open/close 5846 (time 1782609809.27 total 70.07 last 64.45) - open/close 6751 (time 1782609819.27 total 80.07 last 90.44) - open/close 7573 (time 1782609829.28 total 90.08 last 82.18) - open/close 8249 (time 1782609839.29 total 100.09 last 67.54) - open/close 8850 (time 1782609849.29 total 110.09 last 60.05) - open/close 9700 (time 1782609859.29 total 120.09 last 84.99) total: 10000 open/close in 123.20 seconds: 81.17 ops/second - unlinked 0 (time 1782609866 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 843 (time 1782609957.28 total 10.00 last 84.29) - open/close 1535 (time 1782609967.29 total 20.00 last 69.19) - open/close 2506 (time 1782609977.29 total 30.01 last 97.06) - open/close 3459 (time 1782609987.29 total 40.01 last 95.28) - open/close 4282 (time 1782609997.29 total 50.01 last 82.29) - open/close 5246 (time 1782610007.29 total 60.01 last 96.39) - open/close 6094 (time 1782610017.29 total 70.01 last 84.80) - open/close 7007 (time 1782610027.30 total 80.01 last 91.28) - open/close 7824 (time 1782610037.30 total 90.02 last 81.68) - open/close 8701 (time 1782610047.30 total 100.02 last 87.68) - open/close 9537 (time 1782610057.32 total 110.04 last 83.41) total: 10000 open/close in 115.86 seconds: 86.31 ops/second - unlinked 0 (time 1782610068 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 675 (time 1782610164.51 total 10.01 last 67.42) - open/close 1302 (time 1782610174.51 total 20.01 last 62.69) - open/close 2061 (time 1782610184.51 total 30.02 last 75.86) - open/close 2786 (time 1782610194.52 total 40.03 last 72.44) - open/close 3523 (time 1782610204.53 total 50.03 last 73.67) - open/close 4197 (time 1782610214.53 total 60.04 last 67.37) - open/close 4901 (time 1782610224.55 total 70.05 last 70.27) - open/close 5664 (time 1782610234.56 total 80.06 last 76.21) - open/close 6374 (time 1782610244.56 total 90.07 last 70.99) - open/close 7164 (time 1782610254.58 total 100.08 last 78.88) - open/close 7876 (time 1782610264.60 total 110.10 last 71.06) - open/close 8626 (time 1782610274.61 total 120.12 last 74.89) - open/close 9323 (time 1782610284.63 total 130.13 last 69.60) total: 10000 open/close in 138.61 seconds: 72.15 ops/second - unlinked 0 (time 1782610298 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg153-server unloading modules via unload_modules_local on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing unload_modules_local oleg153-server: modules unloaded. pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2