== conf-sanity test 69: replace an OST with the same index ========================================================== 09:40:54 (1782481254) start mds service on oleg414-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg414-server: detected 4 online CPUs by sysfs oleg414-server: MODOPTS_LIBCFS= oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg414-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg414-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg414-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 652 (time 1782481386.88 total 10.01 last 65.11) - open/close 1324 (time 1782481396.89 total 20.02 last 67.14) - open/close 1995 (time 1782481406.90 total 30.03 last 67.06) - open/close 2719 (time 1782481416.91 total 40.03 last 72.36) - open/close 3380 (time 1782481426.91 total 50.04 last 66.08) - open/close 4098 (time 1782481436.91 total 60.04 last 71.76) - open/close 4807 (time 1782481446.93 total 70.06 last 70.78) - open/close 5543 (time 1782481456.94 total 80.07 last 73.54) - open/close 6254 (time 1782481466.95 total 90.08 last 70.98) - open/close 6951 (time 1782481476.96 total 100.09 last 69.67) - open/close 7655 (time 1782481486.96 total 110.09 last 70.36) - open/close 8332 (time 1782481496.97 total 120.09 last 67.69) - open/close 9113 (time 1782481506.97 total 130.10 last 78.09) - open/close 9846 (time 1782481516.98 total 140.11 last 73.22) total: 10000 open/close in 142.21 seconds: 70.32 ops/second - unlinked 0 (time 1782481523 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 754 (time 1782481613.45 total 10.01 last 75.35) - open/close 1485 (time 1782481623.45 total 20.01 last 73.10) - open/close 2236 (time 1782481633.46 total 30.02 last 75.01) - open/close 3011 (time 1782481643.46 total 40.02 last 77.49) - open/close 3761 (time 1782481653.46 total 50.03 last 74.96) - open/close 4541 (time 1782481663.47 total 60.03 last 77.94) - open/close 5280 (time 1782481673.48 total 70.04 last 73.85) - open/close 6087 (time 1782481683.49 total 80.05 last 80.64) - open/close 6912 (time 1782481693.49 total 90.05 last 82.48) - open/close 7699 (time 1782481703.49 total 100.05 last 78.69) - open/close 8423 (time 1782481713.49 total 110.05 last 72.38) - open/close 9193 (time 1782481723.50 total 120.07 last 76.91) - open/close 9965 (time 1782481733.50 total 130.07 last 77.20) total: 10000 open/close in 130.49 seconds: 76.63 ops/second - unlinked 0 (time 1782481737 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 786 (time 1782481819.03 total 10.01 last 78.56) - open/close 1626 (time 1782481829.03 total 20.01 last 83.93) - open/close 2468 (time 1782481839.03 total 30.01 last 84.19) - open/close 3208 (time 1782481849.04 total 40.02 last 73.97) - open/close 4010 (time 1782481859.04 total 50.02 last 80.20) - open/close 4813 (time 1782481869.05 total 60.03 last 80.21) - open/close 5577 (time 1782481879.05 total 70.03 last 76.38) - open/close 6407 (time 1782481889.05 total 80.03 last 82.99) - open/close 7188 (time 1782481899.06 total 90.04 last 78.09) - open/close 7957 (time 1782481909.06 total 100.04 last 76.85) - open/close 8778 (time 1782481919.07 total 110.05 last 82.00) - open/close 9656 (time 1782481929.08 total 120.06 last 87.77) total: 10000 open/close in 124.06 seconds: 80.60 ops/second - unlinked 0 (time 1782481935 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 934 (time 1782482008.70 total 10.01 last 93.35) - open/close 1953 (time 1782482018.71 total 20.01 last 101.85) - open/close 2911 (time 1782482028.71 total 30.01 last 95.77) - open/close 4012 (time 1782482038.72 total 40.02 last 110.03) - open/close 4900 (time 1782482048.72 total 50.03 last 88.73) - open/close 5726 (time 1782482058.72 total 60.03 last 82.59) - open/close 6699 (time 1782482068.73 total 70.03 last 97.25) - open/close 7639 (time 1782482078.73 total 80.04 last 93.98) - open/close 8547 (time 1782482088.74 total 90.05 last 90.70) - open/close 9453 (time 1782482098.74 total 100.05 last 90.59) total: 10000 open/close in 106.02 seconds: 94.32 ops/second - unlinked 0 (time 1782482107 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 929 (time 1782482181.39 total 10.00 last 92.88) - open/close 1782 (time 1782482191.40 total 20.01 last 85.25) - open/close 2761 (time 1782482201.40 total 30.01 last 97.86) - open/close 3589 (time 1782482211.41 total 40.02 last 82.70) - open/close 4410 (time 1782482221.42 total 50.03 last 82.07) - open/close 5319 (time 1782482231.42 total 60.04 last 90.82) - open/close 6180 (time 1782482241.44 total 70.05 last 86.01) - open/close 7042 (time 1782482251.44 total 80.05 last 86.15) - open/close 7869 (time 1782482261.45 total 90.06 last 82.66) - open/close 8752 (time 1782482271.45 total 100.07 last 88.22) - open/close 9599 (time 1782482281.46 total 110.07 last 84.69) total: 10000 open/close in 114.54 seconds: 87.31 ops/second - unlinked 0 (time 1782482289 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 821 (time 1782482369.63 total 10.00 last 82.08) - open/close 1594 (time 1782482379.63 total 20.00 last 77.28) - open/close 2409 (time 1782482389.65 total 30.02 last 81.39) - open/close 3241 (time 1782482399.65 total 40.02 last 83.19) - open/close 4081 (time 1782482409.65 total 50.02 last 83.97) - open/close 4920 (time 1782482419.65 total 60.02 last 83.89) - open/close 5732 (time 1782482429.65 total 70.03 last 81.17) - open/close 6535 (time 1782482439.66 total 80.04 last 80.22) - open/close 7377 (time 1782482449.67 total 90.04 last 84.15) - open/close 8232 (time 1782482459.68 total 100.06 last 85.39) - open/close 9088 (time 1782482469.69 total 110.06 last 85.55) total: 10000 open/close in 119.92 seconds: 83.39 ops/second - unlinked 0 (time 1782482482 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 882 (time 1782482556.89 total 10.00 last 88.16) - open/close 1841 (time 1782482566.90 total 20.00 last 95.89) - open/close 2767 (time 1782482576.90 total 30.01 last 92.55) - open/close 3632 (time 1782482586.90 total 40.01 last 86.48) - open/close 4499 (time 1782482596.91 total 50.02 last 86.67) - open/close 5325 (time 1782482606.92 total 60.03 last 82.51) - open/close 6167 (time 1782482616.92 total 70.03 last 84.19) - open/close 6995 (time 1782482626.92 total 80.03 last 82.76) - open/close 7744 (time 1782482636.94 total 90.05 last 74.77) - open/close 8557 (time 1782482646.95 total 100.06 last 81.25) - open/close 9354 (time 1782482656.95 total 110.06 last 79.67) total: 10000 open/close in 117.75 seconds: 84.93 ops/second - unlinked 0 (time 1782482667 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 852 (time 1782482752.26 total 10.01 last 85.12) - open/close 1703 (time 1782482762.26 total 20.02 last 85.03) - open/close 2607 (time 1782482772.28 total 30.03 last 90.30) - open/close 3517 (time 1782482782.28 total 40.03 last 90.94) - open/close 4402 (time 1782482792.29 total 50.04 last 88.45) - open/close 5291 (time 1782482802.30 total 60.05 last 88.82) - open/close 6269 (time 1782482812.30 total 70.05 last 97.79) - open/close 7121 (time 1782482822.31 total 80.06 last 85.11) - open/close 7913 (time 1782482832.31 total 90.06 last 79.18) - open/close 8793 (time 1782482842.31 total 100.07 last 87.97) - open/close 9712 (time 1782482852.31 total 110.07 last 91.89) total: 10000 open/close in 113.28 seconds: 88.28 ops/second - unlinked 0 (time 1782482858 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 904 (time 1782482929.95 total 10.01 last 90.30) - open/close 1701 (time 1782482939.95 total 20.01 last 79.69) - open/close 2570 (time 1782482949.95 total 30.02 last 86.88) - open/close 3488 (time 1782482959.96 total 40.03 last 91.69) - open/close 4355 (time 1782482969.97 total 50.04 last 86.59) - open/close 5191 (time 1782482979.99 total 60.05 last 83.51) - open/close 6008 (time 1782482989.99 total 70.06 last 81.66) - open/close 6882 (time 1782482999.99 total 80.06 last 87.38) - open/close 7793 (time 1782483010.00 total 90.07 last 91.04) - open/close 8650 (time 1782483020.01 total 100.07 last 85.65) - open/close 9456 (time 1782483030.01 total 110.08 last 80.54) total: 10000 open/close in 116.83 seconds: 85.59 ops/second - unlinked 0 (time 1782483039 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 823 (time 1782483114.75 total 10.01 last 82.24) - open/close 1643 (time 1782483124.76 total 20.02 last 81.91) - open/close 2521 (time 1782483134.77 total 30.03 last 87.74) - open/close 3428 (time 1782483144.77 total 40.03 last 90.68) - open/close 4328 (time 1782483154.77 total 50.03 last 89.98) - open/close 5235 (time 1782483164.77 total 60.03 last 90.70) - open/close 6226 (time 1782483174.78 total 70.03 last 99.06) - open/close 7176 (time 1782483184.78 total 80.04 last 94.99) - open/close 8178 (time 1782483194.80 total 90.05 last 100.03) - open/close 9137 (time 1782483204.81 total 100.06 last 95.80) total: 10000 open/close in 109.90 seconds: 90.99 ops/second - unlinked 0 (time 1782483217 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg414-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg414-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg414-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre On OST0, 19110 used inodes rc=0 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 start ost1 service on oleg414-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg414-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg414-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server unloading modules via unload_modules_local on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local oleg414-server: modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2