== conf-sanity test 69: replace an OST with the same index ========================================================== 04:42:28 (1787560948) start mds service on oleg626-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg626-server' oleg626-server: oleg626-server.virtnet: executing load_modules_local oleg626-server: Loading modules from /home/green/git/lustre-release/lustre oleg626-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg626-server: detected 4 online CPUs by sysfs oleg626-server: MODOPTS_LIBCFS= oleg626-server: Force libcfs to create 2 CPU partitions oleg626-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg626-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg626-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg626-server: oleg626-server.virtnet: executing set_default_debug -1 all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg626-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg626-server: oleg626-server.virtnet: executing set_default_debug -1 all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg626-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg626-server: oleg626-server.virtnet: executing set_default_debug -1 all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Started lustre-OST0000 oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 671 (time 1787561055.29 total 10.04 last 66.86) - open/close 1365 (time 1787561065.30 total 20.05 last 69.31) - open/close 2176 (time 1787561075.32 total 30.07 last 80.96) - open/close 2975 (time 1787561085.32 total 40.07 last 79.89) - open/close 3712 (time 1787561095.33 total 50.07 last 73.66) - open/close 4349 (time 1787561105.34 total 60.09 last 63.61) - open/close 5129 (time 1787561115.35 total 70.09 last 77.97) - open/close 5895 (time 1787561125.35 total 80.10 last 76.54) - open/close 6724 (time 1787561135.36 total 90.10 last 82.85) - open/close 7512 (time 1787561145.37 total 100.11 last 78.72) - open/close 8264 (time 1787561155.39 total 110.13 last 75.06) - open/close 9019 (time 1787561165.40 total 120.14 last 75.42) - open/close 9804 (time 1787561175.41 total 130.15 last 78.45) total: 10000 open/close in 132.53 seconds: 75.46 ops/second - unlinked 0 (time 1787561182 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 731 (time 1787561269.92 total 10.00 last 73.07) - open/close 1532 (time 1787561279.92 total 20.01 last 80.06) - open/close 2157 (time 1787561289.94 total 30.02 last 62.42) - open/close 2760 (time 1787561299.94 total 40.02 last 60.29) - open/close 3436 (time 1787561309.96 total 50.05 last 67.45) - open/close 4161 (time 1787561319.97 total 60.06 last 72.42) - open/close 4800 (time 1787561329.98 total 70.06 last 63.87) - open/close 5460 (time 1787561339.98 total 80.06 last 66.00) - open/close 6092 (time 1787561350.01 total 90.09 last 63.03) - open/close 6877 (time 1787561360.01 total 100.09 last 78.50) - open/close 7666 (time 1787561370.01 total 110.09 last 78.87) - open/close 8449 (time 1787561380.01 total 120.09 last 78.30) - open/close 9186 (time 1787561390.02 total 130.11 last 73.59) - open/close 9884 (time 1787561400.03 total 140.11 last 69.80) total: 10000 open/close in 141.91 seconds: 70.47 ops/second - unlinked 0 (time 1787561406 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 794 (time 1787561492.70 total 10.01 last 79.29) - open/close 1609 (time 1787561502.71 total 20.02 last 81.43) - open/close 2387 (time 1787561512.72 total 30.03 last 77.71) - open/close 3006 (time 1787561522.72 total 40.03 last 61.90) - open/close 3615 (time 1787561532.73 total 50.04 last 60.88) - open/close 4262 (time 1787561542.74 total 60.05 last 64.60) - open/close 5088 (time 1787561552.75 total 70.06 last 82.53) - open/close 5872 (time 1787561562.75 total 80.06 last 78.39) - open/close 6607 (time 1787561572.77 total 90.08 last 73.39) - open/close 7209 (time 1787561582.77 total 100.08 last 60.20) - open/close 7944 (time 1787561592.78 total 110.09 last 73.43) - open/close 8635 (time 1787561602.78 total 120.09 last 69.09) - open/close 9409 (time 1787561612.79 total 130.10 last 77.33) total: 10000 open/close in 136.81 seconds: 73.09 ops/second - unlinked 0 (time 1787561624 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 678 (time 1787561714.71 total 10.00 last 67.79) - open/close 1345 (time 1787561724.72 total 20.01 last 66.66) - open/close 2094 (time 1787561734.73 total 30.02 last 74.80) - open/close 2809 (time 1787561744.74 total 40.03 last 71.42) - open/close 3589 (time 1787561754.74 total 50.04 last 77.98) - open/close 4357 (time 1787561764.75 total 60.04 last 76.76) - open/close 5083 (time 1787561774.75 total 70.04 last 72.57) - open/close 5811 (time 1787561784.75 total 80.04 last 72.80) - open/close 6661 (time 1787561794.76 total 90.05 last 84.95) - open/close 7421 (time 1787561804.77 total 100.06 last 75.94) - open/close 8256 (time 1787561814.77 total 110.07 last 83.44) - open/close 8908 (time 1787561824.78 total 120.07 last 65.17) - open/close 9583 (time 1787561834.79 total 130.08 last 67.41) total: 10000 open/close in 135.41 seconds: 73.85 ops/second - unlinked 0 (time 1787561844 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 717 (time 1787561939.40 total 10.01 last 71.66) - open/close 1439 (time 1787561949.41 total 20.01 last 72.13) - open/close 2187 (time 1787561959.41 total 30.02 last 74.75) - open/close 2842 (time 1787561969.43 total 40.04 last 65.36) - open/close 3667 (time 1787561979.46 total 50.07 last 82.30) - open/close 4477 (time 1787561989.46 total 60.07 last 80.99) - open/close 5255 (time 1787561999.46 total 70.07 last 77.78) - open/close 5943 (time 1787562009.47 total 80.08 last 68.74) - open/close 6640 (time 1787562019.47 total 90.08 last 69.67) - open/close 7363 (time 1787562029.49 total 100.09 last 72.22) - open/close 8186 (time 1787562039.51 total 110.12 last 82.12) - open/close 8943 (time 1787562049.51 total 120.12 last 75.66) - open/close 9758 (time 1787562059.52 total 130.13 last 81.43) total: 10000 open/close in 133.92 seconds: 74.67 ops/second - unlinked 0 (time 1787562067 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 773 (time 1787562152.31 total 10.01 last 77.23) - open/close 1608 (time 1787562162.31 total 20.01 last 83.49) - open/close 2304 (time 1787562172.32 total 30.02 last 69.56) - open/close 3027 (time 1787562182.32 total 40.02 last 72.28) - open/close 3728 (time 1787562192.32 total 50.02 last 70.08) - open/close 4429 (time 1787562202.33 total 60.03 last 70.06) - open/close 5137 (time 1787562212.33 total 70.03 last 70.78) - open/close 5978 (time 1787562222.33 total 80.03 last 84.06) - open/close 6774 (time 1787562232.36 total 90.06 last 79.43) - open/close 7543 (time 1787562242.37 total 100.07 last 76.78) - open/close 8428 (time 1787562252.38 total 110.08 last 88.43) - open/close 9260 (time 1787562262.38 total 120.08 last 83.17) total: 10000 open/close in 128.39 seconds: 77.89 ops/second - unlinked 0 (time 1787562275 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 769 (time 1787562359.89 total 10.01 last 76.85) - open/close 1650 (time 1787562369.89 total 20.01 last 88.08) - open/close 2447 (time 1787562379.89 total 30.01 last 79.66) - open/close 3292 (time 1787562389.90 total 40.02 last 84.48) - open/close 4173 (time 1787562399.90 total 50.02 last 88.07) - open/close 5052 (time 1787562409.90 total 60.02 last 87.89) - open/close 5849 (time 1787562419.90 total 70.02 last 79.69) - open/close 6473 (time 1787562429.91 total 80.03 last 62.35) - open/close 7259 (time 1787562439.92 total 90.04 last 78.57) - open/close 8094 (time 1787562449.92 total 100.04 last 83.45) - open/close 8874 (time 1787562459.93 total 110.05 last 77.92) - open/close 9712 (time 1787562469.95 total 120.07 last 83.68) total: 10000 open/close in 122.63 seconds: 81.54 ops/second - unlinked 0 (time 1787562477 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 788 (time 1787562570.37 total 10.01 last 78.74) - open/close 1598 (time 1787562580.39 total 20.02 last 80.87) - open/close 2437 (time 1787562590.41 total 30.04 last 83.74) - open/close 3233 (time 1787562600.41 total 40.05 last 79.55) - open/close 4102 (time 1787562610.42 total 50.05 last 86.85) - open/close 4984 (time 1787562620.43 total 60.06 last 88.14) - open/close 5829 (time 1787562630.43 total 70.06 last 84.49) - open/close 6657 (time 1787562640.44 total 80.07 last 82.70) - open/close 7503 (time 1787562650.44 total 90.07 last 84.59) - open/close 8306 (time 1787562660.45 total 100.08 last 80.23) - open/close 9008 (time 1787562670.45 total 110.09 last 70.17) - open/close 9690 (time 1787562680.45 total 120.09 last 68.20) total: 10000 open/close in 125.08 seconds: 79.95 ops/second - unlinked 0 (time 1787562690 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 746 (time 1787562773.07 total 10.02 last 74.44) - open/close 1519 (time 1787562783.07 total 20.03 last 77.25) - open/close 2282 (time 1787562793.09 total 30.05 last 76.15) - open/close 3081 (time 1787562803.11 total 40.06 last 79.79) - open/close 3752 (time 1787562813.12 total 50.08 last 67.00) - open/close 4526 (time 1787562823.14 total 60.09 last 77.27) - open/close 5284 (time 1787562833.14 total 70.09 last 75.80) - open/close 6096 (time 1787562843.15 total 80.10 last 81.14) - open/close 6889 (time 1787562853.16 total 90.11 last 79.24) - open/close 7686 (time 1787562863.16 total 100.12 last 79.65) - open/close 8456 (time 1787562873.17 total 110.12 last 76.95) - open/close 9295 (time 1787562883.17 total 120.13 last 83.87) total: 10000 open/close in 128.78 seconds: 77.65 ops/second - unlinked 0 (time 1787562896 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 720 (time 1787562982.06 total 10.01 last 71.93) - open/close 1296 (time 1787562992.07 total 20.01 last 57.58) - open/close 1940 (time 1787563002.08 total 30.03 last 64.31) - open/close 2626 (time 1787563012.09 total 40.04 last 68.52) - open/close 3451 (time 1787563022.09 total 50.04 last 82.50) - open/close 4299 (time 1787563032.09 total 60.04 last 84.78) - open/close 5096 (time 1787563042.12 total 70.06 last 79.51) - open/close 5924 (time 1787563052.12 total 80.06 last 82.80) - open/close 6680 (time 1787563062.12 total 90.07 last 75.58) - open/close 7585 (time 1787563072.12 total 100.07 last 90.48) - open/close 8408 (time 1787563082.14 total 110.08 last 82.19) - open/close 9333 (time 1787563092.14 total 120.08 last 92.50) total: 10000 open/close in 127.56 seconds: 78.40 ops/second - unlinked 0 (time 1787563103 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg626-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg626-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg626-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg626-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg626-server: oleg626-server.virtnet: executing set_default_debug -1 all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Started lustre-OST0000 oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg626-server: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg626-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg626-server: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg626-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg626-client: oleg626-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg626-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg626-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg626-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.126@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.126@tcp sys.timeout=20 start ost1 service on oleg626-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg626-server: oleg626-server.virtnet: executing set_default_debug -1 all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Started lustre-OST0000 oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg626-server: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg626-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg626-server: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg626-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg626-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg626-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg626-server stop mds service on oleg626-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg626-server stop mds service on oleg626-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg626-server unloading modules via unload_modules_local on: 'oleg626-server' oleg626-server: oleg626-server.virtnet: executing unload_modules_local oleg626-server: modules unloaded. pdsh@oleg626-client: oleg626-server: ssh exited with exit code 2 pdsh@oleg626-client: oleg626-client: ssh exited with exit code 2 pdsh@oleg626-client: oleg626-client: ssh exited with exit code 2 pdsh@oleg626-client: oleg626-server: ssh exited with exit code 2