== conf-sanity test 69: replace an OST with the same index ========================================================== 10:44:10 (1787582650) start mds service on oleg613-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg613-server' oleg613-server: oleg613-server.virtnet: executing load_modules_local oleg613-server: Loading modules from /home/green/git/lustre-release/lustre oleg613-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg613-server: detected 4 online CPUs by sysfs oleg613-server: MODOPTS_LIBCFS= oleg613-server: Force libcfs to create 2 CPU partitions oleg613-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg613-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg613-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg613-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg613-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0000 oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 902 (time 1787582753.19 total 10.00 last 90.20) - open/close 1818 (time 1787582763.20 total 20.01 last 91.52) - open/close 2783 (time 1787582773.20 total 30.01 last 96.50) - open/close 3764 (time 1787582783.20 total 40.01 last 98.05) - open/close 4811 (time 1787582793.21 total 50.02 last 104.65) - open/close 5816 (time 1787582803.21 total 60.02 last 100.46) - open/close 6759 (time 1787582813.21 total 70.02 last 94.29) - open/close 7887 (time 1787582823.21 total 80.03 last 112.75) - open/close 8899 (time 1787582833.22 total 90.03 last 101.15) - open/close 9871 (time 1787582843.23 total 100.04 last 97.13) total: 10000 open/close in 101.13 seconds: 98.88 ops/second - unlinked 0 (time 1787582850 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 896 (time 1787582922.97 total 10.02 last 89.46) - open/close 1708 (time 1787582932.97 total 20.02 last 81.19) - open/close 2603 (time 1787582942.98 total 30.03 last 89.39) - open/close 3503 (time 1787582952.98 total 40.03 last 89.98) - open/close 4433 (time 1787582962.99 total 50.04 last 92.94) - open/close 5404 (time 1787582972.99 total 60.04 last 97.08) - open/close 6506 (time 1787582982.99 total 70.04 last 110.17) - open/close 7463 (time 1787582992.99 total 80.04 last 95.69) - open/close 8478 (time 1787583002.99 total 90.04 last 101.49) - open/close 9509 (time 1787583013.00 total 100.05 last 103.06) total: 10000 open/close in 104.65 seconds: 95.55 ops/second - unlinked 0 (time 1787583021 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 850 (time 1787583094.63 total 10.00 last 84.97) - open/close 1805 (time 1787583104.65 total 20.02 last 95.34) - open/close 2755 (time 1787583114.66 total 30.03 last 94.90) - open/close 3754 (time 1787583124.67 total 40.03 last 99.85) - open/close 4771 (time 1787583134.67 total 50.04 last 101.67) - open/close 5766 (time 1787583144.68 total 60.05 last 99.43) - open/close 6822 (time 1787583154.68 total 70.05 last 105.51) - open/close 7778 (time 1787583164.69 total 80.06 last 95.57) - open/close 8753 (time 1787583174.69 total 90.06 last 97.47) - open/close 9756 (time 1787583184.70 total 100.07 last 100.22) total: 10000 open/close in 103.12 seconds: 96.98 ops/second - unlinked 0 (time 1787583191 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 934 (time 1787583261.05 total 10.01 last 93.34) - open/close 1941 (time 1787583271.05 total 20.01 last 100.63) - open/close 2967 (time 1787583281.06 total 30.02 last 102.54) - open/close 3995 (time 1787583291.06 total 40.02 last 102.76) - open/close 4986 (time 1787583301.06 total 50.02 last 99.09) - open/close 6078 (time 1787583311.07 total 60.03 last 109.12) - open/close 7064 (time 1787583321.08 total 70.04 last 98.56) - open/close 8000 (time 1787583331.09 total 80.05 last 93.44) - open/close 8759 (time 1787583341.10 total 90.06 last 75.86) - open/close 9369 (time 1787583351.11 total 100.07 last 60.94) total: 10000 open/close in 106.84 seconds: 93.60 ops/second - unlinked 0 (time 1787583362 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1024 (time 1787583433.09 total 10.02 last 102.22) - open/close 1959 (time 1787583443.09 total 20.02 last 93.48) - open/close 2965 (time 1787583453.11 total 30.03 last 100.45) - open/close 3949 (time 1787583463.11 total 40.04 last 98.32) - open/close 4998 (time 1787583473.12 total 50.05 last 104.83) - open/close 5993 (time 1787583483.14 total 60.06 last 99.36) - open/close 6923 (time 1787583493.14 total 70.07 last 92.96) - open/close 7918 (time 1787583503.15 total 80.07 last 99.42) - open/close 8846 (time 1787583513.16 total 90.08 last 92.72) - open/close 9830 (time 1787583523.16 total 100.09 last 98.35) total: 10000 open/close in 102.56 seconds: 97.50 ops/second - unlinked 0 (time 1787583529 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1000 (time 1787583601.47 total 10.00 last 100.00) - open/close 1919 (time 1787583611.47 total 20.00 last 91.89) - open/close 2869 (time 1787583621.47 total 30.00 last 95.00) - open/close 3867 (time 1787583631.48 total 40.01 last 99.76) - open/close 4921 (time 1787583641.48 total 50.01 last 105.39) - open/close 5978 (time 1787583651.48 total 60.01 last 105.66) - open/close 6946 (time 1787583661.52 total 70.04 last 96.47) - open/close 7931 (time 1787583671.52 total 80.05 last 98.47) - open/close 8890 (time 1787583681.53 total 90.05 last 95.83) - open/close 9903 (time 1787583691.53 total 100.06 last 101.21) total: 10000 open/close in 101.08 seconds: 98.93 ops/second - unlinked 0 (time 1787583696 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 964 (time 1787583771.19 total 10.01 last 96.34) - open/close 1909 (time 1787583781.19 total 20.01 last 94.47) - open/close 2859 (time 1787583791.19 total 30.01 last 94.99) - open/close 3876 (time 1787583801.20 total 40.02 last 101.62) - open/close 4704 (time 1787583811.21 total 50.03 last 82.70) - open/close 5597 (time 1787583821.22 total 60.04 last 89.23) - open/close 6464 (time 1787583831.22 total 70.04 last 86.68) - open/close 7311 (time 1787583841.26 total 80.08 last 84.39) - open/close 8157 (time 1787583851.27 total 90.08 last 84.54) - open/close 9079 (time 1787583861.27 total 100.09 last 92.19) total: 10000 open/close in 109.20 seconds: 91.57 ops/second - unlinked 0 (time 1787583874 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1011 (time 1787583946.99 total 10.00 last 101.09) - open/close 2016 (time 1787583957.00 total 20.01 last 100.45) - open/close 3058 (time 1787583967.01 total 30.02 last 104.07) - open/close 4023 (time 1787583977.02 total 40.02 last 96.44) - open/close 5038 (time 1787583987.02 total 50.03 last 101.45) - open/close 6098 (time 1787583997.03 total 60.03 last 105.97) - open/close 7072 (time 1787584007.03 total 70.04 last 97.34) - open/close 8047 (time 1787584017.03 total 80.04 last 97.47) - open/close 9033 (time 1787584027.04 total 90.05 last 98.54) total: 10000 open/close in 99.21 seconds: 100.80 ops/second - unlinked 0 (time 1787584039 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 978 (time 1787584110.89 total 10.02 last 97.65) - open/close 1764 (time 1787584120.90 total 20.03 last 78.50) - open/close 2617 (time 1787584130.91 total 30.03 last 85.25) - open/close 3496 (time 1787584140.91 total 40.04 last 87.87) - open/close 4578 (time 1787584150.92 total 50.04 last 108.13) - open/close 5578 (time 1787584160.92 total 60.05 last 99.95) - open/close 6579 (time 1787584170.93 total 70.05 last 100.06) - open/close 7519 (time 1787584180.93 total 80.06 last 93.91) - open/close 8471 (time 1787584190.94 total 90.07 last 95.12) - open/close 9551 (time 1787584200.95 total 100.07 last 107.97) total: 10000 open/close in 104.58 seconds: 95.62 ops/second - unlinked 0 (time 1787584211 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1018 (time 1787584285.11 total 10.00 last 101.79) - open/close 1891 (time 1787584295.12 total 20.01 last 87.22) - open/close 2415 (time 1787584305.12 total 30.01 last 52.39) - open/close 2920 (time 1787584315.14 total 40.03 last 50.40) - open/close 3453 (time 1787584325.14 total 50.04 last 53.28) - open/close 4199 (time 1787584335.15 total 60.04 last 74.54) - open/close 5054 (time 1787584345.15 total 70.05 last 85.47) - open/close 6015 (time 1787584355.16 total 80.05 last 96.06) - open/close 7116 (time 1787584365.16 total 90.05 last 110.08) - open/close 8130 (time 1787584375.19 total 100.08 last 101.13) - open/close 9206 (time 1787584385.19 total 110.09 last 107.56) total: 10000 open/close in 117.68 seconds: 84.97 ops/second - unlinked 0 (time 1787584397 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg613-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg613-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.113@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg613-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0000 oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg613-server: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg613-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg613-server: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg613-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg613-client: oleg613-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg613-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg613-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.113@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.113@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.113@tcp sys.timeout=20 start ost1 service on oleg613-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0000 oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg613-server: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg613-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg613-server: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg613-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg613-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg613-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server stop mds service on oleg613-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg613-server stop mds service on oleg613-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg613-server unloading modules via unload_modules_local on: 'oleg613-server' oleg613-server: oleg613-server.virtnet: executing unload_modules_local oleg613-server: modules unloaded. pdsh@oleg613-client: oleg613-client: ssh exited with exit code 2 pdsh@oleg613-client: oleg613-server: ssh exited with exit code 2 pdsh@oleg613-client: oleg613-client: ssh exited with exit code 2 pdsh@oleg613-client: oleg613-server: ssh exited with exit code 2