== conf-sanity test 69: replace an OST with the same index ========================================================== 09:19:09 (1788441549) start mds service on oleg603-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg603-server' oleg603-server: oleg603-server.virtnet: executing load_modules_local oleg603-server: Loading modules from /home/green/git/lustre-release/lustre oleg603-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg603-server: detected 4 online CPUs by sysfs oleg603-server: MODOPTS_LIBCFS= oleg603-server: Force libcfs to create 2 CPU partitions oleg603-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg603-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg603-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg603-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg603-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg603-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 736 (time 1788441660.70 total 10.00 last 73.58) - open/close 1557 (time 1788441670.71 total 20.01 last 82.06) - open/close 2260 (time 1788441680.72 total 30.02 last 70.20) - open/close 2819 (time 1788441690.74 total 40.04 last 55.82) - open/close 3413 (time 1788441700.75 total 50.05 last 59.29) - open/close 4078 (time 1788441710.76 total 60.06 last 66.48) - open/close 4913 (time 1788441720.76 total 70.07 last 83.43) - open/close 5665 (time 1788441730.77 total 80.07 last 75.14) - open/close 6428 (time 1788441740.79 total 90.09 last 76.21) - open/close 7152 (time 1788441750.79 total 100.09 last 72.35) - open/close 7912 (time 1788441760.79 total 110.09 last 76.00) - open/close 8648 (time 1788441770.80 total 120.10 last 73.58) - open/close 9480 (time 1788441780.80 total 130.10 last 83.13) total: 10000 open/close in 136.52 seconds: 73.25 ops/second - unlinked 0 (time 1788441791 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 791 (time 1788441876.37 total 10.00 last 79.10) - open/close 1540 (time 1788441886.38 total 20.01 last 74.82) - open/close 2333 (time 1788441896.39 total 30.01 last 79.29) - open/close 3138 (time 1788441906.40 total 40.03 last 80.37) - open/close 3967 (time 1788441916.41 total 50.04 last 82.85) - open/close 4764 (time 1788441926.42 total 60.05 last 79.60) - open/close 5522 (time 1788441936.43 total 70.05 last 75.77) - open/close 6295 (time 1788441946.43 total 80.06 last 77.28) - open/close 7075 (time 1788441956.47 total 90.10 last 77.66) - open/close 7854 (time 1788441966.48 total 100.10 last 77.86) - open/close 8740 (time 1788441976.49 total 110.12 last 88.47) - open/close 9574 (time 1788441986.50 total 120.12 last 83.36) total: 10000 open/close in 125.49 seconds: 79.69 ops/second - unlinked 0 (time 1788441995 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 796 (time 1788442081.32 total 10.01 last 79.54) - open/close 1597 (time 1788442091.33 total 20.02 last 80.04) - open/close 2370 (time 1788442101.33 total 30.02 last 77.29) - open/close 3209 (time 1788442111.34 total 40.02 last 83.88) - open/close 3967 (time 1788442121.34 total 50.02 last 75.78) - open/close 4734 (time 1788442131.35 total 60.04 last 76.58) - open/close 5567 (time 1788442141.37 total 70.05 last 83.17) - open/close 6345 (time 1788442151.41 total 80.09 last 77.51) - open/close 7057 (time 1788442161.41 total 90.09 last 71.17) - open/close 7897 (time 1788442171.43 total 100.11 last 83.87) - open/close 8722 (time 1788442181.43 total 110.11 last 82.46) - open/close 9522 (time 1788442191.44 total 120.12 last 79.96) total: 10000 open/close in 126.22 seconds: 79.23 ops/second - unlinked 0 (time 1788442201 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 867 (time 1788442286.00 total 10.00 last 86.69) - open/close 1736 (time 1788442296.01 total 20.00 last 86.88) - open/close 2552 (time 1788442306.01 total 30.00 last 81.60) - open/close 3393 (time 1788442316.01 total 40.01 last 84.03) - open/close 4132 (time 1788442326.02 total 50.01 last 73.88) - open/close 4925 (time 1788442336.04 total 60.04 last 79.12) - open/close 5612 (time 1788442346.05 total 70.04 last 68.66) - open/close 6292 (time 1788442356.05 total 80.05 last 67.95) - open/close 7075 (time 1788442366.06 total 90.06 last 78.25) - open/close 7904 (time 1788442376.07 total 100.06 last 82.83) - open/close 8756 (time 1788442386.07 total 110.07 last 85.17) - open/close 9571 (time 1788442396.07 total 120.07 last 81.50) total: 10000 open/close in 125.54 seconds: 79.66 ops/second - unlinked 0 (time 1788442405 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 762 (time 1788442492.72 total 10.00 last 76.19) - open/close 1496 (time 1788442502.73 total 20.01 last 73.33) - open/close 2262 (time 1788442512.74 total 30.02 last 76.55) - open/close 3095 (time 1788442522.74 total 40.02 last 83.27) - open/close 3838 (time 1788442532.75 total 50.03 last 74.22) - open/close 4694 (time 1788442542.75 total 60.03 last 85.59) - open/close 5505 (time 1788442552.75 total 70.03 last 81.09) - open/close 6266 (time 1788442562.76 total 80.04 last 76.08) - open/close 7072 (time 1788442572.77 total 90.05 last 80.50) - open/close 7919 (time 1788442582.77 total 100.05 last 84.68) - open/close 8767 (time 1788442592.78 total 110.07 last 84.68) - open/close 9553 (time 1788442602.79 total 120.07 last 78.57) total: 10000 open/close in 125.75 seconds: 79.52 ops/second - unlinked 0 (time 1788442613 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 674 (time 1788442705.44 total 10.01 last 67.36) - open/close 1383 (time 1788442715.45 total 20.02 last 70.81) - open/close 2162 (time 1788442725.47 total 30.03 last 77.80) - open/close 2867 (time 1788442735.47 total 40.04 last 70.46) - open/close 3658 (time 1788442745.47 total 50.04 last 79.08) - open/close 4504 (time 1788442755.48 total 60.05 last 84.52) - open/close 5316 (time 1788442765.49 total 70.06 last 81.14) - open/close 6068 (time 1788442775.50 total 80.06 last 75.14) - open/close 6865 (time 1788442785.50 total 90.07 last 79.67) - open/close 7658 (time 1788442795.50 total 100.07 last 79.29) - open/close 8453 (time 1788442805.51 total 110.07 last 79.45) - open/close 9180 (time 1788442815.51 total 120.08 last 72.68) - open/close 9986 (time 1788442825.52 total 130.08 last 80.59) total: 10000 open/close in 130.23 seconds: 76.78 ops/second - unlinked 0 (time 1788442829 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 768 (time 1788442919.88 total 10.00 last 76.77) - open/close 1537 (time 1788442929.89 total 20.01 last 76.85) - open/close 2342 (time 1788442939.90 total 30.02 last 80.43) - open/close 3202 (time 1788442949.90 total 40.02 last 85.96) - open/close 3967 (time 1788442959.91 total 50.03 last 76.46) - open/close 4674 (time 1788442969.91 total 60.03 last 70.69) - open/close 5471 (time 1788442979.91 total 70.03 last 79.70) - open/close 6369 (time 1788442989.91 total 80.04 last 89.75) - open/close 7294 (time 1788442999.94 total 90.06 last 92.23) - open/close 8066 (time 1788443009.95 total 100.07 last 77.15) - open/close 8856 (time 1788443019.97 total 110.09 last 78.85) - open/close 9597 (time 1788443029.98 total 120.11 last 73.99) total: 10000 open/close in 125.73 seconds: 79.54 ops/second - unlinked 0 (time 1788443040 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 798 (time 1788443129.23 total 10.01 last 79.73) - open/close 1461 (time 1788443139.24 total 20.02 last 66.25) - open/close 2178 (time 1788443149.24 total 30.02 last 71.68) - open/close 2976 (time 1788443159.25 total 40.03 last 79.73) - open/close 3812 (time 1788443169.26 total 50.04 last 83.50) - open/close 4684 (time 1788443179.27 total 60.05 last 87.11) - open/close 5503 (time 1788443189.29 total 70.06 last 81.79) - open/close 6308 (time 1788443199.30 total 80.08 last 80.40) - open/close 7059 (time 1788443209.30 total 90.08 last 75.10) - open/close 7822 (time 1788443219.30 total 100.08 last 76.30) - open/close 8635 (time 1788443229.30 total 110.08 last 81.27) - open/close 9445 (time 1788443239.31 total 120.08 last 80.98) total: 10000 open/close in 126.58 seconds: 79.00 ops/second - unlinked 0 (time 1788443249 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 799 (time 1788443333.83 total 10.00 last 79.90) - open/close 1607 (time 1788443343.84 total 20.01 last 80.73) - open/close 2450 (time 1788443353.85 total 30.02 last 84.23) - open/close 3307 (time 1788443363.85 total 40.02 last 85.67) - open/close 4131 (time 1788443373.86 total 50.02 last 82.39) - open/close 4966 (time 1788443383.86 total 60.03 last 83.47) - open/close 5826 (time 1788443393.87 total 70.03 last 85.92) - open/close 6740 (time 1788443403.87 total 80.04 last 91.38) - open/close 7568 (time 1788443413.87 total 90.04 last 82.80) - open/close 8410 (time 1788443423.87 total 100.04 last 84.19) - open/close 9259 (time 1788443433.88 total 110.05 last 84.84) total: 10000 open/close in 119.29 seconds: 83.83 ops/second - unlinked 0 (time 1788443447 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 756 (time 1788443533.22 total 10.00 last 75.60) - open/close 1523 (time 1788443543.24 total 20.02 last 76.58) - open/close 2254 (time 1788443553.25 total 30.03 last 73.01) - open/close 3031 (time 1788443563.26 total 40.03 last 77.67) - open/close 3886 (time 1788443573.27 total 50.05 last 85.36) - open/close 4854 (time 1788443583.28 total 60.06 last 96.73) - open/close 5703 (time 1788443593.29 total 70.07 last 84.83) - open/close 6452 (time 1788443603.31 total 80.09 last 74.72) - open/close 7258 (time 1788443613.32 total 90.09 last 80.58) - open/close 7937 (time 1788443623.32 total 100.10 last 67.84) - open/close 8569 (time 1788443633.33 total 110.10 last 63.17) - open/close 9342 (time 1788443643.34 total 120.11 last 77.22) total: 10000 open/close in 127.47 seconds: 78.45 ops/second - unlinked 0 (time 1788443654 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg603-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg603-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg603-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.103@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg603-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg603-server: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg603-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg603-server: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg603-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg603-client: oleg603-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg603-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg603-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg603-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.103@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.103@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.103@tcp sys.timeout=20 start ost1 service on oleg603-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg603-server: oleg603-server.virtnet: executing set_default_debug -1 all pdsh@oleg603-client: oleg603-server: ssh exited with exit code 1 Started lustre-OST0000 oleg603-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg603-server: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg603-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg603-server: oleg603-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg603-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg603-client.virtnet: -o user_xattr,flock 192.168.206.103@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg603-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg603-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg603-server stop mds service on oleg603-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg603-server stop mds service on oleg603-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg603-server unloading modules via unload_modules_local on: 'oleg603-server' oleg603-server: oleg603-server.virtnet: executing unload_modules_local oleg603-server: modules unloaded. pdsh@oleg603-client: oleg603-client: ssh exited with exit code 2 pdsh@oleg603-client: oleg603-server: ssh exited with exit code 2 pdsh@oleg603-client: oleg603-server: ssh exited with exit code 2 pdsh@oleg603-client: oleg603-client: ssh exited with exit code 2