== conf-sanity test 69: replace an OST with the same index ========================================================== 04:59:07 (1788598747) start mds service on oleg223-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg223-server: detected 4 online CPUs by sysfs oleg223-server: MODOPTS_LIBCFS= oleg223-server: Force libcfs to create 2 CPU partitions oleg223-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg223-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg223-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg223-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg223-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 800 (time 1788598862.13 total 10.04 last 79.72) - open/close 1536 (time 1788598872.14 total 20.04 last 73.55) - open/close 2293 (time 1788598882.14 total 30.04 last 75.68) - open/close 3040 (time 1788598892.15 total 40.05 last 74.67) - open/close 3938 (time 1788598902.15 total 50.05 last 89.78) - open/close 4713 (time 1788598912.16 total 60.06 last 77.41) - open/close 5355 (time 1788598922.16 total 70.07 last 64.17) - open/close 6159 (time 1788598932.18 total 80.08 last 80.30) - open/close 6766 (time 1788598942.18 total 90.09 last 60.66) - open/close 7574 (time 1788598952.18 total 100.09 last 80.79) - open/close 8506 (time 1788598962.19 total 110.10 last 93.13) - open/close 9406 (time 1788598972.20 total 120.10 last 89.97) total: 10000 open/close in 125.83 seconds: 79.47 ops/second - unlinked 0 (time 1788598981 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 815 (time 1788599067.24 total 10.01 last 81.43) - open/close 1646 (time 1788599077.25 total 20.01 last 83.05) - open/close 2510 (time 1788599087.27 total 30.04 last 86.19) - open/close 3374 (time 1788599097.28 total 40.04 last 86.34) - open/close 4282 (time 1788599107.29 total 50.05 last 90.73) - open/close 5079 (time 1788599117.31 total 60.07 last 79.55) - open/close 5924 (time 1788599127.31 total 70.08 last 84.46) - open/close 6884 (time 1788599137.31 total 80.08 last 95.99) - open/close 7655 (time 1788599147.32 total 90.09 last 77.02) - open/close 8440 (time 1788599157.33 total 100.09 last 78.49) - open/close 9279 (time 1788599167.33 total 110.09 last 83.89) - open/close 9947 (time 1788599177.33 total 120.09 last 66.80) total: 10000 open/close in 120.96 seconds: 82.67 ops/second - unlinked 0 (time 1788599182 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 680 (time 1788599272.72 total 10.01 last 67.96) - open/close 1503 (time 1788599282.73 total 20.01 last 82.27) - open/close 2361 (time 1788599292.74 total 30.02 last 85.73) - open/close 3262 (time 1788599302.75 total 40.03 last 89.99) - open/close 4033 (time 1788599312.76 total 50.04 last 77.04) - open/close 4785 (time 1788599322.76 total 60.05 last 75.13) - open/close 5531 (time 1788599332.77 total 70.06 last 74.54) - open/close 6296 (time 1788599342.78 total 80.07 last 76.41) - open/close 6956 (time 1788599352.79 total 90.07 last 65.98) - open/close 7665 (time 1788599362.80 total 100.08 last 70.83) - open/close 8412 (time 1788599372.80 total 110.09 last 74.66) - open/close 9238 (time 1788599382.80 total 120.09 last 82.58) total: 10000 open/close in 128.35 seconds: 77.91 ops/second - unlinked 0 (time 1788599394 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 940 (time 1788599471.23 total 10.00 last 94.00) - open/close 1748 (time 1788599481.25 total 20.01 last 80.68) - open/close 2564 (time 1788599491.25 total 30.02 last 81.56) - open/close 3416 (time 1788599501.25 total 40.02 last 85.17) - open/close 4300 (time 1788599511.27 total 50.04 last 88.29) - open/close 5159 (time 1788599521.27 total 60.04 last 85.90) - open/close 6013 (time 1788599531.27 total 70.04 last 85.37) - open/close 6790 (time 1788599541.27 total 80.04 last 77.69) - open/close 7682 (time 1788599551.28 total 90.05 last 89.12) - open/close 8495 (time 1788599561.29 total 100.05 last 81.26) - open/close 9349 (time 1788599571.29 total 110.06 last 85.34) total: 10000 open/close in 117.62 seconds: 85.02 ops/second - unlinked 0 (time 1788599582 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 712 (time 1788599667.13 total 10.00 last 71.17) - open/close 1567 (time 1788599677.13 total 20.00 last 85.50) - open/close 2245 (time 1788599687.13 total 30.01 last 67.80) - open/close 3318 (time 1788599697.13 total 40.01 last 107.24) - open/close 4073 (time 1788599707.14 total 50.02 last 75.45) - open/close 4693 (time 1788599717.15 total 60.03 last 61.95) - open/close 5421 (time 1788599727.16 total 70.04 last 72.71) - open/close 6208 (time 1788599737.17 total 80.04 last 78.66) - open/close 6981 (time 1788599747.17 total 90.05 last 77.26) - open/close 7801 (time 1788599757.18 total 100.05 last 81.96) - open/close 8526 (time 1788599767.18 total 110.06 last 72.46) - open/close 9226 (time 1788599777.19 total 120.07 last 69.93) total: 10000 open/close in 129.84 seconds: 77.02 ops/second - unlinked 0 (time 1788599790 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 834 (time 1788599876.03 total 10.01 last 83.34) - open/close 1543 (time 1788599886.04 total 20.01 last 70.85) - open/close 2290 (time 1788599896.04 total 30.01 last 74.70) - open/close 3075 (time 1788599906.05 total 40.02 last 78.43) - open/close 3706 (time 1788599916.07 total 50.05 last 62.93) - open/close 4555 (time 1788599926.07 total 60.05 last 84.90) - open/close 5332 (time 1788599936.08 total 70.06 last 77.64) - open/close 6078 (time 1788599946.10 total 80.08 last 74.45) - open/close 6823 (time 1788599956.10 total 90.08 last 74.49) - open/close 7683 (time 1788599966.10 total 100.08 last 85.98) - open/close 8548 (time 1788599976.11 total 110.09 last 86.42) - open/close 9426 (time 1788599986.12 total 120.10 last 87.76) total: 10000 open/close in 127.26 seconds: 78.58 ops/second - unlinked 0 (time 1788599998 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 806 (time 1788600087.73 total 10.01 last 80.51) - open/close 1590 (time 1788600097.74 total 20.02 last 78.35) - open/close 2443 (time 1788600107.74 total 30.02 last 85.24) - open/close 3245 (time 1788600117.75 total 40.03 last 80.14) - open/close 4121 (time 1788600127.76 total 50.04 last 87.57) - open/close 4942 (time 1788600137.76 total 60.04 last 82.06) - open/close 5830 (time 1788600147.77 total 70.05 last 88.76) - open/close 6657 (time 1788600157.78 total 80.06 last 82.56) - open/close 7525 (time 1788600167.79 total 90.07 last 86.72) - open/close 8340 (time 1788600177.80 total 100.08 last 81.45) - open/close 9243 (time 1788600187.81 total 110.09 last 90.24) total: 10000 open/close in 119.67 seconds: 83.56 ops/second - unlinked 0 (time 1788600202 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 884 (time 1788600289.61 total 10.00 last 88.36) - open/close 1753 (time 1788600299.61 total 20.01 last 86.85) - open/close 2550 (time 1788600309.62 total 30.01 last 79.68) - open/close 3233 (time 1788600319.62 total 40.02 last 68.28) - open/close 4040 (time 1788600329.64 total 50.04 last 80.53) - open/close 5055 (time 1788600339.64 total 60.04 last 101.46) - open/close 5864 (time 1788600349.64 total 70.04 last 80.90) - open/close 6690 (time 1788600359.66 total 80.05 last 82.48) - open/close 7459 (time 1788600369.66 total 90.06 last 76.88) - open/close 8310 (time 1788600379.66 total 100.06 last 85.08) - open/close 9108 (time 1788600389.67 total 110.07 last 79.72) - open/close 9914 (time 1788600399.68 total 120.07 last 80.58) total: 10000 open/close in 120.98 seconds: 82.66 ops/second - unlinked 0 (time 1788600404 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 809 (time 1788600487.66 total 10.01 last 80.84) - open/close 1587 (time 1788600497.66 total 20.01 last 77.76) - open/close 2482 (time 1788600507.67 total 30.02 last 89.41) - open/close 3336 (time 1788600517.68 total 40.03 last 85.32) - open/close 4201 (time 1788600527.69 total 50.04 last 86.47) - open/close 5139 (time 1788600537.69 total 60.04 last 93.79) - open/close 6013 (time 1788600547.69 total 70.04 last 87.38) - open/close 6743 (time 1788600557.69 total 80.04 last 72.97) - open/close 7373 (time 1788600567.69 total 90.04 last 62.99) - open/close 8215 (time 1788600577.71 total 100.06 last 84.10) - open/close 9029 (time 1788600587.71 total 110.06 last 81.33) - open/close 9867 (time 1788600597.72 total 120.07 last 83.72) total: 10000 open/close in 121.47 seconds: 82.32 ops/second - unlinked 0 (time 1788600603 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 732 (time 1788600696.58 total 10.01 last 73.14) - open/close 1468 (time 1788600706.59 total 20.01 last 73.58) - open/close 2262 (time 1788600716.60 total 30.02 last 79.31) - open/close 3128 (time 1788600726.61 total 40.03 last 86.53) - open/close 3980 (time 1788600736.61 total 50.03 last 85.20) - open/close 4607 (time 1788600746.62 total 60.04 last 62.63) - open/close 5323 (time 1788600756.65 total 70.07 last 71.40) - open/close 6092 (time 1788600766.66 total 80.08 last 76.82) - open/close 6903 (time 1788600776.67 total 90.09 last 81.01) - open/close 7778 (time 1788600786.67 total 100.10 last 87.46) - open/close 8582 (time 1788600796.68 total 110.10 last 80.32) - open/close 9194 (time 1788600806.69 total 120.11 last 61.17) - open/close 9937 (time 1788600816.69 total 130.12 last 74.24) total: 10000 open/close in 131.05 seconds: 76.31 ops/second - unlinked 0 (time 1788600822 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg223-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg223-server: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg223-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg223-server: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg223-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre On OST0, 19970 used inodes rc=0 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.123@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.123@tcp sys.timeout=20 start ost1 service on oleg223-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg223-server: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg223-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg223-server: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg223-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg223-server unloading modules via unload_modules_local on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local oleg223-server: modules unloaded. pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2