== conf-sanity test 69: replace an OST with the same index ========================================================== 23:10:31 (1787368231) start mds service on oleg342-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg342-server: detected 4 online CPUs by sysfs oleg342-server: MODOPTS_LIBCFS= oleg342-server: Force libcfs to create 2 CPU partitions oleg342-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg342-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg342-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg342-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg342-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 783 (time 1787368334.93 total 10.00 last 78.29) - open/close 1516 (time 1787368344.94 total 20.01 last 73.26) - open/close 2271 (time 1787368354.95 total 30.01 last 75.45) - open/close 3011 (time 1787368364.95 total 40.02 last 73.97) - open/close 3758 (time 1787368374.97 total 50.03 last 74.58) - open/close 4518 (time 1787368384.97 total 60.03 last 75.99) - open/close 5333 (time 1787368394.97 total 70.04 last 81.47) - open/close 6133 (time 1787368404.98 total 80.05 last 79.92) - open/close 6868 (time 1787368414.99 total 90.06 last 73.44) - open/close 7540 (time 1787368425.01 total 100.07 last 67.08) - open/close 8280 (time 1787368435.01 total 110.07 last 73.99) - open/close 8939 (time 1787368445.02 total 120.09 last 65.79) - open/close 9587 (time 1787368455.03 total 130.10 last 64.75) total: 10000 open/close in 136.11 seconds: 73.47 ops/second - unlinked 0 (time 1787368465 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 800 (time 1787368555.01 total 10.01 last 79.92) - open/close 1575 (time 1787368565.01 total 20.01 last 77.49) - open/close 2317 (time 1787368575.03 total 30.03 last 74.06) - open/close 3085 (time 1787368585.03 total 40.03 last 76.77) - open/close 3824 (time 1787368595.05 total 50.05 last 73.74) - open/close 4593 (time 1787368605.06 total 60.06 last 76.85) - open/close 5389 (time 1787368615.06 total 70.07 last 79.55) - open/close 6145 (time 1787368625.08 total 80.08 last 75.49) - open/close 6900 (time 1787368635.09 total 90.10 last 75.39) - open/close 7700 (time 1787368645.10 total 100.10 last 79.97) - open/close 8486 (time 1787368655.10 total 110.11 last 78.53) - open/close 9166 (time 1787368665.11 total 120.11 last 67.97) - open/close 9875 (time 1787368675.11 total 130.12 last 70.87) total: 10000 open/close in 131.79 seconds: 75.88 ops/second - unlinked 0 (time 1787368680 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 818 (time 1787368769.07 total 10.01 last 81.75) - open/close 1611 (time 1787368779.09 total 20.02 last 79.17) - open/close 2374 (time 1787368789.09 total 30.03 last 76.25) - open/close 3146 (time 1787368799.11 total 40.05 last 77.04) - open/close 3964 (time 1787368809.13 total 50.07 last 81.65) - open/close 4685 (time 1787368819.14 total 60.07 last 72.06) - open/close 5495 (time 1787368829.14 total 70.08 last 80.97) - open/close 6259 (time 1787368839.15 total 80.08 last 76.37) - open/close 6977 (time 1787368849.15 total 90.09 last 71.75) - open/close 7768 (time 1787368859.16 total 100.10 last 79.02) - open/close 8458 (time 1787368869.18 total 110.11 last 68.89) - open/close 8998 (time 1787368879.19 total 120.12 last 53.95) - open/close 9712 (time 1787368889.19 total 130.12 last 71.39) total: 10000 open/close in 134.34 seconds: 74.44 ops/second - unlinked 0 (time 1787368897 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 636 (time 1787368988.68 total 10.01 last 63.51) - open/close 1296 (time 1787368998.68 total 20.02 last 65.99) - open/close 2033 (time 1787369008.69 total 30.02 last 73.66) - open/close 2735 (time 1787369018.69 total 40.03 last 70.17) - open/close 3516 (time 1787369028.70 total 50.03 last 78.04) - open/close 4304 (time 1787369038.70 total 60.04 last 78.76) - open/close 5098 (time 1787369048.71 total 70.04 last 79.36) - open/close 5826 (time 1787369058.72 total 80.06 last 72.69) - open/close 6543 (time 1787369068.73 total 90.07 last 71.61) - open/close 7213 (time 1787369078.75 total 100.08 last 66.92) - open/close 8001 (time 1787369088.75 total 110.09 last 78.76) - open/close 8728 (time 1787369098.76 total 120.10 last 72.64) - open/close 9525 (time 1787369108.77 total 130.10 last 79.64) total: 10000 open/close in 136.43 seconds: 73.30 ops/second - unlinked 0 (time 1787369119 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 678 (time 1787369205.91 total 10.02 last 67.68) - open/close 1409 (time 1787369215.92 total 20.02 last 73.07) - open/close 2188 (time 1787369225.92 total 30.03 last 77.86) - open/close 2900 (time 1787369235.93 total 40.04 last 71.11) - open/close 3669 (time 1787369245.94 total 50.04 last 76.88) - open/close 4389 (time 1787369255.94 total 60.05 last 71.95) - open/close 5154 (time 1787369265.95 total 70.06 last 76.41) - open/close 5941 (time 1787369275.96 total 80.06 last 78.68) - open/close 6725 (time 1787369285.96 total 90.06 last 78.39) - open/close 7476 (time 1787369295.96 total 100.06 last 75.09) - open/close 8202 (time 1787369305.96 total 110.07 last 72.58) - open/close 8917 (time 1787369315.97 total 120.07 last 71.48) - open/close 9671 (time 1787369325.97 total 130.07 last 75.37) total: 10000 open/close in 134.12 seconds: 74.56 ops/second - unlinked 0 (time 1787369334 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 778 (time 1787369421.29 total 10.03 last 77.60) - open/close 1526 (time 1787369431.29 total 20.03 last 74.78) - open/close 2297 (time 1787369441.31 total 30.04 last 76.99) - open/close 3036 (time 1787369451.32 total 40.05 last 73.84) - open/close 3712 (time 1787369461.32 total 50.05 last 67.57) - open/close 4490 (time 1787369471.33 total 60.07 last 77.70) - open/close 5299 (time 1787369481.34 total 70.07 last 80.87) - open/close 6062 (time 1787369491.34 total 80.07 last 76.29) - open/close 6832 (time 1787369501.34 total 90.08 last 76.96) - open/close 7590 (time 1787369511.35 total 100.08 last 75.78) - open/close 8313 (time 1787369521.35 total 110.08 last 72.29) - open/close 9098 (time 1787369531.35 total 120.08 last 78.48) - open/close 9926 (time 1787369541.35 total 130.08 last 82.79) total: 10000 open/close in 130.92 seconds: 76.38 ops/second - unlinked 0 (time 1787369545 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 718 (time 1787369635.88 total 10.01 last 71.76) - open/close 1456 (time 1787369645.88 total 20.01 last 73.79) - open/close 2225 (time 1787369655.89 total 30.02 last 76.83) - open/close 3034 (time 1787369665.89 total 40.02 last 80.85) - open/close 3792 (time 1787369675.91 total 50.04 last 75.70) - open/close 4486 (time 1787369685.91 total 60.04 last 69.37) - open/close 5284 (time 1787369695.91 total 70.04 last 79.79) - open/close 6034 (time 1787369705.93 total 80.05 last 74.91) - open/close 6769 (time 1787369715.93 total 90.06 last 73.47) - open/close 7614 (time 1787369725.94 total 100.06 last 84.45) - open/close 8383 (time 1787369735.94 total 110.07 last 76.86) - open/close 9214 (time 1787369745.95 total 120.08 last 83.02) - open/close 9973 (time 1787369755.96 total 130.09 last 75.82) total: 10000 open/close in 130.37 seconds: 76.70 ops/second - unlinked 0 (time 1787369759 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 773 (time 1787369847.47 total 10.01 last 77.20) - open/close 1522 (time 1787369857.48 total 20.02 last 74.84) - open/close 2322 (time 1787369867.49 total 30.03 last 79.93) - open/close 3090 (time 1787369877.49 total 40.03 last 76.77) - open/close 3883 (time 1787369887.52 total 50.06 last 79.05) - open/close 4620 (time 1787369897.53 total 60.07 last 73.63) - open/close 5316 (time 1787369907.54 total 70.08 last 69.53) - open/close 6081 (time 1787369917.56 total 80.11 last 76.34) - open/close 6813 (time 1787369927.59 total 90.13 last 73.04) - open/close 7621 (time 1787369937.59 total 100.13 last 80.79) - open/close 8359 (time 1787369947.59 total 110.13 last 73.80) - open/close 9129 (time 1787369957.59 total 120.14 last 76.95) - open/close 9960 (time 1787369967.60 total 130.15 last 83.01) total: 10000 open/close in 130.73 seconds: 76.49 ops/second - unlinked 0 (time 1787369971 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 584 (time 1787370059.79 total 10.01 last 58.37) - open/close 1071 (time 1787370069.80 total 20.02 last 48.64) - open/close 1656 (time 1787370079.82 total 30.03 last 58.43) - open/close 2336 (time 1787370089.83 total 40.05 last 67.89) - open/close 3018 (time 1787370099.85 total 50.06 last 68.09) - open/close 3735 (time 1787370109.88 total 60.09 last 71.50) - open/close 4531 (time 1787370119.88 total 70.09 last 79.59) - open/close 5318 (time 1787370129.89 total 80.10 last 78.62) - open/close 6049 (time 1787370139.89 total 90.10 last 73.09) - open/close 6848 (time 1787370149.90 total 100.11 last 79.82) - open/close 7613 (time 1787370159.91 total 110.12 last 76.45) - open/close 8386 (time 1787370169.91 total 120.12 last 77.27) - open/close 9166 (time 1787370179.92 total 130.13 last 77.92) - open/close 9979 (time 1787370189.93 total 140.15 last 81.20) total: 10000 open/close in 140.44 seconds: 71.21 ops/second - unlinked 0 (time 1787370194 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 729 (time 1787370280.91 total 10.00 last 72.87) - open/close 1489 (time 1787370290.91 total 20.01 last 75.99) - open/close 2267 (time 1787370300.93 total 30.02 last 77.69) - open/close 3002 (time 1787370310.94 total 40.03 last 73.43) - open/close 3789 (time 1787370320.94 total 50.03 last 78.68) - open/close 4533 (time 1787370330.94 total 60.03 last 74.38) - open/close 5310 (time 1787370340.94 total 70.03 last 77.70) - open/close 6026 (time 1787370350.95 total 80.04 last 71.55) - open/close 6766 (time 1787370360.96 total 90.05 last 73.93) - open/close 7516 (time 1787370370.96 total 100.05 last 74.98) - open/close 8184 (time 1787370380.97 total 110.07 last 66.72) - open/close 8844 (time 1787370390.98 total 120.08 last 65.94) - open/close 9516 (time 1787370400.99 total 130.08 last 67.18) total: 10000 open/close in 137.08 seconds: 72.95 ops/second - unlinked 0 (time 1787370413 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.142@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg342-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg342-server: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg342-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg342-server: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg342-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.142@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.142@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.142@tcp sys.timeout=20 start ost1 service on oleg342-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg342-server: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg342-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg342-server: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg342-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server stop mds service on oleg342-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server stop mds service on oleg342-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg342-server unloading modules via unload_modules_local on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing unload_modules_local oleg342-server: modules unloaded. pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2