== conf-sanity test 69: replace an OST with the same index ========================================================== 02:20:08 (1780381208) start mds service on oleg441-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions oleg441-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg441-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg441-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg441-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg441-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 828 (time 1780381307.71 total 10.01 last 82.74) - open/close 1621 (time 1780381317.72 total 20.02 last 79.24) - open/close 2346 (time 1780381327.73 total 30.02 last 72.44) - open/close 3267 (time 1780381337.73 total 40.03 last 92.08) - open/close 4113 (time 1780381347.75 total 50.05 last 84.42) - open/close 4898 (time 1780381357.76 total 60.06 last 78.44) - open/close 5620 (time 1780381367.76 total 70.06 last 72.19) - open/close 6387 (time 1780381377.77 total 80.06 last 76.67) - open/close 7174 (time 1780381387.78 total 90.07 last 78.63) - open/close 7997 (time 1780381397.79 total 100.09 last 82.17) - open/close 8849 (time 1780381407.81 total 110.10 last 85.05) - open/close 9481 (time 1780381417.82 total 120.11 last 63.13) total: 10000 open/close in 127.18 seconds: 78.63 ops/second - unlinked 0 (time 1780381431 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 799 (time 1780381513.73 total 10.02 last 79.76) - open/close 1614 (time 1780381523.73 total 20.02 last 81.50) - open/close 2516 (time 1780381533.73 total 30.02 last 90.19) - open/close 3448 (time 1780381543.73 total 40.02 last 93.19) - open/close 4258 (time 1780381553.73 total 50.02 last 80.98) - open/close 5166 (time 1780381563.73 total 60.02 last 90.79) - open/close 6009 (time 1780381573.73 total 70.02 last 84.30) - open/close 6812 (time 1780381583.73 total 80.02 last 80.30) - open/close 7691 (time 1780381593.73 total 90.03 last 87.89) - open/close 8569 (time 1780381603.77 total 100.06 last 87.52) - open/close 9392 (time 1780381613.78 total 110.07 last 82.20) total: 10000 open/close in 117.10 seconds: 85.40 ops/second - unlinked 0 (time 1780381624 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 820 (time 1780381708.73 total 10.01 last 81.95) - open/close 1599 (time 1780381718.73 total 20.01 last 77.89) - open/close 2437 (time 1780381728.73 total 30.01 last 83.75) - open/close 3271 (time 1780381738.74 total 40.02 last 83.30) - open/close 4178 (time 1780381748.75 total 50.03 last 90.68) - open/close 5103 (time 1780381758.76 total 60.04 last 92.41) - open/close 5918 (time 1780381768.76 total 70.04 last 81.45) - open/close 6480 (time 1780381778.79 total 80.07 last 56.02) - open/close 7226 (time 1780381788.80 total 90.08 last 74.58) - open/close 8009 (time 1780381798.81 total 100.09 last 78.23) - open/close 8841 (time 1780381808.81 total 110.09 last 83.20) - open/close 9581 (time 1780381818.83 total 120.11 last 73.83) total: 10000 open/close in 125.64 seconds: 79.59 ops/second - unlinked 0 (time 1780381829 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 655 (time 1780381916.60 total 10.00 last 65.49) - open/close 1413 (time 1780381926.61 total 20.01 last 75.75) - open/close 2254 (time 1780381936.61 total 30.01 last 84.06) - open/close 3086 (time 1780381946.62 total 40.02 last 83.10) - open/close 3876 (time 1780381956.63 total 50.03 last 78.95) - open/close 4654 (time 1780381966.63 total 60.04 last 77.77) - open/close 5521 (time 1780381976.64 total 70.04 last 86.63) - open/close 6350 (time 1780381986.64 total 80.05 last 82.88) - open/close 7230 (time 1780381996.64 total 90.05 last 87.99) - open/close 8109 (time 1780382006.65 total 100.05 last 87.86) - open/close 8940 (time 1780382016.65 total 110.05 last 83.08) - open/close 9729 (time 1780382026.66 total 120.06 last 78.81) total: 10000 open/close in 122.72 seconds: 81.49 ops/second - unlinked 0 (time 1780382032 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 830 (time 1780382120.48 total 10.00 last 82.98) - open/close 1622 (time 1780382130.49 total 20.01 last 79.14) - open/close 2360 (time 1780382140.49 total 30.01 last 73.80) - open/close 3013 (time 1780382150.50 total 40.02 last 65.23) - open/close 3707 (time 1780382160.50 total 50.02 last 69.39) - open/close 4389 (time 1780382170.52 total 60.04 last 68.08) - open/close 5217 (time 1780382180.52 total 70.05 last 82.76) - open/close 5997 (time 1780382190.53 total 80.05 last 77.98) - open/close 6635 (time 1780382200.54 total 90.06 last 63.70) - open/close 7513 (time 1780382210.54 total 100.07 last 87.78) - open/close 8255 (time 1780382220.55 total 110.07 last 74.17) - open/close 9156 (time 1780382230.56 total 120.09 last 89.96) - open/close 9976 (time 1780382240.59 total 130.11 last 81.83) total: 10000 open/close in 130.54 seconds: 76.61 ops/second - unlinked 0 (time 1780382246 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 724 (time 1780382346.85 total 10.00 last 72.39) - open/close 1626 (time 1780382356.85 total 20.00 last 90.18) - open/close 2495 (time 1780382366.86 total 30.01 last 86.86) - open/close 3342 (time 1780382376.86 total 40.01 last 84.65) - open/close 4230 (time 1780382386.86 total 50.01 last 88.79) - open/close 5037 (time 1780382396.88 total 60.03 last 80.59) - open/close 5833 (time 1780382406.89 total 70.04 last 79.51) - open/close 6620 (time 1780382416.90 total 80.05 last 78.60) - open/close 7354 (time 1780382426.90 total 90.06 last 73.37) - open/close 8137 (time 1780382436.91 total 100.06 last 78.28) - open/close 8883 (time 1780382446.92 total 110.07 last 74.54) - open/close 9677 (time 1780382456.93 total 120.08 last 79.31) total: 10000 open/close in 123.84 seconds: 80.75 ops/second - unlinked 0 (time 1780382465 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 613 (time 1780382559.40 total 10.01 last 61.26) - open/close 1211 (time 1780382569.41 total 20.01 last 59.77) - open/close 1934 (time 1780382579.43 total 30.03 last 72.16) - open/close 2749 (time 1780382589.46 total 40.07 last 81.23) - open/close 3467 (time 1780382599.47 total 50.07 last 71.78) - open/close 4301 (time 1780382609.48 total 60.08 last 83.31) - open/close 5160 (time 1780382619.49 total 70.09 last 85.84) - open/close 5782 (time 1780382629.50 total 80.10 last 62.13) - open/close 6425 (time 1780382639.52 total 90.13 last 64.13) - open/close 7148 (time 1780382649.53 total 100.13 last 72.28) - open/close 7931 (time 1780382659.54 total 110.14 last 78.24) - open/close 8678 (time 1780382669.55 total 120.15 last 74.63) - open/close 9327 (time 1780382679.55 total 130.15 last 64.88) - open/close 9917 (time 1780382689.55 total 140.15 last 59.00) total: 10000 open/close in 141.34 seconds: 70.75 ops/second - unlinked 0 (time 1780382695 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 796 (time 1780382790.43 total 10.02 last 79.44) - open/close 1566 (time 1780382800.43 total 20.02 last 76.99) - open/close 2345 (time 1780382810.43 total 30.03 last 77.87) - open/close 3041 (time 1780382820.45 total 40.04 last 69.49) - open/close 3764 (time 1780382830.46 total 50.06 last 72.20) - open/close 4578 (time 1780382840.49 total 60.08 last 81.16) - open/close 5393 (time 1780382850.50 total 70.09 last 81.45) - open/close 6145 (time 1780382860.50 total 80.09 last 75.19) - open/close 7012 (time 1780382870.50 total 90.09 last 86.70) - open/close 7832 (time 1780382880.51 total 100.10 last 81.93) - open/close 8624 (time 1780382890.51 total 110.11 last 79.17) - open/close 9347 (time 1780382900.52 total 120.11 last 72.25) total: 10000 open/close in 128.65 seconds: 77.73 ops/second - unlinked 0 (time 1780382913 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 865 (time 1780382996.87 total 10.00 last 86.49) - open/close 1660 (time 1780383006.88 total 20.01 last 79.42) - open/close 2364 (time 1780383016.89 total 30.02 last 70.35) - open/close 3170 (time 1780383026.90 total 40.03 last 80.51) - open/close 3826 (time 1780383036.91 total 50.04 last 65.52) - open/close 4384 (time 1780383046.93 total 60.07 last 55.67) - open/close 4954 (time 1780383056.94 total 70.07 last 56.98) - open/close 5691 (time 1780383066.94 total 80.08 last 73.66) - open/close 6436 (time 1780383076.95 total 90.08 last 74.48) - open/close 7210 (time 1780383086.96 total 100.09 last 77.33) - open/close 8043 (time 1780383096.96 total 110.09 last 83.25) - open/close 8819 (time 1780383106.98 total 120.11 last 77.50) - open/close 9621 (time 1780383116.98 total 130.11 last 80.20) total: 10000 open/close in 134.74 seconds: 74.22 ops/second - unlinked 0 (time 1780383125 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 811 (time 1780383210.07 total 10.00 last 81.06) - open/close 1529 (time 1780383220.08 total 20.01 last 71.78) - open/close 2309 (time 1780383230.08 total 30.01 last 77.96) - open/close 3117 (time 1780383240.09 total 40.02 last 80.76) - open/close 3884 (time 1780383250.10 total 50.03 last 76.60) - open/close 4465 (time 1780383260.11 total 60.04 last 58.04) - open/close 5052 (time 1780383270.14 total 70.07 last 58.54) - open/close 5671 (time 1780383280.15 total 80.08 last 61.80) - open/close 6172 (time 1780383290.16 total 90.09 last 50.05) - open/close 6673 (time 1780383300.17 total 100.10 last 50.04) - open/close 7285 (time 1780383310.18 total 110.11 last 61.14) - open/close 7899 (time 1780383320.19 total 120.12 last 61.37) - open/close 8565 (time 1780383330.19 total 130.12 last 66.60) - open/close 9262 (time 1780383340.20 total 140.13 last 69.65) total: 10000 open/close in 149.84 seconds: 66.74 ops/second - unlinked 0 (time 1780383354 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg441-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg441-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg441-server: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg441-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg441-server: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg441-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre On OST0, 17000 used inodes rc=0 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg441-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.141@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.141@tcp sys.timeout=20 start ost1 service on oleg441-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg441-server: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg441-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg441-server: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg441-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg441-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server unloading modules via unload_modules_local on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing unload_modules_local oleg441-server: modules unloaded. pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2