== conf-sanity test 69: replace an OST with the same index ========================================================== 02:21:11 (1786688471) start mds service on oleg321-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing load_modules_local oleg321-server: Loading modules from /home/green/git/lustre-release/lustre oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg321-server: detected 4 online CPUs by sysfs oleg321-server: MODOPTS_LIBCFS= oleg321-server: Force libcfs to create 2 CPU partitions oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg321-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 898 (time 1786688586.01 total 10.01 last 89.73) - open/close 1837 (time 1786688596.02 total 20.02 last 93.77) - open/close 2559 (time 1786688606.03 total 30.03 last 72.13) - open/close 3428 (time 1786688616.03 total 40.03 last 86.88) - open/close 4284 (time 1786688626.03 total 50.04 last 85.58) - open/close 5109 (time 1786688636.04 total 60.04 last 82.46) - open/close 5997 (time 1786688646.04 total 70.05 last 88.76) - open/close 6891 (time 1786688656.05 total 80.05 last 89.31) - open/close 7781 (time 1786688666.08 total 90.08 last 88.81) - open/close 8748 (time 1786688676.08 total 100.08 last 96.64) - open/close 9714 (time 1786688686.11 total 110.11 last 96.31) total: 10000 open/close in 112.98 seconds: 88.51 ops/second - unlinked 0 (time 1786688693 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 765 (time 1786688777.68 total 10.01 last 76.43) - open/close 1623 (time 1786688787.68 total 20.02 last 85.72) - open/close 2513 (time 1786688797.69 total 30.03 last 88.92) - open/close 3456 (time 1786688807.70 total 40.03 last 94.27) - open/close 4257 (time 1786688817.70 total 50.03 last 80.08) - open/close 5088 (time 1786688827.71 total 60.04 last 83.03) - open/close 5935 (time 1786688837.71 total 70.05 last 84.67) - open/close 6822 (time 1786688847.72 total 80.05 last 88.64) - open/close 7617 (time 1786688857.72 total 90.06 last 79.47) - open/close 8534 (time 1786688867.72 total 100.06 last 91.68) - open/close 9514 (time 1786688877.73 total 110.06 last 97.95) total: 10000 open/close in 115.46 seconds: 86.61 ops/second - unlinked 0 (time 1786688886 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 684 (time 1786688968.82 total 10.02 last 68.28) - open/close 1401 (time 1786688978.82 total 20.02 last 71.67) - open/close 2102 (time 1786688988.83 total 30.03 last 70.07) - open/close 2944 (time 1786688998.83 total 40.03 last 84.19) - open/close 3783 (time 1786689008.83 total 50.03 last 83.88) - open/close 4709 (time 1786689018.84 total 60.04 last 92.51) - open/close 5662 (time 1786689028.85 total 70.05 last 95.24) - open/close 6606 (time 1786689038.86 total 80.05 last 94.32) - open/close 7589 (time 1786689048.86 total 90.06 last 98.29) - open/close 8486 (time 1786689058.86 total 100.06 last 89.66) - open/close 9406 (time 1786689068.86 total 110.06 last 92.00) total: 10000 open/close in 117.43 seconds: 85.16 ops/second - unlinked 0 (time 1786689080 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 873 (time 1786689163.92 total 10.00 last 87.30) - open/close 1747 (time 1786689173.95 total 20.03 last 87.15) - open/close 2689 (time 1786689183.96 total 30.04 last 94.09) - open/close 3567 (time 1786689193.97 total 40.05 last 87.75) - open/close 4462 (time 1786689203.97 total 50.05 last 89.49) - open/close 5387 (time 1786689213.98 total 60.05 last 92.43) - open/close 6320 (time 1786689223.99 total 70.07 last 93.18) - open/close 7125 (time 1786689234.00 total 80.08 last 80.41) - open/close 7954 (time 1786689244.01 total 90.09 last 82.83) - open/close 8824 (time 1786689254.03 total 100.10 last 86.85) - open/close 9719 (time 1786689264.03 total 110.11 last 89.46) total: 10000 open/close in 112.79 seconds: 88.66 ops/second - unlinked 0 (time 1786689271 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 701 (time 1786689359.87 total 10.01 last 70.00) - open/close 1586 (time 1786689369.87 total 20.02 last 88.46) - open/close 2532 (time 1786689379.87 total 30.02 last 94.60) - open/close 3412 (time 1786689389.89 total 40.03 last 87.86) - open/close 4297 (time 1786689399.90 total 50.04 last 88.45) - open/close 5198 (time 1786689409.93 total 60.07 last 89.83) - open/close 6015 (time 1786689419.94 total 70.08 last 81.60) - open/close 6773 (time 1786689429.95 total 80.10 last 75.71) - open/close 7621 (time 1786689439.96 total 90.10 last 84.74) - open/close 8589 (time 1786689449.96 total 100.11 last 96.76) - open/close 9372 (time 1786689459.96 total 110.11 last 78.28) total: 10000 open/close in 118.63 seconds: 84.30 ops/second - unlinked 0 (time 1786689473 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 928 (time 1786689560.25 total 10.01 last 92.70) - open/close 1835 (time 1786689570.25 total 20.01 last 90.68) - open/close 2492 (time 1786689580.26 total 30.03 last 65.61) - open/close 3086 (time 1786689590.27 total 40.04 last 59.34) - open/close 3769 (time 1786689600.29 total 50.05 last 68.21) - open/close 4584 (time 1786689610.29 total 60.05 last 81.46) - open/close 5511 (time 1786689620.30 total 70.06 last 92.63) - open/close 6318 (time 1786689630.33 total 80.09 last 80.47) - open/close 7172 (time 1786689640.33 total 90.09 last 85.37) - open/close 7909 (time 1786689650.35 total 100.11 last 73.58) - open/close 8731 (time 1786689660.35 total 110.12 last 82.16) - open/close 9616 (time 1786689670.36 total 120.12 last 88.47) total: 10000 open/close in 124.58 seconds: 80.27 ops/second - unlinked 0 (time 1786689679 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 894 (time 1786689761.50 total 10.02 last 89.26) - open/close 1807 (time 1786689771.50 total 20.02 last 91.29) - open/close 2610 (time 1786689781.51 total 30.02 last 80.25) - open/close 3546 (time 1786689791.51 total 40.03 last 93.55) - open/close 4492 (time 1786689801.52 total 50.04 last 94.52) - open/close 5421 (time 1786689811.53 total 60.04 last 92.83) - open/close 6362 (time 1786689821.53 total 70.05 last 94.08) - open/close 7266 (time 1786689831.53 total 80.05 last 90.39) - open/close 8115 (time 1786689841.54 total 90.06 last 84.81) - open/close 9014 (time 1786689851.55 total 100.06 last 89.88) - open/close 9906 (time 1786689861.55 total 110.06 last 89.20) total: 10000 open/close in 111.14 seconds: 89.98 ops/second - unlinked 0 (time 1786689867 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 864 (time 1786689946.25 total 10.01 last 86.34) - open/close 1808 (time 1786689956.25 total 20.01 last 94.38) - open/close 2686 (time 1786689966.25 total 30.01 last 87.77) - open/close 3586 (time 1786689976.27 total 40.03 last 89.86) - open/close 4529 (time 1786689986.28 total 50.04 last 94.23) - open/close 5492 (time 1786689996.28 total 60.04 last 96.30) - open/close 6395 (time 1786690006.29 total 70.05 last 90.21) - open/close 7313 (time 1786690016.29 total 80.05 last 91.73) - open/close 8159 (time 1786690026.29 total 90.05 last 84.60) - open/close 8965 (time 1786690036.30 total 100.06 last 80.52) - open/close 9697 (time 1786690046.33 total 110.09 last 73.04) total: 10000 open/close in 113.21 seconds: 88.33 ops/second - unlinked 0 (time 1786690054 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 855 (time 1786690139.66 total 10.00 last 85.48) - open/close 1748 (time 1786690149.67 total 20.00 last 89.29) - open/close 2690 (time 1786690159.67 total 30.01 last 94.15) - open/close 3632 (time 1786690169.69 total 40.03 last 94.02) - open/close 4523 (time 1786690179.69 total 50.03 last 89.10) - open/close 5315 (time 1786690189.70 total 60.03 last 79.15) - open/close 6144 (time 1786690199.71 total 70.05 last 82.75) - open/close 6960 (time 1786690209.72 total 80.05 last 81.59) - open/close 7827 (time 1786690219.72 total 90.06 last 86.66) - open/close 8730 (time 1786690229.74 total 100.07 last 90.16) - open/close 9687 (time 1786690239.74 total 110.08 last 95.69) total: 10000 open/close in 113.47 seconds: 88.13 ops/second - unlinked 0 (time 1786690247 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 851 (time 1786690327.07 total 10.02 last 84.93) - open/close 1729 (time 1786690337.07 total 20.02 last 87.79) - open/close 2595 (time 1786690347.09 total 30.04 last 86.47) - open/close 3270 (time 1786690357.10 total 40.04 last 67.46) - open/close 4166 (time 1786690367.10 total 50.05 last 89.54) - open/close 5095 (time 1786690377.11 total 60.06 last 92.80) - open/close 6004 (time 1786690387.13 total 70.08 last 90.72) - open/close 6879 (time 1786690397.15 total 80.09 last 87.38) - open/close 7716 (time 1786690407.15 total 90.10 last 83.64) - open/close 8592 (time 1786690417.16 total 100.11 last 87.55) - open/close 9477 (time 1786690427.17 total 110.12 last 88.41) total: 10000 open/close in 116.19 seconds: 86.07 ops/second - unlinked 0 (time 1786690437 ; total 1 ; last 1) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg321-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg321-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server unloading modules via unload_modules_local on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local oleg321-server: modules unloaded. pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2