== conf-sanity test 69: replace an OST with the same index ========================================================== 17:03:46 (1787864626) start mds service on oleg623-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg623-server' oleg623-server: oleg623-server.virtnet: executing load_modules_local oleg623-server: Loading modules from /home/green/git/lustre-release/lustre oleg623-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg623-server: detected 4 online CPUs by sysfs oleg623-server: MODOPTS_LIBCFS= oleg623-server: Force libcfs to create 2 CPU partitions oleg623-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg623-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg623-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg623-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg623-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg623-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg623-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49502 inodes available. Want 99872. rc=0 - open/close 1000 (time 1787864726.50 total 10.00 last 99.99) - open/close 1945 (time 1787864736.51 total 20.01 last 94.42) - open/close 2896 (time 1787864746.52 total 30.02 last 95.04) - open/close 3849 (time 1787864756.53 total 40.03 last 95.16) - open/close 4807 (time 1787864766.54 total 50.04 last 95.75) - open/close 5752 (time 1787864776.54 total 60.04 last 94.48) - open/close 6725 (time 1787864786.55 total 70.05 last 97.20) - open/close 7703 (time 1787864796.56 total 80.05 last 97.75) - open/close 8686 (time 1787864806.56 total 90.06 last 98.29) - open/close 9698 (time 1787864816.57 total 100.06 last 101.12) total: 10000 open/close in 103.19 seconds: 96.91 ops/second - unlinked 0 (time 1787864822 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 970 (time 1787864894.81 total 10.00 last 96.98) - open/close 1955 (time 1787864904.81 total 20.00 last 98.47) - open/close 2999 (time 1787864914.81 total 30.01 last 104.35) - open/close 3968 (time 1787864924.82 total 40.02 last 96.80) - open/close 4957 (time 1787864934.83 total 50.03 last 98.83) - open/close 5866 (time 1787864944.83 total 60.03 last 90.86) - open/close 6855 (time 1787864954.84 total 70.03 last 98.90) - open/close 7802 (time 1787864964.84 total 80.04 last 94.61) - open/close 8789 (time 1787864974.85 total 90.05 last 98.61) - open/close 9760 (time 1787864984.86 total 100.06 last 96.99) total: 10000 open/close in 102.60 seconds: 97.46 ops/second - unlinked 0 (time 1787864991 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1021 (time 1787865060.72 total 10.01 last 102.04) - open/close 1980 (time 1787865070.74 total 20.03 last 95.66) - open/close 3016 (time 1787865080.75 total 30.03 last 103.57) - open/close 4073 (time 1787865090.75 total 40.04 last 105.67) - open/close 5102 (time 1787865100.75 total 50.04 last 102.86) - open/close 6111 (time 1787865110.76 total 60.05 last 100.86) - open/close 7088 (time 1787865120.76 total 70.05 last 97.69) - open/close 8092 (time 1787865130.77 total 80.06 last 100.31) - open/close 9089 (time 1787865140.77 total 90.06 last 99.70) total: 10000 open/close in 99.20 seconds: 100.81 ops/second - unlinked 0 (time 1787865153 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1019 (time 1787865224.13 total 10.00 last 101.87) - open/close 2073 (time 1787865234.13 total 20.01 last 105.36) - open/close 3042 (time 1787865244.13 total 30.01 last 96.87) - open/close 4055 (time 1787865254.14 total 40.02 last 101.24) - open/close 5154 (time 1787865264.15 total 50.02 last 109.85) - open/close 6143 (time 1787865274.15 total 60.02 last 98.86) - open/close 7187 (time 1787865284.16 total 70.03 last 104.34) - open/close 8129 (time 1787865294.16 total 80.04 last 94.13) - open/close 9060 (time 1787865304.17 total 90.04 last 93.06) - open/close 9953 (time 1787865314.17 total 100.05 last 89.24) total: 10000 open/close in 100.61 seconds: 99.40 ops/second - unlinked 0 (time 1787865318 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 918 (time 1787865392.10 total 10.00 last 91.77) - open/close 1873 (time 1787865402.10 total 20.01 last 95.49) - open/close 2700 (time 1787865412.10 total 30.01 last 82.69) - open/close 3654 (time 1787865422.10 total 40.01 last 95.37) - open/close 4569 (time 1787865432.11 total 50.01 last 91.48) - open/close 5523 (time 1787865442.11 total 60.02 last 95.34) - open/close 6566 (time 1787865452.12 total 70.02 last 104.27) - open/close 7602 (time 1787865462.12 total 80.02 last 103.60) - open/close 8660 (time 1787865472.12 total 90.02 last 105.77) - open/close 9688 (time 1787865482.12 total 100.03 last 102.78) total: 10000 open/close in 103.50 seconds: 96.62 ops/second - unlinked 0 (time 1787865488 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 933 (time 1787865558.73 total 10.00 last 93.29) - open/close 1829 (time 1787865568.73 total 20.01 last 89.55) - open/close 2826 (time 1787865578.74 total 30.02 last 99.59) - open/close 3788 (time 1787865588.75 total 40.03 last 96.13) - open/close 4735 (time 1787865598.77 total 50.04 last 94.57) - open/close 5729 (time 1787865608.77 total 60.05 last 99.32) - open/close 6725 (time 1787865618.78 total 70.05 last 99.56) - open/close 7773 (time 1787865628.78 total 80.05 last 104.77) - open/close 8699 (time 1787865638.78 total 90.06 last 92.59) - open/close 9584 (time 1787865648.78 total 100.06 last 88.49) total: 10000 open/close in 104.33 seconds: 95.85 ops/second - unlinked 0 (time 1787865656 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 868 (time 1787865736.34 total 10.00 last 86.77) - open/close 1680 (time 1787865746.36 total 20.02 last 81.09) - open/close 2680 (time 1787865756.36 total 30.02 last 99.92) - open/close 3674 (time 1787865766.37 total 40.04 last 99.29) - open/close 4558 (time 1787865776.40 total 50.06 last 88.21) - open/close 5305 (time 1787865786.42 total 60.08 last 74.54) - open/close 5806 (time 1787865796.42 total 70.08 last 50.09) - open/close 6669 (time 1787865806.43 total 80.09 last 86.22) - open/close 7511 (time 1787865816.45 total 90.11 last 84.04) - open/close 8394 (time 1787865826.45 total 100.11 last 88.24) - open/close 9293 (time 1787865836.48 total 110.14 last 89.70) total: 10000 open/close in 118.41 seconds: 84.45 ops/second - unlinked 0 (time 1787865848 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 883 (time 1787865933.38 total 10.00 last 88.28) - open/close 1848 (time 1787865943.39 total 20.01 last 96.44) - open/close 2878 (time 1787865953.40 total 30.01 last 102.93) - open/close 3862 (time 1787865963.40 total 40.02 last 98.37) - open/close 4730 (time 1787865973.40 total 50.02 last 86.78) - open/close 5623 (time 1787865983.41 total 60.03 last 89.24) - open/close 6408 (time 1787865993.41 total 70.03 last 78.46) - open/close 7236 (time 1787866003.42 total 80.04 last 82.78) - open/close 8009 (time 1787866013.42 total 90.04 last 77.25) - open/close 8725 (time 1787866023.44 total 100.05 last 71.51) - open/close 9497 (time 1787866033.44 total 110.06 last 77.19) total: 10000 open/close in 114.97 seconds: 86.98 ops/second - unlinked 0 (time 1787866042 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 914 (time 1787866119.75 total 10.01 last 91.32) - open/close 1796 (time 1787866129.75 total 20.01 last 88.15) - open/close 2688 (time 1787866139.76 total 30.02 last 89.14) - open/close 3618 (time 1787866149.77 total 40.03 last 92.93) - open/close 4544 (time 1787866159.77 total 50.03 last 92.56) - open/close 5546 (time 1787866169.78 total 60.04 last 100.17) - open/close 6473 (time 1787866179.78 total 70.04 last 92.67) - open/close 7384 (time 1787866189.79 total 80.05 last 91.03) - open/close 8356 (time 1787866199.79 total 90.05 last 97.17) - open/close 9376 (time 1787866209.79 total 100.06 last 101.94) total: 10000 open/close in 107.16 seconds: 93.32 ops/second - unlinked 0 (time 1787866221 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 919 (time 1787866298.67 total 10.00 last 91.88) - open/close 1874 (time 1787866308.68 total 20.01 last 95.38) - open/close 2841 (time 1787866318.68 total 30.02 last 96.65) - open/close 3843 (time 1787866328.69 total 40.02 last 100.16) - open/close 4812 (time 1787866338.69 total 50.03 last 96.84) - open/close 5795 (time 1787866348.70 total 60.03 last 98.25) - open/close 6759 (time 1787866358.70 total 70.04 last 96.39) - open/close 7665 (time 1787866368.70 total 80.04 last 90.55) - open/close 8588 (time 1787866378.71 total 90.04 last 92.29) - open/close 9588 (time 1787866388.72 total 100.05 last 99.87) total: 10000 open/close in 104.04 seconds: 96.12 ops/second - unlinked 0 (time 1787866396 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg623-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg623-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg623-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg623-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg623-server: oleg623-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg623-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg623-server: oleg623-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg623-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre On OST0, 20431 used inodes rc=0 pdsh@oleg623-client: oleg623-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg623-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg623-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg623-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.123@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.123@tcp sys.timeout=20 start ost1 service on oleg623-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg623-server: oleg623-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg623-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg623-server: oleg623-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg623-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg623-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg623-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg623-server stop mds service on oleg623-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg623-server stop mds service on oleg623-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg623-server unloading modules via unload_modules_local on: 'oleg623-server' oleg623-server: oleg623-server.virtnet: executing unload_modules_local oleg623-server: modules unloaded. pdsh@oleg623-client: oleg623-client: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-server: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-server: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-client: ssh exited with exit code 2