== conf-sanity test 69: replace an OST with the same index ========================================================== 02:28:13 (1786948093) start mds service on oleg238-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: MODOPTS_LIBCFS= oleg238-server: Force libcfs to create 2 CPU partitions oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg238-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg238-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 653 (time 1786948200.57 total 10.02 last 65.15) - open/close 1294 (time 1786948210.58 total 20.03 last 64.06) - open/close 2089 (time 1786948220.58 total 30.04 last 79.45) - open/close 2904 (time 1786948230.59 total 40.04 last 81.47) - open/close 3680 (time 1786948240.61 total 50.06 last 77.43) - open/close 4666 (time 1786948250.62 total 60.07 last 98.53) - open/close 5816 (time 1786948260.62 total 70.08 last 114.91) - open/close 6840 (time 1786948270.63 total 80.08 last 102.37) - open/close 7761 (time 1786948280.63 total 90.08 last 92.09) - open/close 8625 (time 1786948290.63 total 100.08 last 86.39) - open/close 9545 (time 1786948300.64 total 110.09 last 91.90) total: 10000 open/close in 115.98 seconds: 86.22 ops/second - unlinked 0 (time 1786948311 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 903 (time 1786948400.40 total 10.01 last 90.23) - open/close 1753 (time 1786948410.41 total 20.01 last 84.94) - open/close 2445 (time 1786948420.42 total 30.02 last 69.13) - open/close 3197 (time 1786948430.42 total 40.03 last 75.14) - open/close 4152 (time 1786948440.43 total 50.04 last 95.44) - open/close 5189 (time 1786948450.44 total 60.04 last 103.65) - open/close 6223 (time 1786948460.44 total 70.05 last 103.35) - open/close 7308 (time 1786948470.45 total 80.05 last 108.42) - open/close 8302 (time 1786948480.45 total 90.06 last 99.37) - open/close 9442 (time 1786948490.45 total 100.06 last 113.98) total: 10000 open/close in 104.76 seconds: 95.45 ops/second - unlinked 0 (time 1786948500 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 878 (time 1786948575.00 total 10.02 last 87.66) - open/close 1786 (time 1786948585.01 total 20.03 last 90.70) - open/close 2785 (time 1786948595.01 total 30.03 last 99.84) - open/close 3580 (time 1786948605.02 total 40.03 last 79.48) - open/close 4460 (time 1786948615.02 total 50.04 last 87.94) - open/close 5340 (time 1786948625.03 total 60.04 last 87.98) - open/close 6125 (time 1786948635.05 total 70.07 last 78.33) - open/close 6835 (time 1786948645.06 total 80.08 last 70.89) - open/close 7737 (time 1786948655.07 total 90.09 last 90.14) - open/close 8766 (time 1786948665.07 total 100.09 last 102.85) - open/close 9748 (time 1786948675.08 total 110.09 last 98.18) total: 10000 open/close in 112.37 seconds: 89.00 ops/second - unlinked 0 (time 1786948681 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 897 (time 1786948764.86 total 10.00 last 89.66) - open/close 1732 (time 1786948774.86 total 20.01 last 83.49) - open/close 2428 (time 1786948784.88 total 30.03 last 69.46) - open/close 3437 (time 1786948794.88 total 40.03 last 100.86) - open/close 4334 (time 1786948804.89 total 50.03 last 89.67) - open/close 5202 (time 1786948814.89 total 60.04 last 86.75) - open/close 6276 (time 1786948824.90 total 70.05 last 107.29) - open/close 7186 (time 1786948834.90 total 80.05 last 90.98) - open/close 8125 (time 1786948844.90 total 90.05 last 93.89) - open/close 9194 (time 1786948854.91 total 100.06 last 106.85) total: 10000 open/close in 107.84 seconds: 92.73 ops/second - unlinked 0 (time 1786948867 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1040 (time 1786948956.27 total 10.00 last 103.97) - open/close 2007 (time 1786948966.28 total 20.01 last 96.63) - open/close 3150 (time 1786948976.28 total 30.02 last 114.24) - open/close 4286 (time 1786948986.29 total 40.02 last 113.54) - open/close 5445 (time 1786948996.29 total 50.03 last 115.83) - open/close 6469 (time 1786949006.30 total 60.03 last 102.34) - open/close 7404 (time 1786949016.31 total 70.04 last 93.41) - open/close 8469 (time 1786949026.31 total 80.05 last 106.45) - open/close 9243 (time 1786949036.32 total 90.05 last 77.37) total: 10000 open/close in 99.74 seconds: 100.26 ops/second - unlinked 0 (time 1786949050 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 856 (time 1786949133.06 total 10.01 last 85.49) - open/close 1913 (time 1786949143.06 total 20.02 last 105.67) - open/close 2870 (time 1786949153.07 total 30.02 last 95.67) - open/close 3844 (time 1786949163.08 total 40.03 last 97.31) - open/close 4806 (time 1786949173.08 total 50.03 last 96.15) - open/close 5812 (time 1786949183.09 total 60.05 last 100.48) - open/close 6858 (time 1786949193.10 total 70.05 last 104.59) - open/close 7961 (time 1786949203.10 total 80.06 last 110.20) - open/close 8984 (time 1786949213.11 total 90.06 last 102.25) total: 10000 open/close in 99.94 seconds: 100.06 ops/second - unlinked 0 (time 1786949227 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 931 (time 1786949303.63 total 10.00 last 93.08) - open/close 2002 (time 1786949313.65 total 20.02 last 106.88) - open/close 2924 (time 1786949323.65 total 30.02 last 92.19) - open/close 3952 (time 1786949333.65 total 40.02 last 102.80) - open/close 4959 (time 1786949343.65 total 50.03 last 100.68) - open/close 6053 (time 1786949353.66 total 60.03 last 109.33) - open/close 7056 (time 1786949363.68 total 70.05 last 100.16) - open/close 7822 (time 1786949373.68 total 80.05 last 76.56) - open/close 8905 (time 1786949383.68 total 90.05 last 108.26) - open/close 9995 (time 1786949393.68 total 100.05 last 109.00) total: 10000 open/close in 100.08 seconds: 99.92 ops/second - unlinked 0 (time 1786949397 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 967 (time 1786949466.89 total 10.00 last 96.69) - open/close 1824 (time 1786949476.91 total 20.03 last 85.47) - open/close 2933 (time 1786949486.92 total 30.03 last 110.83) - open/close 3766 (time 1786949496.92 total 40.03 last 83.30) - open/close 4532 (time 1786949506.93 total 50.04 last 76.53) - open/close 5599 (time 1786949516.93 total 60.05 last 106.65) - open/close 6709 (time 1786949526.94 total 70.05 last 110.94) - open/close 7677 (time 1786949536.94 total 80.06 last 96.77) - open/close 8736 (time 1786949546.94 total 90.06 last 105.86) - open/close 9692 (time 1786949556.95 total 100.07 last 95.52) total: 10000 open/close in 103.02 seconds: 97.07 ops/second - unlinked 0 (time 1786949563 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1108 (time 1786949631.76 total 10.00 last 110.75) - open/close 2164 (time 1786949641.77 total 20.02 last 105.47) - open/close 3128 (time 1786949651.78 total 30.02 last 96.35) - open/close 4160 (time 1786949661.79 total 40.03 last 103.13) - open/close 5088 (time 1786949671.79 total 50.03 last 92.76) - open/close 6171 (time 1786949681.79 total 60.03 last 108.30) - open/close 6826 (time 1786949691.79 total 70.04 last 65.47) - open/close 7317 (time 1786949701.83 total 80.07 last 48.94) - open/close 7716 (time 1786949711.85 total 90.09 last 39.82) - open/close 8166 (time 1786949721.86 total 100.10 last 44.94) - open/close 8533 (time 1786949731.87 total 110.11 last 36.68) - open/close 9238 (time 1786949741.87 total 120.11 last 70.46) total: 10000 open/close in 129.11 seconds: 77.46 ops/second - unlinked 0 (time 1786949755 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 833 (time 1786949843.65 total 10.01 last 83.23) - open/close 1805 (time 1786949853.66 total 20.02 last 97.12) - open/close 2814 (time 1786949863.66 total 30.02 last 100.88) - open/close 3596 (time 1786949873.66 total 40.02 last 78.17) - open/close 4477 (time 1786949883.67 total 50.03 last 88.06) - open/close 5480 (time 1786949893.67 total 60.03 last 100.27) - open/close 6395 (time 1786949903.68 total 70.04 last 91.42) - open/close 7397 (time 1786949913.69 total 80.05 last 100.14) - open/close 8452 (time 1786949923.69 total 90.05 last 105.42) - open/close 9248 (time 1786949933.71 total 100.07 last 79.49) total: 10000 open/close in 108.08 seconds: 92.53 ops/second - unlinked 0 (time 1786949945 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server unloading modules via unload_modules_local on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: modules unloaded. pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2