== conf-sanity test 69: replace an OST with the same index ========================================================== 06:52:09 (1787395929) start mds service on oleg421-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg421-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg421-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg421-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 485 (time 1787396061.34 total 10.01 last 48.45) - open/close 1104 (time 1787396071.34 total 20.01 last 61.88) - open/close 1743 (time 1787396081.35 total 30.02 last 63.84) - open/close 2402 (time 1787396091.36 total 40.03 last 65.84) - open/close 3075 (time 1787396101.37 total 50.04 last 67.24) - open/close 3688 (time 1787396111.37 total 60.05 last 61.27) - open/close 4295 (time 1787396121.38 total 70.05 last 60.65) - open/close 4942 (time 1787396131.39 total 80.07 last 64.61) - open/close 5621 (time 1787396141.40 total 90.08 last 67.83) - open/close 6328 (time 1787396151.41 total 100.08 last 70.67) - open/close 7040 (time 1787396161.41 total 110.09 last 71.18) - open/close 7692 (time 1787396171.43 total 120.10 last 65.11) - open/close 8359 (time 1787396181.45 total 130.12 last 66.53) - open/close 9015 (time 1787396191.45 total 140.12 last 65.60) - open/close 9777 (time 1787396201.46 total 150.14 last 76.10) total: 10000 open/close in 153.20 seconds: 65.28 ops/second - unlinked 0 (time 1787396209 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 763 (time 1787396309.07 total 10.01 last 76.19) - open/close 1402 (time 1787396319.08 total 20.02 last 63.85) - open/close 2150 (time 1787396329.08 total 30.02 last 74.79) - open/close 2854 (time 1787396339.08 total 40.02 last 70.40) - open/close 3538 (time 1787396349.10 total 50.04 last 68.28) - open/close 4232 (time 1787396359.11 total 60.05 last 69.32) - open/close 4973 (time 1787396369.12 total 70.07 last 73.99) - open/close 5636 (time 1787396379.14 total 80.08 last 66.22) - open/close 6308 (time 1787396389.14 total 90.09 last 67.16) - open/close 6970 (time 1787396399.15 total 100.10 last 66.12) - open/close 7664 (time 1787396409.16 total 110.10 last 69.37) - open/close 8322 (time 1787396419.16 total 120.10 last 65.79) - open/close 9021 (time 1787396429.16 total 130.11 last 69.89) - open/close 9735 (time 1787396439.17 total 140.11 last 71.36) total: 10000 open/close in 144.11 seconds: 69.39 ops/second - unlinked 0 (time 1787396448 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 835 (time 1787396554.42 total 10.00 last 83.46) - open/close 1489 (time 1787396564.44 total 20.02 last 65.27) - open/close 2253 (time 1787396574.47 total 30.05 last 76.23) - open/close 3031 (time 1787396584.47 total 40.05 last 77.78) - open/close 3798 (time 1787396594.48 total 50.06 last 76.61) - open/close 4492 (time 1787396604.48 total 60.06 last 69.38) - open/close 5224 (time 1787396614.50 total 70.08 last 73.10) - open/close 6000 (time 1787396624.51 total 80.09 last 77.50) - open/close 6741 (time 1787396634.52 total 90.10 last 74.04) - open/close 7480 (time 1787396644.53 total 100.11 last 73.80) - open/close 8150 (time 1787396654.55 total 110.13 last 66.91) - open/close 8919 (time 1787396664.55 total 120.13 last 76.88) - open/close 9685 (time 1787396674.56 total 130.14 last 76.55) total: 10000 open/close in 134.35 seconds: 74.43 ops/second - unlinked 0 (time 1787396683 ; total 1 ; last 1) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 591 (time 1787396782.22 total 10.01 last 59.07) - open/close 1312 (time 1787396792.23 total 20.01 last 72.08) - open/close 2033 (time 1787396802.23 total 30.01 last 72.07) - open/close 2744 (time 1787396812.24 total 40.03 last 71.01) - open/close 3511 (time 1787396822.25 total 50.03 last 76.64) - open/close 4234 (time 1787396832.25 total 60.03 last 72.30) - open/close 4939 (time 1787396842.26 total 70.04 last 70.43) - open/close 5685 (time 1787396852.26 total 80.04 last 74.59) - open/close 6351 (time 1787396862.27 total 90.05 last 66.58) - open/close 6861 (time 1787396872.27 total 100.05 last 50.97) - open/close 7574 (time 1787396882.28 total 110.06 last 71.25) - open/close 8280 (time 1787396892.28 total 120.06 last 70.59) - open/close 8998 (time 1787396902.29 total 130.07 last 71.75) - open/close 9752 (time 1787396912.30 total 140.08 last 75.33) total: 10000 open/close in 143.49 seconds: 69.69 ops/second - unlinked 0 (time 1787396920 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 746 (time 1787397013.56 total 10.01 last 74.55) - open/close 1416 (time 1787397023.57 total 20.02 last 66.92) - open/close 2128 (time 1787397033.57 total 30.02 last 71.19) - open/close 2841 (time 1787397043.57 total 40.02 last 71.27) - open/close 3570 (time 1787397053.58 total 50.03 last 72.87) - open/close 4266 (time 1787397063.58 total 60.03 last 69.59) - open/close 4968 (time 1787397073.59 total 70.04 last 70.14) - open/close 5662 (time 1787397083.59 total 80.04 last 69.40) - open/close 6333 (time 1787397093.59 total 90.04 last 67.06) - open/close 6993 (time 1787397103.60 total 100.05 last 65.98) - open/close 7709 (time 1787397113.60 total 110.05 last 71.57) - open/close 8417 (time 1787397123.60 total 120.05 last 70.80) - open/close 9073 (time 1787397133.61 total 130.06 last 65.56) - open/close 9745 (time 1787397143.62 total 140.07 last 67.12) total: 10000 open/close in 143.40 seconds: 69.74 ops/second - unlinked 0 (time 1787397150 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 735 (time 1787397242.43 total 10.01 last 73.43) - open/close 1412 (time 1787397252.44 total 20.03 last 67.59) - open/close 2103 (time 1787397262.45 total 30.04 last 69.02) - open/close 2813 (time 1787397272.46 total 40.04 last 70.98) - open/close 3536 (time 1787397282.48 total 50.06 last 72.12) - open/close 4324 (time 1787397292.49 total 60.07 last 78.73) - open/close 5046 (time 1787397302.50 total 70.08 last 72.13) - open/close 5690 (time 1787397312.50 total 80.09 last 64.38) - open/close 6370 (time 1787397322.51 total 90.10 last 67.94) - open/close 7022 (time 1787397332.51 total 100.10 last 65.20) - open/close 7664 (time 1787397342.53 total 110.12 last 64.08) - open/close 8410 (time 1787397352.54 total 120.13 last 74.53) - open/close 9102 (time 1787397362.54 total 130.13 last 69.19) - open/close 9834 (time 1787397372.56 total 140.15 last 73.05) total: 10000 open/close in 142.80 seconds: 70.03 ops/second - unlinked 0 (time 1787397380 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 706 (time 1787397478.35 total 10.00 last 70.58) - open/close 1360 (time 1787397488.36 total 20.02 last 65.31) - open/close 1944 (time 1787397498.37 total 30.02 last 58.39) - open/close 2584 (time 1787397508.37 total 40.02 last 63.96) - open/close 3195 (time 1787397518.37 total 50.03 last 61.10) - open/close 3875 (time 1787397528.38 total 60.04 last 67.92) - open/close 4597 (time 1787397538.40 total 70.05 last 72.09) - open/close 5277 (time 1787397548.40 total 80.05 last 68.00) - open/close 5921 (time 1787397558.40 total 90.05 last 64.40) - open/close 6540 (time 1787397568.40 total 100.05 last 61.88) - open/close 7195 (time 1787397578.40 total 110.06 last 65.49) - open/close 7777 (time 1787397588.41 total 120.06 last 58.16) - open/close 8434 (time 1787397598.43 total 130.08 last 65.61) - open/close 9076 (time 1787397608.43 total 140.08 last 64.17) - open/close 9667 (time 1787397618.43 total 150.09 last 59.07) total: 10000 open/close in 155.97 seconds: 64.11 ops/second - unlinked 0 (time 1787397629 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 603 (time 1787397735.04 total 10.01 last 60.22) - open/close 1204 (time 1787397745.05 total 20.02 last 60.06) - open/close 1797 (time 1787397755.05 total 30.03 last 59.27) - open/close 2490 (time 1787397765.06 total 40.03 last 69.26) - open/close 3131 (time 1787397775.06 total 50.04 last 64.08) - open/close 3770 (time 1787397785.09 total 60.07 last 63.72) - open/close 4419 (time 1787397795.10 total 70.07 last 64.86) - open/close 5074 (time 1787397805.11 total 80.09 last 65.39) - open/close 5696 (time 1787397815.12 total 90.09 last 62.16) - open/close 6328 (time 1787397825.13 total 100.10 last 63.15) - open/close 7002 (time 1787397835.13 total 110.11 last 67.36) - open/close 7635 (time 1787397845.14 total 120.11 last 63.27) - open/close 8241 (time 1787397855.15 total 130.12 last 60.55) - open/close 8793 (time 1787397865.15 total 140.12 last 55.17) - open/close 9425 (time 1787397875.16 total 150.14 last 63.13) total: 10000 open/close in 158.47 seconds: 63.10 ops/second - unlinked 0 (time 1787397888 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 519 (time 1787397989.31 total 10.02 last 51.82) - open/close 1068 (time 1787397999.32 total 20.03 last 54.85) - open/close 1709 (time 1787398009.33 total 30.03 last 64.07) - open/close 2357 (time 1787398019.34 total 40.04 last 64.72) - open/close 3017 (time 1787398029.34 total 50.05 last 65.97) - open/close 3691 (time 1787398039.35 total 60.06 last 67.33) - open/close 4337 (time 1787398049.39 total 70.10 last 64.35) - open/close 4922 (time 1787398059.42 total 80.12 last 58.36) - open/close 5519 (time 1787398069.42 total 90.12 last 59.70) - open/close 6115 (time 1787398079.42 total 100.13 last 59.56) - open/close 6798 (time 1787398089.42 total 110.13 last 68.30) - open/close 7447 (time 1787398099.43 total 120.13 last 64.86) - open/close 8075 (time 1787398109.43 total 130.14 last 62.77) - open/close 8745 (time 1787398119.44 total 140.15 last 66.95) - open/close 9340 (time 1787398129.45 total 150.15 last 59.45) - open/close 9938 (time 1787398139.45 total 160.16 last 59.76) total: 10000 open/close in 161.09 seconds: 62.08 ops/second - unlinked 0 (time 1787398145 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 609 (time 1787398256.71 total 10.01 last 60.84) - open/close 1208 (time 1787398266.71 total 20.01 last 59.89) - open/close 1790 (time 1787398276.71 total 30.01 last 58.19) - open/close 2422 (time 1787398286.72 total 40.02 last 63.17) - open/close 3026 (time 1787398296.73 total 50.03 last 60.33) - open/close 3642 (time 1787398306.74 total 60.04 last 61.55) - open/close 4252 (time 1787398316.76 total 70.05 last 60.89) - open/close 4924 (time 1787398326.76 total 80.06 last 67.18) - open/close 5508 (time 1787398336.78 total 90.08 last 58.27) - open/close 6019 (time 1787398346.79 total 100.09 last 51.05) - open/close 6526 (time 1787398356.80 total 110.09 last 50.68) - open/close 7140 (time 1787398366.81 total 120.11 last 61.33) - open/close 7843 (time 1787398376.83 total 130.13 last 70.16) - open/close 8444 (time 1787398386.83 total 140.13 last 60.05) - open/close 9048 (time 1787398396.85 total 150.15 last 60.31) - open/close 9636 (time 1787398406.86 total 160.15 last 58.76) total: 10000 open/close in 167.20 seconds: 59.81 ops/second - unlinked 0 (time 1787398419 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.121@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.121@tcp sys.timeout=20 start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: modules unloaded. pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2