== conf-sanity test 69: replace an OST with the same index ========================================================== 06:47:30 (1787136450) start mds service on oleg230-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions oleg230-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg230-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg230-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg230-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg230-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 924 (time 1787136568.17 total 10.01 last 92.33) - open/close 1744 (time 1787136578.18 total 20.01 last 81.95) - open/close 2588 (time 1787136588.19 total 30.02 last 84.33) - open/close 3424 (time 1787136598.19 total 40.03 last 83.57) - open/close 4259 (time 1787136608.20 total 50.03 last 83.43) - open/close 5205 (time 1787136618.20 total 60.04 last 94.57) - open/close 6042 (time 1787136628.21 total 70.05 last 83.64) - open/close 6855 (time 1787136638.22 total 80.05 last 81.24) - open/close 7566 (time 1787136648.22 total 90.05 last 71.09) - open/close 8293 (time 1787136658.22 total 100.06 last 72.65) - open/close 9047 (time 1787136668.23 total 110.07 last 75.36) - open/close 9944 (time 1787136678.24 total 120.08 last 89.60) total: 10000 open/close in 120.75 seconds: 82.81 ops/second - unlinked 0 (time 1787136683 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 763 (time 1787136773.92 total 10.00 last 76.27) - open/close 1606 (time 1787136783.92 total 20.00 last 84.29) - open/close 2472 (time 1787136793.93 total 30.01 last 86.53) - open/close 3326 (time 1787136803.93 total 40.01 last 85.39) - open/close 4154 (time 1787136813.94 total 50.02 last 82.77) - open/close 4998 (time 1787136823.95 total 60.03 last 84.32) - open/close 5786 (time 1787136833.96 total 70.04 last 78.71) - open/close 6624 (time 1787136843.96 total 80.04 last 83.79) - open/close 7227 (time 1787136853.97 total 90.05 last 60.24) - open/close 7919 (time 1787136863.98 total 100.06 last 69.15) - open/close 8520 (time 1787136873.98 total 110.06 last 60.07) - open/close 9169 (time 1787136883.99 total 120.07 last 64.86) - open/close 9947 (time 1787136893.99 total 130.07 last 77.80) total: 10000 open/close in 131.38 seconds: 76.12 ops/second - unlinked 0 (time 1787136900 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 842 (time 1787136983.94 total 10.00 last 84.16) - open/close 1681 (time 1787136993.94 total 20.00 last 83.90) - open/close 2443 (time 1787137003.95 total 30.01 last 76.14) - open/close 3349 (time 1787137013.95 total 40.02 last 90.53) - open/close 4222 (time 1787137023.96 total 50.02 last 87.26) - open/close 4959 (time 1787137033.96 total 60.03 last 73.65) - open/close 5558 (time 1787137043.96 total 70.03 last 59.90) - open/close 6222 (time 1787137053.97 total 80.04 last 66.36) - open/close 6908 (time 1787137063.99 total 90.06 last 68.44) - open/close 7650 (time 1787137073.99 total 100.06 last 74.20) - open/close 8349 (time 1787137084.00 total 110.07 last 69.85) - open/close 9189 (time 1787137094.01 total 120.07 last 83.96) total: 10000 open/close in 129.60 seconds: 77.16 ops/second - unlinked 0 (time 1787137107 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 806 (time 1787137187.51 total 10.00 last 80.57) - open/close 1416 (time 1787137197.51 total 20.00 last 61.00) - open/close 2213 (time 1787137207.52 total 30.02 last 79.60) - open/close 2870 (time 1787137217.53 total 40.02 last 65.66) - open/close 3458 (time 1787137227.53 total 50.02 last 58.79) - open/close 4347 (time 1787137237.56 total 60.06 last 88.59) - open/close 5056 (time 1787137247.57 total 70.06 last 70.89) - open/close 5811 (time 1787137257.59 total 80.09 last 75.28) - open/close 6699 (time 1787137267.60 total 90.09 last 88.77) - open/close 7486 (time 1787137277.60 total 100.09 last 78.69) - open/close 8343 (time 1787137287.60 total 110.10 last 85.66) - open/close 9153 (time 1787137297.61 total 120.10 last 80.95) - open/close 9862 (time 1787137307.62 total 130.11 last 70.84) total: 10000 open/close in 131.83 seconds: 75.85 ops/second - unlinked 0 (time 1787137315 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 873 (time 1787137408.89 total 10.00 last 87.29) - open/close 1728 (time 1787137418.89 total 20.00 last 85.49) - open/close 2499 (time 1787137428.93 total 30.04 last 76.79) - open/close 3242 (time 1787137438.94 total 40.05 last 74.25) - open/close 4012 (time 1787137448.94 total 50.05 last 76.98) - open/close 4834 (time 1787137458.95 total 60.06 last 82.14) - open/close 5590 (time 1787137468.95 total 70.06 last 75.58) - open/close 6384 (time 1787137478.96 total 80.07 last 79.36) - open/close 7142 (time 1787137488.97 total 90.08 last 75.67) - open/close 7805 (time 1787137498.98 total 100.09 last 66.24) - open/close 8564 (time 1787137508.99 total 110.10 last 75.87) - open/close 9443 (time 1787137518.99 total 120.10 last 87.89) total: 10000 open/close in 126.50 seconds: 79.05 ops/second - unlinked 0 (time 1787137531 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 768 (time 1787137617.87 total 10.01 last 76.75) - open/close 1560 (time 1787137627.88 total 20.01 last 79.14) - open/close 2355 (time 1787137637.88 total 30.02 last 79.47) - open/close 3183 (time 1787137647.89 total 40.03 last 82.72) - open/close 4044 (time 1787137657.91 total 50.04 last 85.95) - open/close 4823 (time 1787137667.92 total 60.06 last 77.82) - open/close 5833 (time 1787137677.92 total 70.06 last 100.99) - open/close 6754 (time 1787137687.92 total 80.06 last 92.10) - open/close 7747 (time 1787137697.93 total 90.06 last 99.21) - open/close 8642 (time 1787137707.94 total 100.07 last 89.45) - open/close 9511 (time 1787137717.94 total 110.07 last 86.89) total: 10000 open/close in 115.56 seconds: 86.53 ops/second - unlinked 0 (time 1787137728 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 929 (time 1787137809.04 total 10.00 last 92.90) - open/close 1821 (time 1787137819.04 total 20.00 last 89.17) - open/close 2586 (time 1787137829.04 total 30.00 last 76.49) - open/close 3447 (time 1787137839.05 total 40.01 last 86.06) - open/close 4253 (time 1787137849.05 total 50.01 last 80.59) - open/close 5149 (time 1787137859.06 total 60.03 last 89.47) - open/close 6033 (time 1787137869.07 total 70.03 last 88.35) - open/close 7052 (time 1787137879.07 total 80.03 last 101.88) - open/close 7927 (time 1787137889.07 total 90.04 last 87.48) - open/close 8841 (time 1787137899.07 total 100.04 last 91.38) - open/close 9848 (time 1787137909.08 total 110.04 last 100.68) total: 10000 open/close in 111.63 seconds: 89.58 ops/second - unlinked 0 (time 1787137914 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 717 (time 1787138003.14 total 10.01 last 71.65) - open/close 1502 (time 1787138013.14 total 20.01 last 78.44) - open/close 2275 (time 1787138023.15 total 30.02 last 77.27) - open/close 3089 (time 1787138033.16 total 40.03 last 81.31) - open/close 3963 (time 1787138043.17 total 50.04 last 87.33) - open/close 4710 (time 1787138053.18 total 60.05 last 74.65) - open/close 5371 (time 1787138063.18 total 70.05 last 66.04) - open/close 6223 (time 1787138073.19 total 80.06 last 85.16) - open/close 6926 (time 1787138083.19 total 90.06 last 70.28) - open/close 7782 (time 1787138093.21 total 100.08 last 85.49) - open/close 8561 (time 1787138103.21 total 110.08 last 77.83) - open/close 9488 (time 1787138113.22 total 120.09 last 92.66) total: 10000 open/close in 125.14 seconds: 79.91 ops/second - unlinked 0 (time 1787138122 ; total 1 ; last 1) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 659 (time 1787138200.93 total 10.00 last 65.87) - open/close 1432 (time 1787138210.94 total 20.01 last 77.23) - open/close 2247 (time 1787138220.97 total 30.04 last 81.29) - open/close 3163 (time 1787138230.98 total 40.05 last 91.50) - open/close 4061 (time 1787138240.98 total 50.05 last 89.78) - open/close 4864 (time 1787138250.98 total 60.06 last 80.28) - open/close 5735 (time 1787138260.99 total 70.07 last 87.02) - open/close 6532 (time 1787138270.99 total 80.07 last 79.68) - open/close 7456 (time 1787138281.01 total 90.08 last 92.24) - open/close 8188 (time 1787138291.02 total 100.10 last 73.11) - open/close 9102 (time 1787138301.02 total 110.10 last 91.38) - open/close 9980 (time 1787138311.03 total 120.10 last 87.78) total: 10000 open/close in 120.31 seconds: 83.12 ops/second - unlinked 0 (time 1787138315 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 810 (time 1787138397.77 total 10.00 last 81.00) - open/close 1693 (time 1787138407.78 total 20.01 last 88.25) - open/close 2676 (time 1787138417.78 total 30.01 last 98.24) - open/close 3640 (time 1787138427.79 total 40.02 last 96.35) - open/close 4541 (time 1787138437.79 total 50.02 last 90.10) - open/close 5342 (time 1787138447.79 total 60.02 last 80.06) - open/close 6161 (time 1787138457.80 total 70.03 last 81.86) - open/close 6906 (time 1787138467.80 total 80.03 last 74.46) - open/close 7801 (time 1787138477.81 total 90.03 last 89.49) - open/close 8733 (time 1787138487.81 total 100.03 last 93.18) - open/close 9631 (time 1787138497.81 total 110.04 last 89.76) total: 10000 open/close in 114.24 seconds: 87.53 ops/second - unlinked 0 (time 1787138506 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.130@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg230-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg230-server: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg230-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg230-server: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg230-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.130@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.130@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.130@tcp sys.timeout=20 start ost1 service on oleg230-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg230-server: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg230-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg230-server: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg230-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg230-server unloading modules via unload_modules_local on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing unload_modules_local oleg230-server: modules unloaded. pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2