== conf-sanity test 69: replace an OST with the same index ========================================================== 01:31:05 (1786167065) start mds service on oleg243-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg243-server: detected 4 online CPUs by sysfs oleg243-server: MODOPTS_LIBCFS= oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg243-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg243-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 766 (time 1786167181.38 total 10.01 last 76.52) - open/close 1574 (time 1786167191.39 total 20.02 last 80.70) - open/close 2417 (time 1786167201.41 total 30.04 last 84.16) - open/close 3117 (time 1786167211.41 total 40.05 last 69.94) - open/close 3833 (time 1786167221.42 total 50.05 last 71.58) - open/close 4540 (time 1786167231.43 total 60.06 last 70.60) - open/close 5317 (time 1786167241.44 total 70.07 last 77.65) - open/close 6053 (time 1786167251.44 total 80.08 last 73.56) - open/close 6753 (time 1786167261.45 total 90.08 last 69.98) - open/close 7413 (time 1786167271.46 total 100.09 last 65.94) - open/close 8072 (time 1786167281.46 total 110.09 last 65.87) - open/close 8884 (time 1786167291.47 total 120.10 last 81.15) - open/close 9738 (time 1786167301.47 total 130.11 last 85.34) total: 10000 open/close in 133.60 seconds: 74.85 ops/second - unlinked 0 (time 1786167309 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 585 (time 1786167404.33 total 10.00 last 58.49) - open/close 1313 (time 1786167414.33 total 20.01 last 72.77) - open/close 2093 (time 1786167424.34 total 30.01 last 77.96) - open/close 3008 (time 1786167434.34 total 40.02 last 91.45) - open/close 3910 (time 1786167444.35 total 50.02 last 90.14) - open/close 4707 (time 1786167454.35 total 60.03 last 79.68) - open/close 5459 (time 1786167464.36 total 70.03 last 75.16) - open/close 6287 (time 1786167474.37 total 80.04 last 82.75) - open/close 6896 (time 1786167484.37 total 90.04 last 60.85) - open/close 7724 (time 1786167494.38 total 100.05 last 82.76) - open/close 8477 (time 1786167504.39 total 110.06 last 75.21) - open/close 9107 (time 1786167514.41 total 120.09 last 62.85) - open/close 9673 (time 1786167524.43 total 130.10 last 56.51) total: 10000 open/close in 135.80 seconds: 73.64 ops/second - unlinked 0 (time 1786167535 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 725 (time 1786167641.14 total 10.00 last 72.47) - open/close 1369 (time 1786167651.14 total 20.01 last 64.39) - open/close 2126 (time 1786167661.15 total 30.01 last 75.66) - open/close 2926 (time 1786167671.15 total 40.02 last 79.94) - open/close 3769 (time 1786167681.16 total 50.02 last 84.26) - open/close 4538 (time 1786167691.17 total 60.03 last 76.83) - open/close 5275 (time 1786167701.17 total 70.04 last 73.67) - open/close 6153 (time 1786167711.17 total 80.04 last 87.79) - open/close 6989 (time 1786167721.19 total 90.05 last 83.50) - open/close 7714 (time 1786167731.19 total 100.06 last 72.44) - open/close 8374 (time 1786167741.20 total 110.07 last 65.94) - open/close 9090 (time 1786167751.21 total 120.07 last 71.58) - open/close 9810 (time 1786167761.22 total 130.08 last 71.93) total: 10000 open/close in 133.07 seconds: 75.15 ops/second - unlinked 0 (time 1786167769 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 760 (time 1786167858.59 total 10.01 last 75.94) - open/close 1648 (time 1786167868.59 total 20.01 last 88.80) - open/close 2469 (time 1786167878.60 total 30.02 last 81.99) - open/close 3246 (time 1786167888.61 total 40.02 last 77.67) - open/close 3920 (time 1786167898.61 total 50.03 last 67.38) - open/close 4768 (time 1786167908.63 total 60.05 last 84.65) - open/close 5620 (time 1786167918.64 total 70.05 last 85.12) - open/close 6426 (time 1786167928.65 total 80.07 last 80.50) - open/close 7154 (time 1786167938.66 total 90.07 last 72.75) - open/close 8000 (time 1786167948.66 total 100.08 last 84.59) - open/close 8850 (time 1786167958.67 total 110.08 last 84.94) - open/close 9731 (time 1786167968.67 total 120.09 last 88.07) total: 10000 open/close in 123.15 seconds: 81.20 ops/second - unlinked 0 (time 1786167976 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 733 (time 1786168066.94 total 10.00 last 73.30) - open/close 1607 (time 1786168076.94 total 20.00 last 87.39) - open/close 2380 (time 1786168086.94 total 30.00 last 77.27) - open/close 3262 (time 1786168096.95 total 40.02 last 88.09) - open/close 3992 (time 1786168106.96 total 50.03 last 72.93) - open/close 4777 (time 1786168116.97 total 60.04 last 78.42) - open/close 5479 (time 1786168126.98 total 70.04 last 70.15) - open/close 6184 (time 1786168136.99 total 80.06 last 70.42) - open/close 6917 (time 1786168147.00 total 90.06 last 73.23) - open/close 7702 (time 1786168157.01 total 100.07 last 78.45) - open/close 8306 (time 1786168167.01 total 110.08 last 60.37) - open/close 8935 (time 1786168177.01 total 120.08 last 62.89) - open/close 9679 (time 1786168187.02 total 130.08 last 74.36) total: 10000 open/close in 134.25 seconds: 74.49 ops/second - unlinked 0 (time 1786168196 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 575 (time 1786168292.22 total 10.02 last 57.40) - open/close 1269 (time 1786168302.23 total 20.02 last 69.37) - open/close 2191 (time 1786168312.23 total 30.02 last 92.20) - open/close 2969 (time 1786168322.23 total 40.03 last 77.78) - open/close 3813 (time 1786168332.24 total 50.03 last 84.34) - open/close 4583 (time 1786168342.25 total 60.04 last 76.92) - open/close 5436 (time 1786168352.26 total 70.05 last 85.23) - open/close 6289 (time 1786168362.26 total 80.05 last 85.29) - open/close 7111 (time 1786168372.29 total 90.09 last 81.89) - open/close 7765 (time 1786168382.29 total 100.09 last 65.40) - open/close 8721 (time 1786168392.30 total 110.09 last 95.56) - open/close 9344 (time 1786168402.30 total 120.10 last 62.27) - open/close 9959 (time 1786168412.32 total 130.11 last 61.43) total: 10000 open/close in 130.85 seconds: 76.42 ops/second - unlinked 0 (time 1786168418 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 808 (time 1786168507.84 total 10.01 last 80.72) - open/close 1543 (time 1786168517.85 total 20.02 last 73.46) - open/close 2183 (time 1786168527.85 total 30.02 last 63.97) - open/close 2894 (time 1786168537.86 total 40.03 last 71.07) - open/close 3590 (time 1786168547.86 total 50.03 last 69.59) - open/close 4294 (time 1786168557.87 total 60.04 last 70.29) - open/close 4906 (time 1786168567.88 total 70.05 last 61.16) - open/close 5709 (time 1786168577.89 total 80.05 last 80.26) - open/close 6304 (time 1786168587.89 total 90.06 last 59.46) - open/close 6959 (time 1786168597.91 total 100.08 last 65.40) - open/close 7706 (time 1786168607.92 total 110.09 last 74.63) - open/close 8523 (time 1786168617.92 total 120.09 last 81.68) - open/close 9346 (time 1786168627.93 total 130.10 last 82.21) total: 10000 open/close in 138.32 seconds: 72.30 ops/second - unlinked 0 (time 1786168640 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 776 (time 1786168732.87 total 10.00 last 77.57) - open/close 1492 (time 1786168742.88 total 20.01 last 71.53) - open/close 2106 (time 1786168752.90 total 30.03 last 61.33) - open/close 2609 (time 1786168762.90 total 40.03 last 50.30) - open/close 3190 (time 1786168772.92 total 50.05 last 57.99) - open/close 3842 (time 1786168782.92 total 60.05 last 65.19) - open/close 4547 (time 1786168792.92 total 70.05 last 70.45) - open/close 5299 (time 1786168802.94 total 80.07 last 75.11) - open/close 5996 (time 1786168812.95 total 90.08 last 69.58) - open/close 6594 (time 1786168822.96 total 100.09 last 59.77) - open/close 7301 (time 1786168832.96 total 110.09 last 70.69) - open/close 7934 (time 1786168842.97 total 120.10 last 63.21) - open/close 8612 (time 1786168852.98 total 130.11 last 67.75) - open/close 9368 (time 1786168862.98 total 140.11 last 75.59) total: 10000 open/close in 146.90 seconds: 68.07 ops/second - unlinked 0 (time 1786168874 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 795 (time 1786168963.38 total 10.00 last 79.46) - open/close 1767 (time 1786168973.39 total 20.01 last 97.12) - open/close 2576 (time 1786168983.40 total 30.02 last 80.83) - open/close 3328 (time 1786168993.40 total 40.02 last 75.20) - open/close 4063 (time 1786169003.40 total 50.02 last 73.48) - open/close 4778 (time 1786169013.41 total 60.03 last 71.43) - open/close 5591 (time 1786169023.42 total 70.04 last 81.22) - open/close 6412 (time 1786169033.42 total 80.05 last 82.09) - open/close 7186 (time 1786169043.43 total 90.05 last 77.33) - open/close 7898 (time 1786169053.44 total 100.06 last 71.12) - open/close 8394 (time 1786169063.45 total 110.07 last 49.56) - open/close 8948 (time 1786169073.46 total 120.08 last 55.34) - open/close 9568 (time 1786169083.47 total 130.09 last 61.97) total: 10000 open/close in 136.42 seconds: 73.30 ops/second - unlinked 0 (time 1786169095 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 736 (time 1786169182.71 total 10.02 last 73.47) - open/close 1529 (time 1786169192.71 total 20.02 last 79.30) - open/close 2408 (time 1786169202.72 total 30.03 last 87.77) - open/close 3138 (time 1786169212.72 total 40.03 last 72.99) - open/close 3997 (time 1786169222.72 total 50.03 last 85.90) - open/close 4694 (time 1786169232.73 total 60.04 last 69.66) - open/close 5541 (time 1786169242.75 total 70.06 last 84.50) - open/close 6402 (time 1786169252.77 total 80.08 last 85.95) - open/close 7149 (time 1786169262.78 total 90.09 last 74.63) - open/close 8011 (time 1786169272.79 total 100.10 last 86.12) - open/close 8879 (time 1786169282.80 total 110.11 last 86.72) - open/close 9748 (time 1786169292.80 total 120.12 last 86.83) total: 10000 open/close in 122.93 seconds: 81.34 ops/second - unlinked 0 (time 1786169299 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg243-server unloading modules via unload_modules_local on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2