== conf-sanity test 69: replace an OST with the same index ========================================================== 08:12:58 (1789128778) start mds service on oleg122-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg122-server: detected 4 online CPUs by sysfs oleg122-server: MODOPTS_LIBCFS= oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg122-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg122-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49502 inodes available. Want 99872. rc=0 - open/close 827 (time 1789128914.27 total 10.00 last 82.66) - open/close 1511 (time 1789128924.28 total 20.02 last 68.33) - open/close 2489 (time 1789128934.28 total 30.02 last 97.80) - open/close 3403 (time 1789128944.28 total 40.02 last 91.36) - open/close 4320 (time 1789128954.29 total 50.02 last 91.66) - open/close 4975 (time 1789128964.30 total 60.04 last 65.42) - open/close 5690 (time 1789128974.30 total 70.04 last 71.48) - open/close 6414 (time 1789128984.30 total 80.04 last 72.40) - open/close 7060 (time 1789128994.33 total 90.07 last 64.41) - open/close 7683 (time 1789129004.34 total 100.08 last 62.26) - open/close 8346 (time 1789129014.35 total 110.09 last 66.22) - open/close 9038 (time 1789129024.36 total 120.10 last 69.15) - open/close 9572 (time 1789129034.38 total 130.11 last 53.32) total: 10000 open/close in 137.79 seconds: 72.57 ops/second - unlinked 0 (time 1789129048 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 701 (time 1789129144.44 total 10.00 last 70.08) - open/close 1388 (time 1789129154.45 total 20.01 last 68.66) - open/close 2189 (time 1789129164.45 total 30.01 last 80.07) - open/close 3042 (time 1789129174.45 total 40.01 last 85.28) - open/close 3702 (time 1789129184.45 total 50.02 last 65.98) - open/close 4498 (time 1789129194.46 total 60.02 last 79.58) - open/close 5191 (time 1789129204.46 total 70.03 last 69.26) - open/close 5954 (time 1789129214.46 total 80.03 last 76.30) - open/close 6614 (time 1789129224.47 total 90.03 last 65.97) - open/close 7146 (time 1789129234.48 total 100.04 last 53.14) - open/close 7726 (time 1789129244.48 total 110.04 last 57.99) - open/close 8513 (time 1789129254.49 total 120.05 last 78.64) - open/close 9254 (time 1789129264.49 total 130.05 last 74.08) total: 10000 open/close in 139.32 seconds: 71.78 ops/second - unlinked 0 (time 1789129278 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 727 (time 1789129374.38 total 10.02 last 72.57) - open/close 1425 (time 1789129384.39 total 20.03 last 69.74) - open/close 2224 (time 1789129394.39 total 30.03 last 79.86) - open/close 2975 (time 1789129404.40 total 40.04 last 75.05) - open/close 3712 (time 1789129414.40 total 50.04 last 73.67) - open/close 4431 (time 1789129424.42 total 60.05 last 71.81) - open/close 5218 (time 1789129434.42 total 70.06 last 78.69) - open/close 5892 (time 1789129444.42 total 80.06 last 67.37) - open/close 6626 (time 1789129454.42 total 90.06 last 73.39) - open/close 7413 (time 1789129464.42 total 100.06 last 78.69) - open/close 8330 (time 1789129474.43 total 110.07 last 91.65) - open/close 9067 (time 1789129484.43 total 120.07 last 73.68) - open/close 9731 (time 1789129494.44 total 130.08 last 66.35) total: 10000 open/close in 133.60 seconds: 74.85 ops/second - unlinked 0 (time 1789129503 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 655 (time 1789129598.66 total 10.00 last 65.48) - open/close 1359 (time 1789129608.66 total 20.01 last 70.35) - open/close 2067 (time 1789129618.67 total 30.02 last 70.74) - open/close 2650 (time 1789129628.68 total 40.02 last 58.27) - open/close 3223 (time 1789129638.68 total 50.03 last 57.29) - open/close 3894 (time 1789129648.68 total 60.03 last 67.09) - open/close 4509 (time 1789129658.69 total 70.03 last 61.48) - open/close 5323 (time 1789129668.71 total 80.05 last 81.24) - open/close 6015 (time 1789129678.71 total 90.05 last 69.18) - open/close 6695 (time 1789129688.71 total 100.06 last 67.98) - open/close 7232 (time 1789129698.74 total 110.09 last 53.54) - open/close 7936 (time 1789129708.75 total 120.10 last 70.31) - open/close 8604 (time 1789129718.77 total 130.11 last 66.71) - open/close 9315 (time 1789129728.77 total 140.12 last 71.08) total: 10000 open/close in 149.15 seconds: 67.05 ops/second - unlinked 0 (time 1789129743 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 575 (time 1789129835.22 total 10.02 last 57.38) - open/close 1198 (time 1789129845.23 total 20.03 last 62.26) - open/close 1987 (time 1789129855.24 total 30.04 last 78.79) - open/close 2657 (time 1789129865.25 total 40.05 last 66.96) - open/close 3088 (time 1789129875.26 total 50.06 last 43.06) - open/close 3607 (time 1789129885.27 total 60.07 last 51.83) - open/close 4149 (time 1789129895.29 total 70.09 last 54.11) - open/close 4715 (time 1789129905.30 total 80.10 last 56.53) - open/close 5437 (time 1789129915.33 total 90.13 last 72.02) - open/close 6145 (time 1789129925.35 total 100.15 last 70.66) - open/close 6816 (time 1789129935.36 total 110.16 last 67.03) - open/close 7466 (time 1789129945.38 total 120.18 last 64.86) - open/close 8250 (time 1789129955.39 total 130.19 last 78.28) - open/close 8940 (time 1789129965.40 total 140.19 last 68.99) - open/close 9770 (time 1789129975.41 total 150.20 last 82.90) total: 10000 open/close in 153.27 seconds: 65.25 ops/second - unlinked 0 (time 1789129983 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 530 (time 1789130083.01 total 10.03 last 52.87) - open/close 1051 (time 1789130093.02 total 20.04 last 52.05) - open/close 1815 (time 1789130103.02 total 30.04 last 76.36) - open/close 2519 (time 1789130113.03 total 40.05 last 70.35) - open/close 3354 (time 1789130123.05 total 50.07 last 83.29) - open/close 4077 (time 1789130133.06 total 60.08 last 72.24) - open/close 4757 (time 1789130143.07 total 70.09 last 67.98) - open/close 5350 (time 1789130153.09 total 80.11 last 59.16) - open/close 6097 (time 1789130163.09 total 90.11 last 74.67) - open/close 6718 (time 1789130173.11 total 100.13 last 61.98) - open/close 7426 (time 1789130183.12 total 110.14 last 70.74) - open/close 8089 (time 1789130193.13 total 120.15 last 66.24) - open/close 8742 (time 1789130203.14 total 130.16 last 65.23) - open/close 9573 (time 1789130213.14 total 140.16 last 83.09) total: 10000 open/close in 146.07 seconds: 68.46 ops/second - unlinked 0 (time 1789130223 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 715 (time 1789130313.29 total 10.01 last 71.44) - open/close 1456 (time 1789130323.31 total 20.02 last 74.00) - open/close 2078 (time 1789130333.31 total 30.03 last 62.17) - open/close 2850 (time 1789130343.32 total 40.03 last 77.15) - open/close 3708 (time 1789130353.33 total 50.04 last 85.74) - open/close 4638 (time 1789130363.33 total 60.04 last 93.00) - open/close 5686 (time 1789130373.33 total 70.05 last 104.72) - open/close 6485 (time 1789130383.34 total 80.06 last 79.85) - open/close 7381 (time 1789130393.34 total 90.06 last 89.57) - open/close 8191 (time 1789130403.35 total 100.06 last 80.99) - open/close 8918 (time 1789130413.37 total 110.08 last 72.53) - open/close 9788 (time 1789130423.38 total 120.09 last 86.91) total: 10000 open/close in 122.91 seconds: 81.36 ops/second - unlinked 0 (time 1789130431 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 737 (time 1789130526.66 total 10.02 last 73.58) - open/close 1599 (time 1789130536.66 total 20.02 last 86.19) - open/close 2302 (time 1789130546.67 total 30.03 last 70.23) - open/close 2777 (time 1789130556.69 total 40.04 last 47.44) - open/close 3336 (time 1789130566.69 total 50.05 last 55.87) - open/close 3967 (time 1789130576.70 total 60.05 last 63.07) - open/close 4843 (time 1789130586.70 total 70.06 last 87.54) - open/close 5760 (time 1789130596.71 total 80.06 last 91.65) - open/close 6532 (time 1789130606.71 total 90.07 last 77.16) - open/close 7280 (time 1789130616.73 total 100.08 last 74.70) - open/close 7983 (time 1789130626.74 total 110.09 last 70.23) - open/close 8658 (time 1789130636.74 total 120.10 last 67.45) - open/close 9450 (time 1789130646.75 total 130.10 last 79.16) total: 10000 open/close in 137.57 seconds: 72.69 ops/second - unlinked 0 (time 1789130659 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 700 (time 1789130748.48 total 10.01 last 69.93) - open/close 1466 (time 1789130758.49 total 20.01 last 76.57) - open/close 2038 (time 1789130768.50 total 30.03 last 57.12) - open/close 2575 (time 1789130778.50 total 40.03 last 53.69) - open/close 3142 (time 1789130788.51 total 50.03 last 56.68) - open/close 3735 (time 1789130798.52 total 60.05 last 59.19) - open/close 4360 (time 1789130808.54 total 70.07 last 62.41) - open/close 5010 (time 1789130818.55 total 80.08 last 64.93) - open/close 5779 (time 1789130828.56 total 90.09 last 76.83) - open/close 6551 (time 1789130838.57 total 100.09 last 77.15) - open/close 7262 (time 1789130848.57 total 110.10 last 71.04) - open/close 7947 (time 1789130858.58 total 120.10 last 68.49) - open/close 8727 (time 1789130868.58 total 130.11 last 77.98) - open/close 9418 (time 1789130878.58 total 140.11 last 69.09) - open/close 9992 (time 1789130888.59 total 150.12 last 57.34) total: 10000 open/close in 150.21 seconds: 66.58 ops/second - unlinked 0 (time 1789130893 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 705 (time 1789130989.72 total 10.00 last 70.48) - open/close 1429 (time 1789130999.72 total 20.01 last 72.37) - open/close 2010 (time 1789131009.73 total 30.01 last 58.05) - open/close 2749 (time 1789131019.75 total 40.03 last 73.77) - open/close 3439 (time 1789131029.75 total 50.04 last 68.97) - open/close 4221 (time 1789131039.76 total 60.04 last 78.16) - open/close 4941 (time 1789131049.76 total 70.04 last 71.99) - open/close 5657 (time 1789131059.76 total 80.05 last 71.57) - open/close 6373 (time 1789131069.78 total 90.06 last 71.49) - open/close 7105 (time 1789131079.78 total 100.07 last 73.17) - open/close 7733 (time 1789131089.79 total 110.07 last 62.79) - open/close 8538 (time 1789131099.79 total 120.07 last 80.46) - open/close 9282 (time 1789131109.79 total 130.07 last 74.40) - open/close 9863 (time 1789131119.82 total 140.10 last 57.94) total: 10000 open/close in 142.30 seconds: 70.27 ops/second - unlinked 0 (time 1789131127 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg122-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre On OST0, 20431 used inodes rc=0 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.122@tcp sys.timeout=20 start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg122-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg122-server: oleg122-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg122-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg122-server unloading modules via unload_modules_local on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local oleg122-server: modules unloaded. pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2