== conf-sanity test 69: replace an OST with the same index ========================================================== 22:13:33 (1781230413) start mds service on oleg341-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg341-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg341-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg341-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg341-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 808 (time 1781230521.82 total 10.01 last 80.74) - open/close 1793 (time 1781230531.82 total 20.01 last 98.45) - open/close 2704 (time 1781230541.83 total 30.02 last 91.06) - open/close 3582 (time 1781230551.83 total 40.02 last 87.78) - open/close 4463 (time 1781230561.85 total 50.04 last 87.95) - open/close 5331 (time 1781230571.85 total 60.04 last 86.73) - open/close 6174 (time 1781230581.86 total 70.05 last 84.26) - open/close 7077 (time 1781230591.87 total 80.06 last 90.22) - open/close 7949 (time 1781230601.87 total 90.06 last 87.17) - open/close 8789 (time 1781230611.88 total 100.07 last 83.93) - open/close 9585 (time 1781230621.89 total 110.08 last 79.55) total: 10000 open/close in 115.03 seconds: 86.94 ops/second - unlinked 0 (time 1781230631 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 861 (time 1781230714.98 total 10.01 last 86.05) - open/close 1827 (time 1781230724.98 total 20.01 last 96.59) - open/close 2644 (time 1781230734.98 total 30.01 last 81.64) - open/close 3582 (time 1781230744.99 total 40.02 last 93.78) - open/close 4475 (time 1781230755.01 total 50.04 last 89.06) - open/close 5290 (time 1781230765.02 total 60.05 last 81.47) - open/close 6207 (time 1781230775.02 total 70.05 last 91.65) - open/close 7136 (time 1781230785.02 total 80.05 last 92.89) - open/close 8073 (time 1781230795.02 total 90.05 last 93.69) - open/close 8948 (time 1781230805.02 total 100.05 last 87.49) - open/close 9720 (time 1781230815.03 total 110.06 last 77.16) total: 10000 open/close in 113.38 seconds: 88.20 ops/second - unlinked 0 (time 1781230823 ; total 1 ; last 1) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 828 (time 1781230910.97 total 10.01 last 82.75) - open/close 1718 (time 1781230920.98 total 20.01 last 88.95) - open/close 2540 (time 1781230930.98 total 30.01 last 82.19) - open/close 3415 (time 1781230940.99 total 40.02 last 87.43) - open/close 4307 (time 1781230951.00 total 50.03 last 89.12) - open/close 5140 (time 1781230961.00 total 60.04 last 83.23) - open/close 6043 (time 1781230971.01 total 70.05 last 90.21) - open/close 6900 (time 1781230981.02 total 80.05 last 85.66) - open/close 7751 (time 1781230991.02 total 90.06 last 85.06) - open/close 8559 (time 1781231001.03 total 100.06 last 80.77) - open/close 9448 (time 1781231011.03 total 110.06 last 88.87) total: 10000 open/close in 116.19 seconds: 86.07 ops/second - unlinked 0 (time 1781231021 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 705 (time 1781231107.36 total 10.01 last 70.44) - open/close 1403 (time 1781231117.36 total 20.02 last 69.75) - open/close 2238 (time 1781231127.37 total 30.02 last 83.44) - open/close 3067 (time 1781231137.38 total 40.03 last 82.84) - open/close 3758 (time 1781231147.39 total 50.05 last 68.99) - open/close 4528 (time 1781231157.40 total 60.05 last 76.94) - open/close 5386 (time 1781231167.40 total 70.06 last 85.79) - open/close 6079 (time 1781231177.41 total 80.06 last 69.27) - open/close 6764 (time 1781231187.41 total 90.07 last 68.46) - open/close 7598 (time 1781231197.43 total 100.08 last 83.25) - open/close 8496 (time 1781231207.43 total 110.08 last 89.80) - open/close 9305 (time 1781231217.44 total 120.09 last 80.82) total: 10000 open/close in 128.83 seconds: 77.62 ops/second - unlinked 0 (time 1781231231 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 814 (time 1781231315.47 total 10.00 last 81.38) - open/close 1571 (time 1781231325.48 total 20.01 last 75.65) - open/close 2327 (time 1781231335.49 total 30.02 last 75.55) - open/close 3101 (time 1781231345.50 total 40.03 last 77.31) - open/close 3792 (time 1781231355.50 total 50.03 last 69.08) - open/close 4526 (time 1781231365.51 total 60.04 last 73.35) - open/close 5378 (time 1781231375.52 total 70.05 last 85.11) - open/close 6162 (time 1781231385.52 total 80.05 last 78.34) - open/close 7047 (time 1781231395.53 total 90.06 last 88.43) - open/close 7897 (time 1781231405.55 total 100.08 last 84.85) - open/close 8612 (time 1781231415.55 total 110.08 last 71.49) - open/close 9495 (time 1781231425.56 total 120.09 last 88.27) total: 10000 open/close in 126.46 seconds: 79.07 ops/second - unlinked 0 (time 1781231436 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 923 (time 1781231520.59 total 10.01 last 92.25) - open/close 1771 (time 1781231530.59 total 20.01 last 84.79) - open/close 2606 (time 1781231540.60 total 30.02 last 83.43) - open/close 3537 (time 1781231550.62 total 40.03 last 92.93) - open/close 4392 (time 1781231560.63 total 50.04 last 85.42) - open/close 5321 (time 1781231570.63 total 60.05 last 92.87) - open/close 6144 (time 1781231580.63 total 70.05 last 82.29) - open/close 6910 (time 1781231590.65 total 80.07 last 76.46) - open/close 7743 (time 1781231600.66 total 90.08 last 83.21) - open/close 8634 (time 1781231610.67 total 100.08 last 89.04) - open/close 9606 (time 1781231620.67 total 110.08 last 97.19) total: 10000 open/close in 114.10 seconds: 87.64 ops/second - unlinked 0 (time 1781231628 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 662 (time 1781231711.74 total 10.00 last 66.19) - open/close 1501 (time 1781231721.74 total 20.01 last 83.87) - open/close 2440 (time 1781231731.74 total 30.01 last 93.89) - open/close 3266 (time 1781231741.74 total 40.01 last 82.57) - open/close 3850 (time 1781231751.76 total 50.02 last 58.32) - open/close 4714 (time 1781231761.77 total 60.03 last 86.30) - open/close 5580 (time 1781231771.78 total 70.04 last 86.55) - open/close 6416 (time 1781231781.78 total 80.05 last 83.56) - open/close 7367 (time 1781231791.78 total 90.05 last 95.06) - open/close 8218 (time 1781231801.79 total 100.06 last 85.02) - open/close 9102 (time 1781231811.81 total 110.07 last 88.26) total: 10000 open/close in 119.99 seconds: 83.34 ops/second - unlinked 0 (time 1781231825 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 828 (time 1781231907.79 total 10.01 last 82.72) - open/close 1709 (time 1781231917.80 total 20.02 last 88.01) - open/close 2546 (time 1781231927.82 total 30.04 last 83.54) - open/close 3432 (time 1781231937.82 total 40.04 last 88.58) - open/close 4276 (time 1781231947.82 total 50.04 last 84.39) - open/close 5161 (time 1781231957.83 total 60.05 last 88.44) - open/close 6021 (time 1781231967.83 total 70.05 last 85.96) - open/close 6892 (time 1781231977.83 total 80.05 last 87.09) - open/close 7819 (time 1781231987.83 total 90.06 last 92.70) - open/close 8715 (time 1781231997.84 total 100.06 last 89.58) - open/close 9534 (time 1781232007.86 total 110.08 last 81.72) total: 10000 open/close in 115.72 seconds: 86.42 ops/second - unlinked 0 (time 1781232018 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 842 (time 1781232100.15 total 10.00 last 84.18) - open/close 1752 (time 1781232110.16 total 20.02 last 90.86) - open/close 2637 (time 1781232120.16 total 30.02 last 88.48) - open/close 3484 (time 1781232130.20 total 40.06 last 84.40) - open/close 4192 (time 1781232140.21 total 50.07 last 70.71) - open/close 4709 (time 1781232150.22 total 60.07 last 51.69) - open/close 5480 (time 1781232160.22 total 70.07 last 77.07) - open/close 6286 (time 1781232170.23 total 80.08 last 80.52) - open/close 7121 (time 1781232180.23 total 90.09 last 83.46) - open/close 8023 (time 1781232190.24 total 100.09 last 90.17) - open/close 8883 (time 1781232200.25 total 110.10 last 85.90) - open/close 9771 (time 1781232210.26 total 120.11 last 88.72) total: 10000 open/close in 122.54 seconds: 81.61 ops/second - unlinked 0 (time 1781232216 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 748 (time 1781232305.92 total 10.00 last 74.79) - open/close 1451 (time 1781232315.92 total 20.00 last 70.28) - open/close 2208 (time 1781232325.94 total 30.02 last 75.58) - open/close 2896 (time 1781232335.95 total 40.03 last 68.75) - open/close 3762 (time 1781232345.96 total 50.04 last 86.50) - open/close 4639 (time 1781232355.96 total 60.04 last 87.66) - open/close 5457 (time 1781232365.97 total 70.05 last 81.75) - open/close 6289 (time 1781232375.97 total 80.05 last 83.20) - open/close 7199 (time 1781232385.98 total 90.06 last 90.91) - open/close 8066 (time 1781232395.98 total 100.06 last 86.64) - open/close 8912 (time 1781232405.99 total 110.07 last 84.53) - open/close 9813 (time 1781232416.00 total 120.08 last 90.07) total: 10000 open/close in 122.75 seconds: 81.47 ops/second - unlinked 0 (time 1781232423 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg341-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg341-server: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg341-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg341-server: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg341-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.141@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.141@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.141@tcp sys.timeout=20 start ost1 service on oleg341-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg341-server: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg341-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg341-server: oleg341-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg341-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg341-server unloading modules via unload_modules_local on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local oleg341-server: modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2