== conf-sanity test 69: replace an OST with the same index ========================================================== 09:19:52 (1788441592) start mds service on oleg301-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions oleg301-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg301-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg301-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg301-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 746 (time 1788441704.97 total 10.01 last 74.55) - open/close 1418 (time 1788441714.97 total 20.01 last 67.16) - open/close 2068 (time 1788441724.98 total 30.01 last 64.99) - open/close 2725 (time 1788441734.99 total 40.03 last 65.63) - open/close 3408 (time 1788441745.00 total 50.04 last 68.19) - open/close 4121 (time 1788441755.00 total 60.04 last 71.27) - open/close 4841 (time 1788441765.02 total 70.06 last 71.92) - open/close 5597 (time 1788441775.02 total 80.06 last 75.58) - open/close 6330 (time 1788441785.03 total 90.07 last 73.20) - open/close 7084 (time 1788441795.03 total 100.07 last 75.40) - open/close 7761 (time 1788441805.04 total 110.08 last 67.66) - open/close 8392 (time 1788441815.05 total 120.09 last 63.01) - open/close 8933 (time 1788441825.06 total 130.10 last 54.07) - open/close 9606 (time 1788441835.06 total 140.10 last 67.30) total: 10000 open/close in 146.51 seconds: 68.25 ops/second - unlinked 0 (time 1788441846 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 728 (time 1788441944.48 total 10.00 last 72.77) - open/close 1408 (time 1788441954.49 total 20.01 last 67.96) - open/close 2187 (time 1788441964.50 total 30.03 last 77.78) - open/close 2973 (time 1788441974.51 total 40.03 last 78.59) - open/close 3664 (time 1788441984.51 total 50.03 last 69.10) - open/close 4461 (time 1788441994.51 total 60.03 last 79.65) - open/close 5231 (time 1788442004.52 total 70.04 last 76.98) - open/close 5765 (time 1788442014.52 total 80.05 last 53.35) - open/close 6466 (time 1788442024.53 total 90.05 last 70.05) - open/close 7212 (time 1788442034.53 total 100.05 last 74.58) - open/close 7999 (time 1788442044.53 total 110.06 last 78.70) - open/close 8690 (time 1788442054.56 total 120.08 last 68.92) - open/close 9372 (time 1788442064.57 total 130.09 last 68.13) total: 10000 open/close in 138.60 seconds: 72.15 ops/second - unlinked 0 (time 1788442078 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 741 (time 1788442171.41 total 10.00 last 74.10) - open/close 1480 (time 1788442181.41 total 20.00 last 73.88) - open/close 2236 (time 1788442191.42 total 30.01 last 75.54) - open/close 2944 (time 1788442201.42 total 40.01 last 70.79) - open/close 3723 (time 1788442211.43 total 50.02 last 77.84) - open/close 4464 (time 1788442221.43 total 60.03 last 74.05) - open/close 5182 (time 1788442231.44 total 70.03 last 71.76) - open/close 5877 (time 1788442241.45 total 80.04 last 69.41) - open/close 6583 (time 1788442251.46 total 90.05 last 70.53) - open/close 7294 (time 1788442261.48 total 100.08 last 70.92) - open/close 7960 (time 1788442271.50 total 110.09 last 66.52) - open/close 8656 (time 1788442281.51 total 120.10 last 69.51) - open/close 9377 (time 1788442291.52 total 130.11 last 72.05) total: 10000 open/close in 137.74 seconds: 72.60 ops/second - unlinked 0 (time 1788442303 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 745 (time 1788442398.22 total 10.01 last 74.46) - open/close 1468 (time 1788442408.22 total 20.01 last 72.29) - open/close 2232 (time 1788442418.22 total 30.01 last 76.40) - open/close 2960 (time 1788442428.24 total 40.03 last 72.67) - open/close 3766 (time 1788442438.25 total 50.04 last 80.52) - open/close 4504 (time 1788442448.26 total 60.04 last 73.75) - open/close 5208 (time 1788442458.27 total 70.05 last 70.32) - open/close 5942 (time 1788442468.28 total 80.06 last 73.35) - open/close 6652 (time 1788442478.28 total 90.07 last 70.96) - open/close 7428 (time 1788442488.31 total 100.09 last 77.42) - open/close 8125 (time 1788442498.32 total 110.10 last 69.61) - open/close 8875 (time 1788442508.33 total 120.11 last 74.96) - open/close 9499 (time 1788442518.33 total 130.12 last 62.36) total: 10000 open/close in 137.69 seconds: 72.63 ops/second - unlinked 0 (time 1788442530 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 558 (time 1788442624.27 total 10.01 last 55.77) - open/close 1227 (time 1788442634.28 total 20.02 last 66.80) - open/close 2003 (time 1788442644.29 total 30.03 last 77.53) - open/close 2746 (time 1788442654.30 total 40.04 last 74.24) - open/close 3509 (time 1788442664.31 total 50.04 last 76.24) - open/close 4224 (time 1788442674.32 total 60.06 last 71.42) - open/close 4934 (time 1788442684.34 total 70.07 last 70.87) - open/close 5645 (time 1788442694.34 total 80.08 last 71.08) - open/close 6362 (time 1788442704.34 total 90.08 last 71.68) - open/close 7052 (time 1788442714.34 total 100.08 last 68.99) - open/close 7792 (time 1788442724.35 total 110.08 last 73.97) - open/close 8528 (time 1788442734.36 total 120.10 last 73.47) - open/close 9212 (time 1788442744.38 total 130.11 last 68.32) - open/close 9977 (time 1788442754.39 total 140.13 last 76.39) total: 10000 open/close in 140.42 seconds: 71.21 ops/second - unlinked 0 (time 1788442758 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 721 (time 1788442852.57 total 10.00 last 72.08) - open/close 1439 (time 1788442862.57 total 20.00 last 71.79) - open/close 2107 (time 1788442872.58 total 30.01 last 66.74) - open/close 2745 (time 1788442882.59 total 40.02 last 63.76) - open/close 3454 (time 1788442892.59 total 50.03 last 70.87) - open/close 4213 (time 1788442902.60 total 60.03 last 75.86) - open/close 4927 (time 1788442912.60 total 70.04 last 71.34) - open/close 5511 (time 1788442922.61 total 80.04 last 58.39) - open/close 6076 (time 1788442932.62 total 90.05 last 56.43) - open/close 6750 (time 1788442942.62 total 100.06 last 67.39) - open/close 7371 (time 1788442952.63 total 110.07 last 62.02) - open/close 8068 (time 1788442962.64 total 120.08 last 69.64) - open/close 8735 (time 1788442972.64 total 130.08 last 66.69) - open/close 9370 (time 1788442982.65 total 140.09 last 63.43) total: 10000 open/close in 148.54 seconds: 67.32 ops/second - unlinked 0 (time 1788442995 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 750 (time 1788443097.69 total 10.00 last 74.97) - open/close 1565 (time 1788443107.70 total 20.02 last 81.39) - open/close 2354 (time 1788443117.70 total 30.02 last 78.89) - open/close 3090 (time 1788443127.71 total 40.02 last 73.56) - open/close 3865 (time 1788443137.71 total 50.03 last 77.47) - open/close 4595 (time 1788443147.71 total 60.03 last 72.98) - open/close 5259 (time 1788443157.72 total 70.04 last 66.36) - open/close 5903 (time 1788443167.73 total 80.04 last 64.35) - open/close 6522 (time 1788443177.73 total 90.04 last 61.90) - open/close 7109 (time 1788443187.73 total 100.05 last 58.65) - open/close 7767 (time 1788443197.75 total 110.07 last 65.68) - open/close 8516 (time 1788443207.76 total 120.08 last 74.81) - open/close 9228 (time 1788443217.79 total 130.10 last 71.05) - open/close 9992 (time 1788443227.79 total 140.11 last 76.35) total: 10000 open/close in 140.18 seconds: 71.34 ops/second - unlinked 0 (time 1788443231 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 721 (time 1788443324.70 total 10.01 last 72.06) - open/close 1481 (time 1788443334.72 total 20.02 last 75.90) - open/close 2201 (time 1788443344.73 total 30.03 last 71.93) - open/close 2916 (time 1788443354.73 total 40.03 last 71.47) - open/close 3660 (time 1788443364.73 total 50.03 last 74.40) - open/close 4431 (time 1788443374.74 total 60.04 last 77.05) - open/close 5240 (time 1788443384.74 total 70.04 last 80.88) - open/close 6017 (time 1788443394.75 total 80.05 last 77.67) - open/close 6748 (time 1788443404.75 total 90.05 last 73.04) - open/close 7462 (time 1788443414.75 total 100.05 last 71.40) - open/close 8185 (time 1788443424.77 total 110.07 last 72.21) - open/close 8957 (time 1788443434.78 total 120.08 last 77.08) - open/close 9718 (time 1788443444.79 total 130.09 last 76.07) total: 10000 open/close in 133.44 seconds: 74.94 ops/second - unlinked 0 (time 1788443451 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 525 (time 1788443542.66 total 10.00 last 52.48) - open/close 1180 (time 1788443552.66 total 20.01 last 65.46) - open/close 1825 (time 1788443562.67 total 30.02 last 64.43) - open/close 2522 (time 1788443572.67 total 40.02 last 69.69) - open/close 3098 (time 1788443582.70 total 50.05 last 57.45) - open/close 3677 (time 1788443592.71 total 60.06 last 57.83) - open/close 4377 (time 1788443602.72 total 70.07 last 69.95) - open/close 4964 (time 1788443612.72 total 80.07 last 58.67) - open/close 5611 (time 1788443622.75 total 90.10 last 64.53) - open/close 6353 (time 1788443632.75 total 100.10 last 74.19) - open/close 7078 (time 1788443642.76 total 110.11 last 72.44) - open/close 7768 (time 1788443652.78 total 120.12 last 68.90) - open/close 8454 (time 1788443662.78 total 130.13 last 68.54) - open/close 9068 (time 1788443672.80 total 140.14 last 61.33) - open/close 9626 (time 1788443682.80 total 150.15 last 55.79) total: 10000 open/close in 156.01 seconds: 64.10 ops/second - unlinked 0 (time 1788443693 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 759 (time 1788443797.28 total 10.00 last 75.88) - open/close 1474 (time 1788443807.29 total 20.01 last 71.46) - open/close 2157 (time 1788443817.30 total 30.02 last 68.25) - open/close 2856 (time 1788443827.31 total 40.03 last 69.80) - open/close 3483 (time 1788443837.34 total 50.05 last 62.55) - open/close 4073 (time 1788443847.34 total 60.06 last 58.98) - open/close 4776 (time 1788443857.36 total 70.08 last 70.17) - open/close 5418 (time 1788443867.36 total 80.08 last 64.17) - open/close 6152 (time 1788443877.38 total 90.10 last 73.23) - open/close 6865 (time 1788443887.39 total 100.11 last 71.24) - open/close 7633 (time 1788443897.39 total 110.11 last 76.80) - open/close 8395 (time 1788443907.40 total 120.12 last 76.18) - open/close 9177 (time 1788443917.40 total 130.12 last 78.17) - open/close 9954 (time 1788443927.40 total 140.12 last 77.67) total: 10000 open/close in 140.73 seconds: 71.06 ops/second - unlinked 0 (time 1788443932 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local oleg301-server: modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2