== conf-sanity test 69: replace an OST with the same index ========================================================== 15:28:12 (1787599692) start mds service on oleg448-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg448-server: detected 4 online CPUs by sysfs oleg448-server: MODOPTS_LIBCFS= oleg448-server: Force libcfs to create 2 CPU partitions oleg448-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg448-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg448-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg448-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 721 (time 1787599811.52 total 10.03 last 71.90) - open/close 1517 (time 1787599821.52 total 20.03 last 79.56) - open/close 2286 (time 1787599831.53 total 30.04 last 76.85) - open/close 3071 (time 1787599841.53 total 40.04 last 78.48) - open/close 3789 (time 1787599851.54 total 50.05 last 71.75) - open/close 4549 (time 1787599861.55 total 60.06 last 75.91) - open/close 5373 (time 1787599871.57 total 70.08 last 82.25) - open/close 6113 (time 1787599881.58 total 80.09 last 73.93) - open/close 6760 (time 1787599891.60 total 90.11 last 64.59) - open/close 7428 (time 1787599901.60 total 100.11 last 66.76) - open/close 8181 (time 1787599911.61 total 110.12 last 75.27) - open/close 8927 (time 1787599921.62 total 120.13 last 74.52) - open/close 9595 (time 1787599931.63 total 130.14 last 66.72) total: 10000 open/close in 136.29 seconds: 73.37 ops/second - unlinked 0 (time 1787599943 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 830 (time 1787600036.07 total 10.00 last 82.97) - open/close 1514 (time 1787600046.08 total 20.02 last 68.31) - open/close 2244 (time 1787600056.10 total 30.03 last 72.89) - open/close 3115 (time 1787600066.10 total 40.04 last 87.04) - open/close 3856 (time 1787600076.11 total 50.04 last 74.07) - open/close 4606 (time 1787600086.12 total 60.06 last 74.89) - open/close 5344 (time 1787600096.13 total 70.07 last 73.74) - open/close 6173 (time 1787600106.14 total 80.07 last 82.85) - open/close 6948 (time 1787600116.15 total 90.08 last 77.44) - open/close 7757 (time 1787600126.15 total 100.09 last 80.83) - open/close 8604 (time 1787600136.16 total 110.10 last 84.65) - open/close 9456 (time 1787600146.17 total 120.10 last 85.16) total: 10000 open/close in 127.44 seconds: 78.47 ops/second - unlinked 0 (time 1787600159 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 814 (time 1787600243.11 total 10.00 last 81.39) - open/close 1727 (time 1787600253.12 total 20.00 last 91.28) - open/close 2599 (time 1787600263.12 total 30.01 last 87.13) - open/close 3371 (time 1787600273.13 total 40.01 last 77.18) - open/close 4219 (time 1787600283.14 total 50.02 last 84.72) - open/close 5070 (time 1787600293.14 total 60.03 last 85.08) - open/close 5892 (time 1787600303.15 total 70.03 last 82.15) - open/close 6722 (time 1787600313.15 total 80.04 last 82.95) - open/close 7511 (time 1787600323.16 total 90.05 last 78.81) - open/close 8303 (time 1787600333.17 total 100.06 last 79.14) - open/close 9169 (time 1787600343.17 total 110.06 last 86.59) - open/close 9943 (time 1787600353.18 total 120.07 last 77.33) total: 10000 open/close in 120.94 seconds: 82.69 ops/second - unlinked 0 (time 1787600358 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 845 (time 1787600440.41 total 10.01 last 84.45) - open/close 1759 (time 1787600450.41 total 20.01 last 91.36) - open/close 2641 (time 1787600460.45 total 30.04 last 87.91) - open/close 3504 (time 1787600470.46 total 40.05 last 86.22) - open/close 4325 (time 1787600480.46 total 50.06 last 82.06) - open/close 5133 (time 1787600490.47 total 60.06 last 80.76) - open/close 5784 (time 1787600500.48 total 70.08 last 65.01) - open/close 6528 (time 1787600510.49 total 80.08 last 74.34) - open/close 7347 (time 1787600520.51 total 90.10 last 81.74) - open/close 8224 (time 1787600530.51 total 100.11 last 87.64) - open/close 9083 (time 1787600540.53 total 110.13 last 85.77) - open/close 9915 (time 1787600550.53 total 120.13 last 83.17) total: 10000 open/close in 120.98 seconds: 82.66 ops/second - unlinked 0 (time 1787600555 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 728 (time 1787600639.27 total 10.01 last 72.71) - open/close 1383 (time 1787600649.28 total 20.02 last 65.46) - open/close 1951 (time 1787600659.28 total 30.02 last 56.77) - open/close 2555 (time 1787600669.31 total 40.05 last 60.26) - open/close 3087 (time 1787600679.32 total 50.06 last 53.12) - open/close 3665 (time 1787600689.33 total 60.07 last 57.78) - open/close 4149 (time 1787600699.33 total 70.07 last 48.38) - open/close 4790 (time 1787600709.33 total 80.07 last 64.09) - open/close 5453 (time 1787600719.34 total 90.08 last 66.24) - open/close 6190 (time 1787600729.35 total 100.09 last 73.64) - open/close 6895 (time 1787600739.37 total 110.11 last 70.34) - open/close 7730 (time 1787600749.38 total 120.12 last 83.46) - open/close 8553 (time 1787600759.39 total 130.13 last 82.22) - open/close 9397 (time 1787600769.39 total 140.13 last 84.38) total: 10000 open/close in 147.41 seconds: 67.84 ops/second - unlinked 0 (time 1787600780 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 860 (time 1787600866.18 total 10.01 last 85.95) - open/close 1598 (time 1787600876.18 total 20.01 last 73.77) - open/close 2382 (time 1787600886.18 total 30.01 last 78.39) - open/close 3235 (time 1787600896.18 total 40.01 last 85.30) - open/close 4086 (time 1787600906.19 total 50.02 last 85.02) - open/close 4927 (time 1787600916.20 total 60.03 last 84.05) - open/close 5783 (time 1787600926.20 total 70.03 last 85.56) - open/close 6573 (time 1787600936.20 total 80.03 last 78.98) - open/close 7382 (time 1787600946.20 total 90.03 last 80.90) - open/close 8058 (time 1787600956.20 total 100.04 last 67.59) - open/close 8716 (time 1787600966.21 total 110.04 last 65.79) - open/close 9481 (time 1787600976.21 total 120.04 last 76.49) total: 10000 open/close in 126.29 seconds: 79.18 ops/second - unlinked 0 (time 1787600987 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 766 (time 1787601071.49 total 10.00 last 76.57) - open/close 1459 (time 1787601081.50 total 20.01 last 69.24) - open/close 2138 (time 1787601091.51 total 30.02 last 67.87) - open/close 2772 (time 1787601101.52 total 40.03 last 63.34) - open/close 3516 (time 1787601111.53 total 50.04 last 74.33) - open/close 4377 (time 1787601121.53 total 60.04 last 86.05) - open/close 5150 (time 1787601131.55 total 70.06 last 77.20) - open/close 5968 (time 1787601141.55 total 80.06 last 81.74) - open/close 6662 (time 1787601151.57 total 90.08 last 69.30) - open/close 7412 (time 1787601161.58 total 100.09 last 74.93) - open/close 8181 (time 1787601171.58 total 110.09 last 76.88) - open/close 8701 (time 1787601181.59 total 120.10 last 51.95) - open/close 9329 (time 1787601191.60 total 130.11 last 62.76) total: 10000 open/close in 137.84 seconds: 72.55 ops/second - unlinked 0 (time 1787601205 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 720 (time 1787601297.68 total 10.01 last 71.95) - open/close 1436 (time 1787601307.68 total 20.01 last 71.57) - open/close 2186 (time 1787601317.69 total 30.02 last 74.94) - open/close 2907 (time 1787601327.70 total 40.03 last 72.05) - open/close 3693 (time 1787601337.70 total 50.03 last 78.54) - open/close 4487 (time 1787601347.71 total 60.04 last 79.34) - open/close 5171 (time 1787601357.72 total 70.04 last 68.37) - open/close 5909 (time 1787601367.72 total 80.05 last 73.75) - open/close 6610 (time 1787601377.74 total 90.07 last 69.98) - open/close 7299 (time 1787601387.75 total 100.08 last 68.85) - open/close 7936 (time 1787601397.76 total 110.09 last 63.62) - open/close 8592 (time 1787601407.79 total 120.12 last 65.43) - open/close 9228 (time 1787601417.79 total 130.12 last 63.59) total: 10000 open/close in 138.95 seconds: 71.97 ops/second - unlinked 0 (time 1787601430 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 738 (time 1787601514.45 total 10.00 last 73.79) - open/close 1498 (time 1787601524.45 total 20.01 last 75.96) - open/close 2162 (time 1787601534.46 total 30.01 last 66.40) - open/close 2956 (time 1787601544.47 total 40.02 last 79.26) - open/close 3793 (time 1787601554.48 total 50.03 last 83.65) - open/close 4553 (time 1787601564.48 total 60.03 last 75.97) - open/close 5363 (time 1787601574.48 total 70.04 last 80.99) - open/close 6181 (time 1787601584.50 total 80.05 last 81.70) - open/close 7047 (time 1787601594.51 total 90.06 last 86.51) - open/close 7921 (time 1787601604.51 total 100.06 last 87.40) - open/close 8695 (time 1787601614.51 total 110.06 last 77.40) - open/close 9446 (time 1787601624.52 total 120.07 last 75.04) total: 10000 open/close in 125.84 seconds: 79.46 ops/second - unlinked 0 (time 1787601634 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 728 (time 1787601718.25 total 10.00 last 72.77) - open/close 1492 (time 1787601728.26 total 20.02 last 76.30) - open/close 2349 (time 1787601738.27 total 30.02 last 85.67) - open/close 3198 (time 1787601748.27 total 40.03 last 84.85) - open/close 4035 (time 1787601758.28 total 50.03 last 83.63) - open/close 4811 (time 1787601768.29 total 60.04 last 77.55) - open/close 5464 (time 1787601778.29 total 70.04 last 65.30) - open/close 6164 (time 1787601788.29 total 80.05 last 69.97) - open/close 7006 (time 1787601798.30 total 90.06 last 84.12) - open/close 7923 (time 1787601808.31 total 100.06 last 91.68) - open/close 8747 (time 1787601818.31 total 110.07 last 82.33) - open/close 9503 (time 1787601828.32 total 120.07 last 75.58) total: 10000 open/close in 126.26 seconds: 79.20 ops/second - unlinked 0 (time 1787601838 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.148@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg448-server: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg448-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg448-server: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg448-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.148@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.148@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.148@tcp sys.timeout=20 start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg448-server: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg448-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg448-server: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg448-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg448-server unloading modules via unload_modules_local on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local oleg448-server: modules unloaded. pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2