== conf-sanity test 69: replace an OST with the same index ========================================================== 00:13:53 (1788927233) start mds service on oleg401-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing load_modules_local oleg401-server: Loading modules from /home/green/git/lustre-release/lustre oleg401-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg401-server: detected 4 online CPUs by sysfs oleg401-server: MODOPTS_LIBCFS= oleg401-server: Force libcfs to create 2 CPU partitions oleg401-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg401-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg401-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg401-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg401-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 951 (time 1788927359.47 total 10.01 last 95.04) - open/close 1533 (time 1788927369.49 total 20.02 last 58.10) - open/close 2289 (time 1788927379.50 total 30.03 last 75.54) - open/close 3219 (time 1788927389.50 total 40.03 last 92.98) - open/close 4159 (time 1788927399.51 total 50.04 last 93.93) - open/close 5066 (time 1788927409.51 total 60.05 last 90.64) - open/close 5904 (time 1788927419.52 total 70.06 last 83.72) - open/close 6933 (time 1788927429.53 total 80.07 last 102.82) - open/close 7675 (time 1788927439.54 total 90.07 last 74.17) - open/close 8524 (time 1788927449.54 total 100.08 last 84.85) - open/close 9374 (time 1788927459.55 total 110.09 last 84.89) total: 10000 open/close in 117.93 seconds: 84.80 ops/second - unlinked 0 (time 1788927471 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 813 (time 1788927554.14 total 10.02 last 81.18) - open/close 1604 (time 1788927564.15 total 20.03 last 79.02) - open/close 2400 (time 1788927574.16 total 30.04 last 79.51) - open/close 3192 (time 1788927584.18 total 40.06 last 79.05) - open/close 3993 (time 1788927594.19 total 50.07 last 80.02) - open/close 4781 (time 1788927604.21 total 60.08 last 78.65) - open/close 5619 (time 1788927614.22 total 70.09 last 83.75) - open/close 6511 (time 1788927624.24 total 80.12 last 88.96) - open/close 7357 (time 1788927634.25 total 90.12 last 84.54) - open/close 8213 (time 1788927644.27 total 100.14 last 85.46) - open/close 9082 (time 1788927654.27 total 110.15 last 86.85) - open/close 9936 (time 1788927664.28 total 120.16 last 85.32) total: 10000 open/close in 121.31 seconds: 82.43 ops/second - unlinked 0 (time 1788927669 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 745 (time 1788927751.17 total 10.00 last 74.49) - open/close 1344 (time 1788927761.17 total 20.00 last 59.89) - open/close 2150 (time 1788927771.17 total 30.01 last 80.59) - open/close 2933 (time 1788927781.19 total 40.02 last 78.16) - open/close 3703 (time 1788927791.23 total 50.06 last 76.70) - open/close 4326 (time 1788927801.23 total 60.07 last 62.27) - open/close 5016 (time 1788927811.24 total 70.07 last 68.96) - open/close 5929 (time 1788927821.25 total 80.09 last 91.20) - open/close 6741 (time 1788927831.26 total 90.09 last 81.15) - open/close 7502 (time 1788927841.27 total 100.11 last 76.00) - open/close 8315 (time 1788927851.28 total 110.11 last 81.22) - open/close 9139 (time 1788927861.28 total 120.12 last 82.37) - open/close 9858 (time 1788927871.29 total 130.12 last 71.86) total: 10000 open/close in 131.79 seconds: 75.88 ops/second - unlinked 0 (time 1788927877 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 840 (time 1788927967.45 total 10.01 last 83.90) - open/close 1676 (time 1788927977.45 total 20.02 last 83.57) - open/close 2577 (time 1788927987.46 total 30.02 last 90.07) - open/close 3398 (time 1788927997.46 total 40.02 last 82.06) - open/close 4157 (time 1788928007.48 total 50.04 last 75.76) - open/close 4936 (time 1788928017.49 total 60.05 last 77.80) - open/close 5632 (time 1788928027.51 total 70.08 last 69.45) - open/close 6432 (time 1788928037.52 total 80.08 last 79.97) - open/close 7178 (time 1788928047.52 total 90.09 last 74.54) - open/close 7982 (time 1788928057.53 total 100.09 last 80.34) - open/close 8810 (time 1788928067.54 total 110.10 last 82.72) - open/close 9530 (time 1788928077.56 total 120.13 last 71.84) total: 10000 open/close in 126.92 seconds: 78.79 ops/second - unlinked 0 (time 1788928089 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 859 (time 1788928179.17 total 10.01 last 85.77) - open/close 1716 (time 1788928189.18 total 20.02 last 85.64) - open/close 2520 (time 1788928199.18 total 30.03 last 80.35) - open/close 3180 (time 1788928209.19 total 40.04 last 65.94) - open/close 3893 (time 1788928219.19 total 50.04 last 71.30) - open/close 4552 (time 1788928229.21 total 60.05 last 65.81) - open/close 5368 (time 1788928239.22 total 70.06 last 81.50) - open/close 6023 (time 1788928249.23 total 80.07 last 65.46) - open/close 6774 (time 1788928259.25 total 90.10 last 74.90) - open/close 7337 (time 1788928269.26 total 100.10 last 56.28) - open/close 8030 (time 1788928279.26 total 110.11 last 69.25) - open/close 8727 (time 1788928289.29 total 120.13 last 69.54) - open/close 9403 (time 1788928299.29 total 130.14 last 67.54) total: 10000 open/close in 138.42 seconds: 72.25 ops/second - unlinked 0 (time 1788928312 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 904 (time 1788928406.39 total 10.01 last 90.32) - open/close 1648 (time 1788928416.41 total 20.02 last 74.30) - open/close 2380 (time 1788928426.41 total 30.02 last 73.19) - open/close 2953 (time 1788928436.44 total 40.05 last 57.14) - open/close 3608 (time 1788928446.44 total 50.06 last 65.46) - open/close 4420 (time 1788928456.44 total 60.06 last 81.18) - open/close 5177 (time 1788928466.46 total 70.07 last 75.61) - open/close 5917 (time 1788928476.47 total 80.09 last 73.87) - open/close 6676 (time 1788928486.48 total 90.09 last 75.86) - open/close 7512 (time 1788928496.49 total 100.10 last 83.54) - open/close 8356 (time 1788928506.49 total 110.11 last 84.37) - open/close 9093 (time 1788928516.50 total 120.11 last 73.64) - open/close 9852 (time 1788928526.50 total 130.11 last 75.89) total: 10000 open/close in 131.61 seconds: 75.98 ops/second - unlinked 0 (time 1788928533 ; total 1 ; last 1) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 720 (time 1788928616.08 total 10.01 last 71.95) - open/close 1521 (time 1788928626.09 total 20.01 last 80.04) - open/close 2352 (time 1788928636.10 total 30.03 last 82.99) - open/close 3092 (time 1788928646.10 total 40.03 last 73.99) - open/close 3896 (time 1788928656.11 total 50.04 last 80.32) - open/close 4790 (time 1788928666.12 total 60.04 last 89.38) - open/close 5635 (time 1788928676.12 total 70.05 last 84.43) - open/close 6539 (time 1788928686.13 total 80.06 last 90.34) - open/close 7335 (time 1788928696.15 total 90.07 last 79.46) - open/close 8306 (time 1788928706.15 total 100.08 last 97.05) - open/close 9029 (time 1788928716.16 total 110.08 last 72.28) - open/close 9834 (time 1788928726.16 total 120.09 last 80.46) total: 10000 open/close in 122.66 seconds: 81.53 ops/second - unlinked 0 (time 1788928735 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 643 (time 1788928823.90 total 10.03 last 64.12) - open/close 1350 (time 1788928833.91 total 20.04 last 70.62) - open/close 2154 (time 1788928843.91 total 30.04 last 80.40) - open/close 3038 (time 1788928853.91 total 40.04 last 88.39) - open/close 3957 (time 1788928863.92 total 50.04 last 91.87) - open/close 4816 (time 1788928873.92 total 60.05 last 85.87) - open/close 5708 (time 1788928883.92 total 70.05 last 89.16) - open/close 6555 (time 1788928893.93 total 80.06 last 84.64) - open/close 7368 (time 1788928903.95 total 90.07 last 81.17) - open/close 8202 (time 1788928913.95 total 100.08 last 83.38) - open/close 8893 (time 1788928923.95 total 110.08 last 69.07) - open/close 9691 (time 1788928933.96 total 120.09 last 79.76) total: 10000 open/close in 124.42 seconds: 80.37 ops/second - unlinked 0 (time 1788928943 ; total 1 ; last 1) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 419 (time 1788929022.79 total 10.02 last 41.83) - open/close 1218 (time 1788929032.79 total 20.02 last 79.88) - open/close 2049 (time 1788929042.79 total 30.02 last 83.09) - open/close 2917 (time 1788929052.80 total 40.02 last 86.77) - open/close 3645 (time 1788929062.82 total 50.05 last 72.60) - open/close 4498 (time 1788929072.83 total 60.06 last 85.27) - open/close 5234 (time 1788929082.83 total 70.06 last 73.55) - open/close 5736 (time 1788929092.85 total 80.07 last 50.14) - open/close 6187 (time 1788929102.88 total 90.11 last 44.95) - open/close 6774 (time 1788929112.89 total 100.12 last 58.63) - open/close 7389 (time 1788929122.90 total 110.13 last 61.45) - open/close 8031 (time 1788929132.91 total 120.14 last 64.12) - open/close 8897 (time 1788929142.92 total 130.15 last 86.55) - open/close 9710 (time 1788929152.94 total 140.17 last 81.12) total: 10000 open/close in 144.62 seconds: 69.14 ops/second - unlinked 0 (time 1788929162 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 762 (time 1788929257.19 total 10.02 last 76.07) - open/close 1549 (time 1788929267.20 total 20.02 last 78.65) - open/close 2421 (time 1788929277.21 total 30.03 last 87.13) - open/close 3271 (time 1788929287.21 total 40.04 last 84.93) - open/close 4061 (time 1788929297.24 total 50.07 last 78.80) - open/close 4946 (time 1788929307.24 total 60.07 last 88.47) - open/close 5769 (time 1788929317.25 total 70.07 last 82.28) - open/close 6617 (time 1788929327.25 total 80.08 last 84.76) - open/close 7588 (time 1788929337.25 total 90.08 last 97.06) - open/close 8473 (time 1788929347.26 total 100.08 last 88.48) - open/close 9429 (time 1788929357.26 total 110.08 last 95.57) total: 10000 open/close in 115.99 seconds: 86.21 ops/second - unlinked 0 (time 1788929367 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg401-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.101@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg401-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg401-server: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg401-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg401-server: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg401-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg401-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.101@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.101@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.101@tcp sys.timeout=20 start ost1 service on oleg401-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg401-server: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg401-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg401-server: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg401-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg401-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg401-server unloading modules via unload_modules_local on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing unload_modules_local oleg401-server: modules unloaded. pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2