== conf-sanity test 69: replace an OST with the same index ========================================================== 15:11:20 (1787339480) start mds service on oleg256-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg256-server: detected 4 online CPUs by sysfs oleg256-server: MODOPTS_LIBCFS= oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg256-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg256-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg256-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 733 (time 1787339594.91 total 10.01 last 73.23) - open/close 1438 (time 1787339604.92 total 20.02 last 70.43) - open/close 2241 (time 1787339614.93 total 30.02 last 80.29) - open/close 3000 (time 1787339624.93 total 40.03 last 75.85) - open/close 3754 (time 1787339634.93 total 50.03 last 75.39) - open/close 4515 (time 1787339644.93 total 60.03 last 76.09) - open/close 5190 (time 1787339654.94 total 70.04 last 67.45) - open/close 5874 (time 1787339664.95 total 80.04 last 68.37) - open/close 6534 (time 1787339674.95 total 90.05 last 65.95) - open/close 7325 (time 1787339684.96 total 100.06 last 79.01) - open/close 8075 (time 1787339694.97 total 110.06 last 74.98) - open/close 8751 (time 1787339704.97 total 120.07 last 67.56) - open/close 9519 (time 1787339714.97 total 130.07 last 76.80) total: 10000 open/close in 135.96 seconds: 73.55 ops/second - unlinked 0 (time 1787339725 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 661 (time 1787339818.90 total 10.00 last 66.07) - open/close 1386 (time 1787339828.91 total 20.02 last 72.39) - open/close 2001 (time 1787339838.93 total 30.04 last 61.38) - open/close 2710 (time 1787339848.94 total 40.04 last 70.88) - open/close 3480 (time 1787339858.94 total 50.05 last 76.96) - open/close 4265 (time 1787339868.97 total 60.08 last 78.27) - open/close 5093 (time 1787339878.98 total 70.08 last 82.76) - open/close 5767 (time 1787339889.01 total 80.12 last 67.18) - open/close 6238 (time 1787339899.02 total 90.13 last 47.04) - open/close 6947 (time 1787339909.03 total 100.13 last 70.86) - open/close 7623 (time 1787339919.03 total 110.14 last 67.57) - open/close 8387 (time 1787339929.05 total 120.16 last 76.23) - open/close 9164 (time 1787339939.07 total 130.18 last 77.56) - open/close 9902 (time 1787339949.08 total 140.18 last 73.75) total: 10000 open/close in 141.98 seconds: 70.43 ops/second - unlinked 0 (time 1787339956 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 893 (time 1787340037.79 total 10.01 last 89.25) - open/close 1705 (time 1787340047.80 total 20.01 last 81.12) - open/close 2212 (time 1787340057.80 total 30.02 last 50.68) - open/close 2703 (time 1787340067.81 total 40.02 last 49.07) - open/close 3097 (time 1787340077.82 total 50.04 last 39.35) - open/close 3530 (time 1787340087.82 total 60.04 last 43.28) - open/close 4091 (time 1787340097.83 total 70.05 last 56.05) - open/close 4792 (time 1787340107.84 total 80.06 last 70.01) - open/close 5514 (time 1787340117.85 total 90.07 last 72.17) - open/close 6326 (time 1787340127.85 total 100.07 last 81.16) - open/close 7142 (time 1787340137.87 total 110.09 last 81.49) - open/close 7894 (time 1787340147.87 total 120.09 last 75.18) - open/close 8638 (time 1787340157.88 total 130.10 last 74.34) - open/close 9470 (time 1787340167.88 total 140.10 last 83.16) total: 10000 open/close in 145.64 seconds: 68.66 ops/second - unlinked 0 (time 1787340177 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 794 (time 1787340268.45 total 10.00 last 79.36) - open/close 1536 (time 1787340278.46 total 20.01 last 74.17) - open/close 2319 (time 1787340288.47 total 30.02 last 78.19) - open/close 3219 (time 1787340298.47 total 40.03 last 89.97) - open/close 3999 (time 1787340308.48 total 50.04 last 77.93) - open/close 4878 (time 1787340318.49 total 60.04 last 87.83) - open/close 5757 (time 1787340328.49 total 70.05 last 87.86) - open/close 6545 (time 1787340338.51 total 80.07 last 78.64) - open/close 7327 (time 1787340348.52 total 90.08 last 78.14) - open/close 8179 (time 1787340358.54 total 100.09 last 85.04) - open/close 8893 (time 1787340368.55 total 110.10 last 71.36) - open/close 9621 (time 1787340378.55 total 120.11 last 72.74) total: 10000 open/close in 125.06 seconds: 79.96 ops/second - unlinked 0 (time 1787340388 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 874 (time 1787340477.59 total 10.01 last 87.33) - open/close 1835 (time 1787340487.59 total 20.01 last 96.04) - open/close 2667 (time 1787340497.60 total 30.02 last 83.14) - open/close 3351 (time 1787340507.64 total 40.06 last 68.13) - open/close 4112 (time 1787340517.65 total 50.07 last 76.05) - open/close 4867 (time 1787340527.66 total 60.08 last 75.40) - open/close 5535 (time 1787340537.67 total 70.09 last 66.73) - open/close 6182 (time 1787340547.69 total 80.11 last 64.60) - open/close 6786 (time 1787340557.69 total 90.11 last 60.35) - open/close 7553 (time 1787340567.70 total 100.13 last 76.61) - open/close 8215 (time 1787340577.73 total 110.15 last 66.04) - open/close 8952 (time 1787340587.74 total 120.16 last 73.62) - open/close 9659 (time 1787340597.75 total 130.17 last 70.64) total: 10000 open/close in 137.12 seconds: 72.93 ops/second - unlinked 0 (time 1787340609 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 812 (time 1787340700.44 total 10.00 last 81.19) - open/close 1639 (time 1787340710.45 total 20.01 last 82.64) - open/close 2226 (time 1787340720.46 total 30.02 last 58.64) - open/close 2969 (time 1787340730.47 total 40.03 last 74.25) - open/close 3637 (time 1787340740.48 total 50.04 last 66.70) - open/close 4448 (time 1787340750.50 total 60.06 last 80.98) - open/close 5169 (time 1787340760.50 total 70.06 last 72.06) - open/close 5958 (time 1787340770.51 total 80.07 last 78.85) - open/close 6790 (time 1787340780.51 total 90.07 last 83.14) - open/close 7467 (time 1787340790.52 total 100.08 last 67.64) - open/close 8169 (time 1787340800.53 total 110.09 last 70.14) - open/close 8993 (time 1787340810.53 total 120.09 last 82.39) - open/close 9865 (time 1787340820.55 total 130.11 last 87.04) total: 10000 open/close in 131.53 seconds: 76.03 ops/second - unlinked 0 (time 1787340825 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 499 (time 1787340918.08 total 10.01 last 49.86) - open/close 1388 (time 1787340928.10 total 20.03 last 88.68) - open/close 2151 (time 1787340938.11 total 30.04 last 76.24) - open/close 2855 (time 1787340948.11 total 40.04 last 70.37) - open/close 3637 (time 1787340958.11 total 50.05 last 78.19) - open/close 4528 (time 1787340968.12 total 60.05 last 89.02) - open/close 5322 (time 1787340978.13 total 70.06 last 79.35) - open/close 6189 (time 1787340988.14 total 80.07 last 86.60) - open/close 6952 (time 1787340998.15 total 90.08 last 76.23) - open/close 7706 (time 1787341008.16 total 100.09 last 75.35) - open/close 8586 (time 1787341018.17 total 110.10 last 87.91) - open/close 9478 (time 1787341028.17 total 120.11 last 89.13) total: 10000 open/close in 125.42 seconds: 79.73 ops/second - unlinked 0 (time 1787341037 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 731 (time 1787341127.55 total 10.00 last 73.08) - open/close 1553 (time 1787341137.56 total 20.01 last 82.13) - open/close 2202 (time 1787341147.56 total 30.01 last 64.89) - open/close 2916 (time 1787341157.57 total 40.03 last 71.30) - open/close 3708 (time 1787341167.58 total 50.03 last 79.19) - open/close 4534 (time 1787341177.59 total 60.04 last 82.52) - open/close 5449 (time 1787341187.59 total 70.05 last 91.43) - open/close 6346 (time 1787341197.60 total 80.05 last 89.63) - open/close 7096 (time 1787341207.61 total 90.06 last 74.92) - open/close 7730 (time 1787341217.62 total 100.07 last 63.37) - open/close 8420 (time 1787341227.62 total 110.07 last 68.96) - open/close 9213 (time 1787341237.63 total 120.08 last 79.25) - open/close 9949 (time 1787341247.64 total 130.09 last 73.53) total: 10000 open/close in 130.68 seconds: 76.52 ops/second - unlinked 0 (time 1787341252 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 687 (time 1787341341.07 total 10.01 last 68.63) - open/close 1522 (time 1787341351.09 total 20.03 last 83.34) - open/close 2305 (time 1787341361.09 total 30.03 last 78.29) - open/close 3001 (time 1787341371.10 total 40.04 last 69.54) - open/close 3732 (time 1787341381.11 total 50.05 last 73.04) - open/close 4455 (time 1787341391.12 total 60.06 last 72.20) - open/close 5136 (time 1787341401.13 total 70.07 last 68.06) - open/close 6071 (time 1787341411.14 total 80.07 last 93.45) - open/close 6932 (time 1787341421.14 total 90.07 last 86.10) - open/close 7914 (time 1787341431.14 total 100.08 last 98.16) - open/close 8732 (time 1787341441.15 total 110.08 last 81.75) - open/close 9617 (time 1787341451.15 total 120.09 last 88.45) total: 10000 open/close in 124.45 seconds: 80.36 ops/second - unlinked 0 (time 1787341460 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 945 (time 1787341544.30 total 10.00 last 94.49) - open/close 1794 (time 1787341554.31 total 20.01 last 84.80) - open/close 2702 (time 1787341564.33 total 30.03 last 90.66) - open/close 3711 (time 1787341574.33 total 40.03 last 100.83) - open/close 4615 (time 1787341584.34 total 50.04 last 90.35) - open/close 5510 (time 1787341594.35 total 60.05 last 89.45) - open/close 6164 (time 1787341604.36 total 70.06 last 65.34) - open/close 6849 (time 1787341614.36 total 80.06 last 68.48) - open/close 7629 (time 1787341624.37 total 90.07 last 77.94) - open/close 8542 (time 1787341634.38 total 100.08 last 91.16) - open/close 9558 (time 1787341644.38 total 110.08 last 101.58) total: 10000 open/close in 114.26 seconds: 87.52 ops/second - unlinked 0 (time 1787341652 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg256-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 start ost1 service on oleg256-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server unloading modules via unload_modules_local on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local oleg256-server: modules unloaded. pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2