== conf-sanity test 69: replace an OST with the same index ========================================================== 10:16:38 (1788272198) start mds service on oleg160-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg160-server: detected 4 online CPUs by sysfs oleg160-server: MODOPTS_LIBCFS= oleg160-server: Force libcfs to create 2 CPU partitions oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg160-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg160-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 996 (time 1788272292.15 total 10.00 last 99.60) - open/close 1650 (time 1788272302.15 total 20.00 last 65.39) - open/close 2060 (time 1788272312.16 total 30.01 last 40.95) - open/close 2720 (time 1788272322.17 total 40.02 last 65.97) - open/close 3163 (time 1788272332.18 total 50.03 last 44.25) - open/close 3546 (time 1788272342.21 total 60.06 last 38.18) - open/close 4122 (time 1788272352.22 total 70.07 last 57.56) - open/close 4615 (time 1788272362.26 total 80.10 last 49.12) - open/close 5137 (time 1788272372.27 total 90.12 last 52.11) - open/close 5566 (time 1788272382.28 total 100.13 last 42.86) - open/close 6158 (time 1788272392.30 total 110.15 last 59.08) - open/close 6856 (time 1788272402.31 total 120.15 last 69.78) - open/close 7666 (time 1788272412.31 total 130.15 last 80.99) - open/close 8517 (time 1788272422.32 total 140.16 last 85.02) - open/close 9440 (time 1788272432.32 total 150.16 last 92.30) total: 10000 open/close in 157.59 seconds: 63.46 ops/second - unlinked 0 (time 1788272444 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 805 (time 1788272535.96 total 10.01 last 80.43) - open/close 1495 (time 1788272545.96 total 20.01 last 68.96) - open/close 2373 (time 1788272555.97 total 30.02 last 87.72) - open/close 3106 (time 1788272565.99 total 40.04 last 73.17) - open/close 3855 (time 1788272576.00 total 50.05 last 74.85) - open/close 4656 (time 1788272586.01 total 60.06 last 79.98) - open/close 5405 (time 1788272596.02 total 70.07 last 74.82) - open/close 6034 (time 1788272606.02 total 80.07 last 62.90) - open/close 6785 (time 1788272616.03 total 90.08 last 75.07) - open/close 7552 (time 1788272626.03 total 100.09 last 76.64) - open/close 8382 (time 1788272636.05 total 110.10 last 82.89) - open/close 8977 (time 1788272646.06 total 120.11 last 59.44) - open/close 9596 (time 1788272656.06 total 130.12 last 61.86) total: 10000 open/close in 134.74 seconds: 74.22 ops/second - unlinked 0 (time 1788272665 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 791 (time 1788272758.41 total 10.00 last 79.08) - open/close 1522 (time 1788272768.43 total 20.02 last 72.97) - open/close 2273 (time 1788272778.44 total 30.03 last 75.03) - open/close 3087 (time 1788272788.45 total 40.04 last 81.29) - open/close 3805 (time 1788272798.46 total 50.05 last 71.74) - open/close 4614 (time 1788272808.47 total 60.06 last 80.83) - open/close 5455 (time 1788272818.49 total 70.08 last 83.96) - open/close 6326 (time 1788272828.51 total 80.10 last 86.93) - open/close 6968 (time 1788272838.52 total 90.10 last 64.15) - open/close 7613 (time 1788272848.52 total 100.11 last 64.46) - open/close 8308 (time 1788272858.52 total 110.11 last 69.50) - open/close 9020 (time 1788272868.52 total 120.11 last 71.20) - open/close 9808 (time 1788272878.53 total 130.12 last 78.73) total: 10000 open/close in 132.97 seconds: 75.20 ops/second - unlinked 0 (time 1788272886 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 853 (time 1788272980.16 total 10.00 last 85.29) - open/close 1701 (time 1788272990.17 total 20.01 last 84.73) - open/close 2463 (time 1788273000.17 total 30.01 last 76.18) - open/close 3281 (time 1788273010.18 total 40.02 last 81.75) - open/close 3981 (time 1788273020.18 total 50.02 last 69.99) - open/close 4931 (time 1788273030.18 total 60.02 last 94.97) - open/close 5803 (time 1788273040.18 total 70.02 last 87.19) - open/close 6441 (time 1788273050.19 total 80.03 last 63.78) - open/close 7180 (time 1788273060.19 total 90.03 last 73.86) - open/close 7856 (time 1788273070.20 total 100.04 last 67.53) - open/close 8562 (time 1788273080.21 total 110.05 last 70.53) - open/close 9362 (time 1788273090.21 total 120.05 last 80.00) total: 10000 open/close in 127.46 seconds: 78.45 ops/second - unlinked 0 (time 1788273102 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 761 (time 1788273190.29 total 10.01 last 76.05) - open/close 1488 (time 1788273200.29 total 20.01 last 72.66) - open/close 2187 (time 1788273210.30 total 30.02 last 69.87) - open/close 2698 (time 1788273220.30 total 40.02 last 51.08) - open/close 3284 (time 1788273230.31 total 50.03 last 58.52) - open/close 3964 (time 1788273240.33 total 60.05 last 67.92) - open/close 4856 (time 1788273250.33 total 70.05 last 89.17) - open/close 5758 (time 1788273260.33 total 80.05 last 90.17) - open/close 6589 (time 1788273270.34 total 90.06 last 83.06) - open/close 7416 (time 1788273280.34 total 100.06 last 82.69) - open/close 8121 (time 1788273290.34 total 110.06 last 70.46) - open/close 8852 (time 1788273300.36 total 120.08 last 72.98) - open/close 9590 (time 1788273310.37 total 130.09 last 73.74) total: 10000 open/close in 135.58 seconds: 73.76 ops/second - unlinked 0 (time 1788273321 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 733 (time 1788273411.12 total 10.01 last 73.26) - open/close 1485 (time 1788273421.12 total 20.01 last 75.19) - open/close 2201 (time 1788273431.12 total 30.01 last 71.59) - open/close 2902 (time 1788273441.13 total 40.02 last 70.04) - open/close 3600 (time 1788273451.15 total 50.04 last 69.67) - open/close 4373 (time 1788273461.15 total 60.04 last 77.30) - open/close 5236 (time 1788273471.16 total 70.04 last 86.24) - open/close 5958 (time 1788273481.16 total 80.05 last 72.16) - open/close 6749 (time 1788273491.17 total 90.05 last 79.06) - open/close 7466 (time 1788273501.18 total 100.07 last 71.58) - open/close 8170 (time 1788273511.19 total 110.08 last 70.35) - open/close 8934 (time 1788273521.21 total 120.10 last 76.25) - open/close 9716 (time 1788273531.21 total 130.10 last 78.20) total: 10000 open/close in 134.40 seconds: 74.40 ops/second - unlinked 0 (time 1788273541 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 791 (time 1788273629.09 total 10.00 last 79.07) - open/close 1494 (time 1788273639.10 total 20.01 last 70.25) - open/close 2306 (time 1788273649.11 total 30.02 last 81.09) - open/close 3145 (time 1788273659.12 total 40.03 last 83.88) - open/close 4016 (time 1788273669.12 total 50.03 last 87.03) - open/close 4786 (time 1788273679.12 total 60.03 last 77.00) - open/close 5653 (time 1788273689.14 total 70.04 last 86.61) - open/close 6537 (time 1788273699.14 total 80.05 last 88.38) - open/close 7329 (time 1788273709.14 total 90.05 last 79.19) - open/close 8098 (time 1788273719.15 total 100.06 last 76.81) - open/close 8891 (time 1788273729.15 total 110.06 last 79.26) - open/close 9767 (time 1788273739.16 total 120.07 last 87.58) total: 10000 open/close in 122.64 seconds: 81.54 ops/second - unlinked 0 (time 1788273746 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 927 (time 1788273828.84 total 10.01 last 92.64) - open/close 1795 (time 1788273838.84 total 20.01 last 86.79) - open/close 2682 (time 1788273848.84 total 30.01 last 88.69) - open/close 3488 (time 1788273858.85 total 40.02 last 80.53) - open/close 4260 (time 1788273868.85 total 50.02 last 77.18) - open/close 5100 (time 1788273878.86 total 60.02 last 84.00) - open/close 5933 (time 1788273888.86 total 70.02 last 83.28) - open/close 6803 (time 1788273898.87 total 80.03 last 86.91) - open/close 7655 (time 1788273908.88 total 90.05 last 85.11) - open/close 8521 (time 1788273918.88 total 100.05 last 86.57) - open/close 9390 (time 1788273928.88 total 110.05 last 86.88) total: 10000 open/close in 117.34 seconds: 85.22 ops/second - unlinked 0 (time 1788273940 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 880 (time 1788274017.99 total 10.00 last 87.97) - open/close 1771 (time 1788274028.00 total 20.02 last 88.97) - open/close 2727 (time 1788274038.01 total 30.02 last 95.56) - open/close 3622 (time 1788274048.02 total 40.03 last 89.43) - open/close 4460 (time 1788274058.02 total 50.04 last 83.75) - open/close 5356 (time 1788274068.02 total 60.04 last 89.59) - open/close 6285 (time 1788274078.04 total 70.05 last 92.78) - open/close 7186 (time 1788274088.04 total 80.05 last 90.08) - open/close 8025 (time 1788274098.05 total 90.06 last 83.83) - open/close 8882 (time 1788274108.05 total 100.06 last 85.66) - open/close 9429 (time 1788274118.09 total 110.10 last 54.49) total: 10000 open/close in 119.97 seconds: 83.35 ops/second - unlinked 0 (time 1788274136 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 702 (time 1788274219.59 total 10.00 last 70.19) - open/close 1269 (time 1788274229.60 total 20.01 last 56.65) - open/close 1976 (time 1788274239.60 total 30.01 last 70.70) - open/close 2799 (time 1788274249.60 total 40.01 last 82.28) - open/close 3581 (time 1788274259.61 total 50.02 last 78.16) - open/close 4369 (time 1788274269.62 total 60.03 last 78.71) - open/close 5134 (time 1788274279.62 total 70.03 last 76.49) - open/close 5934 (time 1788274289.64 total 80.04 last 79.89) - open/close 6636 (time 1788274299.64 total 90.05 last 70.17) - open/close 7376 (time 1788274309.65 total 100.06 last 73.95) - open/close 8284 (time 1788274319.66 total 110.06 last 90.73) - open/close 9142 (time 1788274329.67 total 120.08 last 85.65) - open/close 9893 (time 1788274339.68 total 130.09 last 75.03) total: 10000 open/close in 131.33 seconds: 76.15 ops/second - unlinked 0 (time 1788274345 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg160-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg160-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server unloading modules via unload_modules_local on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: modules unloaded. pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2