== conf-sanity test 69: replace an OST with the same index ========================================================== 17:54:07 (1784584447) start mds service on oleg334-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg334-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg334-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 699 (time 1784584556.64 total 10.03 last 69.72) - open/close 1434 (time 1784584566.67 total 20.05 last 73.30) - open/close 2163 (time 1784584576.67 total 30.06 last 72.88) - open/close 2991 (time 1784584586.68 total 40.07 last 82.72) - open/close 3780 (time 1784584596.68 total 50.07 last 78.89) - open/close 4549 (time 1784584606.68 total 60.07 last 76.88) - open/close 5301 (time 1784584616.68 total 70.07 last 75.20) - open/close 5979 (time 1784584626.69 total 80.08 last 67.76) - open/close 6738 (time 1784584636.71 total 90.10 last 75.74) - open/close 7533 (time 1784584646.72 total 100.11 last 79.41) - open/close 8326 (time 1784584656.73 total 110.12 last 79.24) - open/close 9118 (time 1784584666.73 total 120.12 last 79.19) - open/close 9912 (time 1784584676.74 total 130.12 last 79.36) total: 10000 open/close in 131.33 seconds: 76.14 ops/second - unlinked 0 (time 1784584682 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 708 (time 1784584772.21 total 10.00 last 70.79) - open/close 1548 (time 1784584782.21 total 20.00 last 83.99) - open/close 2300 (time 1784584792.22 total 30.01 last 75.15) - open/close 3085 (time 1784584802.23 total 40.02 last 78.44) - open/close 3920 (time 1784584812.23 total 50.03 last 83.43) - open/close 4632 (time 1784584822.24 total 60.03 last 71.17) - open/close 5273 (time 1784584832.24 total 70.04 last 64.07) - open/close 6061 (time 1784584842.26 total 80.05 last 78.70) - open/close 6817 (time 1784584852.26 total 90.06 last 75.55) - open/close 7597 (time 1784584862.28 total 100.07 last 77.86) - open/close 8367 (time 1784584872.29 total 110.08 last 76.92) - open/close 9114 (time 1784584882.30 total 120.09 last 74.64) - open/close 9855 (time 1784584892.30 total 130.10 last 74.06) total: 10000 open/close in 132.04 seconds: 75.73 ops/second - unlinked 0 (time 1784584898 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 724 (time 1784584992.69 total 10.00 last 72.38) - open/close 1522 (time 1784585002.70 total 20.02 last 79.69) - open/close 2262 (time 1784585012.71 total 30.02 last 73.95) - open/close 2963 (time 1784585022.71 total 40.03 last 70.05) - open/close 3650 (time 1784585032.72 total 50.04 last 68.66) - open/close 4299 (time 1784585042.74 total 60.06 last 64.75) - open/close 5011 (time 1784585052.74 total 70.06 last 71.18) - open/close 5728 (time 1784585062.75 total 80.07 last 71.64) - open/close 6419 (time 1784585072.75 total 90.07 last 69.09) - open/close 7148 (time 1784585082.76 total 100.07 last 72.89) - open/close 7925 (time 1784585092.76 total 110.08 last 77.65) - open/close 8648 (time 1784585102.76 total 120.08 last 72.29) - open/close 9325 (time 1784585112.78 total 130.09 last 67.61) total: 10000 open/close in 139.14 seconds: 71.87 ops/second - unlinked 0 (time 1784585125 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 794 (time 1784585214.25 total 10.01 last 79.32) - open/close 1609 (time 1784585224.25 total 20.01 last 81.48) - open/close 2406 (time 1784585234.26 total 30.02 last 79.66) - open/close 3131 (time 1784585244.27 total 40.03 last 72.40) - open/close 3873 (time 1784585254.28 total 50.04 last 74.12) - open/close 4643 (time 1784585264.29 total 60.05 last 76.93) - open/close 5277 (time 1784585274.30 total 70.06 last 63.34) - open/close 6002 (time 1784585284.32 total 80.08 last 72.38) - open/close 6730 (time 1784585294.33 total 90.09 last 72.74) - open/close 7388 (time 1784585304.34 total 100.10 last 65.69) - open/close 8072 (time 1784585314.35 total 110.11 last 68.34) - open/close 8846 (time 1784585324.35 total 120.11 last 77.40) - open/close 9575 (time 1784585334.36 total 130.12 last 72.83) total: 10000 open/close in 136.06 seconds: 73.50 ops/second - unlinked 0 (time 1784585345 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 781 (time 1784585435.37 total 10.01 last 78.01) - open/close 1499 (time 1784585445.38 total 20.02 last 71.73) - open/close 2213 (time 1784585455.39 total 30.03 last 71.36) - open/close 2928 (time 1784585465.40 total 40.04 last 71.42) - open/close 3759 (time 1784585475.40 total 50.04 last 83.07) - open/close 4535 (time 1784585485.42 total 60.06 last 77.48) - open/close 5255 (time 1784585495.43 total 70.07 last 71.92) - open/close 6020 (time 1784585505.44 total 80.08 last 76.38) - open/close 6756 (time 1784585515.45 total 90.09 last 73.57) - open/close 7509 (time 1784585525.46 total 100.10 last 75.20) - open/close 8236 (time 1784585535.47 total 110.11 last 72.65) - open/close 8923 (time 1784585545.48 total 120.12 last 68.61) - open/close 9630 (time 1784585555.49 total 130.13 last 70.63) total: 10000 open/close in 135.62 seconds: 73.73 ops/second - unlinked 0 (time 1784585565 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 564 (time 1784585665.10 total 10.01 last 56.36) - open/close 1216 (time 1784585675.10 total 20.01 last 65.17) - open/close 1900 (time 1784585685.11 total 30.02 last 68.35) - open/close 2593 (time 1784585695.11 total 40.02 last 69.27) - open/close 3365 (time 1784585705.13 total 50.04 last 77.07) - open/close 4077 (time 1784585715.13 total 60.04 last 71.19) - open/close 4746 (time 1784585725.13 total 70.04 last 66.89) - open/close 5317 (time 1784585735.13 total 80.04 last 57.10) - open/close 5955 (time 1784585745.14 total 90.05 last 63.76) - open/close 6591 (time 1784585755.15 total 100.06 last 63.53) - open/close 7243 (time 1784585765.16 total 110.07 last 65.14) - open/close 8033 (time 1784585775.17 total 120.08 last 78.92) - open/close 8757 (time 1784585785.17 total 130.08 last 72.36) - open/close 9496 (time 1784585795.17 total 140.09 last 73.88) total: 10000 open/close in 146.76 seconds: 68.14 ops/second - unlinked 0 (time 1784585806 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 552 (time 1784585898.15 total 10.01 last 55.13) - open/close 1174 (time 1784585908.17 total 20.03 last 62.12) - open/close 1928 (time 1784585918.17 total 30.03 last 75.38) - open/close 2675 (time 1784585928.17 total 40.03 last 74.70) - open/close 3463 (time 1784585938.17 total 50.03 last 78.78) - open/close 4196 (time 1784585948.19 total 60.05 last 73.19) - open/close 4727 (time 1784585958.20 total 70.06 last 53.05) - open/close 5323 (time 1784585968.21 total 80.07 last 59.56) - open/close 6113 (time 1784585978.22 total 90.07 last 78.93) - open/close 6935 (time 1784585988.23 total 100.08 last 82.12) - open/close 7663 (time 1784585998.23 total 110.09 last 72.76) - open/close 8372 (time 1784586008.24 total 120.10 last 70.83) - open/close 9047 (time 1784586018.25 total 130.11 last 67.41) - open/close 9824 (time 1784586028.26 total 140.12 last 77.67) total: 10000 open/close in 142.51 seconds: 70.17 ops/second - unlinked 0 (time 1784586034 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 667 (time 1784586134.29 total 10.00 last 66.67) - open/close 1330 (time 1784586144.30 total 20.01 last 66.24) - open/close 2070 (time 1784586154.31 total 30.02 last 73.96) - open/close 2796 (time 1784586164.31 total 40.03 last 72.54) - open/close 3509 (time 1784586174.32 total 50.03 last 71.28) - open/close 4260 (time 1784586184.32 total 60.04 last 75.06) - open/close 5029 (time 1784586194.32 total 70.04 last 76.89) - open/close 5755 (time 1784586204.33 total 80.04 last 72.55) - open/close 6472 (time 1784586214.35 total 90.06 last 71.59) - open/close 7227 (time 1784586224.35 total 100.06 last 75.49) - open/close 8024 (time 1784586234.35 total 110.07 last 79.65) - open/close 8814 (time 1784586244.36 total 120.07 last 78.98) - open/close 9618 (time 1784586254.36 total 130.07 last 80.35) total: 10000 open/close in 135.55 seconds: 73.77 ops/second - unlinked 0 (time 1784586264 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 720 (time 1784586358.42 total 10.01 last 71.94) - open/close 1422 (time 1784586368.42 total 20.01 last 70.16) - open/close 2183 (time 1784586378.43 total 30.02 last 76.02) - open/close 2968 (time 1784586388.44 total 40.03 last 78.47) - open/close 3630 (time 1784586398.44 total 50.03 last 66.19) - open/close 4198 (time 1784586408.46 total 60.05 last 56.69) - open/close 4773 (time 1784586418.47 total 70.05 last 57.47) - open/close 5354 (time 1784586428.47 total 80.06 last 58.07) - open/close 5950 (time 1784586438.47 total 90.06 last 59.59) - open/close 6505 (time 1784586448.48 total 100.07 last 55.47) - open/close 7042 (time 1784586458.48 total 110.07 last 53.65) - open/close 7721 (time 1784586468.51 total 120.10 last 67.73) - open/close 8309 (time 1784586478.53 total 130.12 last 58.69) - open/close 8810 (time 1784586488.54 total 140.13 last 50.06) - open/close 9425 (time 1784586498.56 total 150.15 last 61.36) total: 10000 open/close in 159.68 seconds: 62.62 ops/second - unlinked 0 (time 1784586512 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 685 (time 1784586604.82 total 10.00 last 68.47) - open/close 1439 (time 1784586614.83 total 20.01 last 75.32) - open/close 2160 (time 1784586624.84 total 30.02 last 72.05) - open/close 2852 (time 1784586634.84 total 40.03 last 69.17) - open/close 3575 (time 1784586644.85 total 50.03 last 72.26) - open/close 4204 (time 1784586654.86 total 60.04 last 62.83) - open/close 4812 (time 1784586664.87 total 70.05 last 60.74) - open/close 5452 (time 1784586674.88 total 80.07 last 63.90) - open/close 6219 (time 1784586684.89 total 90.07 last 76.66) - open/close 6991 (time 1784586694.90 total 100.08 last 77.09) - open/close 7675 (time 1784586704.91 total 110.09 last 68.33) - open/close 8311 (time 1784586714.93 total 120.11 last 63.51) - open/close 8928 (time 1784586724.94 total 130.12 last 61.64) - open/close 9585 (time 1784586734.94 total 140.12 last 65.68) total: 10000 open/close in 146.13 seconds: 68.43 ops/second - unlinked 0 (time 1784586745 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.134@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.134@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.134@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.134@tcp sys.timeout=20 start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local oleg334-server: modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2