== conf-sanity test 69: replace an OST with the same index ========================================================== 11:10:01 (1781622601) start mds service on oleg152-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg152-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg152-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 737 (time 1781622719.76 total 10.00 last 73.67) - open/close 1404 (time 1781622729.76 total 20.01 last 66.68) - open/close 2242 (time 1781622739.77 total 30.02 last 83.74) - open/close 3025 (time 1781622749.77 total 40.02 last 78.29) - open/close 3874 (time 1781622759.79 total 50.03 last 84.74) - open/close 4646 (time 1781622769.79 total 60.04 last 77.15) - open/close 5462 (time 1781622779.81 total 70.06 last 81.44) - open/close 6266 (time 1781622789.82 total 80.07 last 80.34) - open/close 7087 (time 1781622799.83 total 90.08 last 82.04) - open/close 7886 (time 1781622809.83 total 100.08 last 79.88) - open/close 8646 (time 1781622819.84 total 110.09 last 75.95) - open/close 9381 (time 1781622829.86 total 120.11 last 73.33) total: 10000 open/close in 128.72 seconds: 77.69 ops/second - unlinked 0 (time 1781622842 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 769 (time 1781622922.65 total 10.01 last 76.80) - open/close 1565 (time 1781622932.65 total 20.02 last 79.55) - open/close 2322 (time 1781622942.65 total 30.02 last 75.69) - open/close 3211 (time 1781622952.65 total 40.02 last 88.90) - open/close 4101 (time 1781622962.67 total 50.04 last 88.86) - open/close 4944 (time 1781622972.67 total 60.04 last 84.26) - open/close 5844 (time 1781622982.68 total 70.05 last 89.96) - open/close 6662 (time 1781622992.68 total 80.05 last 81.79) - open/close 7485 (time 1781623002.68 total 90.05 last 82.29) - open/close 8233 (time 1781623012.68 total 100.05 last 74.77) - open/close 9072 (time 1781623022.69 total 110.06 last 83.84) - open/close 9908 (time 1781623032.71 total 120.07 last 83.49) total: 10000 open/close in 121.10 seconds: 82.58 ops/second - unlinked 0 (time 1781623037 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 975 (time 1781623119.17 total 10.01 last 97.43) - open/close 1875 (time 1781623129.18 total 20.01 last 89.93) - open/close 2764 (time 1781623139.21 total 30.04 last 88.67) - open/close 3617 (time 1781623149.23 total 40.06 last 85.10) - open/close 4573 (time 1781623159.23 total 50.07 last 95.55) - open/close 5456 (time 1781623169.24 total 60.08 last 88.25) - open/close 6405 (time 1781623179.24 total 70.08 last 94.89) - open/close 7360 (time 1781623189.25 total 80.08 last 95.44) - open/close 8287 (time 1781623199.25 total 90.09 last 92.68) - open/close 9253 (time 1781623209.25 total 100.09 last 96.58) total: 10000 open/close in 107.86 seconds: 92.71 ops/second - unlinked 0 (time 1781623220 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1043 (time 1781623290.00 total 10.01 last 104.19) - open/close 2111 (time 1781623300.01 total 20.01 last 106.77) - open/close 3106 (time 1781623310.02 total 30.02 last 99.41) - open/close 4020 (time 1781623320.02 total 40.02 last 91.38) - open/close 5088 (time 1781623330.03 total 50.03 last 106.69) - open/close 6150 (time 1781623340.04 total 60.04 last 106.13) - open/close 7248 (time 1781623350.04 total 70.05 last 109.75) - open/close 8231 (time 1781623360.04 total 80.05 last 98.29) - open/close 9202 (time 1781623370.05 total 90.05 last 97.04) total: 10000 open/close in 99.29 seconds: 100.72 ops/second - unlinked 0 (time 1781623382 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1039 (time 1781623454.04 total 10.01 last 103.80) - open/close 2024 (time 1781623464.05 total 20.02 last 98.44) - open/close 3064 (time 1781623474.06 total 30.02 last 103.93) - open/close 4037 (time 1781623484.06 total 40.03 last 97.26) - open/close 5068 (time 1781623494.06 total 50.03 last 103.07) - open/close 6090 (time 1781623504.06 total 60.03 last 102.19) - open/close 7181 (time 1781623514.08 total 70.04 last 108.95) - open/close 8192 (time 1781623524.09 total 80.06 last 100.96) - open/close 9238 (time 1781623534.10 total 90.06 last 104.55) total: 10000 open/close in 97.04 seconds: 103.05 ops/second - unlinked 0 (time 1781623543 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 994 (time 1781623610.36 total 10.01 last 99.27) - open/close 2089 (time 1781623620.38 total 20.02 last 109.39) - open/close 3214 (time 1781623630.38 total 30.03 last 112.43) - open/close 4232 (time 1781623640.39 total 40.04 last 101.69) - open/close 5205 (time 1781623650.40 total 50.05 last 97.21) - open/close 6224 (time 1781623660.41 total 60.06 last 101.81) - open/close 7301 (time 1781623670.42 total 70.07 last 107.62) - open/close 8255 (time 1781623680.42 total 80.07 last 95.36) - open/close 9283 (time 1781623690.43 total 90.08 last 102.73) total: 10000 open/close in 97.48 seconds: 102.59 ops/second - unlinked 0 (time 1781623701 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1065 (time 1781623771.78 total 10.00 last 106.48) - open/close 2021 (time 1781623781.79 total 20.01 last 95.49) - open/close 3083 (time 1781623791.79 total 30.01 last 106.18) - open/close 4177 (time 1781623801.80 total 40.02 last 109.33) - open/close 5202 (time 1781623811.80 total 50.03 last 102.44) - open/close 6245 (time 1781623821.81 total 60.04 last 104.21) - open/close 7155 (time 1781623831.82 total 70.04 last 90.95) - open/close 8121 (time 1781623841.83 total 80.05 last 96.48) - open/close 9159 (time 1781623851.83 total 90.06 last 103.77) total: 10000 open/close in 98.05 seconds: 101.98 ops/second - unlinked 0 (time 1781623862 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 868 (time 1781623930.98 total 10.00 last 86.77) - open/close 1813 (time 1781623940.99 total 20.01 last 94.40) - open/close 2753 (time 1781623950.99 total 30.01 last 94.00) - open/close 3779 (time 1781623960.99 total 40.02 last 102.55) - open/close 4876 (time 1781623970.99 total 50.02 last 109.69) - open/close 5930 (time 1781623981.00 total 60.02 last 105.36) - open/close 6863 (time 1781623991.01 total 70.03 last 93.20) - open/close 7837 (time 1781624001.01 total 80.04 last 97.38) - open/close 8876 (time 1781624011.02 total 90.04 last 103.83) - open/close 9830 (time 1781624021.02 total 100.05 last 95.35) total: 10000 open/close in 101.49 seconds: 98.54 ops/second - unlinked 0 (time 1781624025 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 962 (time 1781624095.72 total 10.01 last 96.14) - open/close 1928 (time 1781624105.72 total 20.01 last 96.54) - open/close 2967 (time 1781624115.72 total 30.01 last 103.89) - open/close 3980 (time 1781624125.73 total 40.02 last 101.27) - open/close 5079 (time 1781624135.73 total 50.02 last 109.85) - open/close 6277 (time 1781624145.73 total 60.02 last 119.78) - open/close 7457 (time 1781624155.75 total 70.03 last 117.85) - open/close 8547 (time 1781624165.76 total 80.04 last 108.90) - open/close 9573 (time 1781624175.76 total 90.04 last 102.59) total: 10000 open/close in 94.92 seconds: 105.36 ops/second - unlinked 0 (time 1781624183 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1014 (time 1781624249.40 total 10.00 last 101.39) - open/close 2029 (time 1781624259.40 total 20.00 last 101.49) - open/close 2994 (time 1781624269.40 total 30.00 last 96.48) - open/close 4047 (time 1781624279.40 total 40.01 last 105.24) - open/close 5013 (time 1781624289.41 total 50.01 last 96.59) - open/close 6124 (time 1781624299.41 total 60.02 last 111.02) - open/close 7271 (time 1781624309.41 total 70.02 last 114.69) - open/close 8385 (time 1781624319.42 total 80.02 last 111.36) - open/close 9454 (time 1781624329.42 total 90.03 last 106.86) total: 10000 open/close in 94.97 seconds: 105.30 ops/second - unlinked 0 (time 1781624337 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre On OST0, 19966 used inodes rc=0 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2