== conf-sanity test 69: replace an OST with the same index ========================================================== 09:23:38 (1787837018) start mds service on oleg205-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg205-server: detected 4 online CPUs by sysfs oleg205-server: MODOPTS_LIBCFS= oleg205-server: Force libcfs to create 2 CPU partitions oleg205-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg205-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg205-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg205-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg205-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1061 (time 1787837130.32 total 10.00 last 106.10) - open/close 2003 (time 1787837140.33 total 20.01 last 94.14) - open/close 3000 (time 1787837150.33 total 30.01 last 99.64) - open/close 3928 (time 1787837160.37 total 40.05 last 92.48) - open/close 4901 (time 1787837170.38 total 50.05 last 97.22) - open/close 6041 (time 1787837180.38 total 60.06 last 114.00) - open/close 7198 (time 1787837190.38 total 70.06 last 115.66) - open/close 8154 (time 1787837200.40 total 80.08 last 95.44) - open/close 9222 (time 1787837210.40 total 90.08 last 106.76) total: 10000 open/close in 98.07 seconds: 101.97 ops/second - unlinked 0 (time 1787837222 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 988 (time 1787837296.36 total 10.00 last 98.76) - open/close 1685 (time 1787837306.39 total 20.03 last 69.52) - open/close 2380 (time 1787837316.41 total 30.05 last 69.34) - open/close 3059 (time 1787837326.42 total 40.06 last 67.85) - open/close 3936 (time 1787837336.42 total 50.06 last 87.66) - open/close 4916 (time 1787837346.44 total 60.08 last 97.82) - open/close 5911 (time 1787837356.45 total 70.09 last 99.43) - open/close 6935 (time 1787837366.46 total 80.10 last 102.31) - open/close 7850 (time 1787837376.46 total 90.10 last 91.46) - open/close 8837 (time 1787837386.46 total 100.10 last 98.69) - open/close 9900 (time 1787837396.47 total 110.11 last 106.27) total: 10000 open/close in 111.24 seconds: 89.89 ops/second - unlinked 0 (time 1787837401 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 923 (time 1787837478.06 total 10.00 last 92.28) - open/close 1915 (time 1787837488.08 total 20.02 last 99.04) - open/close 2850 (time 1787837498.08 total 30.02 last 93.45) - open/close 3695 (time 1787837508.12 total 40.06 last 84.20) - open/close 4658 (time 1787837518.12 total 50.07 last 96.24) - open/close 5163 (time 1787837528.15 total 60.09 last 50.39) - open/close 6246 (time 1787837538.16 total 70.10 last 108.13) - open/close 7317 (time 1787837548.17 total 80.11 last 107.03) - open/close 8435 (time 1787837558.18 total 90.13 last 111.63) - open/close 9315 (time 1787837568.20 total 100.14 last 87.90) - open/close 9904 (time 1787837578.21 total 110.15 last 58.84) total: 10000 open/close in 111.27 seconds: 89.87 ops/second - unlinked 0 (time 1787837585 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 843 (time 1787837674.44 total 10.00 last 84.29) - open/close 1681 (time 1787837684.47 total 20.03 last 83.59) - open/close 2595 (time 1787837694.47 total 30.03 last 91.34) - open/close 3660 (time 1787837704.48 total 40.04 last 106.45) - open/close 4583 (time 1787837714.48 total 50.04 last 92.27) - open/close 5511 (time 1787837724.50 total 60.05 last 92.67) - open/close 6234 (time 1787837734.50 total 70.06 last 72.29) - open/close 7026 (time 1787837744.50 total 80.06 last 79.19) - open/close 8038 (time 1787837754.51 total 90.06 last 101.13) - open/close 9096 (time 1787837764.51 total 100.07 last 105.78) total: 10000 open/close in 109.60 seconds: 91.24 ops/second - unlinked 0 (time 1787837778 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 954 (time 1787837855.68 total 10.01 last 95.35) - open/close 2072 (time 1787837865.69 total 20.01 last 111.72) - open/close 3158 (time 1787837875.69 total 30.02 last 108.53) - open/close 4207 (time 1787837885.70 total 40.03 last 104.82) - open/close 5324 (time 1787837895.71 total 50.03 last 111.66) - open/close 6354 (time 1787837905.71 total 60.03 last 102.97) - open/close 7468 (time 1787837915.72 total 70.04 last 111.31) - open/close 8516 (time 1787837925.73 total 80.05 last 104.65) - open/close 9516 (time 1787837935.73 total 90.05 last 100.00) total: 10000 open/close in 95.34 seconds: 104.89 ops/second - unlinked 0 (time 1787837944 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 986 (time 1787838014.97 total 10.00 last 98.58) - open/close 2024 (time 1787838024.98 total 20.01 last 103.71) - open/close 3083 (time 1787838034.98 total 30.01 last 105.87) - open/close 3975 (time 1787838044.99 total 40.02 last 89.12) - open/close 4815 (time 1787838054.99 total 50.03 last 83.98) - open/close 5523 (time 1787838065.00 total 60.03 last 70.78) - open/close 6381 (time 1787838075.01 total 70.04 last 85.70) - open/close 7182 (time 1787838085.03 total 80.06 last 79.95) - open/close 7758 (time 1787838095.03 total 90.06 last 57.58) - open/close 8405 (time 1787838105.03 total 100.06 last 64.68) - open/close 9227 (time 1787838115.04 total 110.07 last 82.16) total: 10000 open/close in 118.27 seconds: 84.56 ops/second - unlinked 0 (time 1787838127 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 877 (time 1787838202.80 total 10.01 last 87.59) - open/close 1737 (time 1787838212.80 total 20.02 last 85.96) - open/close 2500 (time 1787838222.80 total 30.02 last 76.28) - open/close 3463 (time 1787838232.80 total 40.02 last 96.29) - open/close 4546 (time 1787838242.81 total 50.03 last 108.22) - open/close 5489 (time 1787838252.81 total 60.03 last 94.28) - open/close 6466 (time 1787838262.81 total 70.03 last 97.70) - open/close 7319 (time 1787838272.81 total 80.03 last 85.28) - open/close 8275 (time 1787838282.82 total 90.04 last 95.53) - open/close 9243 (time 1787838292.82 total 100.04 last 96.79) total: 10000 open/close in 107.57 seconds: 92.96 ops/second - unlinked 0 (time 1787838304 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 929 (time 1787838377.27 total 10.01 last 92.80) - open/close 1980 (time 1787838387.29 total 20.03 last 104.86) - open/close 2851 (time 1787838397.30 total 30.04 last 87.04) - open/close 3718 (time 1787838407.30 total 40.04 last 86.67) - open/close 4715 (time 1787838417.31 total 50.06 last 99.58) - open/close 5830 (time 1787838427.32 total 60.06 last 111.45) - open/close 6810 (time 1787838437.32 total 70.07 last 97.95) - open/close 7680 (time 1787838447.34 total 80.08 last 86.90) - open/close 8630 (time 1787838457.35 total 90.09 last 94.87) - open/close 9499 (time 1787838467.35 total 100.09 last 86.89) total: 10000 open/close in 105.79 seconds: 94.53 ops/second - unlinked 0 (time 1787838477 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 916 (time 1787838551.53 total 10.00 last 91.56) - open/close 1894 (time 1787838561.53 total 20.01 last 97.79) - open/close 2820 (time 1787838571.53 total 30.01 last 92.55) - open/close 3870 (time 1787838581.53 total 40.01 last 104.98) - open/close 4774 (time 1787838591.54 total 50.02 last 90.37) - open/close 5755 (time 1787838601.55 total 60.02 last 98.02) - open/close 6409 (time 1787838611.56 total 70.04 last 65.31) - open/close 7476 (time 1787838621.56 total 80.04 last 106.65) - open/close 8457 (time 1787838631.57 total 90.05 last 98.04) - open/close 9561 (time 1787838641.57 total 100.05 last 110.35) total: 10000 open/close in 103.78 seconds: 96.35 ops/second - unlinked 0 (time 1787838648 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 871 (time 1787838725.30 total 10.01 last 87.03) - open/close 1910 (time 1787838735.30 total 20.01 last 103.89) - open/close 2973 (time 1787838745.30 total 30.01 last 106.28) - open/close 3906 (time 1787838755.31 total 40.02 last 93.25) - open/close 4787 (time 1787838765.31 total 50.02 last 88.09) - open/close 5642 (time 1787838775.31 total 60.02 last 85.49) - open/close 6580 (time 1787838785.31 total 70.02 last 93.80) - open/close 7577 (time 1787838795.33 total 80.04 last 99.53) - open/close 8531 (time 1787838805.34 total 90.04 last 95.34) - open/close 9500 (time 1787838815.34 total 100.05 last 96.86) total: 10000 open/close in 106.56 seconds: 93.84 ops/second - unlinked 0 (time 1787838826 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.105@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg205-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg205-server: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg205-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg205-server: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg205-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.105@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.105@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.105@tcp sys.timeout=20 start ost1 service on oleg205-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg205-server: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg205-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg205-server: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg205-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server unloading modules via unload_modules_local on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing unload_modules_local oleg205-server: modules unloaded. pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2