== conf-sanity test 69: replace an OST with the same index ========================================================== 16:29:01 (1780086541) start mds service on oleg452-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing load_modules_local oleg452-server: Loading modules from /home/green/git/lustre-release/lustre oleg452-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg452-server: detected 4 online CPUs by sysfs oleg452-server: MODOPTS_LIBCFS= oleg452-server: Force libcfs to create 2 CPU partitions oleg452-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg452-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg452-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg452-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg452-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1229 (time 1780086612.97 total 10.00 last 122.87) - open/close 2343 (time 1780086622.98 total 20.01 last 111.37) - open/close 3414 (time 1780086632.98 total 30.01 last 107.03) - open/close 4330 (time 1780086642.99 total 40.02 last 91.55) - open/close 5284 (time 1780086653.00 total 50.03 last 95.32) - open/close 6210 (time 1780086663.00 total 60.03 last 92.54) - open/close 7139 (time 1780086673.00 total 70.03 last 92.89) - open/close 8094 (time 1780086683.01 total 80.04 last 95.46) - open/close 9025 (time 1780086693.01 total 90.04 last 93.09) - open/close 9971 (time 1780086703.02 total 100.05 last 94.52) total: 10000 open/close in 100.36 seconds: 99.64 ops/second - unlinked 0 (time 1780086705 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 950 (time 1780086770.18 total 10.01 last 94.93) - open/close 1933 (time 1780086780.18 total 20.01 last 98.29) - open/close 3037 (time 1780086790.19 total 30.01 last 110.38) - open/close 4177 (time 1780086800.20 total 40.02 last 113.89) - open/close 5367 (time 1780086810.21 total 50.03 last 118.86) - open/close 6247 (time 1780086820.21 total 60.04 last 87.97) - open/close 7195 (time 1780086830.21 total 70.04 last 94.80) - open/close 8224 (time 1780086840.21 total 80.04 last 102.90) - open/close 9395 (time 1780086850.22 total 90.04 last 117.02) total: 10000 open/close in 94.37 seconds: 105.96 ops/second - unlinked 0 (time 1780086856 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1152 (time 1780086919.95 total 10.00 last 115.16) - open/close 2331 (time 1780086929.95 total 20.01 last 117.87) - open/close 3519 (time 1780086939.96 total 30.01 last 118.76) - open/close 4693 (time 1780086949.96 total 40.01 last 117.35) - open/close 6045 (time 1780086959.96 total 50.01 last 135.19) - open/close 7533 (time 1780086969.96 total 60.02 last 148.77) - open/close 8955 (time 1780086979.96 total 70.02 last 142.18) total: 10000 open/close in 75.55 seconds: 132.36 ops/second - unlinked 0 (time 1780086987 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1527 (time 1780087034.13 total 10.00 last 152.66) - open/close 2992 (time 1780087044.13 total 20.01 last 146.41) - open/close 4294 (time 1780087054.13 total 30.01 last 130.18) - open/close 5690 (time 1780087064.14 total 40.01 last 139.54) - open/close 7829 (time 1780087074.14 total 50.01 last 213.90) - open/close 9701 (time 1780087084.14 total 60.02 last 187.10) total: 10000 open/close in 61.73 seconds: 162.00 ops/second - unlinked 0 (time 1780087087 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1305 (time 1780087138.96 total 10.00 last 130.49) - open/close 2654 (time 1780087148.96 total 20.01 last 134.79) - open/close 3742 (time 1780087158.97 total 30.01 last 108.75) - open/close 5092 (time 1780087168.97 total 40.02 last 134.92) - open/close 6315 (time 1780087178.98 total 50.02 last 122.28) - open/close 7600 (time 1780087188.98 total 60.02 last 128.47) - open/close 9001 (time 1780087198.98 total 70.03 last 140.06) total: 10000 open/close in 78.40 seconds: 127.56 ops/second - unlinked 0 (time 1780087209 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1178 (time 1780087267.43 total 10.01 last 117.66) - open/close 2368 (time 1780087277.45 total 20.03 last 118.83) - open/close 4147 (time 1780087287.45 total 30.03 last 177.84) - open/close 5615 (time 1780087297.46 total 40.04 last 146.67) - open/close 6766 (time 1780087307.46 total 50.04 last 115.07) - open/close 7945 (time 1780087317.46 total 60.04 last 117.89) - open/close 9085 (time 1780087327.46 total 70.04 last 113.99) total: 10000 open/close in 78.36 seconds: 127.62 ops/second - unlinked 0 (time 1780087337 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1721 (time 1780087385.63 total 10.01 last 172.01) - open/close 3751 (time 1780087395.63 total 20.01 last 202.96) - open/close 5688 (time 1780087405.64 total 30.01 last 193.61) - open/close 7673 (time 1780087415.64 total 40.01 last 198.49) total: 10000 open/close in 49.22 seconds: 203.19 ops/second - unlinked 0 (time 1780087426 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1447 (time 1780087471.99 total 10.00 last 144.69) - open/close 3530 (time 1780087481.99 total 20.00 last 208.27) - open/close 5446 (time 1780087491.99 total 30.01 last 191.52) - open/close 7029 (time 1780087502.00 total 40.01 last 158.26) - open/close 8689 (time 1780087512.00 total 50.01 last 165.90) total: 10000 open/close in 59.09 seconds: 169.23 ops/second - unlinked 0 (time 1780087522 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1301 (time 1780087566.85 total 10.01 last 130.02) - open/close 2551 (time 1780087576.86 total 20.01 last 124.92) - open/close 3930 (time 1780087586.87 total 30.02 last 137.81) - open/close 5262 (time 1780087596.87 total 40.03 last 133.12) - open/close 6956 (time 1780087606.88 total 50.03 last 169.28) - open/close 8950 (time 1780087616.88 total 60.04 last 199.32) total: 10000 open/close in 63.01 seconds: 158.71 ops/second - unlinked 0 (time 1780087621 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 1698 (time 1780087656.03 total 10.00 last 169.73) - open/close 3865 (time 1780087666.04 total 20.01 last 216.48) - open/close 5776 (time 1780087676.04 total 30.02 last 191.03) - open/close 7835 (time 1780087686.05 total 40.02 last 205.89) - open/close 9648 (time 1780087696.05 total 50.02 last 181.26) total: 10000 open/close in 51.22 seconds: 195.25 ops/second - unlinked 0 (time 1780087698 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg452-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg452-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg452-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.152@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg452-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg452-server: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg452-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec oleg452-server: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg452-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre On OST0, 18148 used inodes rc=0 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg452-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg452-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg452-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.152@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.152@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.152@tcp sys.timeout=20 start ost1 service on oleg452-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg452-server: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg452-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec oleg452-server: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg452-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg452-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg452-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg452-server unloading modules via unload_modules_local on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing unload_modules_local oleg452-server: modules unloaded. pdsh@oleg452-client: oleg452-client: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-client: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2