== conf-sanity test 69: replace an OST with the same index ========================================================== 14:05:29 (1781028329) start mds service on oleg114-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg114-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg114-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre kernel.panic_on_warn = 0 seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 964 (time 1781028431.18 total 10.01 last 96.29) - open/close 1900 (time 1781028441.18 total 20.01 last 93.59) - open/close 2937 (time 1781028451.20 total 30.03 last 103.51) - open/close 3918 (time 1781028461.20 total 40.03 last 98.05) - open/close 4836 (time 1781028471.21 total 50.04 last 91.78) - open/close 5840 (time 1781028481.21 total 60.04 last 100.36) - open/close 6837 (time 1781028491.22 total 70.05 last 99.60) - open/close 7873 (time 1781028501.23 total 80.06 last 103.55) - open/close 8817 (time 1781028511.24 total 90.07 last 94.30) - open/close 9724 (time 1781028521.24 total 100.07 last 90.68) total: 10000 open/close in 103.19 seconds: 96.91 ops/second - unlinked 0 (time 1781028527 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 975 (time 1781028598.67 total 10.00 last 97.49) - open/close 2009 (time 1781028608.67 total 20.00 last 103.37) - open/close 3099 (time 1781028618.68 total 30.01 last 108.96) - open/close 4179 (time 1781028628.69 total 40.02 last 107.89) - open/close 5156 (time 1781028638.70 total 50.03 last 97.60) - open/close 6305 (time 1781028648.70 total 60.03 last 114.90) - open/close 7475 (time 1781028658.70 total 70.03 last 116.96) - open/close 8569 (time 1781028668.71 total 80.04 last 109.33) - open/close 9722 (time 1781028678.72 total 90.05 last 115.22) total: 10000 open/close in 92.29 seconds: 108.36 ops/second - unlinked 0 (time 1781028683 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1245 (time 1781028741.38 total 10.00 last 124.49) - open/close 2367 (time 1781028751.39 total 20.01 last 112.07) - open/close 3576 (time 1781028761.40 total 30.01 last 120.86) - open/close 4795 (time 1781028771.40 total 40.02 last 121.89) - open/close 6023 (time 1781028781.40 total 50.02 last 122.79) - open/close 7134 (time 1781028791.40 total 60.02 last 111.06) - open/close 8243 (time 1781028801.41 total 70.03 last 110.82) - open/close 9380 (time 1781028811.41 total 80.03 last 113.70) total: 10000 open/close in 85.43 seconds: 117.05 ops/second - unlinked 0 (time 1781028819 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1170 (time 1781028877.97 total 10.01 last 116.92) - open/close 2220 (time 1781028887.98 total 20.01 last 104.97) - open/close 3411 (time 1781028897.98 total 30.01 last 119.06) - open/close 4855 (time 1781028907.99 total 40.03 last 144.20) - open/close 6184 (time 1781028918.00 total 50.03 last 132.83) - open/close 7465 (time 1781028928.00 total 60.04 last 128.04) - open/close 8880 (time 1781028938.01 total 70.04 last 141.46) total: 10000 open/close in 79.19 seconds: 126.28 ops/second - unlinked 0 (time 1781028949 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1182 (time 1781029001.98 total 10.00 last 118.15) - open/close 2355 (time 1781029011.99 total 20.01 last 117.21) - open/close 3747 (time 1781029022.00 total 30.02 last 139.09) - open/close 5072 (time 1781029032.00 total 40.02 last 132.48) - open/close 6479 (time 1781029042.00 total 50.03 last 140.63) - open/close 7923 (time 1781029052.01 total 60.03 last 144.36) - open/close 9396 (time 1781029062.01 total 70.03 last 147.24) total: 10000 open/close in 73.81 seconds: 135.49 ops/second - unlinked 0 (time 1781029067 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1230 (time 1781029121.11 total 10.00 last 122.96) - open/close 2438 (time 1781029131.11 total 20.01 last 120.75) - open/close 3658 (time 1781029141.12 total 30.01 last 121.97) - open/close 4958 (time 1781029151.12 total 40.01 last 130.00) - open/close 6158 (time 1781029161.12 total 50.02 last 119.94) - open/close 7416 (time 1781029171.12 total 60.02 last 125.78) - open/close 8795 (time 1781029181.13 total 70.02 last 137.81) total: 10000 open/close in 78.71 seconds: 127.05 ops/second - unlinked 0 (time 1781029191 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1415 (time 1781029243.32 total 10.00 last 141.47) - open/close 2808 (time 1781029253.32 total 20.01 last 139.25) - open/close 4043 (time 1781029263.33 total 30.01 last 123.42) - open/close 5643 (time 1781029273.33 total 40.02 last 159.96) - open/close 7373 (time 1781029283.34 total 50.02 last 172.87) - open/close 8903 (time 1781029293.34 total 60.02 last 152.98) total: 10000 open/close in 67.30 seconds: 148.59 ops/second - unlinked 0 (time 1781029302 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1512 (time 1781029352.32 total 10.00 last 151.17) - open/close 3043 (time 1781029362.32 total 20.01 last 153.03) - open/close 4491 (time 1781029372.33 total 30.01 last 144.79) - open/close 6161 (time 1781029382.33 total 40.01 last 166.95) - open/close 7812 (time 1781029392.33 total 50.02 last 165.01) - open/close 9434 (time 1781029402.34 total 60.02 last 162.13) total: 10000 open/close in 64.07 seconds: 156.08 ops/second - unlinked 0 (time 1781029408 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1839 (time 1781029453.13 total 10.00 last 183.84) - open/close 3448 (time 1781029463.14 total 20.01 last 160.80) - open/close 5306 (time 1781029473.15 total 30.01 last 185.71) - open/close 6926 (time 1781029483.15 total 40.02 last 161.99) - open/close 8564 (time 1781029493.15 total 50.02 last 163.70) total: 10000 open/close in 56.95 seconds: 175.58 ops/second - unlinked 0 (time 1781029501 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2623 (time 1781029539.58 total 10.00 last 262.21) - open/close 4644 (time 1781029549.58 total 20.00 last 202.10) - open/close 6489 (time 1781029559.58 total 30.00 last 184.49) - open/close 8113 (time 1781029569.59 total 40.01 last 162.34) - open/close 9804 (time 1781029579.59 total 50.01 last 169.07) total: 10000 open/close in 51.01 seconds: 196.05 ops/second - unlinked 0 (time 1781029582 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre kernel.panic_on_warn = 0 On OST0, 14126 used inodes rc=0 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre kernel.panic_on_warn = 0 umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server unloading modules via unload_modules_local on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: modules unloaded. pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2