== conf-sanity test 69: replace an OST with the same index ========================================================== 16:17:52 (1784233072) start mds service on oleg429-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing load_modules_local oleg429-server: Loading modules from /home/green/git/lustre-release/lustre oleg429-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg429-server: detected 4 online CPUs by sysfs oleg429-server: MODOPTS_LIBCFS= oleg429-server: Force libcfs to create 2 CPU partitions oleg429-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg429-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg429-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 1156 (time 1784233128.13 total 10.01 last 115.54) - open/close 2334 (time 1784233138.13 total 20.01 last 117.79) - open/close 3537 (time 1784233148.13 total 30.01 last 120.25) - open/close 4936 (time 1784233158.14 total 40.01 last 139.85) - open/close 6542 (time 1784233168.14 total 50.02 last 160.53) - open/close 8202 (time 1784233178.14 total 60.02 last 166.00) - open/close 9937 (time 1784233188.14 total 70.02 last 173.45) total: 10000 open/close in 70.39 seconds: 142.07 ops/second - unlinked 0 (time 1784233190 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1054 (time 1784233250.99 total 10.00 last 105.36) - open/close 2049 (time 1784233261.00 total 20.01 last 99.40) - open/close 2863 (time 1784233271.02 total 30.04 last 81.19) - open/close 3675 (time 1784233281.03 total 40.05 last 81.13) - open/close 4335 (time 1784233291.04 total 50.06 last 65.96) - open/close 5002 (time 1784233301.04 total 60.06 last 66.67) - open/close 5743 (time 1784233311.05 total 70.06 last 74.06) - open/close 6442 (time 1784233321.06 total 80.08 last 69.78) - open/close 7166 (time 1784233331.07 total 90.09 last 72.35) - open/close 7948 (time 1784233341.08 total 100.10 last 78.15) - open/close 8748 (time 1784233351.08 total 110.10 last 79.98) - open/close 9595 (time 1784233361.09 total 120.11 last 84.63) total: 10000 open/close in 124.56 seconds: 80.28 ops/second - unlinked 0 (time 1784233368 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 988 (time 1784233436.31 total 10.01 last 98.74) - open/close 1917 (time 1784233446.32 total 20.01 last 92.88) - open/close 2911 (time 1784233456.32 total 30.01 last 99.38) - open/close 3926 (time 1784233466.32 total 40.01 last 101.50) - open/close 4971 (time 1784233476.32 total 50.01 last 104.45) - open/close 5993 (time 1784233486.32 total 60.02 last 102.18) - open/close 7048 (time 1784233496.33 total 70.02 last 105.46) - open/close 8237 (time 1784233506.33 total 80.02 last 118.84) - open/close 9361 (time 1784233516.34 total 90.03 last 112.30) total: 10000 open/close in 96.15 seconds: 104.01 ops/second - unlinked 0 (time 1784233524 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1151 (time 1784233579.89 total 10.00 last 115.04) - open/close 2382 (time 1784233589.89 total 20.01 last 123.05) - open/close 3917 (time 1784233599.90 total 30.02 last 153.37) - open/close 5927 (time 1784233609.90 total 40.02 last 200.93) - open/close 7481 (time 1784233619.91 total 50.02 last 155.35) - open/close 9471 (time 1784233629.91 total 60.02 last 198.99) total: 10000 open/close in 62.51 seconds: 159.96 ops/second - unlinked 0 (time 1784233634 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1482 (time 1784233674.62 total 10.01 last 148.08) - open/close 3147 (time 1784233684.63 total 20.02 last 166.36) - open/close 4996 (time 1784233694.64 total 30.02 last 184.83) - open/close 6766 (time 1784233704.64 total 40.02 last 176.99) - open/close 8678 (time 1784233714.64 total 50.02 last 191.15) total: 10000 open/close in 56.21 seconds: 177.89 ops/second - unlinked 0 (time 1784233722 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 2014 (time 1784233763.26 total 10.00 last 201.38) - open/close 4206 (time 1784233773.26 total 20.00 last 219.19) - open/close 6796 (time 1784233783.26 total 30.01 last 258.89) - open/close 8890 (time 1784233793.26 total 40.01 last 209.39) total: 10000 open/close in 44.15 seconds: 226.51 ops/second - unlinked 0 (time 1784233799 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 3479 (time 1784233835.43 total 10.00 last 347.84) - open/close 7637 (time 1784233845.43 total 20.00 last 415.74) total: 10000 open/close in 27.50 seconds: 363.66 ops/second - unlinked 0 (time 1784233854 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 4198 (time 1784233882.12 total 10.00 last 419.75) - open/close 8770 (time 1784233892.12 total 20.00 last 457.15) total: 10000 open/close in 22.29 seconds: 448.61 ops/second - unlinked 0 (time 1784233895 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4242 (time 1784233920.89 total 10.00 last 424.12) - open/close 9234 (time 1784233930.89 total 20.00 last 499.18) total: 10000 open/close in 21.55 seconds: 464.10 ops/second - unlinked 0 (time 1784233933 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4462 (time 1784233959.06 total 10.00 last 446.12) - open/close 9053 (time 1784233969.06 total 20.00 last 459.02) total: 10000 open/close in 22.08 seconds: 452.86 ops/second - unlinked 0 (time 1784233972 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.129@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg429-server: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg429-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 12 sec mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre On OST0, 19462 used inodes rc=0 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.129@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.129@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.129@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.129@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg429-server: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg429-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 13 sec mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server unloading modules via unload_modules_local on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing unload_modules_local oleg429-server: modules unloaded. pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2