== conf-sanity test 69: replace an OST with the same index ========================================================== 20:48:34 (1763344114) start mds service on oleg258-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing load_modules_local oleg258-server: Loading modules from /home/green/git/lustre-release/lustre oleg258-server: detected 4 online CPUs by sysfs oleg258-server: MODOPTS_LIBCFS= oleg258-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg258-server: Force libcfs to create 2 CPU partitions oleg258-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg258-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg258-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg258-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 634 (time 1763344188.14 total 10.01 last 63.34) - open/close 1257 (time 1763344198.15 total 20.02 last 62.23) - open/close 1912 (time 1763344208.16 total 30.03 last 65.46) - open/close 2508 (time 1763344218.17 total 40.05 last 59.49) - open/close 3153 (time 1763344228.18 total 50.05 last 64.49) - open/close 3832 (time 1763344238.18 total 60.05 last 67.89) - open/close 4503 (time 1763344248.19 total 70.06 last 67.04) - open/close 5221 (time 1763344258.20 total 80.07 last 71.68) - open/close 5851 (time 1763344268.23 total 90.10 last 62.84) - open/close 6506 (time 1763344278.23 total 100.10 last 65.47) - open/close 7206 (time 1763344288.24 total 110.11 last 69.95) - open/close 7871 (time 1763344298.24 total 120.11 last 66.49) - open/close 8449 (time 1763344308.28 total 130.15 last 57.60) - open/close 8960 (time 1763344318.28 total 140.15 last 51.08) - open/close 9520 (time 1763344328.28 total 150.16 last 55.98) total: 10000 open/close in 158.08 seconds: 63.26 ops/second - unlinked 0 (time 1763344340 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 745 (time 1763344439.14 total 10.01 last 74.45) - open/close 1403 (time 1763344449.15 total 20.01 last 65.78) - open/close 2176 (time 1763344459.15 total 30.02 last 77.25) - open/close 2855 (time 1763344469.15 total 40.02 last 67.90) - open/close 3590 (time 1763344479.15 total 50.02 last 73.50) - open/close 4279 (time 1763344489.16 total 60.03 last 68.84) - open/close 4909 (time 1763344499.16 total 70.03 last 62.99) - open/close 5530 (time 1763344509.18 total 80.04 last 62.02) - open/close 6246 (time 1763344519.19 total 90.06 last 71.47) - open/close 7070 (time 1763344529.21 total 100.07 last 82.29) - open/close 7809 (time 1763344539.22 total 110.08 last 73.84) - open/close 8544 (time 1763344549.23 total 120.09 last 73.41) - open/close 9260 (time 1763344559.24 total 130.11 last 71.49) total: 10000 open/close in 139.86 seconds: 71.50 ops/second - unlinked 0 (time 1763344572 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 739 (time 1763344660.92 total 10.01 last 73.84) - open/close 1416 (time 1763344670.92 total 20.01 last 67.68) - open/close 2049 (time 1763344680.92 total 30.01 last 63.29) - open/close 2750 (time 1763344690.93 total 40.02 last 70.04) - open/close 3474 (time 1763344700.94 total 50.03 last 72.31) - open/close 4179 (time 1763344710.96 total 60.05 last 70.40) - open/close 4851 (time 1763344720.96 total 70.06 last 67.15) - open/close 5560 (time 1763344730.98 total 80.07 last 70.77) - open/close 6281 (time 1763344741.00 total 90.09 last 72.01) - open/close 6927 (time 1763344751.01 total 100.10 last 64.52) - open/close 7658 (time 1763344761.02 total 110.11 last 73.03) - open/close 8405 (time 1763344771.03 total 120.12 last 74.64) - open/close 9141 (time 1763344781.03 total 130.13 last 73.54) - open/close 9901 (time 1763344791.04 total 140.13 last 75.93) total: 10000 open/close in 141.55 seconds: 70.65 ops/second - unlinked 0 (time 1763344796 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 696 (time 1763344898.45 total 10.00 last 69.58) - open/close 1352 (time 1763344908.46 total 20.01 last 65.53) - open/close 2018 (time 1763344918.47 total 30.02 last 66.56) - open/close 2676 (time 1763344928.48 total 40.02 last 65.76) - open/close 3319 (time 1763344938.48 total 50.03 last 64.28) - open/close 3937 (time 1763344948.49 total 60.04 last 61.72) - open/close 4562 (time 1763344958.50 total 70.05 last 62.45) - open/close 5244 (time 1763344968.51 total 80.06 last 68.15) - open/close 5918 (time 1763344978.52 total 90.07 last 67.33) - open/close 6611 (time 1763344988.52 total 100.07 last 69.27) - open/close 7243 (time 1763344998.53 total 110.08 last 63.14) - open/close 7889 (time 1763345008.54 total 120.09 last 64.55) - open/close 8535 (time 1763345018.55 total 130.10 last 64.50) - open/close 9225 (time 1763345028.57 total 140.12 last 68.91) - open/close 9934 (time 1763345038.57 total 150.12 last 70.89) total: 10000 open/close in 151.07 seconds: 66.20 ops/second - unlinked 0 (time 1763345042 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 692 (time 1763345139.14 total 10.01 last 69.15) - open/close 1372 (time 1763345149.14 total 20.02 last 67.94) - open/close 2062 (time 1763345159.15 total 30.02 last 68.99) - open/close 2754 (time 1763345169.15 total 40.03 last 69.14) - open/close 3476 (time 1763345179.16 total 50.03 last 72.19) - open/close 4183 (time 1763345189.17 total 60.04 last 70.63) - open/close 4904 (time 1763345199.17 total 70.04 last 72.05) - open/close 5526 (time 1763345209.17 total 80.04 last 62.20) - open/close 6203 (time 1763345219.18 total 90.05 last 67.67) - open/close 6870 (time 1763345229.18 total 100.05 last 66.66) - open/close 7510 (time 1763345239.19 total 110.06 last 63.95) - open/close 8176 (time 1763345249.19 total 120.06 last 66.59) - open/close 8836 (time 1763345259.20 total 130.07 last 65.94) - open/close 9536 (time 1763345269.21 total 140.08 last 69.93) total: 10000 open/close in 146.61 seconds: 68.21 ops/second - unlinked 0 (time 1763345279 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 650 (time 1763345379.62 total 10.01 last 64.94) - open/close 1333 (time 1763345389.62 total 20.01 last 68.30) - open/close 2007 (time 1763345399.63 total 30.02 last 67.34) - open/close 2667 (time 1763345409.64 total 40.03 last 65.95) - open/close 3357 (time 1763345419.65 total 50.04 last 68.93) - open/close 4052 (time 1763345429.65 total 60.04 last 69.47) - open/close 4710 (time 1763345439.66 total 70.05 last 65.74) - open/close 5394 (time 1763345449.67 total 80.06 last 68.31) - open/close 5979 (time 1763345459.69 total 90.08 last 58.41) - open/close 6565 (time 1763345469.70 total 100.09 last 58.54) - open/close 7102 (time 1763345479.71 total 110.10 last 53.65) - open/close 7637 (time 1763345489.71 total 120.10 last 53.49) - open/close 8279 (time 1763345499.72 total 130.11 last 64.11) - open/close 8930 (time 1763345509.73 total 140.12 last 65.03) - open/close 9594 (time 1763345519.74 total 150.13 last 66.39) total: 10000 open/close in 157.10 seconds: 63.66 ops/second - unlinked 0 (time 1763345531 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 712 (time 1763345635.24 total 10.01 last 71.11) - open/close 1450 (time 1763345645.25 total 20.03 last 73.70) - open/close 2138 (time 1763345655.26 total 30.03 last 68.76) - open/close 2823 (time 1763345665.26 total 40.03 last 68.50) - open/close 3501 (time 1763345675.26 total 50.03 last 67.80) - open/close 4234 (time 1763345685.27 total 60.04 last 73.26) - open/close 4955 (time 1763345695.27 total 70.05 last 72.05) - open/close 5667 (time 1763345705.27 total 80.05 last 71.20) - open/close 6400 (time 1763345715.28 total 90.06 last 73.23) - open/close 7090 (time 1763345725.29 total 100.06 last 68.96) - open/close 7788 (time 1763345735.30 total 110.08 last 69.70) - open/close 8453 (time 1763345745.32 total 120.09 last 66.41) - open/close 9078 (time 1763345755.34 total 130.11 last 62.37) - open/close 9764 (time 1763345765.34 total 140.11 last 68.57) total: 10000 open/close in 143.52 seconds: 69.68 ops/second - unlinked 0 (time 1763345772 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 701 (time 1763345878.36 total 10.00 last 70.07) - open/close 1331 (time 1763345888.37 total 20.01 last 62.94) - open/close 2013 (time 1763345898.37 total 30.02 last 68.16) - open/close 2706 (time 1763345908.38 total 40.03 last 69.25) - open/close 3373 (time 1763345918.39 total 50.03 last 66.67) - open/close 3978 (time 1763345928.41 total 60.06 last 60.33) - open/close 4653 (time 1763345938.42 total 70.07 last 67.43) - open/close 5346 (time 1763345948.43 total 80.08 last 69.25) - open/close 6042 (time 1763345958.44 total 90.09 last 69.54) - open/close 6735 (time 1763345968.44 total 100.09 last 69.28) - open/close 7314 (time 1763345978.44 total 110.09 last 57.90) - open/close 7971 (time 1763345988.45 total 120.10 last 65.64) - open/close 8693 (time 1763345998.46 total 130.11 last 72.15) - open/close 9361 (time 1763346008.47 total 140.12 last 66.72) total: 10000 open/close in 149.58 seconds: 66.85 ops/second - unlinked 0 (time 1763346021 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 680 (time 1763346122.83 total 10.02 last 67.84) - open/close 1384 (time 1763346132.84 total 20.03 last 70.34) - open/close 2032 (time 1763346142.84 total 30.03 last 64.79) - open/close 2714 (time 1763346152.84 total 40.04 last 68.19) - open/close 3326 (time 1763346162.85 total 50.05 last 61.14) - open/close 3929 (time 1763346172.85 total 60.05 last 60.30) - open/close 4553 (time 1763346182.86 total 70.05 last 62.36) - open/close 5219 (time 1763346192.86 total 80.05 last 66.59) - open/close 5877 (time 1763346202.87 total 90.06 last 65.77) - open/close 6499 (time 1763346212.87 total 100.06 last 62.20) - open/close 7190 (time 1763346222.87 total 110.07 last 69.05) - open/close 7872 (time 1763346232.87 total 120.07 last 68.19) - open/close 8573 (time 1763346242.88 total 130.07 last 70.07) - open/close 9283 (time 1763346252.89 total 140.08 last 70.91) - open/close 9909 (time 1763346262.89 total 150.09 last 62.58) total: 10000 open/close in 151.28 seconds: 66.10 ops/second - unlinked 0 (time 1763346267 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 693 (time 1763346374.29 total 10.01 last 69.22) - open/close 1357 (time 1763346384.30 total 20.02 last 66.36) - open/close 2043 (time 1763346394.30 total 30.02 last 68.58) - open/close 2688 (time 1763346404.32 total 40.04 last 64.39) - open/close 3349 (time 1763346414.33 total 50.05 last 66.04) - open/close 4045 (time 1763346424.34 total 60.06 last 69.48) - open/close 4808 (time 1763346434.35 total 70.07 last 76.22) - open/close 5525 (time 1763346444.37 total 80.09 last 71.60) - open/close 6111 (time 1763346454.39 total 90.11 last 58.49) - open/close 6742 (time 1763346464.40 total 100.12 last 63.03) - open/close 7427 (time 1763346474.40 total 110.12 last 68.50) - open/close 8100 (time 1763346484.41 total 120.13 last 67.22) - open/close 8791 (time 1763346494.42 total 130.14 last 69.00) - open/close 9474 (time 1763346504.44 total 140.16 last 68.20) total: 10000 open/close in 148.18 seconds: 67.49 ops/second - unlinked 0 (time 1763346516 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg258-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg258-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.158@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.158@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg258-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg258-server: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg258-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg258-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg258-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.158@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.158@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.158@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.158@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.158@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg258-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg258-server: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg258-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg258-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg258-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server stop mds service on oleg258-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg258-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing unload_modules_local oleg258-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg258-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2