== conf-sanity test 69: replace an OST with the same index ========================================================== 17:27:39 (1786310859) start mds service on oleg246-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg246-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg246-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg246-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 570 (time 1786310960.38 total 10.00 last 56.97) - open/close 1072 (time 1786310970.38 total 20.01 last 50.19) - open/close 1552 (time 1786310980.41 total 30.03 last 47.87) - open/close 2208 (time 1786310990.41 total 40.04 last 65.56) - open/close 2988 (time 1786311000.42 total 50.05 last 77.96) - open/close 3728 (time 1786311010.43 total 60.05 last 73.96) - open/close 4391 (time 1786311020.43 total 70.06 last 66.26) - open/close 5072 (time 1786311030.43 total 80.06 last 68.10) - open/close 5751 (time 1786311040.44 total 90.06 last 67.87) - open/close 6573 (time 1786311050.45 total 100.07 last 82.10) - open/close 7357 (time 1786311060.46 total 110.08 last 78.33) - open/close 8041 (time 1786311070.46 total 120.09 last 68.36) - open/close 8716 (time 1786311080.47 total 130.09 last 67.48) - open/close 9422 (time 1786311090.47 total 140.10 last 70.54) total: 10000 open/close in 148.63 seconds: 67.28 ops/second - unlinked 0 (time 1786311104 ; total 1 ; last 1) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 614 (time 1786311213.16 total 10.03 last 61.20) - open/close 1219 (time 1786311223.18 total 20.05 last 60.37) - open/close 1828 (time 1786311233.19 total 30.06 last 60.85) - open/close 2489 (time 1786311243.19 total 40.07 last 66.07) - open/close 3141 (time 1786311253.20 total 50.07 last 65.17) - open/close 3727 (time 1786311263.21 total 60.08 last 58.53) - open/close 4487 (time 1786311273.21 total 70.09 last 75.96) - open/close 5183 (time 1786311283.22 total 80.09 last 69.57) - open/close 5888 (time 1786311293.23 total 90.11 last 70.42) - open/close 6581 (time 1786311303.23 total 100.11 last 69.29) - open/close 7322 (time 1786311313.23 total 110.11 last 74.07) - open/close 7955 (time 1786311323.26 total 120.14 last 63.12) - open/close 8648 (time 1786311333.27 total 130.15 last 69.26) - open/close 9272 (time 1786311343.29 total 140.17 last 62.26) - open/close 9865 (time 1786311353.30 total 150.18 last 59.22) total: 10000 open/close in 152.04 seconds: 65.77 ops/second - unlinked 0 (time 1786311360 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 657 (time 1786311470.39 total 10.00 last 65.68) - open/close 1244 (time 1786311480.40 total 20.01 last 58.67) - open/close 1961 (time 1786311490.42 total 30.03 last 71.58) - open/close 2514 (time 1786311500.44 total 40.05 last 55.19) - open/close 3217 (time 1786311510.45 total 50.06 last 70.19) - open/close 3802 (time 1786311520.46 total 60.07 last 58.43) - open/close 4546 (time 1786311530.47 total 70.08 last 74.36) - open/close 5169 (time 1786311540.50 total 80.11 last 62.11) - open/close 5757 (time 1786311550.51 total 90.12 last 58.76) - open/close 6395 (time 1786311560.52 total 100.13 last 63.69) - open/close 7088 (time 1786311570.55 total 110.16 last 69.10) - open/close 7613 (time 1786311580.56 total 120.17 last 52.46) - open/close 8257 (time 1786311590.57 total 130.17 last 64.37) - open/close 8915 (time 1786311600.58 total 140.19 last 65.70) - open/close 9577 (time 1786311610.59 total 150.20 last 66.12) total: 10000 open/close in 156.82 seconds: 63.77 ops/second - unlinked 0 (time 1786311622 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 706 (time 1786311735.95 total 10.00 last 70.57) - open/close 1392 (time 1786311745.96 total 20.01 last 68.55) - open/close 2101 (time 1786311755.96 total 30.02 last 70.87) - open/close 2781 (time 1786311765.96 total 40.02 last 67.97) - open/close 3452 (time 1786311775.97 total 50.03 last 67.06) - open/close 4089 (time 1786311785.98 total 60.04 last 63.63) - open/close 4796 (time 1786311795.98 total 70.04 last 70.70) - open/close 5411 (time 1786311805.98 total 80.04 last 61.49) - open/close 6029 (time 1786311816.00 total 90.05 last 61.71) - open/close 6655 (time 1786311826.01 total 100.07 last 62.51) - open/close 7296 (time 1786311836.02 total 110.08 last 64.04) - open/close 7698 (time 1786311846.03 total 120.09 last 40.16) - open/close 8233 (time 1786311856.04 total 130.09 last 53.49) - open/close 8957 (time 1786311866.04 total 140.10 last 72.37) - open/close 9619 (time 1786311876.05 total 150.10 last 66.14) total: 10000 open/close in 156.17 seconds: 64.03 ops/second - unlinked 0 (time 1786311887 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 452 (time 1786311995.26 total 10.00 last 45.20) - open/close 987 (time 1786312005.26 total 20.01 last 53.48) - open/close 1534 (time 1786312015.29 total 30.03 last 54.56) - open/close 2100 (time 1786312025.30 total 40.04 last 56.55) - open/close 2677 (time 1786312035.30 total 50.04 last 57.69) - open/close 3310 (time 1786312045.31 total 60.05 last 63.25) - open/close 3970 (time 1786312055.33 total 70.07 last 65.85) - open/close 4536 (time 1786312065.34 total 80.08 last 56.55) - open/close 5231 (time 1786312075.36 total 90.10 last 69.40) - open/close 5870 (time 1786312085.36 total 100.10 last 63.86) - open/close 6550 (time 1786312095.36 total 110.10 last 67.98) - open/close 7160 (time 1786312105.37 total 120.11 last 60.97) - open/close 7733 (time 1786312115.37 total 130.11 last 57.28) - open/close 8357 (time 1786312125.38 total 140.12 last 62.38) - open/close 9073 (time 1786312135.39 total 150.13 last 71.51) - open/close 9744 (time 1786312145.42 total 160.16 last 66.92) total: 10000 open/close in 164.62 seconds: 60.75 ops/second - unlinked 0 (time 1786312155 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 694 (time 1786312268.08 total 10.00 last 69.38) - open/close 1284 (time 1786312278.09 total 20.01 last 58.97) - open/close 1961 (time 1786312288.10 total 30.02 last 67.65) - open/close 2653 (time 1786312298.10 total 40.02 last 69.16) - open/close 3427 (time 1786312308.10 total 50.02 last 77.39) - open/close 4191 (time 1786312318.11 total 60.03 last 76.36) - open/close 4904 (time 1786312328.12 total 70.04 last 71.20) - open/close 5516 (time 1786312338.13 total 80.05 last 61.15) - open/close 6181 (time 1786312348.13 total 90.05 last 66.49) - open/close 6710 (time 1786312358.14 total 100.06 last 52.83) - open/close 7223 (time 1786312368.17 total 110.08 last 51.19) - open/close 7764 (time 1786312378.17 total 120.08 last 54.10) - open/close 8410 (time 1786312388.18 total 130.10 last 64.50) - open/close 9029 (time 1786312398.19 total 140.11 last 61.81) - open/close 9662 (time 1786312408.19 total 150.11 last 63.30) total: 10000 open/close in 154.67 seconds: 64.65 ops/second - unlinked 0 (time 1786312417 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 618 (time 1786312532.83 total 10.01 last 61.73) - open/close 1301 (time 1786312542.85 total 20.04 last 68.12) - open/close 1826 (time 1786312552.89 total 30.07 last 52.33) - open/close 2439 (time 1786312562.89 total 40.07 last 61.28) - open/close 3018 (time 1786312572.90 total 50.09 last 57.81) - open/close 3732 (time 1786312582.91 total 60.09 last 71.36) - open/close 4526 (time 1786312592.91 total 70.09 last 79.39) - open/close 5159 (time 1786312602.91 total 80.10 last 63.29) - open/close 5625 (time 1786312612.91 total 90.10 last 46.59) - open/close 6236 (time 1786312622.93 total 100.11 last 61.02) - open/close 6847 (time 1786312632.94 total 110.12 last 61.03) - open/close 7435 (time 1786312642.95 total 120.13 last 58.77) - open/close 8141 (time 1786312652.95 total 130.14 last 70.55) - open/close 8833 (time 1786312662.98 total 140.17 last 68.98) - open/close 9486 (time 1786312673.00 total 150.18 last 65.23) total: 10000 open/close in 157.78 seconds: 63.38 ops/second - unlinked 0 (time 1786312686 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 664 (time 1786312795.93 total 10.00 last 66.37) - open/close 1373 (time 1786312805.94 total 20.01 last 70.84) - open/close 1951 (time 1786312815.95 total 30.03 last 57.71) - open/close 2615 (time 1786312825.98 total 40.05 last 66.25) - open/close 3203 (time 1786312835.99 total 50.07 last 58.71) - open/close 3832 (time 1786312846.00 total 60.08 last 62.84) - open/close 4460 (time 1786312856.01 total 70.08 last 62.76) - open/close 5167 (time 1786312866.01 total 80.09 last 70.64) - open/close 5888 (time 1786312876.04 total 90.12 last 71.92) - open/close 6573 (time 1786312886.05 total 100.13 last 68.41) - open/close 7259 (time 1786312896.05 total 110.13 last 68.60) - open/close 8021 (time 1786312906.07 total 120.14 last 76.09) - open/close 8693 (time 1786312916.07 total 130.15 last 67.19) - open/close 9347 (time 1786312926.09 total 140.17 last 65.25) total: 10000 open/close in 149.56 seconds: 66.86 ops/second - unlinked 0 (time 1786312940 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 591 (time 1786313048.81 total 10.01 last 59.04) - open/close 1349 (time 1786313058.82 total 20.01 last 75.77) - open/close 2122 (time 1786313068.82 total 30.02 last 77.28) - open/close 2871 (time 1786313078.83 total 40.02 last 74.87) - open/close 3574 (time 1786313088.83 total 50.03 last 70.23) - open/close 4305 (time 1786313098.84 total 60.04 last 73.04) - open/close 4984 (time 1786313108.85 total 70.05 last 67.84) - open/close 5688 (time 1786313118.86 total 80.06 last 70.32) - open/close 6405 (time 1786313128.87 total 90.07 last 71.63) - open/close 7036 (time 1786313138.87 total 100.07 last 63.09) - open/close 7824 (time 1786313148.89 total 110.08 last 78.68) - open/close 8526 (time 1786313158.90 total 120.10 last 70.12) - open/close 9136 (time 1786313168.92 total 130.12 last 60.88) - open/close 9820 (time 1786313178.93 total 140.12 last 68.37) total: 10000 open/close in 142.58 seconds: 70.13 ops/second - unlinked 0 (time 1786313186 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 688 (time 1786313288.17 total 10.01 last 68.70) - open/close 1409 (time 1786313298.17 total 20.02 last 72.05) - open/close 2112 (time 1786313308.19 total 30.03 last 70.21) - open/close 2697 (time 1786313318.19 total 40.04 last 58.47) - open/close 3327 (time 1786313328.20 total 50.05 last 62.94) - open/close 4042 (time 1786313338.21 total 60.05 last 71.45) - open/close 4841 (time 1786313348.22 total 70.07 last 79.81) - open/close 5612 (time 1786313358.23 total 80.07 last 77.04) - open/close 6350 (time 1786313368.23 total 90.07 last 73.79) - open/close 7072 (time 1786313378.24 total 100.08 last 72.13) - open/close 7739 (time 1786313388.24 total 110.09 last 66.64) - open/close 8316 (time 1786313398.25 total 120.10 last 57.68) - open/close 9061 (time 1786313408.26 total 130.11 last 74.42) - open/close 9807 (time 1786313418.27 total 140.12 last 74.52) total: 10000 open/close in 142.66 seconds: 70.10 ops/second - unlinked 0 (time 1786313424 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.146@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.146@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg246-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg246-server: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg246-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre On OST0, 12430 used inodes rc=0 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.146@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.146@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.146@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.146@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.202.146@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg246-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg246-server: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg246-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local oleg246-server: modules unloaded. pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2