== conf-sanity test 69: replace an OST with the same index ========================================================== 12:46:07 (1761237967) start mds service on oleg605-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg605-server' oleg605-server: oleg605-server.virtnet: executing load_modules_local oleg605-server: Loading modules from /home/green/git/lustre-release/lustre oleg605-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg605-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg605-server: detected 4 online CPUs by sysfs oleg605-server: MODOPTS_LIBCFS= oleg605-server: Force libcfs to create 2 CPU partitions oleg605-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg605-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg605-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 751 (time 1761238038.86 total 10.01 last 75.06) - open/close 1423 (time 1761238048.87 total 20.02 last 67.08) - open/close 2141 (time 1761238058.90 total 30.04 last 71.65) - open/close 2840 (time 1761238068.91 total 40.06 last 69.82) - open/close 3590 (time 1761238078.91 total 50.06 last 74.94) - open/close 4304 (time 1761238088.92 total 60.07 last 71.36) - open/close 5005 (time 1761238098.93 total 70.07 last 70.06) - open/close 5708 (time 1761238108.94 total 80.09 last 70.18) - open/close 6374 (time 1761238118.95 total 90.10 last 66.52) - open/close 7061 (time 1761238128.96 total 100.11 last 68.67) - open/close 7743 (time 1761238138.98 total 110.12 last 68.08) - open/close 8386 (time 1761238148.99 total 120.13 last 64.23) - open/close 9078 (time 1761238158.99 total 130.14 last 69.18) - open/close 9771 (time 1761238169.00 total 140.15 last 69.22) total: 10000 open/close in 143.60 seconds: 69.64 ops/second - unlinked 0 (time 1761238176 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 688 (time 1761238273.74 total 10.01 last 68.74) - open/close 1378 (time 1761238283.74 total 20.01 last 68.97) - open/close 2057 (time 1761238293.75 total 30.02 last 67.84) - open/close 2718 (time 1761238303.76 total 40.04 last 65.99) - open/close 3427 (time 1761238313.77 total 50.04 last 70.89) - open/close 4137 (time 1761238323.77 total 60.04 last 70.97) - open/close 4837 (time 1761238333.77 total 70.04 last 69.98) - open/close 5523 (time 1761238343.78 total 80.05 last 68.57) - open/close 6186 (time 1761238353.78 total 90.05 last 66.26) - open/close 6858 (time 1761238363.78 total 100.06 last 67.19) - open/close 7560 (time 1761238373.79 total 110.06 last 70.16) - open/close 8270 (time 1761238383.80 total 120.07 last 70.91) - open/close 8982 (time 1761238393.81 total 130.09 last 71.12) - open/close 9641 (time 1761238403.83 total 140.10 last 65.81) total: 10000 open/close in 144.96 seconds: 68.98 ops/second - unlinked 0 (time 1761238411 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 684 (time 1761238507.93 total 10.01 last 68.32) - open/close 1396 (time 1761238517.95 total 20.03 last 71.05) - open/close 2064 (time 1761238527.96 total 30.05 last 66.68) - open/close 2790 (time 1761238537.97 total 40.05 last 72.59) - open/close 3519 (time 1761238547.97 total 50.05 last 72.90) - open/close 4209 (time 1761238557.97 total 60.06 last 68.96) - open/close 4908 (time 1761238567.98 total 70.06 last 69.86) - open/close 5635 (time 1761238577.98 total 80.07 last 72.67) - open/close 6333 (time 1761238587.98 total 90.07 last 69.79) - open/close 7110 (time 1761238598.00 total 100.08 last 77.61) - open/close 7873 (time 1761238608.01 total 110.10 last 76.17) - open/close 8605 (time 1761238618.02 total 120.10 last 73.17) - open/close 9359 (time 1761238628.03 total 130.12 last 75.31) total: 10000 open/close in 138.85 seconds: 72.02 ops/second - unlinked 0 (time 1761238640 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 742 (time 1761238724.54 total 10.01 last 74.10) - open/close 1467 (time 1761238734.54 total 20.02 last 72.49) - open/close 2242 (time 1761238744.55 total 30.03 last 77.42) - open/close 3033 (time 1761238754.55 total 40.03 last 79.09) - open/close 3859 (time 1761238764.56 total 50.03 last 82.54) - open/close 4691 (time 1761238774.57 total 60.04 last 83.13) - open/close 5491 (time 1761238784.58 total 70.05 last 79.94) - open/close 6295 (time 1761238794.58 total 80.05 last 80.37) - open/close 7131 (time 1761238804.58 total 90.06 last 83.56) - open/close 8065 (time 1761238814.59 total 100.06 last 93.39) - open/close 8736 (time 1761238824.59 total 110.06 last 67.07) - open/close 9457 (time 1761238834.59 total 120.06 last 72.09) total: 10000 open/close in 127.82 seconds: 78.23 ops/second - unlinked 0 (time 1761238845 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 695 (time 1761238927.74 total 10.01 last 69.42) - open/close 1430 (time 1761238937.74 total 20.02 last 73.46) - open/close 2201 (time 1761238947.74 total 30.02 last 77.10) - open/close 3051 (time 1761238957.77 total 40.04 last 84.79) - open/close 3830 (time 1761238967.77 total 50.05 last 77.85) - open/close 4472 (time 1761238977.78 total 60.06 last 64.14) - open/close 5239 (time 1761238987.79 total 70.06 last 76.68) - open/close 6001 (time 1761238997.79 total 80.06 last 76.20) - open/close 6769 (time 1761239007.79 total 90.06 last 76.79) - open/close 7527 (time 1761239017.79 total 100.06 last 75.78) - open/close 8345 (time 1761239027.79 total 110.07 last 81.77) - open/close 9126 (time 1761239037.80 total 120.07 last 78.05) - open/close 9852 (time 1761239047.82 total 130.09 last 72.47) total: 10000 open/close in 132.27 seconds: 75.60 ops/second - unlinked 0 (time 1761239053 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 750 (time 1761239143.38 total 10.01 last 74.92) - open/close 1547 (time 1761239153.38 total 20.02 last 79.67) - open/close 2317 (time 1761239163.39 total 30.02 last 76.96) - open/close 3006 (time 1761239173.39 total 40.02 last 68.89) - open/close 3704 (time 1761239183.39 total 50.02 last 69.79) - open/close 4364 (time 1761239193.39 total 60.03 last 65.97) - open/close 5061 (time 1761239203.40 total 70.04 last 69.64) - open/close 5772 (time 1761239213.41 total 80.05 last 71.02) - open/close 6451 (time 1761239223.43 total 90.06 last 67.79) - open/close 7205 (time 1761239233.43 total 100.06 last 75.40) - open/close 7954 (time 1761239243.43 total 110.07 last 74.87) - open/close 8672 (time 1761239253.44 total 120.07 last 71.79) - open/close 9364 (time 1761239263.44 total 130.08 last 69.14) total: 10000 open/close in 138.93 seconds: 71.98 ops/second - unlinked 0 (time 1761239275 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 722 (time 1761239365.89 total 10.01 last 72.15) - open/close 1413 (time 1761239375.90 total 20.01 last 69.07) - open/close 2079 (time 1761239385.91 total 30.02 last 66.53) - open/close 2852 (time 1761239395.91 total 40.02 last 77.29) - open/close 3644 (time 1761239405.92 total 50.03 last 79.13) - open/close 4417 (time 1761239415.92 total 60.04 last 77.24) - open/close 5138 (time 1761239425.93 total 70.05 last 72.05) - open/close 5802 (time 1761239435.94 total 80.06 last 66.32) - open/close 6512 (time 1761239445.95 total 90.06 last 70.97) - open/close 7211 (time 1761239455.95 total 100.07 last 69.88) - open/close 7920 (time 1761239465.96 total 110.08 last 70.80) - open/close 8596 (time 1761239475.97 total 120.08 last 67.57) - open/close 9295 (time 1761239485.97 total 130.09 last 69.86) - open/close 9961 (time 1761239495.99 total 140.11 last 66.50) total: 10000 open/close in 140.67 seconds: 71.09 ops/second - unlinked 0 (time 1761239501 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 835 (time 1761239585.24 total 10.01 last 83.43) - open/close 1667 (time 1761239595.26 total 20.03 last 83.03) - open/close 2386 (time 1761239605.28 total 30.04 last 71.80) - open/close 3137 (time 1761239615.29 total 40.05 last 75.05) - open/close 3874 (time 1761239625.29 total 50.05 last 73.70) - open/close 4606 (time 1761239635.29 total 60.05 last 73.19) - open/close 5332 (time 1761239645.29 total 70.06 last 72.57) - open/close 6057 (time 1761239655.29 total 80.06 last 72.49) - open/close 6791 (time 1761239665.30 total 90.07 last 73.31) - open/close 7480 (time 1761239675.31 total 100.08 last 68.86) - open/close 8196 (time 1761239685.32 total 110.08 last 71.57) - open/close 8923 (time 1761239695.32 total 120.09 last 72.64) - open/close 9647 (time 1761239705.34 total 130.10 last 72.30) total: 10000 open/close in 134.74 seconds: 74.22 ops/second - unlinked 0 (time 1761239713 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 748 (time 1761239804.34 total 10.00 last 74.78) - open/close 1517 (time 1761239814.34 total 20.00 last 76.90) - open/close 2297 (time 1761239824.34 total 30.01 last 77.94) - open/close 3063 (time 1761239834.35 total 40.02 last 76.54) - open/close 3794 (time 1761239844.36 total 50.03 last 73.03) - open/close 4511 (time 1761239854.37 total 60.03 last 71.66) - open/close 5222 (time 1761239864.37 total 70.03 last 71.09) - open/close 6062 (time 1761239874.37 total 80.04 last 83.96) - open/close 6780 (time 1761239884.40 total 90.06 last 71.63) - open/close 7493 (time 1761239894.40 total 100.07 last 71.28) - open/close 8400 (time 1761239904.41 total 110.07 last 90.64) - open/close 9177 (time 1761239914.41 total 120.08 last 77.65) - open/close 9936 (time 1761239924.42 total 130.08 last 75.89) total: 10000 open/close in 130.92 seconds: 76.38 ops/second - unlinked 0 (time 1761239928 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 729 (time 1761240010.36 total 10.00 last 72.87) - open/close 1624 (time 1761240020.36 total 20.01 last 89.44) - open/close 2386 (time 1761240030.37 total 30.02 last 76.16) - open/close 3086 (time 1761240040.38 total 40.02 last 69.94) - open/close 3822 (time 1761240050.38 total 50.03 last 73.58) - open/close 4653 (time 1761240060.39 total 60.03 last 83.04) - open/close 5573 (time 1761240070.39 total 70.04 last 91.92) - open/close 6327 (time 1761240080.40 total 80.05 last 75.36) - open/close 7066 (time 1761240090.40 total 90.05 last 73.90) - open/close 7866 (time 1761240100.42 total 100.07 last 79.83) - open/close 8687 (time 1761240110.42 total 110.07 last 82.09) - open/close 9479 (time 1761240120.44 total 120.09 last 79.06) total: 10000 open/close in 127.14 seconds: 78.65 ops/second - unlinked 0 (time 1761240130 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg605-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg605-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg605-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.105@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.105@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg605-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg605-server: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg605-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre On OST0, 19477 used inodes rc=0 pdsh@oleg605-client: oleg605-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg605-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg605-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg605-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.105@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.105@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.206.105@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.206.105@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.206.105@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg605-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug -1 all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg605-server: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg605-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg605-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg605-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg605-server stop mds service on oleg605-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg605-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg605-server' oleg605-server: oleg605-server.virtnet: executing unload_modules_local oleg605-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg605-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg605-client: oleg605-client: ssh exited with exit code 2 pdsh@oleg605-client: oleg605-server: ssh exited with exit code 2 pdsh@oleg605-client: oleg605-client: ssh exited with exit code 2 pdsh@oleg605-client: oleg605-server: ssh exited with exit code 2