== conf-sanity test 69: replace an OST with the same index ========================================================== 21:10:04 (1786756204) start mds service on oleg417-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg417-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg417-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 552 (time 1786756290.07 total 10.01 last 55.15) - open/close 1164 (time 1786756300.09 total 20.03 last 61.05) - open/close 1783 (time 1786756310.10 total 30.03 last 61.89) - open/close 2456 (time 1786756320.10 total 40.04 last 67.28) - open/close 3008 (time 1786756330.10 total 50.04 last 55.20) - open/close 3589 (time 1786756340.10 total 60.04 last 58.10) - open/close 4202 (time 1786756350.10 total 70.04 last 61.30) - open/close 4862 (time 1786756360.11 total 80.05 last 65.96) - open/close 5508 (time 1786756370.11 total 90.05 last 64.58) - open/close 6186 (time 1786756380.12 total 100.06 last 67.75) - open/close 6846 (time 1786756390.12 total 110.06 last 65.95) - open/close 7532 (time 1786756400.13 total 120.07 last 68.58) - open/close 8196 (time 1786756410.14 total 130.08 last 66.33) - open/close 8837 (time 1786756420.14 total 140.08 last 64.08) - open/close 9469 (time 1786756430.15 total 150.09 last 63.15) total: 10000 open/close in 158.42 seconds: 63.12 ops/second - unlinked 0 (time 1786756442 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 599 (time 1786756554.76 total 10.00 last 59.88) - open/close 1204 (time 1786756564.78 total 20.02 last 60.40) - open/close 1769 (time 1786756574.78 total 30.03 last 56.45) - open/close 2362 (time 1786756584.79 total 40.04 last 59.24) - open/close 2871 (time 1786756594.82 total 50.06 last 50.78) - open/close 3332 (time 1786756604.82 total 60.07 last 46.07) - open/close 3961 (time 1786756614.83 total 70.08 last 62.84) - open/close 4620 (time 1786756624.87 total 80.12 last 65.64) - open/close 5257 (time 1786756634.88 total 90.12 last 63.69) - open/close 5869 (time 1786756644.88 total 100.13 last 61.17) - open/close 6515 (time 1786756654.88 total 110.13 last 64.60) - open/close 7109 (time 1786756664.89 total 120.13 last 59.36) - open/close 7769 (time 1786756674.89 total 130.13 last 65.99) - open/close 8423 (time 1786756684.90 total 140.14 last 65.33) - open/close 9030 (time 1786756694.90 total 150.15 last 60.68) - open/close 9627 (time 1786756704.91 total 160.16 last 59.65) total: 10000 open/close in 166.63 seconds: 60.01 ops/second - unlinked 0 (time 1786756716 ; total 1 ; last 1) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 389 (time 1786756825.26 total 10.02 last 38.83) - open/close 824 (time 1786756835.28 total 20.03 last 43.43) - open/close 1404 (time 1786756845.29 total 30.05 last 57.94) - open/close 2082 (time 1786756855.30 total 40.05 last 67.76) - open/close 2768 (time 1786756865.30 total 50.06 last 68.56) - open/close 3419 (time 1786756875.32 total 60.07 last 64.99) - open/close 4103 (time 1786756885.35 total 70.10 last 68.19) - open/close 4791 (time 1786756895.35 total 80.11 last 68.79) - open/close 5536 (time 1786756905.38 total 90.14 last 74.25) - open/close 6224 (time 1786756915.40 total 100.16 last 68.68) - open/close 6799 (time 1786756925.41 total 110.16 last 57.47) - open/close 7433 (time 1786756935.41 total 120.16 last 63.38) - open/close 8075 (time 1786756945.42 total 130.18 last 64.12) - open/close 8696 (time 1786756955.44 total 140.19 last 61.99) - open/close 9299 (time 1786756965.47 total 150.23 last 60.10) - open/close 9941 (time 1786756975.48 total 160.23 last 64.19) total: 10000 open/close in 161.01 seconds: 62.11 ops/second - unlinked 0 (time 1786756980 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 668 (time 1786757099.84 total 10.01 last 66.77) - open/close 1378 (time 1786757109.85 total 20.01 last 70.94) - open/close 2094 (time 1786757119.86 total 30.02 last 71.52) - open/close 2739 (time 1786757129.87 total 40.03 last 64.44) - open/close 3398 (time 1786757139.87 total 50.04 last 65.86) - open/close 4124 (time 1786757149.88 total 60.04 last 72.59) - open/close 4868 (time 1786757159.88 total 70.04 last 74.40) - open/close 5591 (time 1786757169.88 total 80.05 last 72.24) - open/close 6168 (time 1786757179.89 total 90.06 last 57.64) - open/close 6760 (time 1786757189.92 total 100.08 last 59.06) - open/close 7395 (time 1786757199.92 total 110.09 last 63.49) - open/close 8121 (time 1786757209.92 total 120.09 last 72.57) - open/close 8895 (time 1786757219.94 total 130.10 last 77.32) - open/close 9518 (time 1786757229.95 total 140.12 last 62.19) total: 10000 open/close in 147.20 seconds: 67.93 ops/second - unlinked 0 (time 1786757242 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 576 (time 1786757365.41 total 10.01 last 57.56) - open/close 1238 (time 1786757375.42 total 20.01 last 66.17) - open/close 1880 (time 1786757385.42 total 30.02 last 64.15) - open/close 2528 (time 1786757395.43 total 40.03 last 64.75) - open/close 3180 (time 1786757405.44 total 50.04 last 65.14) - open/close 3767 (time 1786757415.45 total 60.05 last 58.63) - open/close 4218 (time 1786757425.47 total 70.07 last 45.01) - open/close 4683 (time 1786757435.50 total 80.09 last 46.39) - open/close 5241 (time 1786757445.51 total 90.11 last 55.70) - open/close 5858 (time 1786757455.51 total 100.11 last 61.70) - open/close 6488 (time 1786757465.52 total 110.12 last 62.95) - open/close 7138 (time 1786757475.53 total 120.12 last 64.97) - open/close 7755 (time 1786757485.53 total 130.12 last 61.70) - open/close 8432 (time 1786757495.53 total 140.13 last 67.68) - open/close 9121 (time 1786757505.54 total 150.14 last 68.84) - open/close 9746 (time 1786757515.55 total 160.14 last 62.45) total: 10000 open/close in 163.70 seconds: 61.09 ops/second - unlinked 0 (time 1786757523 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 562 (time 1786757647.21 total 10.00 last 56.18) - open/close 1198 (time 1786757657.21 total 20.01 last 63.58) - open/close 1820 (time 1786757667.22 total 30.01 last 62.16) - open/close 2405 (time 1786757677.24 total 40.04 last 58.37) - open/close 2975 (time 1786757687.25 total 50.05 last 56.93) - open/close 3551 (time 1786757697.26 total 60.05 last 57.60) - open/close 4333 (time 1786757707.26 total 70.05 last 78.17) - open/close 4832 (time 1786757717.26 total 80.05 last 49.89) - open/close 5399 (time 1786757727.29 total 90.08 last 56.54) - open/close 5937 (time 1786757737.30 total 100.09 last 53.74) - open/close 6558 (time 1786757747.31 total 110.11 last 62.03) - open/close 7228 (time 1786757757.33 total 120.12 last 66.88) - open/close 7939 (time 1786757767.33 total 130.13 last 71.08) - open/close 8635 (time 1786757777.34 total 140.13 last 69.54) - open/close 9315 (time 1786757787.35 total 150.14 last 67.93) - open/close 9958 (time 1786757797.36 total 160.15 last 64.27) total: 10000 open/close in 160.62 seconds: 62.26 ops/second - unlinked 0 (time 1786757803 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 577 (time 1786757914.74 total 10.01 last 57.63) - open/close 1268 (time 1786757924.75 total 20.02 last 69.04) - open/close 2077 (time 1786757934.76 total 30.03 last 80.85) - open/close 2761 (time 1786757944.76 total 40.03 last 68.36) - open/close 3433 (time 1786757954.77 total 50.03 last 67.18) - open/close 4100 (time 1786757964.79 total 60.05 last 66.57) - open/close 4675 (time 1786757974.80 total 70.06 last 57.45) - open/close 5231 (time 1786757984.81 total 80.08 last 55.53) - open/close 5873 (time 1786757994.83 total 90.09 last 64.09) - open/close 6456 (time 1786758004.84 total 100.11 last 58.23) - open/close 6902 (time 1786758014.84 total 110.11 last 44.58) - open/close 7375 (time 1786758024.85 total 120.12 last 47.28) - open/close 7952 (time 1786758034.86 total 130.13 last 57.61) - open/close 8588 (time 1786758044.88 total 140.14 last 63.52) - open/close 9142 (time 1786758054.88 total 150.15 last 55.37) - open/close 9842 (time 1786758064.89 total 160.15 last 69.97) total: 10000 open/close in 162.38 seconds: 61.58 ops/second - unlinked 0 (time 1786758072 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 669 (time 1786758182.40 total 10.04 last 66.61) - open/close 1336 (time 1786758192.40 total 20.04 last 66.70) - open/close 1982 (time 1786758202.42 total 30.06 last 64.50) - open/close 2514 (time 1786758212.43 total 40.07 last 53.16) - open/close 3090 (time 1786758222.44 total 50.08 last 57.55) - open/close 3614 (time 1786758232.44 total 60.08 last 52.38) - open/close 4259 (time 1786758242.45 total 70.09 last 64.43) - open/close 4959 (time 1786758252.47 total 80.11 last 69.88) - open/close 5674 (time 1786758262.49 total 90.13 last 71.33) - open/close 6432 (time 1786758272.49 total 100.13 last 75.79) - open/close 6995 (time 1786758282.51 total 110.15 last 56.21) - open/close 7345 (time 1786758292.52 total 120.16 last 34.95) - open/close 7860 (time 1786758302.54 total 130.18 last 51.41) - open/close 8471 (time 1786758312.55 total 140.19 last 61.03) - open/close 9178 (time 1786758322.56 total 150.20 last 70.64) - open/close 9789 (time 1786758332.62 total 160.26 last 60.75) total: 10000 open/close in 164.83 seconds: 60.67 ops/second - unlinked 0 (time 1786758341 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 587 (time 1786758452.88 total 10.03 last 58.54) - open/close 1289 (time 1786758462.88 total 20.03 last 70.15) - open/close 1965 (time 1786758472.89 total 30.04 last 67.55) - open/close 2635 (time 1786758482.90 total 40.05 last 66.94) - open/close 3293 (time 1786758492.90 total 50.05 last 65.79) - open/close 3868 (time 1786758502.93 total 60.08 last 57.35) - open/close 4594 (time 1786758512.93 total 70.08 last 72.55) - open/close 5117 (time 1786758522.94 total 80.09 last 52.25) - open/close 5758 (time 1786758532.95 total 90.10 last 64.05) - open/close 6464 (time 1786758542.96 total 100.11 last 70.51) - open/close 7140 (time 1786758552.97 total 110.12 last 67.55) - open/close 7811 (time 1786758562.98 total 120.13 last 67.07) - open/close 8490 (time 1786758572.99 total 130.14 last 67.82) - open/close 9217 (time 1786758582.99 total 140.14 last 72.68) - open/close 9950 (time 1786758592.99 total 150.14 last 73.28) total: 10000 open/close in 150.91 seconds: 66.26 ops/second - unlinked 0 (time 1786758598 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 539 (time 1786758712.91 total 10.01 last 53.84) - open/close 1122 (time 1786758722.96 total 20.05 last 58.04) - open/close 1813 (time 1786758732.98 total 30.07 last 68.98) - open/close 2420 (time 1786758742.99 total 40.09 last 60.62) - open/close 2984 (time 1786758753.00 total 50.10 last 56.34) - open/close 3625 (time 1786758763.00 total 60.10 last 64.06) - open/close 4217 (time 1786758773.01 total 70.11 last 59.15) - open/close 4907 (time 1786758783.03 total 80.12 last 68.90) - open/close 5595 (time 1786758793.03 total 90.13 last 68.77) - open/close 6280 (time 1786758803.04 total 100.14 last 68.45) - open/close 6944 (time 1786758813.05 total 110.15 last 66.31) - open/close 7512 (time 1786758823.05 total 120.15 last 56.79) - open/close 8057 (time 1786758833.07 total 130.17 last 54.41) - open/close 8603 (time 1786758843.07 total 140.17 last 54.59) - open/close 9256 (time 1786758853.08 total 150.17 last 65.26) - open/close 9955 (time 1786758863.09 total 160.19 last 69.81) total: 10000 open/close in 160.96 seconds: 62.13 ops/second - unlinked 0 (time 1786758868 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.117@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.117@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local oleg417-server: modules unloaded. pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2