== conf-sanity test 69: replace an OST with the same index ========================================================== 17:53:24 (1786744404) start mds service on oleg207-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg207-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 747 (time 1786744499.62 total 10.02 last 74.55) - open/close 1360 (time 1786744509.63 total 20.03 last 61.25) - open/close 2008 (time 1786744519.65 total 30.05 last 64.67) - open/close 2809 (time 1786744529.66 total 40.05 last 80.08) - open/close 3539 (time 1786744539.66 total 50.06 last 72.95) - open/close 4232 (time 1786744549.66 total 60.06 last 69.29) - open/close 4930 (time 1786744559.68 total 70.08 last 69.68) - open/close 5613 (time 1786744569.69 total 80.08 last 68.25) - open/close 6353 (time 1786744579.69 total 90.09 last 73.95) - open/close 7066 (time 1786744589.70 total 100.10 last 71.26) - open/close 7774 (time 1786744599.71 total 110.11 last 70.71) - open/close 8496 (time 1786744609.74 total 120.13 last 72.04) - open/close 9036 (time 1786744619.74 total 130.13 last 53.98) - open/close 9713 (time 1786744629.74 total 140.14 last 67.69) total: 10000 open/close in 146.15 seconds: 68.42 ops/second - unlinked 0 (time 1786744641 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 523 (time 1786744743.00 total 10.01 last 52.26) - open/close 1354 (time 1786744753.00 total 20.01 last 83.09) - open/close 2082 (time 1786744763.01 total 30.02 last 72.72) - open/close 2774 (time 1786744773.02 total 40.03 last 69.15) - open/close 3415 (time 1786744783.05 total 50.06 last 63.87) - open/close 4092 (time 1786744793.05 total 60.07 last 67.70) - open/close 4769 (time 1786744803.06 total 70.07 last 67.64) - open/close 5475 (time 1786744813.06 total 80.08 last 70.59) - open/close 6124 (time 1786744823.06 total 90.08 last 64.89) - open/close 6743 (time 1786744833.07 total 100.08 last 61.87) - open/close 7389 (time 1786744843.10 total 110.11 last 64.42) - open/close 8117 (time 1786744853.11 total 120.13 last 72.68) - open/close 8731 (time 1786744863.13 total 130.14 last 61.34) - open/close 9356 (time 1786744873.13 total 140.15 last 62.44) - open/close 9940 (time 1786744883.15 total 150.16 last 58.31) total: 10000 open/close in 151.45 seconds: 66.03 ops/second - unlinked 0 (time 1786744889 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 475 (time 1786745003.87 total 10.02 last 47.39) - open/close 941 (time 1786745013.88 total 20.03 last 46.56) - open/close 1606 (time 1786745023.88 total 30.04 last 66.48) - open/close 2208 (time 1786745033.89 total 40.05 last 60.14) - open/close 2963 (time 1786745043.89 total 50.05 last 75.48) - open/close 3695 (time 1786745053.90 total 60.05 last 73.15) - open/close 4462 (time 1786745063.91 total 70.06 last 76.66) - open/close 5125 (time 1786745073.92 total 80.07 last 66.22) - open/close 5962 (time 1786745083.93 total 90.08 last 83.62) - open/close 6770 (time 1786745093.93 total 100.09 last 80.76) - open/close 7641 (time 1786745103.94 total 110.10 last 87.01) - open/close 8404 (time 1786745113.96 total 120.11 last 76.20) - open/close 9116 (time 1786745123.96 total 130.11 last 71.17) - open/close 9846 (time 1786745133.96 total 140.12 last 72.98) total: 10000 open/close in 142.35 seconds: 70.25 ops/second - unlinked 0 (time 1786745140 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 719 (time 1786745245.42 total 10.02 last 71.77) - open/close 1316 (time 1786745255.42 total 20.02 last 59.69) - open/close 2023 (time 1786745265.44 total 30.03 last 70.61) - open/close 2708 (time 1786745275.44 total 40.04 last 68.48) - open/close 3397 (time 1786745285.46 total 50.06 last 68.72) - open/close 4124 (time 1786745295.48 total 60.07 last 72.61) - open/close 4901 (time 1786745305.48 total 70.07 last 77.70) - open/close 5632 (time 1786745315.48 total 80.08 last 73.07) - open/close 6374 (time 1786745325.50 total 90.10 last 74.03) - open/close 6967 (time 1786745335.51 total 100.11 last 59.25) - open/close 7385 (time 1786745345.52 total 110.12 last 41.78) - open/close 8070 (time 1786745355.53 total 120.13 last 68.43) - open/close 8725 (time 1786745365.53 total 130.13 last 65.50) - open/close 9423 (time 1786745375.53 total 140.13 last 69.78) total: 10000 open/close in 148.62 seconds: 67.29 ops/second - unlinked 0 (time 1786745388 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 719 (time 1786745492.07 total 10.00 last 71.90) - open/close 1434 (time 1786745502.07 total 20.00 last 71.48) - open/close 2172 (time 1786745512.07 total 30.00 last 73.79) - open/close 2815 (time 1786745522.08 total 40.01 last 64.29) - open/close 3335 (time 1786745532.08 total 50.01 last 51.97) - open/close 3937 (time 1786745542.09 total 60.02 last 60.15) - open/close 4761 (time 1786745552.10 total 70.04 last 82.27) - open/close 5530 (time 1786745562.11 total 80.04 last 76.87) - open/close 6287 (time 1786745572.11 total 90.04 last 75.67) - open/close 7034 (time 1786745582.13 total 100.06 last 74.59) - open/close 7662 (time 1786745592.14 total 110.07 last 62.74) - open/close 8335 (time 1786745602.14 total 120.07 last 67.30) - open/close 9107 (time 1786745612.15 total 130.08 last 77.14) - open/close 9836 (time 1786745622.16 total 140.09 last 72.82) total: 10000 open/close in 142.43 seconds: 70.21 ops/second - unlinked 0 (time 1786745628 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 752 (time 1786745732.95 total 10.01 last 75.15) - open/close 1489 (time 1786745742.97 total 20.03 last 73.51) - open/close 2161 (time 1786745752.98 total 30.04 last 67.14) - open/close 2811 (time 1786745762.99 total 40.05 last 64.94) - open/close 3523 (time 1786745773.00 total 50.06 last 71.12) - open/close 4185 (time 1786745783.01 total 60.07 last 66.15) - open/close 4690 (time 1786745793.02 total 70.08 last 50.43) - open/close 5200 (time 1786745803.03 total 80.09 last 50.99) - open/close 5892 (time 1786745813.03 total 90.09 last 69.16) - open/close 6794 (time 1786745823.06 total 100.12 last 89.99) - open/close 7490 (time 1786745833.06 total 110.12 last 69.57) - open/close 7897 (time 1786745843.09 total 120.15 last 40.59) - open/close 8535 (time 1786745853.10 total 130.16 last 63.72) - open/close 9173 (time 1786745863.12 total 140.18 last 63.65) - open/close 9563 (time 1786745873.14 total 150.20 last 38.94) total: 10000 open/close in 159.66 seconds: 62.63 ops/second - unlinked 0 (time 1786745888 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 599 (time 1786745998.91 total 10.01 last 59.85) - open/close 1489 (time 1786746008.91 total 20.01 last 88.96) - open/close 2253 (time 1786746018.93 total 30.03 last 76.29) - open/close 3059 (time 1786746028.95 total 40.05 last 80.43) - open/close 3731 (time 1786746038.96 total 50.06 last 67.12) - open/close 4438 (time 1786746048.96 total 60.07 last 70.66) - open/close 5044 (time 1786746058.97 total 70.07 last 60.58) - open/close 5564 (time 1786746068.97 total 80.07 last 51.99) - open/close 6067 (time 1786746078.97 total 90.08 last 50.27) - open/close 6841 (time 1786746088.98 total 100.08 last 77.38) - open/close 7507 (time 1786746098.98 total 110.09 last 66.55) - open/close 8121 (time 1786746108.99 total 120.09 last 61.38) - open/close 8690 (time 1786746118.99 total 130.09 last 56.89) - open/close 9403 (time 1786746129.00 total 140.10 last 71.22) - open/close 9955 (time 1786746139.02 total 150.12 last 55.10) total: 10000 open/close in 150.66 seconds: 66.37 ops/second - unlinked 0 (time 1786746143 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 710 (time 1786746252.81 total 10.02 last 70.89) - open/close 1318 (time 1786746262.81 total 20.02 last 60.79) - open/close 1959 (time 1786746272.81 total 30.02 last 64.09) - open/close 2650 (time 1786746282.81 total 40.02 last 69.07) - open/close 3304 (time 1786746292.83 total 50.04 last 65.31) - open/close 3962 (time 1786746302.83 total 60.04 last 65.76) - open/close 4643 (time 1786746312.87 total 70.08 last 67.84) - open/close 5329 (time 1786746322.87 total 80.08 last 68.58) - open/close 6021 (time 1786746332.88 total 90.09 last 69.18) - open/close 6675 (time 1786746342.89 total 100.10 last 65.32) - open/close 7384 (time 1786746352.90 total 110.11 last 70.85) - open/close 8000 (time 1786746362.93 total 120.14 last 61.42) - open/close 8728 (time 1786746372.95 total 130.16 last 72.64) - open/close 9376 (time 1786746382.97 total 140.18 last 64.69) - open/close 9981 (time 1786746392.97 total 150.18 last 60.50) total: 10000 open/close in 150.40 seconds: 66.49 ops/second - unlinked 0 (time 1786746398 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 679 (time 1786746502.25 total 10.01 last 67.83) - open/close 1372 (time 1786746512.26 total 20.02 last 69.21) - open/close 2037 (time 1786746522.28 total 30.04 last 66.36) - open/close 2778 (time 1786746532.29 total 40.05 last 74.03) - open/close 3325 (time 1786746542.31 total 50.07 last 54.62) - open/close 3910 (time 1786746552.33 total 60.09 last 58.38) - open/close 4466 (time 1786746562.35 total 70.12 last 55.43) - open/close 5054 (time 1786746572.36 total 80.12 last 58.76) - open/close 5691 (time 1786746582.36 total 90.13 last 63.68) - open/close 6323 (time 1786746592.37 total 100.14 last 63.14) - open/close 7043 (time 1786746602.38 total 110.14 last 71.94) - open/close 7742 (time 1786746612.39 total 120.16 last 69.81) - open/close 8403 (time 1786746622.40 total 130.16 last 66.09) - open/close 9103 (time 1786746632.40 total 140.17 last 69.96) - open/close 9859 (time 1786746642.41 total 150.17 last 75.58) total: 10000 open/close in 151.97 seconds: 65.80 ops/second - unlinked 0 (time 1786746648 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 538 (time 1786746755.80 total 10.01 last 53.76) - open/close 1382 (time 1786746765.82 total 20.02 last 84.28) - open/close 1972 (time 1786746775.82 total 30.02 last 58.99) - open/close 2392 (time 1786746785.85 total 40.06 last 41.86) - open/close 2893 (time 1786746795.86 total 50.06 last 50.06) - open/close 3553 (time 1786746805.87 total 60.07 last 65.98) - open/close 4186 (time 1786746815.88 total 70.08 last 63.22) - open/close 4891 (time 1786746825.88 total 80.08 last 70.49) - open/close 5584 (time 1786746835.89 total 90.09 last 69.24) - open/close 6301 (time 1786746845.90 total 100.11 last 71.58) - open/close 6944 (time 1786746855.90 total 110.11 last 64.29) - open/close 7558 (time 1786746865.91 total 120.11 last 61.37) - open/close 8197 (time 1786746875.91 total 130.11 last 63.90) - open/close 8730 (time 1786746885.92 total 140.12 last 53.27) - open/close 9368 (time 1786746895.93 total 150.13 last 63.71) - open/close 9958 (time 1786746905.94 total 160.14 last 58.96) total: 10000 open/close in 160.73 seconds: 62.22 ops/second - unlinked 0 (time 1786746912 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.107@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre On OST0, 17902 used inodes rc=0 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.107@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.107@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.107@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.107@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2