== conf-sanity test 69: replace an OST with the same index ========================================================== 19:15:57 (1787440557) start mds service on oleg604-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg604-server' oleg604-server: oleg604-server.virtnet: executing load_modules_local oleg604-server: Loading modules from /home/green/git/lustre-release/lustre oleg604-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg604-server: detected 4 online CPUs by sysfs oleg604-server: MODOPTS_LIBCFS= oleg604-server: Force libcfs to create 2 CPU partitions oleg604-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg604-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg604-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg604-server: oleg604-server.virtnet: executing set_default_debug -1 all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg604-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg604-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug -1 all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0000 oleg604-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 650 (time 1787440647.95 total 10.02 last 64.85) - open/close 1292 (time 1787440657.96 total 20.03 last 64.13) - open/close 1866 (time 1787440667.97 total 30.05 last 57.33) - open/close 2520 (time 1787440677.98 total 40.05 last 65.35) - open/close 3170 (time 1787440687.98 total 50.06 last 64.98) - open/close 3824 (time 1787440697.98 total 60.06 last 65.40) - open/close 4476 (time 1787440707.99 total 70.06 last 65.16) - open/close 5107 (time 1787440718.00 total 80.07 last 63.05) - open/close 5750 (time 1787440728.01 total 90.09 last 64.18) - open/close 6438 (time 1787440738.02 total 100.10 last 68.73) - open/close 7101 (time 1787440748.03 total 110.11 last 66.24) - open/close 7788 (time 1787440758.05 total 120.13 last 68.58) - open/close 8495 (time 1787440768.06 total 130.14 last 70.61) - open/close 9234 (time 1787440778.07 total 140.14 last 73.87) - open/close 9960 (time 1787440788.07 total 150.14 last 72.60) total: 10000 open/close in 150.63 seconds: 66.39 ops/second - unlinked 0 (time 1787440792 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 609 (time 1787440895.79 total 10.00 last 60.87) - open/close 1200 (time 1787440905.80 total 20.01 last 59.05) - open/close 1736 (time 1787440915.80 total 30.02 last 53.58) - open/close 2426 (time 1787440925.80 total 40.02 last 68.98) - open/close 3071 (time 1787440935.81 total 50.03 last 64.43) - open/close 3758 (time 1787440945.82 total 60.04 last 68.64) - open/close 4478 (time 1787440955.84 total 70.06 last 71.87) - open/close 5151 (time 1787440965.84 total 80.06 last 67.28) - open/close 5878 (time 1787440975.86 total 90.07 last 72.61) - open/close 6517 (time 1787440985.86 total 100.08 last 63.87) - open/close 7198 (time 1787440995.86 total 110.08 last 68.09) - open/close 7875 (time 1787441005.86 total 120.08 last 67.69) - open/close 8562 (time 1787441015.87 total 130.09 last 68.65) - open/close 9142 (time 1787441025.88 total 140.09 last 57.96) - open/close 9730 (time 1787441035.89 total 150.11 last 58.73) total: 10000 open/close in 154.29 seconds: 64.81 ops/second - unlinked 0 (time 1787441043 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 744 (time 1787441143.81 total 10.00 last 74.39) - open/close 1458 (time 1787441153.82 total 20.01 last 71.37) - open/close 2178 (time 1787441163.82 total 30.01 last 71.99) - open/close 2925 (time 1787441173.84 total 40.03 last 74.51) - open/close 3652 (time 1787441183.85 total 50.04 last 72.62) - open/close 4284 (time 1787441193.86 total 60.05 last 63.18) - open/close 4924 (time 1787441203.86 total 70.05 last 63.98) - open/close 5623 (time 1787441213.87 total 80.06 last 69.84) - open/close 6315 (time 1787441223.90 total 90.09 last 69.02) - open/close 7005 (time 1787441233.90 total 100.09 last 68.95) - open/close 7611 (time 1787441243.91 total 110.10 last 60.55) - open/close 8327 (time 1787441253.92 total 120.11 last 71.56) - open/close 8973 (time 1787441263.92 total 130.11 last 64.58) - open/close 9621 (time 1787441273.93 total 140.12 last 64.74) total: 10000 open/close in 145.61 seconds: 68.68 ops/second - unlinked 0 (time 1787441283 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 639 (time 1787441389.19 total 10.01 last 63.85) - open/close 1299 (time 1787441399.19 total 20.01 last 65.99) - open/close 1998 (time 1787441409.20 total 30.02 last 69.84) - open/close 2765 (time 1787441419.21 total 40.03 last 76.60) - open/close 3540 (time 1787441429.21 total 50.03 last 77.50) - open/close 4294 (time 1787441439.22 total 60.04 last 75.32) - open/close 5031 (time 1787441449.23 total 70.05 last 73.66) - open/close 5786 (time 1787441459.24 total 80.06 last 75.44) - open/close 6541 (time 1787441469.25 total 90.07 last 75.40) - open/close 7275 (time 1787441479.25 total 100.07 last 73.40) - open/close 7969 (time 1787441489.26 total 110.08 last 69.35) - open/close 8737 (time 1787441499.27 total 120.09 last 76.74) - open/close 9509 (time 1787441509.28 total 130.10 last 77.11) total: 10000 open/close in 136.74 seconds: 73.13 ops/second - unlinked 0 (time 1787441519 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 750 (time 1787441613.50 total 10.00 last 75.00) - open/close 1486 (time 1787441623.50 total 20.00 last 73.59) - open/close 2259 (time 1787441633.51 total 30.01 last 77.27) - open/close 2969 (time 1787441643.52 total 40.01 last 70.95) - open/close 3743 (time 1787441653.52 total 50.01 last 77.39) - open/close 4470 (time 1787441663.53 total 60.02 last 72.62) - open/close 5221 (time 1787441673.54 total 70.03 last 75.04) - open/close 5964 (time 1787441683.54 total 80.03 last 74.28) - open/close 6738 (time 1787441693.55 total 90.05 last 77.28) - open/close 7519 (time 1787441703.56 total 100.06 last 78.04) - open/close 8220 (time 1787441713.58 total 110.07 last 70.00) - open/close 8884 (time 1787441723.59 total 120.08 last 66.32) - open/close 9592 (time 1787441733.61 total 130.10 last 70.67) total: 10000 open/close in 136.59 seconds: 73.21 ops/second - unlinked 0 (time 1787441746 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 734 (time 1787441849.78 total 10.00 last 73.40) - open/close 1491 (time 1787441859.79 total 20.01 last 75.63) - open/close 2117 (time 1787441869.81 total 30.03 last 62.46) - open/close 2790 (time 1787441879.82 total 40.05 last 67.21) - open/close 3510 (time 1787441889.83 total 50.06 last 71.94) - open/close 4232 (time 1787441899.84 total 60.07 last 72.13) - open/close 4931 (time 1787441909.84 total 70.07 last 69.89) - open/close 5708 (time 1787441919.85 total 80.07 last 77.68) - open/close 6427 (time 1787441929.85 total 90.07 last 71.88) - open/close 7177 (time 1787441939.87 total 100.09 last 74.88) - open/close 7911 (time 1787441949.87 total 110.09 last 73.37) - open/close 8642 (time 1787441959.87 total 120.10 last 73.07) - open/close 9373 (time 1787441969.89 total 130.11 last 72.99) total: 10000 open/close in 138.85 seconds: 72.02 ops/second - unlinked 0 (time 1787441982 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 617 (time 1787442087.80 total 10.00 last 61.68) - open/close 1329 (time 1787442097.81 total 20.01 last 71.12) - open/close 2098 (time 1787442107.82 total 30.02 last 76.83) - open/close 2763 (time 1787442117.84 total 40.04 last 66.39) - open/close 3495 (time 1787442127.86 total 50.06 last 73.03) - open/close 4235 (time 1787442137.87 total 60.07 last 73.92) - open/close 5019 (time 1787442147.88 total 70.08 last 78.37) - open/close 5718 (time 1787442157.90 total 80.11 last 69.71) - open/close 6381 (time 1787442167.91 total 90.11 last 66.27) - open/close 7112 (time 1787442177.91 total 100.12 last 73.05) - open/close 7833 (time 1787442187.92 total 110.12 last 72.10) - open/close 8592 (time 1787442197.92 total 120.12 last 75.84) - open/close 9341 (time 1787442207.93 total 130.13 last 74.82) total: 10000 open/close in 138.99 seconds: 71.95 ops/second - unlinked 0 (time 1787442220 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 746 (time 1787442318.96 total 10.01 last 74.51) - open/close 1492 (time 1787442328.97 total 20.01 last 74.58) - open/close 2215 (time 1787442338.97 total 30.02 last 72.25) - open/close 2943 (time 1787442348.98 total 40.03 last 72.78) - open/close 3683 (time 1787442358.98 total 50.03 last 73.95) - open/close 4410 (time 1787442368.99 total 60.03 last 72.68) - open/close 5131 (time 1787442378.99 total 70.04 last 72.08) - open/close 5858 (time 1787442388.99 total 80.04 last 72.70) - open/close 6615 (time 1787442398.99 total 90.04 last 75.68) - open/close 7380 (time 1787442409.00 total 100.04 last 76.47) - open/close 8134 (time 1787442419.00 total 110.05 last 75.34) - open/close 8828 (time 1787442429.01 total 120.06 last 69.35) - open/close 9579 (time 1787442439.03 total 130.08 last 74.98) total: 10000 open/close in 135.85 seconds: 73.61 ops/second - unlinked 0 (time 1787442448 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 623 (time 1787442555.30 total 10.01 last 62.23) - open/close 1289 (time 1787442565.31 total 20.02 last 66.55) - open/close 1931 (time 1787442575.34 total 30.05 last 64.01) - open/close 2577 (time 1787442585.35 total 40.06 last 64.51) - open/close 3285 (time 1787442595.37 total 50.09 last 70.64) - open/close 4029 (time 1787442605.38 total 60.10 last 74.32) - open/close 4713 (time 1787442615.38 total 70.10 last 68.40) - open/close 5283 (time 1787442625.40 total 80.11 last 56.93) - open/close 5993 (time 1787442635.41 total 90.12 last 70.91) - open/close 6677 (time 1787442645.42 total 100.13 last 68.31) - open/close 7386 (time 1787442655.45 total 110.16 last 70.70) - open/close 8103 (time 1787442665.45 total 120.16 last 71.70) - open/close 8828 (time 1787442675.45 total 130.17 last 72.47) - open/close 9542 (time 1787442685.46 total 140.17 last 71.36) total: 10000 open/close in 147.31 seconds: 67.88 ops/second - unlinked 0 (time 1787442698 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 717 (time 1787442798.63 total 10.00 last 71.69) - open/close 1404 (time 1787442808.63 total 20.01 last 68.67) - open/close 2183 (time 1787442818.63 total 30.01 last 77.89) - open/close 2967 (time 1787442828.64 total 40.02 last 78.31) - open/close 3651 (time 1787442838.65 total 50.03 last 68.33) - open/close 4370 (time 1787442848.66 total 60.04 last 71.85) - open/close 4954 (time 1787442858.67 total 70.04 last 58.37) - open/close 5589 (time 1787442868.67 total 80.04 last 63.48) - open/close 6193 (time 1787442878.67 total 90.05 last 60.38) - open/close 6813 (time 1787442888.67 total 100.05 last 61.99) - open/close 7386 (time 1787442898.68 total 110.05 last 57.29) - open/close 7945 (time 1787442908.68 total 120.05 last 55.89) - open/close 8680 (time 1787442918.69 total 130.06 last 73.44) - open/close 9285 (time 1787442928.70 total 140.07 last 60.45) - open/close 9847 (time 1787442938.71 total 150.09 last 56.11) total: 10000 open/close in 152.89 seconds: 65.41 ops/second - unlinked 0 (time 1787442946 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg604-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg604-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg604-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.104@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.104@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg604-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug -1 all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0000 oleg604-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg604-server: oleg604-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg604-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre On OST0, 14198 used inodes rc=0 pdsh@oleg604-client: oleg604-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg604-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg604-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg604-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.104@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.104@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.104@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.104@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.104@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg604-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug -1 all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0000 oleg604-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg604-server: oleg604-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg604-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg604-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg604-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg604-server stop mds service on oleg604-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg604-server unloading modules via unload_modules_local on: 'oleg604-server' oleg604-server: oleg604-server.virtnet: executing unload_modules_local oleg604-server: modules unloaded. pdsh@oleg604-client: oleg604-server: ssh exited with exit code 2 pdsh@oleg604-client: oleg604-client: ssh exited with exit code 2 pdsh@oleg604-client: oleg604-server: ssh exited with exit code 2 pdsh@oleg604-client: oleg604-client: ssh exited with exit code 2