== conf-sanity test 69: replace an OST with the same index ========================================================== 23:48:49 (1783136929) start mds service on oleg207-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg207-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 641 (time 1783137018.55 total 10.01 last 64.06) - open/close 1325 (time 1783137028.57 total 20.03 last 68.24) - open/close 1969 (time 1783137038.58 total 30.03 last 64.36) - open/close 2569 (time 1783137048.59 total 40.04 last 59.94) - open/close 3158 (time 1783137058.59 total 50.05 last 58.89) - open/close 3760 (time 1783137068.61 total 60.07 last 60.04) - open/close 4303 (time 1783137078.61 total 70.07 last 54.30) - open/close 4919 (time 1783137088.62 total 80.08 last 61.58) - open/close 5576 (time 1783137098.63 total 90.09 last 65.62) - open/close 6227 (time 1783137108.63 total 100.09 last 65.09) - open/close 6896 (time 1783137118.64 total 110.10 last 66.82) - open/close 7583 (time 1783137128.64 total 120.10 last 68.70) - open/close 8274 (time 1783137138.66 total 130.12 last 68.97) - open/close 8862 (time 1783137148.67 total 140.13 last 58.73) - open/close 9459 (time 1783137158.68 total 150.14 last 59.65) total: 10000 open/close in 159.75 seconds: 62.60 ops/second - unlinked 0 (time 1783137173 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 661 (time 1783137290.55 total 10.01 last 66.01) - open/close 1217 (time 1783137300.55 total 20.02 last 55.58) - open/close 1849 (time 1783137310.56 total 30.03 last 63.14) - open/close 2574 (time 1783137320.56 total 40.03 last 72.46) - open/close 3211 (time 1783137330.57 total 50.04 last 63.63) - open/close 3918 (time 1783137340.59 total 60.06 last 70.59) - open/close 4612 (time 1783137350.59 total 70.06 last 69.38) - open/close 5305 (time 1783137360.60 total 80.07 last 69.27) - open/close 5968 (time 1783137370.66 total 90.13 last 65.90) - open/close 6552 (time 1783137380.67 total 100.14 last 58.35) - open/close 7118 (time 1783137390.67 total 110.14 last 56.56) - open/close 7756 (time 1783137400.68 total 120.15 last 63.74) - open/close 8396 (time 1783137410.69 total 130.16 last 63.96) - open/close 9140 (time 1783137420.70 total 140.17 last 74.29) - open/close 9827 (time 1783137430.71 total 150.18 last 68.68) total: 10000 open/close in 153.49 seconds: 65.15 ops/second - unlinked 0 (time 1783137439 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 659 (time 1783137558.24 total 10.01 last 65.87) - open/close 1385 (time 1783137568.26 total 20.02 last 72.51) - open/close 2079 (time 1783137578.26 total 30.02 last 69.35) - open/close 2780 (time 1783137588.27 total 40.04 last 70.02) - open/close 3463 (time 1783137598.28 total 50.04 last 68.26) - open/close 4032 (time 1783137608.29 total 60.05 last 56.85) - open/close 4680 (time 1783137618.30 total 70.06 last 64.74) - open/close 5374 (time 1783137628.30 total 80.07 last 69.36) - open/close 6128 (time 1783137638.31 total 90.07 last 75.38) - open/close 6759 (time 1783137648.33 total 100.09 last 62.97) - open/close 7298 (time 1783137658.34 total 110.10 last 53.84) - open/close 7944 (time 1783137668.34 total 120.10 last 64.60) - open/close 8546 (time 1783137678.35 total 130.11 last 60.14) - open/close 9293 (time 1783137688.35 total 140.12 last 74.66) - open/close 9962 (time 1783137698.36 total 150.12 last 66.87) total: 10000 open/close in 150.48 seconds: 66.46 ops/second - unlinked 0 (time 1783137703 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 584 (time 1783137819.06 total 10.00 last 58.38) - open/close 1241 (time 1783137829.07 total 20.01 last 65.67) - open/close 1943 (time 1783137839.08 total 30.02 last 70.14) - open/close 2540 (time 1783137849.08 total 40.02 last 59.68) - open/close 3256 (time 1783137859.09 total 50.03 last 71.52) - open/close 3805 (time 1783137869.12 total 60.06 last 54.75) - open/close 4518 (time 1783137879.13 total 70.07 last 71.25) - open/close 5153 (time 1783137889.13 total 80.07 last 63.44) - open/close 5786 (time 1783137899.14 total 90.08 last 63.24) - open/close 6423 (time 1783137909.17 total 100.11 last 63.56) - open/close 7026 (time 1783137919.17 total 110.11 last 60.30) - open/close 7734 (time 1783137929.17 total 120.11 last 70.77) - open/close 8247 (time 1783137939.19 total 130.13 last 51.23) - open/close 8847 (time 1783137949.19 total 140.13 last 59.96) - open/close 9535 (time 1783137959.20 total 150.14 last 68.76) total: 10000 open/close in 156.04 seconds: 64.09 ops/second - unlinked 0 (time 1783137969 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 643 (time 1783138085.79 total 10.01 last 64.26) - open/close 1160 (time 1783138095.81 total 20.03 last 51.60) - open/close 1603 (time 1783138105.82 total 30.03 last 44.27) - open/close 2029 (time 1783138115.82 total 40.04 last 42.59) - open/close 2535 (time 1783138125.83 total 50.05 last 50.54) - open/close 3279 (time 1783138135.85 total 60.06 last 74.29) - open/close 3902 (time 1783138145.85 total 70.06 last 62.29) - open/close 4608 (time 1783138155.85 total 80.07 last 70.55) - open/close 5326 (time 1783138165.86 total 90.08 last 71.75) - open/close 5988 (time 1783138175.87 total 100.09 last 66.14) - open/close 6609 (time 1783138185.89 total 110.11 last 61.97) - open/close 7236 (time 1783138195.90 total 120.12 last 62.66) - open/close 7834 (time 1783138205.91 total 130.13 last 59.73) - open/close 8518 (time 1783138215.93 total 140.15 last 68.28) - open/close 9273 (time 1783138225.93 total 150.15 last 75.46) - open/close 9947 (time 1783138235.94 total 160.16 last 67.37) total: 10000 open/close in 160.90 seconds: 62.15 ops/second - unlinked 0 (time 1783138240 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 621 (time 1783138358.55 total 10.04 last 61.87) - open/close 1130 (time 1783138368.56 total 20.04 last 50.88) - open/close 1785 (time 1783138378.56 total 30.04 last 65.49) - open/close 2568 (time 1783138388.57 total 40.05 last 78.25) - open/close 3150 (time 1783138398.57 total 50.05 last 58.20) - open/close 3799 (time 1783138408.58 total 60.06 last 64.83) - open/close 4348 (time 1783138418.58 total 70.07 last 54.86) - open/close 4941 (time 1783138428.59 total 80.07 last 59.26) - open/close 5532 (time 1783138438.60 total 90.08 last 59.05) - open/close 6187 (time 1783138448.61 total 100.09 last 65.42) - open/close 6842 (time 1783138458.61 total 110.10 last 65.48) - open/close 7561 (time 1783138468.62 total 120.11 last 71.82) - open/close 8090 (time 1783138478.64 total 130.13 last 52.79) - open/close 8622 (time 1783138488.65 total 140.13 last 53.19) - open/close 9344 (time 1783138498.65 total 150.13 last 72.18) total: 10000 open/close in 159.09 seconds: 62.86 ops/second - unlinked 0 (time 1783138512 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 602 (time 1783138628.82 total 10.00 last 60.20) - open/close 1274 (time 1783138638.83 total 20.01 last 67.16) - open/close 1858 (time 1783138648.84 total 30.02 last 58.31) - open/close 2513 (time 1783138658.84 total 40.02 last 65.49) - open/close 3248 (time 1783138668.85 total 50.03 last 73.42) - open/close 3916 (time 1783138678.87 total 60.05 last 66.69) - open/close 4642 (time 1783138688.88 total 70.06 last 72.52) - open/close 5272 (time 1783138698.89 total 80.07 last 62.97) - open/close 5808 (time 1783138708.90 total 90.08 last 53.53) - open/close 6477 (time 1783138718.91 total 100.09 last 66.86) - open/close 7158 (time 1783138728.91 total 110.09 last 68.09) - open/close 7722 (time 1783138738.91 total 120.09 last 56.37) - open/close 8344 (time 1783138748.93 total 130.11 last 62.07) - open/close 8794 (time 1783138758.94 total 140.12 last 44.97) - open/close 9431 (time 1783138768.96 total 150.15 last 63.54) total: 10000 open/close in 159.39 seconds: 62.74 ops/second - unlinked 0 (time 1783138783 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 717 (time 1783138883.18 total 10.00 last 71.69) - open/close 1428 (time 1783138893.18 total 20.01 last 71.08) - open/close 2165 (time 1783138903.19 total 30.01 last 73.69) - open/close 2911 (time 1783138913.19 total 40.01 last 74.57) - open/close 3601 (time 1783138923.20 total 50.02 last 68.95) - open/close 4367 (time 1783138933.21 total 60.03 last 76.51) - open/close 5155 (time 1783138943.22 total 70.04 last 78.70) - open/close 5901 (time 1783138953.23 total 80.05 last 74.54) - open/close 6610 (time 1783138963.23 total 90.05 last 70.87) - open/close 7394 (time 1783138973.24 total 100.06 last 78.38) - open/close 8086 (time 1783138983.26 total 110.08 last 69.07) - open/close 8842 (time 1783138993.26 total 120.08 last 75.56) - open/close 9625 (time 1783139003.27 total 130.09 last 78.24) total: 10000 open/close in 134.59 seconds: 74.30 ops/second - unlinked 0 (time 1783139012 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 744 (time 1783139111.12 total 10.02 last 74.22) - open/close 1449 (time 1783139121.13 total 20.03 last 70.46) - open/close 2216 (time 1783139131.13 total 30.03 last 76.68) - open/close 2986 (time 1783139141.14 total 40.04 last 76.93) - open/close 3594 (time 1783139151.14 total 50.04 last 60.79) - open/close 4350 (time 1783139161.14 total 60.05 last 75.58) - open/close 5107 (time 1783139171.14 total 70.05 last 75.69) - open/close 5858 (time 1783139181.15 total 80.05 last 75.04) - open/close 6534 (time 1783139191.17 total 90.07 last 67.49) - open/close 7228 (time 1783139201.17 total 100.07 last 69.39) - open/close 7884 (time 1783139211.17 total 110.08 last 65.55) - open/close 8564 (time 1783139221.18 total 120.08 last 67.98) - open/close 9167 (time 1783139231.21 total 130.11 last 60.14) - open/close 9857 (time 1783139241.21 total 140.12 last 68.95) total: 10000 open/close in 141.78 seconds: 70.53 ops/second - unlinked 0 (time 1783139247 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 655 (time 1783139355.63 total 10.01 last 65.44) - open/close 1348 (time 1783139365.63 total 20.01 last 69.27) - open/close 2159 (time 1783139375.65 total 30.02 last 81.01) - open/close 2880 (time 1783139385.66 total 40.04 last 72.01) - open/close 3540 (time 1783139395.66 total 50.04 last 65.96) - open/close 4248 (time 1783139405.67 total 60.05 last 70.72) - open/close 4955 (time 1783139415.68 total 70.06 last 70.66) - open/close 5670 (time 1783139425.69 total 80.07 last 71.44) - open/close 6426 (time 1783139435.70 total 90.08 last 75.51) - open/close 7215 (time 1783139445.70 total 100.08 last 78.90) - open/close 7830 (time 1783139455.71 total 110.09 last 61.43) - open/close 8543 (time 1783139465.72 total 120.10 last 71.24) - open/close 9217 (time 1783139475.72 total 130.10 last 67.39) - open/close 9829 (time 1783139485.73 total 140.11 last 61.15) total: 10000 open/close in 142.24 seconds: 70.31 ops/second - unlinked 0 (time 1783139492 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.107@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.107@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.107@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.107@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.107@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.202.107@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: modules unloaded. pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2