== conf-sanity test 69: replace an OST with the same index ========================================================== 17:14:48 (1777497288) start mds service on oleg433-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: MODOPTS_LIBCFS= oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg433-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg433-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 561 (time 1777497370.05 total 10.00 last 56.08) - open/close 1127 (time 1777497380.05 total 20.01 last 56.58) - open/close 1667 (time 1777497390.06 total 30.01 last 53.96) - open/close 2264 (time 1777497400.08 total 40.04 last 59.55) - open/close 2816 (time 1777497410.10 total 50.06 last 55.10) - open/close 3417 (time 1777497420.10 total 60.06 last 60.09) - open/close 3971 (time 1777497430.11 total 70.07 last 55.33) - open/close 4486 (time 1777497440.16 total 80.12 last 51.25) - open/close 5042 (time 1777497450.19 total 90.15 last 55.45) - open/close 5572 (time 1777497460.21 total 100.16 last 52.90) - open/close 6022 (time 1777497470.23 total 110.19 last 44.90) - open/close 6515 (time 1777497480.25 total 120.20 last 49.20) - open/close 7058 (time 1777497490.27 total 130.22 last 54.21) - open/close 7562 (time 1777497500.32 total 140.27 last 50.13) - open/close 7995 (time 1777497510.33 total 150.28 last 43.27) - open/close 8506 (time 1777497520.34 total 160.29 last 51.04) - open/close 9029 (time 1777497530.34 total 170.29 last 52.29) - open/close 9608 (time 1777497540.36 total 180.31 last 57.79) total: 10000 open/close in 187.59 seconds: 53.31 ops/second - unlinked 0 (time 1777497552 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second - open/close 569 (time 1777497679.72 total 10.01 last 56.85) - open/close 1103 (time 1777497689.72 total 20.01 last 53.39) - open/close 1651 (time 1777497699.73 total 30.02 last 54.73) - open/close 2280 (time 1777497709.74 total 40.03 last 62.89) - open/close 2864 (time 1777497719.74 total 50.03 last 58.37) - open/close 3431 (time 1777497729.75 total 60.04 last 56.66) - open/close 4008 (time 1777497739.75 total 70.04 last 57.67) - open/close 4573 (time 1777497749.76 total 80.04 last 56.49) - open/close 5107 (time 1777497759.76 total 90.05 last 53.37) - open/close 5667 (time 1777497769.78 total 100.07 last 55.90) - open/close 6261 (time 1777497779.79 total 110.08 last 59.35) - open/close 6843 (time 1777497789.79 total 120.08 last 58.17) - open/close 7452 (time 1777497799.81 total 130.09 last 60.81) - open/close 8019 (time 1777497809.81 total 140.10 last 56.67) - open/close 8586 (time 1777497819.82 total 150.11 last 56.63) - open/close 9170 (time 1777497829.84 total 160.13 last 58.31) - open/close 9724 (time 1777497839.85 total 170.14 last 55.33) total: 10000 open/close in 174.97 seconds: 57.15 ops/second - unlinked 0 (time 1777497849 ; total 0 ; last 0) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 550 (time 1777497973.05 total 10.01 last 54.96) - open/close 1160 (time 1777497983.06 total 20.02 last 60.94) - open/close 1798 (time 1777497993.06 total 30.02 last 63.77) - open/close 2373 (time 1777498003.07 total 40.03 last 57.48) - open/close 3015 (time 1777498013.07 total 50.03 last 64.18) - open/close 3652 (time 1777498023.08 total 60.04 last 63.65) - open/close 4205 (time 1777498033.09 total 70.05 last 55.24) - open/close 4875 (time 1777498043.09 total 80.05 last 66.95) - open/close 5448 (time 1777498053.10 total 90.06 last 57.28) - open/close 6129 (time 1777498063.13 total 100.09 last 67.88) - open/close 6747 (time 1777498073.13 total 110.09 last 61.79) - open/close 7276 (time 1777498083.14 total 120.10 last 52.86) - open/close 7849 (time 1777498093.16 total 130.12 last 57.16) - open/close 8455 (time 1777498103.17 total 140.13 last 60.55) - open/close 9097 (time 1777498113.18 total 150.14 last 64.17) - open/close 9759 (time 1777498123.18 total 160.14 last 66.16) total: 10000 open/close in 163.49 seconds: 61.16 ops/second - unlinked 0 (time 1777498130 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 709 (time 1777498235.81 total 10.01 last 70.86) - open/close 1403 (time 1777498245.81 total 20.01 last 69.40) - open/close 2097 (time 1777498255.83 total 30.03 last 69.23) - open/close 2735 (time 1777498265.84 total 40.04 last 63.72) - open/close 3426 (time 1777498275.85 total 50.05 last 69.05) - open/close 4143 (time 1777498285.87 total 60.06 last 71.60) - open/close 4823 (time 1777498295.88 total 70.08 last 67.89) - open/close 5492 (time 1777498305.90 total 80.09 last 66.81) - open/close 6218 (time 1777498315.91 total 90.11 last 72.49) - open/close 6957 (time 1777498325.91 total 100.11 last 73.89) - open/close 7605 (time 1777498335.92 total 110.11 last 64.77) - open/close 8258 (time 1777498345.92 total 120.12 last 65.26) - open/close 8996 (time 1777498355.92 total 130.12 last 73.80) - open/close 9774 (time 1777498365.93 total 140.13 last 77.75) total: 10000 open/close in 142.86 seconds: 70.00 ops/second - unlinked 0 (time 1777498371 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 727 (time 1777498461.18 total 10.01 last 72.64) - open/close 1469 (time 1777498471.18 total 20.01 last 74.17) - open/close 2246 (time 1777498481.19 total 30.02 last 77.63) - open/close 3097 (time 1777498491.19 total 40.02 last 85.10) - open/close 3906 (time 1777498501.20 total 50.03 last 80.83) - open/close 4725 (time 1777498511.21 total 60.04 last 81.82) - open/close 5600 (time 1777498521.22 total 70.05 last 87.41) - open/close 6488 (time 1777498531.23 total 80.06 last 88.70) - open/close 7471 (time 1777498541.23 total 90.06 last 98.29) - open/close 8424 (time 1777498551.24 total 100.07 last 95.19) - open/close 9329 (time 1777498561.25 total 110.08 last 90.43) total: 10000 open/close in 118.11 seconds: 84.67 ops/second - unlinked 0 (time 1777498572 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 867 (time 1777498652.51 total 10.01 last 86.61) - open/close 1626 (time 1777498662.53 total 20.03 last 75.77) - open/close 2379 (time 1777498672.55 total 30.05 last 75.14) - open/close 3201 (time 1777498682.55 total 40.05 last 82.15) - open/close 4054 (time 1777498692.56 total 50.06 last 85.28) - open/close 4915 (time 1777498702.56 total 60.06 last 86.09) - open/close 5777 (time 1777498712.56 total 70.06 last 86.18) - open/close 6619 (time 1777498722.56 total 80.06 last 84.19) - open/close 7488 (time 1777498732.56 total 90.06 last 86.88) - open/close 8449 (time 1777498742.57 total 100.07 last 96.08) - open/close 9323 (time 1777498752.57 total 110.07 last 87.36) total: 10000 open/close in 117.63 seconds: 85.01 ops/second - unlinked 0 (time 1777498762 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1277 (time 1777498832.65 total 10.00 last 127.68) - open/close 2302 (time 1777498842.66 total 20.01 last 102.42) - open/close 2636 (time 1777498852.70 total 30.05 last 33.27) - open/close 3095 (time 1777498862.70 total 40.05 last 45.89) - open/close 3538 (time 1777498872.74 total 50.09 last 44.11) - open/close 3968 (time 1777498882.75 total 60.10 last 42.98) - open/close 4429 (time 1777498892.81 total 70.16 last 45.81) - open/close 4883 (time 1777498902.83 total 80.18 last 45.33) - open/close 5264 (time 1777498912.84 total 90.19 last 38.05) - open/close 5614 (time 1777498922.84 total 100.19 last 34.99) - open/close 6040 (time 1777498932.85 total 110.21 last 42.55) - open/close 6576 (time 1777498942.86 total 120.21 last 53.58) - open/close 7143 (time 1777498952.87 total 130.22 last 56.63) - open/close 7693 (time 1777498962.89 total 140.24 last 54.91) - open/close 8266 (time 1777498972.90 total 150.25 last 57.25) - open/close 8852 (time 1777498982.90 total 160.25 last 58.58) - open/close 9427 (time 1777498992.90 total 170.26 last 57.48) total: 10000 open/close in 180.23 seconds: 55.48 ops/second - unlinked 0 (time 1777499006 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 612 (time 1777499126.02 total 10.00 last 61.18) - open/close 1240 (time 1777499136.03 total 20.02 last 62.72) - open/close 1809 (time 1777499146.04 total 30.02 last 56.86) - open/close 2358 (time 1777499156.04 total 40.03 last 54.89) - open/close 2918 (time 1777499166.05 total 50.03 last 55.95) - open/close 3503 (time 1777499176.06 total 60.04 last 58.47) - open/close 4103 (time 1777499186.07 total 70.05 last 59.94) - open/close 4655 (time 1777499196.07 total 80.05 last 55.18) - open/close 5225 (time 1777499206.09 total 90.08 last 56.87) - open/close 5752 (time 1777499216.10 total 100.08 last 52.67) - open/close 6287 (time 1777499226.12 total 110.10 last 53.39) - open/close 6833 (time 1777499236.12 total 120.11 last 54.59) - open/close 7383 (time 1777499246.14 total 130.12 last 54.91) - open/close 7954 (time 1777499256.15 total 140.13 last 57.04) - open/close 8486 (time 1777499266.16 total 150.15 last 53.14) - open/close 9058 (time 1777499276.18 total 160.16 last 57.11) - open/close 9601 (time 1777499286.19 total 170.17 last 54.24) total: 10000 open/close in 178.21 seconds: 56.11 ops/second - unlinked 0 (time 1777499299 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 568 (time 1777499427.16 total 10.02 last 56.67) - open/close 1150 (time 1777499437.18 total 20.05 last 58.05) - open/close 1762 (time 1777499447.19 total 30.05 last 61.19) - open/close 2327 (time 1777499457.19 total 40.06 last 56.46) - open/close 2924 (time 1777499467.20 total 50.07 last 59.64) - open/close 3542 (time 1777499477.21 total 60.07 last 61.77) - open/close 4152 (time 1777499487.22 total 70.09 last 60.92) - open/close 4783 (time 1777499497.24 total 80.10 last 63.01) - open/close 5362 (time 1777499507.25 total 90.12 last 57.80) - open/close 5938 (time 1777499517.27 total 100.13 last 57.52) - open/close 6546 (time 1777499527.28 total 110.15 last 60.71) - open/close 7116 (time 1777499537.30 total 120.17 last 56.88) - open/close 7703 (time 1777499547.32 total 130.18 last 58.61) - open/close 8284 (time 1777499557.33 total 140.19 last 58.05) - open/close 8868 (time 1777499567.34 total 150.21 last 58.30) - open/close 9489 (time 1777499577.36 total 160.23 last 61.97) total: 10000 open/close in 167.95 seconds: 59.54 ops/second - unlinked 0 (time 1777499588 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 623 (time 1777499698.61 total 10.02 last 62.18) - open/close 1220 (time 1777499708.62 total 20.03 last 59.64) - open/close 1805 (time 1777499718.62 total 30.03 last 58.49) - open/close 2415 (time 1777499728.63 total 40.03 last 60.98) - open/close 3034 (time 1777499738.64 total 50.04 last 61.84) - open/close 3683 (time 1777499748.64 total 60.05 last 64.84) - open/close 4324 (time 1777499758.65 total 70.06 last 64.06) - open/close 4943 (time 1777499768.66 total 80.07 last 61.85) - open/close 5625 (time 1777499778.67 total 90.08 last 68.14) - open/close 6313 (time 1777499788.69 total 100.10 last 68.62) - open/close 6998 (time 1777499798.71 total 110.12 last 68.38) - open/close 7655 (time 1777499808.72 total 120.13 last 65.66) - open/close 8317 (time 1777499818.73 total 130.13 last 66.16) - open/close 9010 (time 1777499828.73 total 140.14 last 69.27) - open/close 9696 (time 1777499838.76 total 150.17 last 68.42) total: 10000 open/close in 154.60 seconds: 64.68 ops/second - unlinked 0 (time 1777499846 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.133@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.133@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 autodegrade=on Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.133@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server unloading modules via unload_modules_local on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local oleg433-server: modules unloaded. pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2