== conf-sanity test 69: replace an OST with the same index ========================================================== 16:56:29 (1779310589) start mds service on oleg432-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions oleg432-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg432-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg432-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 833 (time 1779310654.52 total 10.01 last 83.24) - open/close 1681 (time 1779310664.53 total 20.01 last 84.75) - open/close 2498 (time 1779310674.54 total 30.02 last 81.62) - open/close 3239 (time 1779310684.55 total 40.04 last 74.00) - open/close 4089 (time 1779310694.56 total 50.05 last 84.92) - open/close 4965 (time 1779310704.57 total 60.05 last 87.53) - open/close 5813 (time 1779310714.57 total 70.06 last 84.79) - open/close 6713 (time 1779310724.57 total 80.06 last 90.00) - open/close 7567 (time 1779310734.57 total 90.06 last 85.38) - open/close 8487 (time 1779310744.58 total 100.06 last 91.94) - open/close 9384 (time 1779310754.59 total 110.07 last 89.63) total: 10000 open/close in 117.82 seconds: 84.87 ops/second - unlinked 0 (time 1779310765 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1038 (time 1779310837.00 total 10.01 last 103.67) - open/close 2105 (time 1779310847.01 total 20.02 last 106.66) - open/close 3097 (time 1779310857.01 total 30.02 last 99.18) - open/close 4105 (time 1779310867.01 total 40.02 last 100.75) - open/close 5146 (time 1779310877.03 total 50.04 last 103.96) - open/close 6132 (time 1779310887.03 total 60.04 last 98.52) - open/close 7099 (time 1779310897.04 total 70.05 last 96.67) - open/close 7973 (time 1779310907.04 total 80.05 last 87.38) - open/close 8919 (time 1779310917.05 total 90.06 last 94.52) - open/close 9944 (time 1779310927.05 total 100.06 last 102.45) total: 10000 open/close in 100.69 seconds: 99.31 ops/second - unlinked 0 (time 1779310930 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1004 (time 1779310999.76 total 10.00 last 100.38) - open/close 1955 (time 1779311009.78 total 20.01 last 94.98) - open/close 2917 (time 1779311019.79 total 30.02 last 96.12) - open/close 3854 (time 1779311029.79 total 40.03 last 93.62) - open/close 4864 (time 1779311039.80 total 50.03 last 100.98) - open/close 5822 (time 1779311049.80 total 60.04 last 95.71) - open/close 6888 (time 1779311059.81 total 70.04 last 106.59) - open/close 8015 (time 1779311069.81 total 80.05 last 112.65) - open/close 9258 (time 1779311079.81 total 90.05 last 124.30) total: 10000 open/close in 96.05 seconds: 104.11 ops/second - unlinked 0 (time 1779311087 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1086 (time 1779311146.77 total 10.01 last 108.49) - open/close 2158 (time 1779311156.77 total 20.01 last 107.20) - open/close 3296 (time 1779311166.77 total 30.01 last 113.78) - open/close 4309 (time 1779311176.78 total 40.03 last 101.15) - open/close 5553 (time 1779311186.79 total 50.03 last 124.36) - open/close 6647 (time 1779311196.79 total 60.03 last 109.36) - open/close 7696 (time 1779311206.79 total 70.04 last 104.86) - open/close 8885 (time 1779311216.80 total 80.04 last 118.88) total: 10000 open/close in 89.61 seconds: 111.59 ops/second - unlinked 0 (time 1779311228 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1582 (time 1779311284.06 total 10.00 last 158.13) - open/close 3203 (time 1779311294.06 total 20.01 last 162.05) - open/close 4806 (time 1779311304.06 total 30.01 last 160.28) - open/close 6369 (time 1779311314.07 total 40.01 last 156.25) - open/close 7986 (time 1779311324.07 total 50.01 last 161.69) - open/close 9725 (time 1779311334.07 total 60.02 last 173.84) total: 10000 open/close in 62.19 seconds: 160.79 ops/second - unlinked 0 (time 1779311338 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1262 (time 1779311388.32 total 10.00 last 126.18) - open/close 2476 (time 1779311398.32 total 20.01 last 121.32) - open/close 3614 (time 1779311408.32 total 30.01 last 113.79) - open/close 4855 (time 1779311418.33 total 40.01 last 124.07) - open/close 5966 (time 1779311428.33 total 50.02 last 111.03) - open/close 6992 (time 1779311438.34 total 60.02 last 102.57) - open/close 8153 (time 1779311448.34 total 70.03 last 116.03) - open/close 9291 (time 1779311458.34 total 80.03 last 113.76) total: 10000 open/close in 85.17 seconds: 117.41 ops/second - unlinked 0 (time 1779311465 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1349 (time 1779311524.78 total 10.00 last 134.86) - open/close 2639 (time 1779311534.79 total 20.01 last 128.93) - open/close 3882 (time 1779311544.80 total 30.01 last 124.22) - open/close 5343 (time 1779311554.80 total 40.02 last 146.07) - open/close 6767 (time 1779311564.80 total 50.02 last 142.30) - open/close 7984 (time 1779311574.81 total 60.03 last 121.67) - open/close 9631 (time 1779311584.81 total 70.03 last 164.64) total: 10000 open/close in 72.37 seconds: 138.18 ops/second - unlinked 0 (time 1779311588 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1868 (time 1779311640.65 total 10.01 last 186.71) - open/close 3337 (time 1779311650.66 total 20.01 last 146.87) - open/close 3863 (time 1779311660.67 total 30.03 last 52.50) - open/close 4409 (time 1779311670.68 total 40.03 last 54.59) - open/close 4781 (time 1779311680.68 total 50.03 last 37.19) - open/close 5173 (time 1779311690.68 total 60.03 last 39.19) - open/close 5622 (time 1779311700.69 total 70.04 last 44.86) - open/close 6036 (time 1779311710.73 total 80.08 last 41.25) - open/close 6439 (time 1779311720.73 total 90.08 last 40.29) - open/close 6880 (time 1779311730.77 total 100.12 last 43.93) - open/close 7455 (time 1779311740.77 total 110.12 last 57.49) - open/close 8011 (time 1779311750.77 total 120.12 last 55.59) - open/close 8565 (time 1779311760.78 total 130.13 last 55.35) - open/close 9105 (time 1779311770.80 total 140.15 last 53.89) - open/close 9754 (time 1779311780.81 total 150.16 last 64.84) total: 10000 open/close in 153.74 seconds: 65.05 ops/second - unlinked 0 (time 1779311789 ; total 1 ; last 1) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 696 (time 1779311895.01 total 10.00 last 69.58) - open/close 1380 (time 1779311905.02 total 20.01 last 68.35) - open/close 2075 (time 1779311915.04 total 30.03 last 69.35) - open/close 2706 (time 1779311925.05 total 40.03 last 63.09) - open/close 3414 (time 1779311935.06 total 50.04 last 70.73) - open/close 4139 (time 1779311945.07 total 60.05 last 72.42) - open/close 4836 (time 1779311955.08 total 70.07 last 69.62) - open/close 5514 (time 1779311965.09 total 80.08 last 67.73) - open/close 6194 (time 1779311975.12 total 90.11 last 67.79) - open/close 6820 (time 1779311985.13 total 100.11 last 62.56) - open/close 7482 (time 1779311995.14 total 110.12 last 66.13) - open/close 8183 (time 1779312005.14 total 120.13 last 70.05) - open/close 8822 (time 1779312015.16 total 130.15 last 63.81) - open/close 9494 (time 1779312025.23 total 140.22 last 66.74) total: 10000 open/close in 147.72 seconds: 67.70 ops/second - unlinked 0 (time 1779312036 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 689 (time 1779312140.85 total 10.01 last 68.80) - open/close 1390 (time 1779312150.86 total 20.02 last 70.07) - open/close 2116 (time 1779312160.86 total 30.02 last 72.58) - open/close 2871 (time 1779312170.86 total 40.02 last 75.48) - open/close 3597 (time 1779312180.86 total 50.02 last 72.59) - open/close 4284 (time 1779312190.88 total 60.04 last 68.60) - open/close 5018 (time 1779312200.88 total 70.04 last 73.36) - open/close 5793 (time 1779312210.90 total 80.06 last 77.39) - open/close 6413 (time 1779312220.90 total 90.06 last 61.96) - open/close 7085 (time 1779312230.90 total 100.07 last 67.19) - open/close 7827 (time 1779312240.91 total 110.07 last 74.14) - open/close 8498 (time 1779312250.91 total 120.07 last 67.09) - open/close 9303 (time 1779312260.92 total 130.08 last 80.45) total: 10000 open/close in 139.78 seconds: 71.54 ops/second - unlinked 0 (time 1779312274 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.132@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.132@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg432-server: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg432-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.132@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.132@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.132@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.132@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.204.132@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg432-server: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg432-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local oleg432-server: modules unloaded. pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2