== conf-sanity test 69: replace an OST with the same index ========================================================== 14:12:26 (1758823946) start mds service on oleg259-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing load_modules_local oleg259-server: Loading modules from /home/green/git/lustre-release/lustre oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg259-server: detected 4 online CPUs by sysfs oleg259-server: MODOPTS_LIBCFS= oleg259-server: Force libcfs to create 2 CPU partitions oleg259-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg259-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1822 (time 1758823992.53 total 10.00 last 182.20) - open/close 3654 (time 1758824002.53 total 20.00 last 183.17) - open/close 6291 (time 1758824012.54 total 30.01 last 263.52) - open/close 8511 (time 1758824022.54 total 40.01 last 222.00) total: 10000 open/close in 48.35 seconds: 206.81 ops/second - unlinked 0 (time 1758824033 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1037 (time 1758824088.04 total 10.01 last 103.62) - open/close 2299 (time 1758824098.04 total 20.01 last 126.16) - open/close 3418 (time 1758824108.05 total 30.02 last 111.85) - open/close 4595 (time 1758824118.06 total 40.02 last 117.59) - open/close 5942 (time 1758824128.07 total 50.03 last 134.60) - open/close 7931 (time 1758824138.07 total 60.03 last 198.88) - open/close 9729 (time 1758824148.07 total 70.04 last 179.77) total: 10000 open/close in 71.27 seconds: 140.31 ops/second - unlinked 0 (time 1758824150 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1684 (time 1758824193.26 total 10.01 last 168.31) - open/close 3136 (time 1758824203.26 total 20.01 last 145.12) - open/close 4694 (time 1758824213.26 total 30.01 last 155.77) - open/close 6262 (time 1758824223.27 total 40.02 last 156.65) - open/close 7794 (time 1758824233.28 total 50.02 last 153.18) - open/close 9265 (time 1758824243.28 total 60.03 last 147.03) total: 10000 open/close in 65.07 seconds: 153.67 ops/second - unlinked 0 (time 1758824250 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1924 (time 1758824293.76 total 10.01 last 192.24) - open/close 3719 (time 1758824303.76 total 20.01 last 179.45) - open/close 5620 (time 1758824313.77 total 30.01 last 190.07) - open/close 7514 (time 1758824323.77 total 40.01 last 189.37) - open/close 9211 (time 1758824333.77 total 50.01 last 169.70) total: 10000 open/close in 53.92 seconds: 185.46 ops/second - unlinked 0 (time 1758824339 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1891 (time 1758824377.04 total 10.00 last 189.07) - open/close 4316 (time 1758824387.04 total 20.00 last 242.45) - open/close 6664 (time 1758824397.04 total 30.01 last 234.75) - open/close 8965 (time 1758824407.04 total 40.01 last 230.05) total: 10000 open/close in 44.97 seconds: 222.36 ops/second - unlinked 0 (time 1758824413 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2512 (time 1758824449.56 total 10.00 last 251.13) - open/close 4849 (time 1758824459.56 total 20.00 last 233.68) - open/close 6906 (time 1758824469.56 total 30.01 last 205.64) - open/close 9620 (time 1758824479.57 total 40.01 last 271.33) total: 10000 open/close in 41.79 seconds: 239.31 ops/second - unlinked 0 (time 1758824482 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2701 (time 1758824518.59 total 10.00 last 270.03) - open/close 4680 (time 1758824528.59 total 20.00 last 197.87) - open/close 6447 (time 1758824538.59 total 30.01 last 176.68) - open/close 8342 (time 1758824548.59 total 40.01 last 189.47) - open/close 9986 (time 1758824558.59 total 50.01 last 164.37) total: 10000 open/close in 50.08 seconds: 199.67 ops/second - unlinked 0 (time 1758824560 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2141 (time 1758824595.88 total 10.00 last 214.08) - open/close 4718 (time 1758824605.88 total 20.00 last 257.64) - open/close 7463 (time 1758824615.88 total 30.01 last 274.39) total: 10000 open/close in 39.85 seconds: 250.92 ops/second - unlinked 0 (time 1758824627 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2113 (time 1758824663.27 total 10.00 last 211.29) - open/close 4863 (time 1758824673.27 total 20.00 last 274.94) - open/close 7566 (time 1758824683.28 total 30.00 last 270.28) total: 10000 open/close in 39.36 seconds: 254.09 ops/second - unlinked 0 (time 1758824694 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2903 (time 1758824724.39 total 10.00 last 290.28) - open/close 6144 (time 1758824734.39 total 20.00 last 324.09) total: 10000 open/close in 29.23 seconds: 342.15 ops/second - unlinked 0 (time 1758824744 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.159@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.159@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg259-server: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg259-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.159@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.159@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.159@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.159@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.202.159@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg259-server: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg259-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing unload_modules_local oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2