== conf-sanity test 69: replace an OST with the same index ========================================================== 14:07:53 (1784225273)
start mds service on oleg357-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg357-server'
oleg357-server: oleg357-server.virtnet: executing load_modules_local
oleg357-server: Loading modules from /home/green/git/lustre-release/lustre
oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg357-server: detected 4 online CPUs by sysfs
oleg357-server: MODOPTS_LIBCFS=
oleg357-server: Force libcfs to create 2 CPU partitions
oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg357-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg357-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-OST0000
oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg357-client.virtnet:  -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 1561 (time 1784225320.45 total 10.00 last 156.03)
 - open/close 3410 (time 1784225330.45 total 20.01 last 184.83)
 - open/close 5327 (time 1784225340.46 total 30.02 last 191.57)
 - open/close 7243 (time 1784225350.46 total 40.02 last 191.59)
 - open/close 9015 (time 1784225360.46 total 50.02 last 177.15)
total: 10000 open/close in 55.39 seconds: 180.53 ops/second
 - unlinked 0 (time 1784225367 ; total 0 ; last 0)
total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second
 - open/close 1874 (time 1784225411.39 total 10.00 last 187.38)
 - open/close 3469 (time 1784225421.39 total 20.01 last 159.43)
 - open/close 5181 (time 1784225431.40 total 30.01 last 171.18)
 - open/close 7155 (time 1784225441.40 total 40.01 last 197.33)
 - open/close 8717 (time 1784225451.40 total 50.01 last 156.14)
total: 10000 open/close in 58.97 seconds: 169.57 ops/second
 - unlinked 0 (time 1784225462 ; total 0 ; last 0)
total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second
 - open/close 1961 (time 1784225507.34 total 10.00 last 196.04)
 - open/close 3353 (time 1784225517.35 total 20.01 last 139.13)
 - open/close 4227 (time 1784225527.35 total 30.01 last 87.35)
 - open/close 5045 (time 1784225537.36 total 40.02 last 81.75)
 - open/close 5874 (time 1784225547.36 total 50.02 last 82.89)
 - open/close 6666 (time 1784225557.37 total 60.03 last 79.14)
 - open/close 7369 (time 1784225567.38 total 70.04 last 70.21)
 - open/close 8186 (time 1784225577.38 total 80.04 last 81.66)
 - open/close 9104 (time 1784225587.38 total 90.04 last 91.80)
 - open/close 9962 (time 1784225597.39 total 100.05 last 85.76)
total: 10000 open/close in 100.49 seconds: 99.52 ops/second
 - unlinked 0 (time 1784225600 ; total 0 ; last 0)
total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second
 - open/close 978 (time 1784225670.82 total 10.01 last 97.67)
 - open/close 1860 (time 1784225680.83 total 20.03 last 88.09)
 - open/close 2741 (time 1784225690.84 total 30.03 last 88.07)
 - open/close 3666 (time 1784225700.84 total 40.03 last 92.48)
 - open/close 4461 (time 1784225710.85 total 50.04 last 79.43)
 - open/close 5260 (time 1784225720.86 total 60.05 last 79.82)
 - open/close 6117 (time 1784225730.86 total 70.06 last 85.64)
 - open/close 7003 (time 1784225740.87 total 80.06 last 88.58)
 - open/close 7930 (time 1784225750.87 total 90.06 last 92.69)
 - open/close 8854 (time 1784225760.87 total 100.06 last 92.37)
 - open/close 9854 (time 1784225770.88 total 110.07 last 99.94)
total: 10000 open/close in 111.40 seconds: 89.77 ops/second
 - unlinked 0 (time 1784225774 ; total 0 ; last 0)
total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second
 - open/close 1142 (time 1784225849.35 total 10.00 last 114.16)
 - open/close 2262 (time 1784225859.36 total 20.01 last 111.95)
 - open/close 3364 (time 1784225869.36 total 30.01 last 110.15)
 - open/close 4605 (time 1784225879.37 total 40.02 last 124.04)
 - open/close 5886 (time 1784225889.38 total 50.03 last 127.97)
 - open/close 7298 (time 1784225899.38 total 60.03 last 141.15)
 - open/close 8747 (time 1784225909.38 total 70.04 last 144.84)
total: 10000 open/close in 79.54 seconds: 125.72 ops/second
 - unlinked 0 (time 1784225921 ; total 0 ; last 0)
total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second
 - open/close 1277 (time 1784225983.60 total 10.00 last 127.64)
 - open/close 2399 (time 1784225993.60 total 20.00 last 112.20)
 - open/close 3837 (time 1784226003.61 total 30.01 last 143.75)
 - open/close 5062 (time 1784226013.62 total 40.02 last 122.41)
 - open/close 6164 (time 1784226023.63 total 50.03 last 110.05)
 - open/close 7331 (time 1784226033.63 total 60.03 last 116.64)
 - open/close 8348 (time 1784226043.64 total 70.04 last 101.59)
 - open/close 9271 (time 1784226053.66 total 80.06 last 92.16)
total: 10000 open/close in 87.20 seconds: 114.68 ops/second
 - unlinked 0 (time 1784226063 ; total 0 ; last 0)
total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second
 - open/close 1231 (time 1784226127.47 total 10.00 last 123.04)
 - open/close 2554 (time 1784226137.47 total 20.01 last 132.21)
 - open/close 3884 (time 1784226147.48 total 30.02 last 132.91)
 - open/close 5166 (time 1784226157.48 total 40.02 last 128.18)
 - open/close 6780 (time 1784226167.49 total 50.02 last 161.37)
 - open/close 8083 (time 1784226177.49 total 60.03 last 130.22)
 - open/close 9452 (time 1784226187.50 total 70.04 last 136.80)
total: 10000 open/close in 74.24 seconds: 134.69 ops/second
 - unlinked 0 (time 1784226193 ; total 0 ; last 0)
total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second
 - open/close 1424 (time 1784226253.82 total 10.00 last 142.38)
 - open/close 3127 (time 1784226263.83 total 20.00 last 170.25)
 - open/close 4629 (time 1784226273.83 total 30.01 last 150.08)
 - open/close 6014 (time 1784226283.84 total 40.02 last 138.44)
 - open/close 7602 (time 1784226293.85 total 50.02 last 158.69)
 - open/close 8852 (time 1784226303.85 total 60.03 last 124.93)
total: 10000 open/close in 68.30 seconds: 146.40 ops/second
 - unlinked 0 (time 1784226314 ; total 0 ; last 0)
total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second
 - open/close 1627 (time 1784226363.71 total 10.00 last 162.70)
 - open/close 2991 (time 1784226373.72 total 20.01 last 136.32)
 - open/close 4626 (time 1784226383.72 total 30.01 last 163.47)
 - open/close 5949 (time 1784226393.72 total 40.01 last 132.21)
 - open/close 7191 (time 1784226403.73 total 50.02 last 124.14)
 - open/close 8694 (time 1784226413.74 total 60.03 last 150.21)
total: 10000 open/close in 69.29 seconds: 144.32 ops/second
 - unlinked 0 (time 1784226425 ; total 0 ; last 0)
total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second
 - open/close 1524 (time 1784226479.06 total 10.01 last 152.32)
 - open/close 3503 (time 1784226489.06 total 20.01 last 197.84)
 - open/close 5209 (time 1784226499.07 total 30.01 last 170.51)
 - open/close 6956 (time 1784226509.07 total 40.02 last 174.65)
 - open/close 8370 (time 1784226519.07 total 50.02 last 141.38)
 - open/close 9799 (time 1784226529.07 total 60.02 last 142.90)
total: 10000 open/close in 61.38 seconds: 162.92 ops/second
 - unlinked 0 (time 1784226532 ; total 0 ; last 0)
total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg357-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg357-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.157@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg357-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-OST0000
oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg357-client.virtnet:  -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre
On OST0, 19438 used inodes rc=0
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg357-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg357-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.157@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.157@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.157@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.203.157@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg357-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-OST0000
oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg357-client.virtnet:  -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg357-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg357-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server
stop mds service on oleg357-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server

unloading modules via unload_modules_local on: 'oleg357-server'
oleg357-server: oleg357-server.virtnet: executing unload_modules_local
oleg357-server: 
modules unloaded.
pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2
pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2