== conf-sanity test 69: replace an OST with the same index ========================================================== 02:29:47 (1771399787)
start mds service on oleg143-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg143-server'
oleg143-server: oleg143-server.virtnet: executing load_modules_local
oleg143-server: Loading modules from /home/green/git/lustre-release/lustre
oleg143-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg143-server: detected 4 online CPUs by sysfs
oleg143-server: MODOPTS_LIBCFS=
oleg143-server: Force libcfs to create 2 CPU partitions
oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg143-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-OST0000
oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg143-client.virtnet:  -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11872 inodes available. Want 99872. rc=0
 - open/close 556 (time 1771399876.84 total 10.01 last 55.55)
 - open/close 1296 (time 1771399886.84 total 20.01 last 73.99)
 - open/close 2090 (time 1771399896.84 total 30.01 last 79.37)
 - open/close 2898 (time 1771399906.84 total 40.02 last 80.79)
 - open/close 3651 (time 1771399916.85 total 50.02 last 75.25)
 - open/close 4441 (time 1771399926.85 total 60.02 last 78.98)
 - open/close 5150 (time 1771399936.86 total 70.03 last 70.85)
 - open/close 5952 (time 1771399946.87 total 80.04 last 80.13)
 - open/close 6667 (time 1771399956.87 total 90.04 last 71.48)
 - open/close 7480 (time 1771399966.87 total 100.05 last 81.28)
 - open/close 8199 (time 1771399976.88 total 110.05 last 71.87)
 - open/close 8851 (time 1771399986.88 total 120.05 last 65.18)
 - open/close 9621 (time 1771399996.89 total 130.06 last 76.93)
total: 10000 open/close in 134.66 seconds: 74.26 ops/second
 - unlinked 0 (time 1771400006 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 683 (time 1771400111.06 total 10.02 last 68.15)
 - open/close 1467 (time 1771400121.06 total 20.02 last 78.39)
 - open/close 2113 (time 1771400131.06 total 30.03 last 64.56)
 - open/close 2774 (time 1771400141.06 total 40.03 last 66.10)
 - open/close 3536 (time 1771400151.07 total 50.04 last 76.13)
 - open/close 4239 (time 1771400161.08 total 60.04 last 70.28)
 - open/close 5003 (time 1771400171.08 total 70.04 last 76.39)
 - open/close 5760 (time 1771400181.08 total 80.05 last 75.66)
 - open/close 6579 (time 1771400191.09 total 90.06 last 81.83)
 - open/close 7456 (time 1771400201.09 total 100.06 last 87.67)
 - open/close 8293 (time 1771400211.10 total 110.07 last 83.65)
 - open/close 9110 (time 1771400221.10 total 120.07 last 81.69)
 - open/close 9850 (time 1771400231.11 total 130.07 last 73.95)
total: 10000 open/close in 131.68 seconds: 75.94 ops/second
 - unlinked 0 (time 1771400237 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 776 (time 1771400328.69 total 10.02 last 77.48)
 - open/close 1517 (time 1771400338.70 total 20.03 last 73.99)
 - open/close 2275 (time 1771400348.71 total 30.03 last 75.77)
 - open/close 2983 (time 1771400358.72 total 40.05 last 70.67)
 - open/close 3756 (time 1771400368.73 total 50.05 last 77.28)
 - open/close 4527 (time 1771400378.75 total 60.07 last 76.96)
 - open/close 5436 (time 1771400388.77 total 70.10 last 90.69)
 - open/close 6112 (time 1771400398.78 total 80.11 last 67.53)
 - open/close 6898 (time 1771400408.78 total 90.11 last 78.59)
 - open/close 7752 (time 1771400418.80 total 100.13 last 85.24)
 - open/close 8622 (time 1771400428.80 total 110.13 last 86.99)
 - open/close 9479 (time 1771400438.82 total 120.15 last 85.52)
total: 10000 open/close in 127.32 seconds: 78.54 ops/second
 - unlinked 0 (time 1771400450 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 893 (time 1771400547.03 total 10.00 last 89.26)
 - open/close 1646 (time 1771400557.03 total 20.01 last 75.29)
 - open/close 2407 (time 1771400567.04 total 30.01 last 76.07)
 - open/close 3171 (time 1771400577.06 total 40.03 last 76.26)
 - open/close 3990 (time 1771400587.06 total 50.03 last 81.86)
 - open/close 4656 (time 1771400597.06 total 60.03 last 66.59)
 - open/close 5412 (time 1771400607.07 total 70.04 last 75.52)
 - open/close 5999 (time 1771400617.07 total 80.05 last 58.69)
 - open/close 6596 (time 1771400627.08 total 90.05 last 59.66)
 - open/close 7219 (time 1771400637.08 total 100.06 last 62.28)
 - open/close 7954 (time 1771400647.09 total 110.06 last 73.47)
 - open/close 8596 (time 1771400657.09 total 120.06 last 64.19)
 - open/close 9469 (time 1771400667.11 total 130.08 last 87.13)
total: 10000 open/close in 136.47 seconds: 73.28 ops/second
 - unlinked 0 (time 1771400677 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 528 (time 1771400775.86 total 10.00 last 52.79)
 - open/close 1356 (time 1771400785.87 total 20.01 last 82.77)
 - open/close 2154 (time 1771400795.87 total 30.01 last 79.76)
 - open/close 2884 (time 1771400805.88 total 40.02 last 72.95)
 - open/close 3663 (time 1771400815.88 total 50.02 last 77.88)
 - open/close 4334 (time 1771400825.89 total 60.03 last 67.04)
 - open/close 4846 (time 1771400835.90 total 70.04 last 51.12)
 - open/close 5560 (time 1771400845.91 total 80.05 last 71.33)
 - open/close 6362 (time 1771400855.92 total 90.06 last 80.17)
 - open/close 7120 (time 1771400865.92 total 100.06 last 75.78)
 - open/close 7918 (time 1771400875.93 total 110.07 last 79.74)
 - open/close 8767 (time 1771400885.93 total 120.07 last 84.85)
 - open/close 9544 (time 1771400895.95 total 130.09 last 77.57)
total: 10000 open/close in 134.92 seconds: 74.12 ops/second
 - unlinked 0 (time 1771400905 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 764 (time 1771400992.32 total 10.01 last 76.33)
 - open/close 1498 (time 1771401002.33 total 20.03 last 73.28)
 - open/close 2338 (time 1771401012.34 total 30.03 last 83.94)
 - open/close 3218 (time 1771401022.35 total 40.04 last 87.94)
 - open/close 3897 (time 1771401032.38 total 50.07 last 67.68)
 - open/close 4641 (time 1771401042.39 total 60.08 last 74.36)
 - open/close 5375 (time 1771401052.39 total 70.08 last 73.35)
 - open/close 6279 (time 1771401062.40 total 80.09 last 90.34)
 - open/close 7180 (time 1771401072.40 total 90.09 last 90.10)
 - open/close 8022 (time 1771401082.40 total 100.09 last 84.18)
 - open/close 8857 (time 1771401092.41 total 110.10 last 83.42)
 - open/close 9614 (time 1771401102.41 total 120.10 last 75.69)
total: 10000 open/close in 124.46 seconds: 80.35 ops/second
 - unlinked 0 (time 1771401110 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 528 (time 1771401204.89 total 10.01 last 52.72)
 - open/close 1314 (time 1771401214.89 total 20.02 last 78.58)
 - open/close 2105 (time 1771401224.89 total 30.02 last 79.08)
 - open/close 2863 (time 1771401234.89 total 40.02 last 75.80)
 - open/close 3568 (time 1771401244.89 total 50.02 last 70.49)
 - open/close 4223 (time 1771401254.89 total 60.02 last 65.49)
 - open/close 4950 (time 1771401264.91 total 70.04 last 72.58)
 - open/close 5707 (time 1771401274.91 total 80.04 last 75.70)
 - open/close 6362 (time 1771401284.91 total 90.04 last 65.48)
 - open/close 7089 (time 1771401294.92 total 100.05 last 72.66)
 - open/close 7867 (time 1771401304.92 total 110.05 last 77.78)
 - open/close 8538 (time 1771401314.94 total 120.07 last 66.99)
 - open/close 9309 (time 1771401324.95 total 130.08 last 77.03)
total: 10000 open/close in 139.25 seconds: 71.81 ops/second
 - unlinked 0 (time 1771401339 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 821 (time 1771401436.74 total 10.02 last 81.91)
 - open/close 1623 (time 1771401446.75 total 20.03 last 80.11)
 - open/close 2478 (time 1771401456.75 total 30.04 last 85.47)
 - open/close 3262 (time 1771401466.75 total 40.04 last 78.40)
 - open/close 4220 (time 1771401476.77 total 50.05 last 95.69)
 - open/close 5078 (time 1771401486.77 total 60.05 last 85.77)
 - open/close 5904 (time 1771401496.78 total 70.06 last 82.54)
 - open/close 6715 (time 1771401506.78 total 80.06 last 81.06)
 - open/close 7587 (time 1771401516.79 total 90.08 last 87.09)
 - open/close 8496 (time 1771401526.80 total 100.08 last 90.87)
 - open/close 9405 (time 1771401536.80 total 110.08 last 90.85)
total: 10000 open/close in 116.00 seconds: 86.20 ops/second
 - unlinked 0 (time 1771401546 ; total 0 ; last 0)
total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second
 - open/close 865 (time 1771401630.72 total 10.00 last 86.50)
 - open/close 1756 (time 1771401640.73 total 20.00 last 89.10)
 - open/close 2706 (time 1771401650.73 total 30.00 last 95.00)
 - open/close 3736 (time 1771401660.73 total 40.01 last 102.95)
 - open/close 4598 (time 1771401670.74 total 50.02 last 86.11)
 - open/close 5564 (time 1771401680.76 total 60.03 last 96.46)
 - open/close 6528 (time 1771401690.76 total 70.04 last 96.35)
 - open/close 7489 (time 1771401700.76 total 80.04 last 96.05)
 - open/close 8465 (time 1771401710.77 total 90.05 last 97.53)
 - open/close 9553 (time 1771401720.78 total 100.05 last 108.74)
total: 10000 open/close in 104.80 seconds: 95.42 ops/second
 - unlinked 0 (time 1771401729 ; total 0 ; last 0)
total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second
 - open/close 1025 (time 1771401801.59 total 10.00 last 102.48)
 - open/close 2057 (time 1771401811.60 total 20.01 last 103.13)
 - open/close 2922 (time 1771401821.61 total 30.02 last 86.41)
 - open/close 3914 (time 1771401831.62 total 40.02 last 99.14)
 - open/close 5008 (time 1771401841.64 total 50.05 last 109.14)
 - open/close 5957 (time 1771401851.65 total 60.06 last 94.76)
 - open/close 6908 (time 1771401861.67 total 70.07 last 95.00)
 - open/close 7812 (time 1771401871.67 total 80.08 last 90.35)
 - open/close 8849 (time 1771401881.68 total 90.09 last 103.58)
 - open/close 9824 (time 1771401891.69 total 100.10 last 97.44)
total: 10000 open/close in 102.20 seconds: 97.85 ops/second
 - unlinked 0 (time 1771401896 ; total 0 ; last 0)
total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg143-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg143-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.143@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.143@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg143-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-OST0000
oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg143-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg143-client.virtnet:  -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre
On OST0, 17341 used inodes rc=0
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg143-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg143-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.143@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.143@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.143@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.143@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.143@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg143-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-OST0000
oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg143-server: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg143-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg143-client.virtnet:  -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg143-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg143-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server
stop mds service on oleg143-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server

unloading modules via unload_modules_local on: 'oleg143-server'
oleg143-server: oleg143-server.virtnet: executing unload_modules_local
oleg143-server: 
modules unloaded.
pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2
pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2