== conf-sanity test 69: replace an OST with the same index ========================================================== 02:24:28 (1772436268)
start mds service on oleg146-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg146-server'
oleg146-server: oleg146-server.virtnet: executing load_modules_local
oleg146-server: Loading modules from /home/green/git/lustre-release/lustre
oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg146-server: detected 4 online CPUs by sysfs
oleg146-server: MODOPTS_LIBCFS=
oleg146-server: Force libcfs to create 2 CPU partitions
oleg146-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg146-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg146-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all
pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg146-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all
pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1
Started lustre-OST0000
oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg146-client.virtnet:  -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11872 inodes available. Want 99872. rc=0
 - open/close 762 (time 1772436339.33 total 10.00 last 76.17)
 - open/close 1558 (time 1772436349.34 total 20.01 last 79.55)
 - open/close 2358 (time 1772436359.35 total 30.02 last 79.94)
 - open/close 3165 (time 1772436369.35 total 40.02 last 80.66)
 - open/close 4005 (time 1772436379.36 total 50.03 last 83.95)
 - open/close 4952 (time 1772436389.37 total 60.04 last 94.60)
 - open/close 5753 (time 1772436399.38 total 70.05 last 80.00)
 - open/close 6628 (time 1772436409.39 total 80.07 last 87.39)
 - open/close 7513 (time 1772436419.40 total 90.07 last 88.45)
 - open/close 8449 (time 1772436429.40 total 100.07 last 93.58)
 - open/close 9342 (time 1772436439.40 total 110.08 last 89.29)
total: 10000 open/close in 117.17 seconds: 85.35 ops/second
 - unlinked 0 (time 1772436449 ; total 0 ; last 0)
total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second
 - open/close 906 (time 1772436524.04 total 10.01 last 90.55)
 - open/close 1826 (time 1772436534.05 total 20.02 last 91.90)
 - open/close 2686 (time 1772436544.06 total 30.03 last 85.91)
 - open/close 3645 (time 1772436554.07 total 40.03 last 95.86)
 - open/close 4568 (time 1772436564.07 total 50.03 last 92.26)
 - open/close 5430 (time 1772436574.07 total 60.04 last 86.19)
 - open/close 6280 (time 1772436584.08 total 70.04 last 84.97)
 - open/close 7142 (time 1772436594.08 total 80.04 last 86.19)
 - open/close 8093 (time 1772436604.08 total 90.04 last 95.09)
 - open/close 9013 (time 1772436614.09 total 100.05 last 91.92)
 - open/close 9874 (time 1772436624.10 total 110.06 last 86.03)
total: 10000 open/close in 111.36 seconds: 89.80 ops/second
 - unlinked 0 (time 1772436628 ; total 0 ; last 0)
total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second
 - open/close 984 (time 1772436699.71 total 10.01 last 98.31)
 - open/close 2069 (time 1772436709.72 total 20.02 last 108.39)
 - open/close 3030 (time 1772436719.72 total 30.02 last 96.05)
 - open/close 4048 (time 1772436729.73 total 40.03 last 101.75)
 - open/close 5019 (time 1772436739.73 total 50.03 last 97.06)
 - open/close 5869 (time 1772436749.74 total 60.04 last 84.91)
 - open/close 6911 (time 1772436759.74 total 70.04 last 104.20)
 - open/close 7782 (time 1772436769.75 total 80.05 last 87.06)
 - open/close 8866 (time 1772436779.75 total 90.05 last 108.34)
 - open/close 9981 (time 1772436789.75 total 100.06 last 111.49)
total: 10000 open/close in 100.23 seconds: 99.77 ops/second
 - unlinked 0 (time 1772436792 ; total 0 ; last 0)
total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second
 - open/close 990 (time 1772436856.53 total 10.01 last 98.86)
 - open/close 1984 (time 1772436866.54 total 20.02 last 99.32)
 - open/close 2947 (time 1772436876.54 total 30.03 last 96.24)
 - open/close 3861 (time 1772436886.55 total 40.04 last 91.32)
 - open/close 4848 (time 1772436896.56 total 50.05 last 98.62)
 - open/close 5839 (time 1772436906.57 total 60.06 last 98.99)
 - open/close 6846 (time 1772436916.58 total 70.07 last 100.62)
 - open/close 7737 (time 1772436926.59 total 80.08 last 89.00)
 - open/close 8795 (time 1772436936.60 total 90.08 last 105.74)
 - open/close 9859 (time 1772436946.60 total 100.08 last 106.39)
total: 10000 open/close in 101.41 seconds: 98.61 ops/second
 - unlinked 0 (time 1772436950 ; total 0 ; last 0)
total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second
 - open/close 1000 (time 1772437021.57 total 10.01 last 99.93)
 - open/close 1922 (time 1772437031.57 total 20.01 last 92.18)
 - open/close 2863 (time 1772437041.57 total 30.01 last 94.09)
 - open/close 3911 (time 1772437051.58 total 40.02 last 104.71)
 - open/close 4935 (time 1772437061.59 total 50.03 last 102.33)
 - open/close 5879 (time 1772437071.59 total 60.03 last 94.38)
 - open/close 6837 (time 1772437081.60 total 70.03 last 95.78)
 - open/close 7922 (time 1772437091.60 total 80.03 last 108.48)
 - open/close 8869 (time 1772437101.61 total 90.04 last 94.63)
total: 10000 open/close in 99.68 seconds: 100.32 ops/second
 - unlinked 0 (time 1772437113 ; total 0 ; last 0)
total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second
 - open/close 1369 (time 1772437176.99 total 10.02 last 136.65)
 - open/close 2366 (time 1772437186.99 total 20.02 last 99.66)
 - open/close 3350 (time 1772437197.00 total 30.03 last 98.34)
 - open/close 4279 (time 1772437207.01 total 40.04 last 92.79)
 - open/close 5312 (time 1772437217.01 total 50.04 last 103.28)
 - open/close 6341 (time 1772437227.02 total 60.04 last 102.90)
 - open/close 7468 (time 1772437237.02 total 70.04 last 112.68)
 - open/close 8667 (time 1772437247.03 total 80.06 last 119.76)
 - open/close 9695 (time 1772437257.03 total 90.06 last 102.75)
total: 10000 open/close in 93.63 seconds: 106.81 ops/second
 - unlinked 0 (time 1772437263 ; total 0 ; last 0)
total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second
 - open/close 936 (time 1772437334.59 total 10.01 last 93.52)
 - open/close 1912 (time 1772437344.60 total 20.02 last 97.52)
 - open/close 2948 (time 1772437354.60 total 30.02 last 103.57)
 - open/close 3943 (time 1772437364.60 total 40.02 last 99.47)
 - open/close 4835 (time 1772437374.61 total 50.03 last 89.15)
 - open/close 5773 (time 1772437384.62 total 60.04 last 93.72)
 - open/close 6675 (time 1772437394.62 total 70.04 last 90.17)
 - open/close 7681 (time 1772437404.63 total 80.05 last 100.55)
 - open/close 8715 (time 1772437414.64 total 90.06 last 103.29)
 - open/close 9695 (time 1772437424.65 total 100.06 last 97.91)
total: 10000 open/close in 103.30 seconds: 96.80 ops/second
 - unlinked 0 (time 1772437430 ; total 0 ; last 0)
total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second
 - open/close 1158 (time 1772437493.33 total 10.01 last 115.74)
 - open/close 2476 (time 1772437503.34 total 20.01 last 131.70)
 - open/close 3707 (time 1772437513.34 total 30.01 last 123.09)
 - open/close 4992 (time 1772437523.35 total 40.02 last 128.40)
 - open/close 6087 (time 1772437533.35 total 50.02 last 109.48)
 - open/close 7186 (time 1772437543.36 total 60.04 last 109.74)
 - open/close 8268 (time 1772437553.36 total 70.04 last 108.19)
 - open/close 9465 (time 1772437563.37 total 80.04 last 119.67)
total: 10000 open/close in 84.00 seconds: 119.05 ops/second
 - unlinked 0 (time 1772437569 ; total 0 ; last 0)
total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second
 - open/close 1320 (time 1772437620.39 total 10.00 last 131.97)
 - open/close 2711 (time 1772437630.40 total 20.01 last 139.06)
 - open/close 4605 (time 1772437640.40 total 30.01 last 189.38)
 - open/close 6334 (time 1772437650.40 total 40.01 last 172.89)
 - open/close 7790 (time 1772437660.40 total 50.01 last 145.59)
 - open/close 9201 (time 1772437670.40 total 60.01 last 141.04)
total: 10000 open/close in 64.23 seconds: 155.68 ops/second
 - unlinked 0 (time 1772437676 ; total 0 ; last 0)
total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second
 - open/close 1462 (time 1772437719.35 total 10.00 last 146.16)
 - open/close 2745 (time 1772437729.35 total 20.01 last 128.22)
 - open/close 4105 (time 1772437739.36 total 30.01 last 135.95)
 - open/close 5963 (time 1772437749.36 total 40.02 last 185.72)
 - open/close 7711 (time 1772437759.36 total 50.02 last 174.76)
 - open/close 9326 (time 1772437769.37 total 60.02 last 161.46)
total: 10000 open/close in 65.02 seconds: 153.81 ops/second
 - unlinked 0 (time 1772437776 ; total 0 ; last 0)
total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg146-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg146-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.146@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.146@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg146-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all
pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1
Started lustre-OST0000
oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg146-server: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg146-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 12 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg146-client.virtnet:  -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre
On OST0, 18765 used inodes rc=0
pdsh@oleg146-client: oleg146-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg146-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg146-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.146@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.146@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.146@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.146@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.146@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg146-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all
pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1
Started lustre-OST0000
oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg146-server: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg146-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg146-client.virtnet:  -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg146-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg146-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server
stop mds service on oleg146-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server

unloading modules via unload_modules_local on: 'oleg146-server'
oleg146-server: oleg146-server.virtnet: executing unload_modules_local
oleg146-server: 
modules unloaded.
pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2
pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2
pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2
pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2