== conf-sanity test 69: replace an OST with the same index ========================================================== 19:00:43 (1785452443)
start mds service on oleg613-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg613-server'
oleg613-server: oleg613-server.virtnet: executing load_modules_local
oleg613-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg613-server: Loading modules from /home/green/git/lustre-release/lustre
oleg613-server: detected 4 online CPUs by sysfs
oleg613-server: MODOPTS_LIBCFS=
oleg613-server: Force libcfs to create 2 CPU partitions
oleg613-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg613-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg613-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all
pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg613-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all
pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1
Started lustre-OST0000
oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg613-client.virtnet:  -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 775 (time 1785452529.67 total 10.01 last 77.40)
 - open/close 1538 (time 1785452539.67 total 20.01 last 76.29)
 - open/close 2281 (time 1785452549.68 total 30.03 last 74.22)
 - open/close 3083 (time 1785452559.68 total 40.03 last 80.17)
 - open/close 3797 (time 1785452569.70 total 50.04 last 71.29)
 - open/close 4508 (time 1785452579.70 total 60.05 last 71.07)
 - open/close 5189 (time 1785452589.71 total 70.06 last 68.01)
 - open/close 5994 (time 1785452599.72 total 80.06 last 80.47)
 - open/close 6785 (time 1785452609.73 total 90.08 last 79.00)
 - open/close 7532 (time 1785452619.73 total 100.08 last 74.67)
 - open/close 8245 (time 1785452629.75 total 110.09 last 71.22)
 - open/close 9019 (time 1785452639.75 total 120.10 last 77.33)
 - open/close 9746 (time 1785452649.76 total 130.11 last 72.64)
total: 10000 open/close in 133.62 seconds: 74.84 ops/second
 - unlinked 0 (time 1785452656 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 749 (time 1785452753.80 total 10.01 last 74.84)
 - open/close 1463 (time 1785452763.80 total 20.01 last 71.38)
 - open/close 2219 (time 1785452773.80 total 30.01 last 75.58)
 - open/close 2973 (time 1785452783.82 total 40.03 last 75.31)
 - open/close 3700 (time 1785452793.83 total 50.04 last 72.61)
 - open/close 4425 (time 1785452803.84 total 60.05 last 72.42)
 - open/close 5178 (time 1785452813.84 total 70.05 last 75.26)
 - open/close 5928 (time 1785452823.85 total 80.06 last 74.94)
 - open/close 6677 (time 1785452833.85 total 90.06 last 74.89)
 - open/close 7408 (time 1785452843.87 total 100.08 last 73.01)
 - open/close 8103 (time 1785452853.88 total 110.09 last 69.43)
 - open/close 8818 (time 1785452863.89 total 120.10 last 71.41)
 - open/close 9558 (time 1785452873.91 total 130.11 last 73.88)
total: 10000 open/close in 136.68 seconds: 73.16 ops/second
 - unlinked 0 (time 1785452885 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 739 (time 1785452987.52 total 10.00 last 73.88)
 - open/close 1461 (time 1785452997.53 total 20.02 last 72.11)
 - open/close 2242 (time 1785453007.54 total 30.02 last 78.06)
 - open/close 2970 (time 1785453017.55 total 40.03 last 72.74)
 - open/close 3705 (time 1785453027.56 total 50.04 last 73.40)
 - open/close 4486 (time 1785453037.57 total 60.05 last 78.05)
 - open/close 5205 (time 1785453047.58 total 70.06 last 71.84)
 - open/close 5949 (time 1785453057.58 total 80.06 last 74.40)
 - open/close 6689 (time 1785453067.58 total 90.06 last 73.97)
 - open/close 7412 (time 1785453077.59 total 100.07 last 72.26)
 - open/close 8156 (time 1785453087.60 total 110.08 last 74.32)
 - open/close 8934 (time 1785453097.61 total 120.09 last 77.74)
 - open/close 9691 (time 1785453107.61 total 130.09 last 75.65)
total: 10000 open/close in 133.90 seconds: 74.68 ops/second
 - unlinked 0 (time 1785453114 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 752 (time 1785453210.23 total 10.02 last 75.07)
 - open/close 1462 (time 1785453220.24 total 20.03 last 70.95)
 - open/close 2087 (time 1785453230.24 total 30.03 last 62.49)
 - open/close 2743 (time 1785453240.26 total 40.04 last 65.50)
 - open/close 3423 (time 1785453250.26 total 50.05 last 67.97)
 - open/close 4162 (time 1785453260.27 total 60.06 last 73.81)
 - open/close 4892 (time 1785453270.28 total 70.06 last 72.96)
 - open/close 5630 (time 1785453280.29 total 80.07 last 73.73)
 - open/close 6308 (time 1785453290.29 total 90.07 last 67.80)
 - open/close 7022 (time 1785453300.29 total 100.08 last 71.38)
 - open/close 7755 (time 1785453310.30 total 110.08 last 73.25)
 - open/close 8454 (time 1785453320.31 total 120.10 last 69.78)
 - open/close 9171 (time 1785453330.33 total 130.12 last 71.57)
 - open/close 9850 (time 1785453340.33 total 140.12 last 67.89)
total: 10000 open/close in 142.02 seconds: 70.41 ops/second
 - unlinked 0 (time 1785453346 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 727 (time 1785453449.01 total 10.03 last 72.52)
 - open/close 1466 (time 1785453459.01 total 20.03 last 73.83)
 - open/close 2089 (time 1785453469.04 total 30.06 last 62.13)
 - open/close 2714 (time 1785453479.08 total 40.10 last 62.26)
 - open/close 3353 (time 1785453489.08 total 50.10 last 63.88)
 - open/close 4079 (time 1785453499.09 total 60.11 last 72.54)
 - open/close 4727 (time 1785453509.12 total 70.14 last 64.63)
 - open/close 5432 (time 1785453519.14 total 80.16 last 70.37)
 - open/close 6151 (time 1785453529.14 total 90.16 last 71.89)
 - open/close 6906 (time 1785453539.14 total 100.16 last 75.47)
 - open/close 7537 (time 1785453549.16 total 110.18 last 62.97)
 - open/close 8249 (time 1785453559.16 total 120.18 last 71.19)
 - open/close 8974 (time 1785453569.17 total 130.19 last 72.47)
 - open/close 9652 (time 1785453579.18 total 140.20 last 67.71)
total: 10000 open/close in 145.09 seconds: 68.92 ops/second
 - unlinked 0 (time 1785453588 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 685 (time 1785453698.95 total 10.00 last 68.49)
 - open/close 1365 (time 1785453708.95 total 20.00 last 68.00)
 - open/close 2058 (time 1785453718.95 total 30.00 last 69.29)
 - open/close 2795 (time 1785453728.96 total 40.01 last 73.64)
 - open/close 3439 (time 1785453738.96 total 50.01 last 64.39)
 - open/close 4134 (time 1785453748.97 total 60.02 last 69.44)
 - open/close 4827 (time 1785453758.98 total 70.03 last 69.26)
 - open/close 5484 (time 1785453769.00 total 80.05 last 65.54)
 - open/close 6172 (time 1785453779.01 total 90.06 last 68.75)
 - open/close 6853 (time 1785453789.01 total 100.06 last 68.10)
 - open/close 7561 (time 1785453799.02 total 110.07 last 70.73)
 - open/close 8249 (time 1785453809.02 total 120.08 last 68.75)
 - open/close 8983 (time 1785453819.03 total 130.08 last 73.39)
 - open/close 9679 (time 1785453829.04 total 140.09 last 69.54)
total: 10000 open/close in 145.03 seconds: 68.95 ops/second
 - unlinked 0 (time 1785453838 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 672 (time 1785453944.78 total 10.00 last 67.19)
 - open/close 1374 (time 1785453954.78 total 20.00 last 70.18)
 - open/close 2033 (time 1785453964.79 total 30.01 last 65.85)
 - open/close 2702 (time 1785453974.79 total 40.02 last 66.87)
 - open/close 3329 (time 1785453984.80 total 50.03 last 62.62)
 - open/close 4027 (time 1785453994.81 total 60.03 last 69.78)
 - open/close 4765 (time 1785454004.81 total 70.04 last 73.78)
 - open/close 5488 (time 1785454014.82 total 80.04 last 72.25)
 - open/close 6210 (time 1785454024.83 total 90.05 last 72.11)
 - open/close 6905 (time 1785454034.84 total 100.06 last 69.45)
 - open/close 7633 (time 1785454044.84 total 110.07 last 72.75)
 - open/close 8337 (time 1785454054.85 total 120.07 last 70.37)
 - open/close 8990 (time 1785454064.85 total 130.08 last 65.28)
 - open/close 9620 (time 1785454074.86 total 140.09 last 62.92)
total: 10000 open/close in 147.56 seconds: 67.77 ops/second
 - unlinked 0 (time 1785454087 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 679 (time 1785454190.07 total 10.00 last 67.88)
 - open/close 1406 (time 1785454200.07 total 20.00 last 72.69)
 - open/close 2152 (time 1785454210.08 total 30.01 last 74.52)
 - open/close 2896 (time 1785454220.09 total 40.03 last 74.29)
 - open/close 3629 (time 1785454230.11 total 50.05 last 73.18)
 - open/close 4347 (time 1785454240.12 total 60.06 last 71.73)
 - open/close 5055 (time 1785454250.13 total 70.06 last 70.74)
 - open/close 5662 (time 1785454260.13 total 80.07 last 60.69)
 - open/close 6318 (time 1785454270.13 total 90.07 last 65.59)
 - open/close 6982 (time 1785454280.13 total 100.07 last 66.39)
 - open/close 7678 (time 1785454290.14 total 110.08 last 69.55)
 - open/close 8457 (time 1785454300.15 total 120.09 last 77.77)
 - open/close 9133 (time 1785454310.16 total 130.10 last 67.58)
 - open/close 9786 (time 1785454320.16 total 140.10 last 65.27)
total: 10000 open/close in 143.30 seconds: 69.78 ops/second
 - unlinked 0 (time 1785454327 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 618 (time 1785454432.28 total 10.01 last 61.73)
 - open/close 1321 (time 1785454442.29 total 20.01 last 70.28)
 - open/close 1998 (time 1785454452.30 total 30.02 last 67.64)
 - open/close 2714 (time 1785454462.30 total 40.03 last 71.56)
 - open/close 3419 (time 1785454472.31 total 50.04 last 70.43)
 - open/close 4004 (time 1785454482.33 total 60.06 last 58.38)
 - open/close 4636 (time 1785454492.34 total 70.07 last 63.16)
 - open/close 5284 (time 1785454502.34 total 80.07 last 64.78)
 - open/close 5945 (time 1785454512.35 total 90.08 last 66.05)
 - open/close 6707 (time 1785454522.36 total 100.09 last 76.11)
 - open/close 7361 (time 1785454532.39 total 110.12 last 65.23)
 - open/close 8048 (time 1785454542.40 total 120.12 last 68.64)
 - open/close 8692 (time 1785454552.40 total 130.13 last 64.36)
 - open/close 9318 (time 1785454562.41 total 140.13 last 62.57)
 - open/close 9894 (time 1785454572.42 total 150.15 last 57.52)
total: 10000 open/close in 152.20 seconds: 65.70 ops/second
 - unlinked 0 (time 1785454579 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 713 (time 1785454681.26 total 10.01 last 71.25)
 - open/close 1391 (time 1785454691.29 total 20.04 last 67.60)
 - open/close 2114 (time 1785454701.29 total 30.04 last 72.28)
 - open/close 2816 (time 1785454711.29 total 40.04 last 70.18)
 - open/close 3560 (time 1785454721.29 total 50.04 last 74.38)
 - open/close 4283 (time 1785454731.30 total 60.05 last 72.22)
 - open/close 5028 (time 1785454741.31 total 70.06 last 74.48)
 - open/close 5766 (time 1785454751.31 total 80.06 last 73.76)
 - open/close 6491 (time 1785454761.32 total 90.07 last 72.44)
 - open/close 7153 (time 1785454771.32 total 100.07 last 66.19)
 - open/close 7822 (time 1785454781.32 total 110.07 last 66.90)
 - open/close 8549 (time 1785454791.33 total 120.08 last 72.65)
 - open/close 9213 (time 1785454801.33 total 130.08 last 66.37)
 - open/close 9887 (time 1785454811.35 total 140.10 last 67.29)
total: 10000 open/close in 141.51 seconds: 70.67 ops/second
 - unlinked 0 (time 1785454817 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg613-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg613-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.113@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.113@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg613-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all
pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1
Started lustre-OST0000
oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg613-server: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg613-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg613-client.virtnet:  -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg613-client: oleg613-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg613-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg613-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.113@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.113@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.206.113@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.206.113@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.206.113@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg613-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all
pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1
Started lustre-OST0000
oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg613-server: oleg613-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg613-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg613-client.virtnet:  -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg613-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg613-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server
stop mds service on oleg613-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg613-server

unloading modules via unload_modules_local on: 'oleg613-server'
oleg613-server: oleg613-server.virtnet: executing unload_modules_local
oleg613-server: 
modules unloaded.
pdsh@oleg613-client: oleg613-client: ssh exited with exit code 2
pdsh@oleg613-client: oleg613-server: ssh exited with exit code 2
pdsh@oleg613-client: oleg613-client: ssh exited with exit code 2
pdsh@oleg613-client: oleg613-server: ssh exited with exit code 2