== conf-sanity test 69: replace an OST with the same index ========================================================== 14:35:45 (1787855745) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 670 (time 1787855823.99 total 10.00 last 66.98) - open/close 1323 (time 1787855834.00 total 20.01 last 65.22) - open/close 1989 (time 1787855844.01 total 30.02 last 66.54) - open/close 2648 (time 1787855854.02 total 40.04 last 65.81) - open/close 3369 (time 1787855864.03 total 50.04 last 72.06) - open/close 4040 (time 1787855874.03 total 60.04 last 67.09) - open/close 4690 (time 1787855884.04 total 70.05 last 64.96) - open/close 5319 (time 1787855894.04 total 80.05 last 62.89) - open/close 5999 (time 1787855904.05 total 90.06 last 67.93) - open/close 6783 (time 1787855914.07 total 100.09 last 78.19) - open/close 7628 (time 1787855924.08 total 110.09 last 84.46) - open/close 8311 (time 1787855934.08 total 120.09 last 68.29) - open/close 8922 (time 1787855944.09 total 130.10 last 61.05) - open/close 9570 (time 1787855954.10 total 140.12 last 64.71) total: 10000 open/close in 146.78 seconds: 68.13 ops/second - unlinked 0 (time 1787855966 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 705 (time 1787856070.77 total 10.00 last 70.49) - open/close 1365 (time 1787856080.78 total 20.01 last 65.97) - open/close 2037 (time 1787856090.79 total 30.02 last 67.10) - open/close 2707 (time 1787856100.80 total 40.03 last 66.98) - open/close 3389 (time 1787856110.80 total 50.03 last 68.15) - open/close 4089 (time 1787856120.81 total 60.04 last 69.96) - open/close 4782 (time 1787856130.83 total 70.06 last 69.16) - open/close 5455 (time 1787856140.84 total 80.07 last 67.23) - open/close 6104 (time 1787856150.84 total 90.07 last 64.89) - open/close 6859 (time 1787856160.84 total 100.07 last 75.47) - open/close 7587 (time 1787856170.85 total 110.08 last 72.76) - open/close 8255 (time 1787856180.85 total 120.08 last 66.80) - open/close 8915 (time 1787856190.85 total 130.08 last 65.99) - open/close 9630 (time 1787856200.86 total 140.08 last 71.47) total: 10000 open/close in 145.34 seconds: 68.80 ops/second - unlinked 0 (time 1787856209 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 622 (time 1787856321.70 total 10.00 last 62.19) - open/close 1305 (time 1787856331.71 total 20.01 last 68.26) - open/close 2022 (time 1787856341.71 total 30.01 last 71.69) - open/close 2683 (time 1787856351.71 total 40.01 last 66.07) - open/close 3356 (time 1787856361.72 total 50.02 last 67.26) - open/close 3936 (time 1787856371.73 total 60.03 last 57.95) - open/close 4590 (time 1787856381.75 total 70.05 last 65.26) - open/close 5256 (time 1787856391.76 total 80.05 last 66.56) - open/close 5989 (time 1787856401.76 total 90.06 last 73.27) - open/close 6635 (time 1787856411.77 total 100.07 last 64.55) - open/close 7311 (time 1787856421.78 total 110.08 last 67.53) - open/close 7940 (time 1787856431.78 total 120.08 last 62.89) - open/close 8606 (time 1787856441.78 total 130.08 last 66.60) - open/close 9269 (time 1787856451.78 total 140.08 last 66.30) - open/close 9932 (time 1787856461.78 total 150.08 last 66.28) total: 10000 open/close in 151.03 seconds: 66.21 ops/second - unlinked 0 (time 1787856466 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 686 (time 1787856577.90 total 10.01 last 68.53) - open/close 1349 (time 1787856587.90 total 20.02 last 66.27) - open/close 2048 (time 1787856597.92 total 30.03 last 69.78) - open/close 2671 (time 1787856607.92 total 40.04 last 62.28) - open/close 3362 (time 1787856617.93 total 50.04 last 69.04) - open/close 4029 (time 1787856627.93 total 60.05 last 66.69) - open/close 4685 (time 1787856637.93 total 70.05 last 65.60) - open/close 5325 (time 1787856647.94 total 80.05 last 63.95) - open/close 6022 (time 1787856657.95 total 90.06 last 69.66) - open/close 6668 (time 1787856667.96 total 100.07 last 64.52) - open/close 7373 (time 1787856677.98 total 110.09 last 70.38) - open/close 8022 (time 1787856687.99 total 120.10 last 64.83) - open/close 8624 (time 1787856697.99 total 130.10 last 60.19) - open/close 9194 (time 1787856708.00 total 140.11 last 56.96) - open/close 9787 (time 1787856718.01 total 150.12 last 59.23) total: 10000 open/close in 153.76 seconds: 65.04 ops/second - unlinked 0 (time 1787856725 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 591 (time 1787856840.08 total 10.01 last 59.07) - open/close 1187 (time 1787856850.09 total 20.01 last 59.56) - open/close 1807 (time 1787856860.09 total 30.02 last 61.98) - open/close 2431 (time 1787856870.10 total 40.02 last 62.35) - open/close 3096 (time 1787856880.10 total 50.02 last 66.49) - open/close 3705 (time 1787856890.10 total 60.02 last 60.90) - open/close 4252 (time 1787856900.11 total 70.03 last 54.68) - open/close 4838 (time 1787856910.12 total 80.04 last 58.54) - open/close 5380 (time 1787856920.14 total 90.06 last 54.06) - open/close 5979 (time 1787856930.15 total 100.07 last 59.85) - open/close 6546 (time 1787856940.16 total 110.08 last 56.65) - open/close 7105 (time 1787856950.19 total 120.11 last 55.75) - open/close 7746 (time 1787856960.19 total 130.11 last 64.07) - open/close 8351 (time 1787856970.20 total 140.12 last 60.43) - open/close 8953 (time 1787856980.22 total 150.14 last 60.13) - open/close 9556 (time 1787856990.22 total 160.14 last 60.27) total: 10000 open/close in 167.12 seconds: 59.84 ops/second - unlinked 0 (time 1787857002 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 600 (time 1787857108.70 total 10.01 last 59.92) - open/close 1236 (time 1787857118.73 total 20.04 last 63.46) - open/close 1870 (time 1787857128.73 total 30.04 last 63.40) - open/close 2476 (time 1787857138.73 total 40.04 last 60.57) - open/close 3064 (time 1787857148.74 total 50.05 last 58.77) - open/close 3648 (time 1787857158.74 total 60.05 last 58.37) - open/close 4257 (time 1787857168.76 total 70.06 last 60.83) - open/close 4851 (time 1787857178.77 total 80.07 last 59.34) - open/close 5440 (time 1787857188.78 total 90.08 last 58.84) - open/close 6057 (time 1787857198.78 total 100.09 last 61.66) - open/close 6681 (time 1787857208.79 total 110.09 last 62.37) - open/close 7278 (time 1787857218.80 total 120.10 last 59.65) - open/close 7956 (time 1787857228.80 total 130.11 last 67.75) - open/close 8603 (time 1787857238.81 total 140.11 last 64.68) - open/close 9282 (time 1787857248.82 total 150.13 last 67.79) - open/close 9952 (time 1787857258.85 total 160.15 last 66.84) total: 10000 open/close in 160.81 seconds: 62.19 ops/second - unlinked 0 (time 1787857263 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 698 (time 1787857373.89 total 10.00 last 69.77) - open/close 1303 (time 1787857383.90 total 20.01 last 60.48) - open/close 1916 (time 1787857393.91 total 30.02 last 61.22) - open/close 2551 (time 1787857403.93 total 40.04 last 63.41) - open/close 3195 (time 1787857413.95 total 50.06 last 64.27) - open/close 3772 (time 1787857423.95 total 60.06 last 57.68) - open/close 4381 (time 1787857433.95 total 70.06 last 60.88) - open/close 5063 (time 1787857443.96 total 80.07 last 68.14) - open/close 5735 (time 1787857453.97 total 90.08 last 67.17) - open/close 6419 (time 1787857463.98 total 100.09 last 68.31) - open/close 7136 (time 1787857473.98 total 110.09 last 71.68) - open/close 7878 (time 1787857484.01 total 120.12 last 73.99) - open/close 8573 (time 1787857494.01 total 130.12 last 69.48) - open/close 9249 (time 1787857504.04 total 140.15 last 67.45) - open/close 9904 (time 1787857514.04 total 150.15 last 65.45) total: 10000 open/close in 151.78 seconds: 65.89 ops/second - unlinked 0 (time 1787857521 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 640 (time 1787857626.95 total 10.02 last 63.90) - open/close 1253 (time 1787857636.95 total 20.02 last 61.30) - open/close 1935 (time 1787857646.97 total 30.03 last 68.12) - open/close 2544 (time 1787857656.97 total 40.04 last 60.85) - open/close 3131 (time 1787857666.98 total 50.04 last 58.65) - open/close 3799 (time 1787857677.00 total 60.06 last 66.70) - open/close 4403 (time 1787857687.00 total 70.07 last 60.36) - open/close 5098 (time 1787857697.01 total 80.08 last 69.43) - open/close 5773 (time 1787857707.03 total 90.09 last 67.41) - open/close 6461 (time 1787857717.03 total 100.10 last 68.75) - open/close 7151 (time 1787857727.03 total 110.10 last 68.99) - open/close 7872 (time 1787857737.05 total 120.11 last 72.02) - open/close 8544 (time 1787857747.05 total 130.12 last 67.16) - open/close 9265 (time 1787857757.06 total 140.12 last 72.07) - open/close 9816 (time 1787857767.08 total 150.15 last 54.95) total: 10000 open/close in 152.90 seconds: 65.40 ops/second - unlinked 0 (time 1787857774 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 658 (time 1787857881.89 total 10.03 last 65.61) - open/close 1321 (time 1787857891.89 total 20.03 last 66.26) - open/close 1967 (time 1787857901.90 total 30.04 last 64.57) - open/close 2665 (time 1787857911.90 total 40.04 last 69.80) - open/close 3305 (time 1787857921.90 total 50.05 last 63.97) - open/close 3930 (time 1787857931.91 total 60.05 last 62.46) - open/close 4578 (time 1787857941.91 total 70.05 last 64.80) - open/close 5239 (time 1787857951.92 total 80.06 last 66.04) - open/close 5789 (time 1787857961.95 total 90.09 last 54.85) - open/close 6464 (time 1787857971.96 total 100.10 last 67.41) - open/close 7145 (time 1787857981.97 total 110.11 last 68.06) - open/close 7736 (time 1787857991.98 total 120.12 last 59.02) - open/close 8374 (time 1787858001.99 total 130.13 last 63.74) - open/close 9008 (time 1787858012.00 total 140.14 last 63.35) - open/close 9668 (time 1787858022.00 total 150.15 last 65.94) total: 10000 open/close in 154.83 seconds: 64.59 ops/second - unlinked 0 (time 1787858031 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 650 (time 1787858140.47 total 10.01 last 64.94) - open/close 1263 (time 1787858150.48 total 20.02 last 61.22) - open/close 1822 (time 1787858160.49 total 30.03 last 55.84) - open/close 2432 (time 1787858170.50 total 40.04 last 60.93) - open/close 3019 (time 1787858180.51 total 50.05 last 58.66) - open/close 3566 (time 1787858190.53 total 60.07 last 54.61) - open/close 4223 (time 1787858200.54 total 70.08 last 65.64) - open/close 4912 (time 1787858210.55 total 80.09 last 68.79) - open/close 5562 (time 1787858220.56 total 90.10 last 64.97) - open/close 6193 (time 1787858230.57 total 100.11 last 63.03) - open/close 6840 (time 1787858240.58 total 110.12 last 64.64) - open/close 7501 (time 1787858250.58 total 120.12 last 66.06) - open/close 8162 (time 1787858260.60 total 130.14 last 66.00) - open/close 8901 (time 1787858270.60 total 140.14 last 73.86) - open/close 9558 (time 1787858280.61 total 150.15 last 65.66) total: 10000 open/close in 156.57 seconds: 63.87 ops/second - unlinked 0 (time 1787858290 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.139@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.139@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2