== conf-sanity test 69: replace an OST with the same index ========================================================== 20:18:34 (1775866714) start mds service on oleg137-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg137-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg137-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 579 (time 1775866800.99 total 10.01 last 57.85) - open/close 1237 (time 1775866810.99 total 20.01 last 65.76) - open/close 1847 (time 1775866821.01 total 30.04 last 60.86) - open/close 2503 (time 1775866831.02 total 40.04 last 65.59) - open/close 3102 (time 1775866841.02 total 50.05 last 59.86) - open/close 3700 (time 1775866851.03 total 60.05 last 59.77) - open/close 4249 (time 1775866861.03 total 70.05 last 54.89) - open/close 4824 (time 1775866871.04 total 80.07 last 57.42) - open/close 5474 (time 1775866881.05 total 90.07 last 64.99) - open/close 6038 (time 1775866891.05 total 100.07 last 56.38) - open/close 6685 (time 1775866901.05 total 110.08 last 64.67) - open/close 7327 (time 1775866911.06 total 120.08 last 64.18) - open/close 7929 (time 1775866921.06 total 130.09 last 60.17) - open/close 8469 (time 1775866931.06 total 140.09 last 53.99) - open/close 9197 (time 1775866941.07 total 150.09 last 72.77) - open/close 9854 (time 1775866951.09 total 160.11 last 65.56) total: 10000 open/close in 162.14 seconds: 61.67 ops/second - unlinked 0 (time 1775866957 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 700 (time 1775867067.76 total 10.00 last 70.00) - open/close 1371 (time 1775867077.77 total 20.01 last 67.02) - open/close 2066 (time 1775867087.78 total 30.02 last 69.45) - open/close 2746 (time 1775867097.79 total 40.03 last 67.92) - open/close 3474 (time 1775867107.82 total 50.07 last 72.54) - open/close 4147 (time 1775867117.83 total 60.07 last 67.26) - open/close 4801 (time 1775867127.84 total 70.08 last 65.35) - open/close 5510 (time 1775867137.86 total 80.10 last 70.77) - open/close 6251 (time 1775867147.86 total 90.10 last 74.09) - open/close 6893 (time 1775867157.87 total 100.11 last 64.12) - open/close 7553 (time 1775867167.87 total 110.12 last 65.97) - open/close 8207 (time 1775867177.88 total 120.12 last 65.40) - open/close 8940 (time 1775867187.88 total 130.12 last 73.25) - open/close 9648 (time 1775867197.90 total 140.14 last 70.66) total: 10000 open/close in 145.01 seconds: 68.96 ops/second - unlinked 0 (time 1775867206 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 752 (time 1775867300.45 total 10.02 last 75.09) - open/close 1248 (time 1775867310.45 total 20.02 last 49.60) - open/close 1656 (time 1775867320.46 total 30.03 last 40.76) - open/close 2088 (time 1775867330.48 total 40.04 last 43.12) - open/close 2468 (time 1775867340.50 total 50.07 last 37.91) - open/close 2988 (time 1775867350.56 total 60.12 last 51.73) - open/close 3413 (time 1775867360.56 total 70.13 last 42.47) - open/close 4071 (time 1775867370.57 total 80.13 last 65.76) - open/close 4693 (time 1775867380.57 total 90.14 last 62.17) - open/close 5286 (time 1775867390.58 total 100.15 last 59.25) - open/close 6088 (time 1775867400.59 total 110.15 last 80.16) - open/close 6824 (time 1775867410.60 total 120.17 last 73.50) - open/close 7569 (time 1775867420.60 total 130.17 last 74.49) - open/close 8174 (time 1775867430.61 total 140.18 last 60.43) - open/close 8861 (time 1775867440.63 total 150.19 last 68.62) - open/close 9600 (time 1775867450.63 total 160.19 last 73.89) total: 10000 open/close in 166.78 seconds: 59.96 ops/second - unlinked 0 (time 1775867461 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 692 (time 1775867561.29 total 10.00 last 69.19) - open/close 1231 (time 1775867571.31 total 20.01 last 53.84) - open/close 1927 (time 1775867581.31 total 30.01 last 69.60) - open/close 2634 (time 1775867591.32 total 40.02 last 70.63) - open/close 3227 (time 1775867601.32 total 50.03 last 59.29) - open/close 3976 (time 1775867611.32 total 60.03 last 74.87) - open/close 4681 (time 1775867621.33 total 70.04 last 70.45) - open/close 5274 (time 1775867631.34 total 80.05 last 59.25) - open/close 6019 (time 1775867641.35 total 90.06 last 74.39) - open/close 6795 (time 1775867651.36 total 100.07 last 77.51) - open/close 7488 (time 1775867661.37 total 110.08 last 69.24) - open/close 8213 (time 1775867671.39 total 120.09 last 72.41) - open/close 8813 (time 1775867681.39 total 130.10 last 59.96) - open/close 9342 (time 1775867691.40 total 140.11 last 52.83) total: 10000 open/close in 149.36 seconds: 66.95 ops/second - unlinked 0 (time 1775867705 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 528 (time 1775867809.22 total 10.01 last 52.76) - open/close 973 (time 1775867819.23 total 20.02 last 44.43) - open/close 1526 (time 1775867829.25 total 30.04 last 55.21) - open/close 2021 (time 1775867839.25 total 40.04 last 49.50) - open/close 2644 (time 1775867849.26 total 50.05 last 62.28) - open/close 3243 (time 1775867859.28 total 60.07 last 59.75) - open/close 3845 (time 1775867869.29 total 70.08 last 60.16) - open/close 4595 (time 1775867879.29 total 80.08 last 74.96) - open/close 5266 (time 1775867889.30 total 90.09 last 67.03) - open/close 6026 (time 1775867899.31 total 100.10 last 75.93) - open/close 6709 (time 1775867909.32 total 110.11 last 68.23) - open/close 7366 (time 1775867919.33 total 120.12 last 65.68) - open/close 7991 (time 1775867929.33 total 130.12 last 62.48) - open/close 8653 (time 1775867939.33 total 140.12 last 66.18) - open/close 9183 (time 1775867949.36 total 150.15 last 52.86) - open/close 9763 (time 1775867959.36 total 160.15 last 57.99) total: 10000 open/close in 164.11 seconds: 60.94 ops/second - unlinked 0 (time 1775867967 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 640 (time 1775868090.92 total 10.01 last 63.95) - open/close 1260 (time 1775868100.92 total 20.01 last 61.96) - open/close 1834 (time 1775868110.95 total 30.04 last 57.24) - open/close 2469 (time 1775868120.96 total 40.05 last 63.45) - open/close 3131 (time 1775868130.97 total 50.06 last 66.12) - open/close 3737 (time 1775868141.00 total 60.10 last 60.40) - open/close 4473 (time 1775868151.02 total 70.11 last 73.52) - open/close 5153 (time 1775868161.03 total 80.12 last 67.90) - open/close 5821 (time 1775868171.03 total 90.12 last 66.79) - open/close 6466 (time 1775868181.03 total 100.12 last 64.49) - open/close 7165 (time 1775868191.04 total 110.14 last 69.83) - open/close 7805 (time 1775868201.05 total 120.14 last 63.96) - open/close 8372 (time 1775868211.06 total 130.15 last 56.66) - open/close 9012 (time 1775868221.06 total 140.15 last 63.99) - open/close 9695 (time 1775868231.06 total 150.15 last 68.27) total: 10000 open/close in 154.54 seconds: 64.71 ops/second - unlinked 0 (time 1775868240 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 482 (time 1775868358.17 total 10.01 last 48.15) - open/close 977 (time 1775868368.21 total 20.05 last 49.32) - open/close 1521 (time 1775868378.21 total 30.05 last 54.36) - open/close 2200 (time 1775868388.23 total 40.07 last 67.80) - open/close 2854 (time 1775868398.24 total 50.08 last 65.35) - open/close 3463 (time 1775868408.25 total 60.09 last 60.81) - open/close 4106 (time 1775868418.27 total 70.11 last 64.19) - open/close 4770 (time 1775868428.27 total 80.11 last 66.40) - open/close 5311 (time 1775868438.27 total 90.12 last 54.07) - open/close 5785 (time 1775868448.29 total 100.13 last 47.33) - open/close 6377 (time 1775868458.29 total 110.13 last 59.20) - open/close 6970 (time 1775868468.30 total 120.14 last 59.27) - open/close 7479 (time 1775868478.31 total 130.15 last 50.83) - open/close 7969 (time 1775868488.32 total 140.16 last 48.93) - open/close 8528 (time 1775868498.33 total 150.17 last 55.87) - open/close 9061 (time 1775868508.33 total 160.18 last 53.27) - open/close 9598 (time 1775868518.35 total 170.20 last 53.59) total: 10000 open/close in 176.06 seconds: 56.80 ops/second - unlinked 0 (time 1775868529 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 482 (time 1775868657.91 total 10.01 last 48.17) - open/close 1048 (time 1775868667.91 total 20.01 last 56.57) - open/close 1589 (time 1775868677.95 total 30.05 last 53.90) - open/close 2036 (time 1775868687.96 total 40.06 last 44.66) - open/close 2706 (time 1775868697.96 total 50.06 last 66.99) - open/close 3399 (time 1775868707.97 total 60.07 last 69.23) - open/close 4077 (time 1775868717.98 total 70.08 last 67.72) - open/close 4800 (time 1775868727.99 total 80.08 last 72.28) - open/close 5528 (time 1775868737.99 total 90.09 last 72.74) - open/close 6196 (time 1775868748.01 total 100.11 last 66.68) - open/close 6864 (time 1775868758.01 total 110.11 last 66.80) - open/close 7484 (time 1775868768.01 total 120.11 last 61.98) - open/close 8118 (time 1775868778.04 total 130.14 last 63.24) - open/close 8636 (time 1775868788.06 total 140.16 last 51.69) - open/close 9257 (time 1775868798.06 total 150.16 last 62.10) - open/close 9809 (time 1775868808.07 total 160.17 last 55.18) total: 10000 open/close in 163.46 seconds: 61.18 ops/second - unlinked 0 (time 1775868815 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 597 (time 1775868929.28 total 10.01 last 59.65) - open/close 1159 (time 1775868939.28 total 20.01 last 56.17) - open/close 1901 (time 1775868949.29 total 30.02 last 74.15) - open/close 2659 (time 1775868959.29 total 40.02 last 75.80) - open/close 3292 (time 1775868969.33 total 50.06 last 63.09) - open/close 4003 (time 1775868979.35 total 60.08 last 70.96) - open/close 4563 (time 1775868989.36 total 70.09 last 55.94) - open/close 5208 (time 1775868999.38 total 80.11 last 64.36) - open/close 5872 (time 1775869009.39 total 90.12 last 66.34) - open/close 6504 (time 1775869019.40 total 100.13 last 63.14) - open/close 7004 (time 1775869029.40 total 110.13 last 49.96) - open/close 7571 (time 1775869039.41 total 120.14 last 56.68) - open/close 8140 (time 1775869049.41 total 130.14 last 56.89) - open/close 8706 (time 1775869059.42 total 140.15 last 56.51) - open/close 9405 (time 1775869069.44 total 150.17 last 69.81) - open/close 9968 (time 1775869079.45 total 160.18 last 56.21) total: 10000 open/close in 161.24 seconds: 62.02 ops/second - unlinked 0 (time 1775869088 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 615 (time 1775869194.18 total 10.01 last 61.45) - open/close 1299 (time 1775869204.19 total 20.02 last 68.33) - open/close 1998 (time 1775869214.20 total 30.03 last 69.84) - open/close 2819 (time 1775869224.21 total 40.03 last 82.05) - open/close 3591 (time 1775869234.21 total 50.04 last 77.15) - open/close 4355 (time 1775869244.21 total 60.04 last 76.40) - open/close 5056 (time 1775869254.22 total 70.05 last 70.02) - open/close 5747 (time 1775869264.22 total 80.05 last 69.10) - open/close 6595 (time 1775869274.22 total 90.05 last 84.80) - open/close 7482 (time 1775869284.24 total 100.07 last 88.55) - open/close 8300 (time 1775869294.25 total 110.07 last 81.75) - open/close 9164 (time 1775869304.26 total 120.09 last 86.29) total: 10000 open/close in 129.90 seconds: 76.98 ops/second - unlinked 0 (time 1775869317 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg137-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg137-server: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg137-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre On OST0, 16557 used inodes rc=0 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.137@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.137@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.201.137@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg137-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg137-server: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg137-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server unloading modules via unload_modules_local on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local oleg137-server: modules unloaded. pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2