== conf-sanity test 69: replace an OST with the same index ========================================================== 21:37:29 (1767321449) start mds service on oleg417-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg417-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg417-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 852 (time 1767321573.60 total 10.00 last 85.17) - open/close 1657 (time 1767321583.61 total 20.01 last 80.44) - open/close 2434 (time 1767321593.61 total 30.02 last 77.64) - open/close 3225 (time 1767321603.62 total 40.03 last 79.04) - open/close 4011 (time 1767321613.63 total 50.04 last 78.50) - open/close 4676 (time 1767321623.65 total 60.05 last 66.40) - open/close 5168 (time 1767321633.65 total 70.06 last 49.19) - open/close 5795 (time 1767321643.66 total 80.07 last 62.63) - open/close 6455 (time 1767321653.67 total 90.08 last 65.92) - open/close 7146 (time 1767321663.69 total 100.09 last 69.02) - open/close 7899 (time 1767321673.69 total 110.09 last 75.28) - open/close 8687 (time 1767321683.69 total 120.10 last 78.75) - open/close 9372 (time 1767321693.70 total 130.11 last 68.46) total: 10000 open/close in 139.19 seconds: 71.85 ops/second - unlinked 0 (time 1767321707 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 672 (time 1767321804.27 total 10.02 last 67.10) - open/close 1479 (time 1767321814.27 total 20.02 last 80.65) - open/close 2234 (time 1767321824.30 total 30.05 last 75.27) - open/close 2977 (time 1767321834.31 total 40.06 last 74.24) - open/close 3704 (time 1767321844.32 total 50.07 last 72.62) - open/close 4480 (time 1767321854.33 total 60.08 last 77.54) - open/close 5195 (time 1767321864.34 total 70.08 last 71.46) - open/close 6012 (time 1767321874.34 total 80.09 last 81.63) - open/close 6790 (time 1767321884.35 total 90.10 last 77.73) - open/close 7575 (time 1767321894.35 total 100.10 last 78.49) - open/close 8327 (time 1767321904.36 total 110.10 last 75.18) - open/close 9044 (time 1767321914.36 total 120.11 last 71.67) - open/close 9701 (time 1767321924.36 total 130.11 last 65.68) total: 10000 open/close in 136.39 seconds: 73.32 ops/second - unlinked 0 (time 1767321936 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 633 (time 1767322044.61 total 10.02 last 63.18) - open/close 1228 (time 1767322054.62 total 20.03 last 59.42) - open/close 2003 (time 1767322064.62 total 30.04 last 77.47) - open/close 2579 (time 1767322074.65 total 40.06 last 57.45) - open/close 3229 (time 1767322084.67 total 50.08 last 64.89) - open/close 3856 (time 1767322094.67 total 60.08 last 62.68) - open/close 4583 (time 1767322104.68 total 70.09 last 72.67) - open/close 5262 (time 1767322114.69 total 80.10 last 67.83) - open/close 5866 (time 1767322124.69 total 90.10 last 60.40) - open/close 6538 (time 1767322134.69 total 100.10 last 67.17) - open/close 7221 (time 1767322144.70 total 110.11 last 68.24) - open/close 7917 (time 1767322154.72 total 120.13 last 69.46) - open/close 8534 (time 1767322164.74 total 130.16 last 61.56) - open/close 9207 (time 1767322174.76 total 140.17 last 67.21) - open/close 9888 (time 1767322184.76 total 150.17 last 68.08) total: 10000 open/close in 151.67 seconds: 65.93 ops/second - unlinked 0 (time 1767322190 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 641 (time 1767322298.81 total 10.01 last 64.00) - open/close 1414 (time 1767322308.81 total 20.02 last 77.29) - open/close 2055 (time 1767322318.83 total 30.04 last 63.95) - open/close 2721 (time 1767322328.84 total 40.05 last 66.53) - open/close 3298 (time 1767322338.85 total 50.05 last 57.68) - open/close 3878 (time 1767322348.85 total 60.06 last 57.96) - open/close 4556 (time 1767322358.85 total 70.06 last 67.80) - open/close 5265 (time 1767322368.87 total 80.08 last 70.80) - open/close 5930 (time 1767322378.89 total 90.10 last 66.35) - open/close 6545 (time 1767322388.90 total 100.11 last 61.46) - open/close 7269 (time 1767322398.91 total 110.11 last 72.34) - open/close 7870 (time 1767322408.91 total 120.12 last 60.06) - open/close 8649 (time 1767322418.92 total 130.13 last 77.84) - open/close 9354 (time 1767322428.93 total 140.14 last 70.42) - open/close 9963 (time 1767322438.95 total 150.16 last 60.79) total: 10000 open/close in 150.64 seconds: 66.38 ops/second - unlinked 0 (time 1767322444 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 720 (time 1767322542.24 total 10.00 last 71.97) - open/close 1424 (time 1767322552.25 total 20.01 last 70.34) - open/close 2127 (time 1767322562.27 total 30.03 last 70.19) - open/close 2791 (time 1767322572.27 total 40.03 last 66.37) - open/close 3481 (time 1767322582.28 total 50.04 last 68.94) - open/close 4291 (time 1767322592.28 total 60.05 last 80.97) - open/close 5030 (time 1767322602.29 total 70.06 last 73.82) - open/close 5764 (time 1767322612.30 total 80.06 last 73.39) - open/close 6521 (time 1767322622.31 total 90.07 last 75.62) - open/close 7302 (time 1767322632.33 total 100.09 last 77.95) - open/close 8042 (time 1767322642.33 total 110.09 last 74.00) - open/close 8814 (time 1767322652.33 total 120.09 last 77.16) - open/close 9601 (time 1767322662.33 total 130.09 last 78.70) total: 10000 open/close in 135.06 seconds: 74.04 ops/second - unlinked 0 (time 1767322671 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 801 (time 1767322762.61 total 10.01 last 80.06) - open/close 1669 (time 1767322772.61 total 20.01 last 86.78) - open/close 2474 (time 1767322782.62 total 30.02 last 80.42) - open/close 3401 (time 1767322792.63 total 40.02 last 92.64) - open/close 4039 (time 1767322802.64 total 50.03 last 63.76) - open/close 4820 (time 1767322812.64 total 60.03 last 78.10) - open/close 5670 (time 1767322822.65 total 70.04 last 84.92) - open/close 6513 (time 1767322832.65 total 80.04 last 84.26) - open/close 7368 (time 1767322842.65 total 90.05 last 85.48) - open/close 8288 (time 1767322852.66 total 100.05 last 91.94) - open/close 9063 (time 1767322862.67 total 110.06 last 77.42) - open/close 9852 (time 1767322872.67 total 120.07 last 78.87) total: 10000 open/close in 122.10 seconds: 81.90 ops/second - unlinked 0 (time 1767322878 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 834 (time 1767322954.93 total 10.01 last 83.33) - open/close 1745 (time 1767322964.94 total 20.02 last 91.02) - open/close 2477 (time 1767322974.94 total 30.02 last 73.16) - open/close 3166 (time 1767322984.95 total 40.03 last 68.85) - open/close 4011 (time 1767322994.96 total 50.03 last 84.47) - open/close 4856 (time 1767323004.98 total 60.06 last 84.32) - open/close 5633 (time 1767323014.99 total 70.07 last 77.61) - open/close 6488 (time 1767323025.00 total 80.07 last 85.45) - open/close 7401 (time 1767323035.00 total 90.07 last 91.29) - open/close 8275 (time 1767323045.01 total 100.08 last 87.32) - open/close 9175 (time 1767323055.01 total 110.08 last 89.99) total: 10000 open/close in 119.64 seconds: 83.58 ops/second - unlinked 0 (time 1767323068 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 775 (time 1767323156.13 total 10.01 last 77.45) - open/close 1641 (time 1767323166.14 total 20.02 last 86.48) - open/close 2515 (time 1767323176.15 total 30.03 last 87.29) - open/close 3456 (time 1767323186.17 total 40.05 last 93.97) - open/close 4401 (time 1767323196.17 total 50.05 last 94.49) - open/close 5294 (time 1767323206.18 total 60.07 last 89.14) - open/close 6042 (time 1767323216.19 total 70.07 last 74.74) - open/close 6822 (time 1767323226.21 total 80.09 last 77.90) - open/close 7631 (time 1767323236.22 total 90.10 last 80.82) - open/close 8443 (time 1767323246.23 total 100.11 last 81.13) - open/close 9311 (time 1767323256.23 total 110.11 last 86.78) total: 10000 open/close in 118.44 seconds: 84.43 ops/second - unlinked 0 (time 1767323268 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1061 (time 1767323345.92 total 10.01 last 106.00) - open/close 1916 (time 1767323355.92 total 20.01 last 85.49) - open/close 2896 (time 1767323365.92 total 30.01 last 98.00) - open/close 3724 (time 1767323375.92 total 40.01 last 82.80) - open/close 4528 (time 1767323385.93 total 50.02 last 80.36) - open/close 5409 (time 1767323395.93 total 60.02 last 88.08) - open/close 6169 (time 1767323405.94 total 70.03 last 75.93) - open/close 7078 (time 1767323415.96 total 80.05 last 90.73) - open/close 7760 (time 1767323425.96 total 90.05 last 68.17) - open/close 8593 (time 1767323435.97 total 100.06 last 83.26) - open/close 9358 (time 1767323445.97 total 110.06 last 76.48) total: 10000 open/close in 117.66 seconds: 84.99 ops/second - unlinked 0 (time 1767323457 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 865 (time 1767323538.10 total 10.00 last 86.50) - open/close 1682 (time 1767323548.10 total 20.01 last 81.64) - open/close 2597 (time 1767323558.11 total 30.01 last 91.44) - open/close 3413 (time 1767323568.12 total 40.02 last 81.56) - open/close 4402 (time 1767323578.12 total 50.02 last 98.88) - open/close 5346 (time 1767323588.12 total 60.03 last 94.35) - open/close 6210 (time 1767323598.12 total 70.03 last 86.39) - open/close 7167 (time 1767323608.13 total 80.03 last 95.66) - open/close 8104 (time 1767323618.13 total 90.03 last 93.67) - open/close 9077 (time 1767323628.14 total 100.04 last 97.21) - open/close 9964 (time 1767323638.15 total 110.05 last 88.61) total: 10000 open/close in 110.50 seconds: 90.49 ops/second - unlinked 0 (time 1767323641 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre On OST0, 11885 used inodes rc=0 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.117@tcp autodegrade=on sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.117@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.117@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local oleg417-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg417-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2