== conf-sanity test 69: replace an OST with the same index ========================================================== 16:58:18 (1779310698) start mds service on oleg433-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: MODOPTS_LIBCFS= oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg433-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg433-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg433-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 834 (time 1779310772.45 total 10.01 last 83.35) - open/close 1789 (time 1779310782.45 total 20.01 last 95.43) - open/close 2962 (time 1779310792.47 total 30.03 last 117.15) - open/close 3981 (time 1779310802.48 total 40.03 last 101.81) - open/close 4875 (time 1779310812.49 total 50.05 last 89.25) - open/close 5848 (time 1779310822.49 total 60.05 last 97.28) - open/close 6883 (time 1779310832.50 total 70.06 last 103.47) - open/close 7893 (time 1779310842.50 total 80.06 last 100.95) - open/close 8858 (time 1779310852.51 total 90.07 last 96.41) - open/close 9801 (time 1779310862.52 total 100.08 last 94.25) total: 10000 open/close in 102.31 seconds: 97.74 ops/second - unlinked 0 (time 1779310867 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 975 (time 1779310941.70 total 10.01 last 97.38) - open/close 1972 (time 1779310951.70 total 20.01 last 99.69) - open/close 3029 (time 1779310961.70 total 30.01 last 105.69) - open/close 4067 (time 1779310971.71 total 40.02 last 103.77) - open/close 5060 (time 1779310981.71 total 50.02 last 99.26) - open/close 6099 (time 1779310991.71 total 60.02 last 103.89) - open/close 7050 (time 1779311001.72 total 70.03 last 95.00) - open/close 8018 (time 1779311011.73 total 80.04 last 96.77) - open/close 8947 (time 1779311021.73 total 90.04 last 92.85) - open/close 9889 (time 1779311031.74 total 100.05 last 94.09) total: 10000 open/close in 101.16 seconds: 98.85 ops/second - unlinked 0 (time 1779311035 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1093 (time 1779311102.18 total 10.01 last 109.23) - open/close 2235 (time 1779311112.19 total 20.01 last 114.11) - open/close 3466 (time 1779311122.19 total 30.02 last 123.04) - open/close 4634 (time 1779311132.19 total 40.02 last 116.74) - open/close 5851 (time 1779311142.20 total 50.03 last 121.68) - open/close 6851 (time 1779311152.20 total 60.03 last 99.99) - open/close 7982 (time 1779311162.21 total 70.04 last 112.99) - open/close 8989 (time 1779311172.21 total 80.04 last 100.69) total: 10000 open/close in 89.15 seconds: 112.17 ops/second - unlinked 0 (time 1779311183 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1266 (time 1779311249.43 total 10.01 last 126.53) - open/close 2592 (time 1779311259.44 total 20.02 last 132.43) - open/close 3999 (time 1779311269.44 total 30.02 last 140.68) - open/close 5504 (time 1779311279.45 total 40.02 last 150.43) - open/close 7108 (time 1779311289.45 total 50.03 last 160.35) - open/close 8571 (time 1779311299.45 total 60.03 last 146.28) total: 10000 open/close in 69.08 seconds: 144.77 ops/second - unlinked 0 (time 1779311310 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1512 (time 1779311361.86 total 10.00 last 151.18) - open/close 3010 (time 1779311371.86 total 20.00 last 149.79) - open/close 4193 (time 1779311381.86 total 30.00 last 118.28) - open/close 5417 (time 1779311391.86 total 40.01 last 122.35) - open/close 6564 (time 1779311401.87 total 50.01 last 114.63) - open/close 7722 (time 1779311411.88 total 60.02 last 115.72) - open/close 8906 (time 1779311421.88 total 70.02 last 118.37) - open/close 9988 (time 1779311431.89 total 80.03 last 108.11) total: 10000 open/close in 80.17 seconds: 124.74 ops/second - unlinked 0 (time 1779311434 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1160 (time 1779311496.57 total 10.00 last 115.99) - open/close 2290 (time 1779311506.58 total 20.01 last 112.90) - open/close 3457 (time 1779311516.58 total 30.01 last 116.65) - open/close 4740 (time 1779311526.59 total 40.02 last 128.18) - open/close 5966 (time 1779311536.59 total 50.03 last 122.58) - open/close 7174 (time 1779311546.60 total 60.03 last 120.74) - open/close 8550 (time 1779311556.60 total 70.03 last 137.59) - open/close 9856 (time 1779311566.61 total 80.04 last 130.45) total: 10000 open/close in 81.32 seconds: 122.97 ops/second - unlinked 0 (time 1779311569 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1400 (time 1779311624.47 total 10.00 last 139.99) - open/close 3137 (time 1779311634.48 total 20.00 last 173.67) - open/close 4824 (time 1779311644.48 total 30.01 last 168.63) - open/close 5931 (time 1779311654.50 total 40.03 last 110.43) - open/close 6300 (time 1779311664.54 total 50.07 last 36.76) - open/close 6555 (time 1779311674.56 total 60.08 last 25.46) - open/close 7235 (time 1779311684.56 total 70.09 last 67.98) - open/close 7764 (time 1779311694.80 total 80.33 last 51.67) - open/close 8012 (time 1779311704.81 total 90.34 last 24.77) - open/close 8413 (time 1779311714.82 total 100.34 last 40.08) - open/close 8799 (time 1779311724.84 total 110.36 last 38.52) - open/close 9347 (time 1779311734.86 total 120.38 last 54.71) - open/close 9941 (time 1779311744.87 total 130.40 last 59.31) total: 10000 open/close in 131.02 seconds: 76.33 ops/second - unlinked 0 (time 1779311749 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 747 (time 1779311853.14 total 10.02 last 74.57) - open/close 1451 (time 1779311863.15 total 20.03 last 70.33) - open/close 2113 (time 1779311873.16 total 30.04 last 66.15) - open/close 2805 (time 1779311883.16 total 40.04 last 69.16) - open/close 3569 (time 1779311893.17 total 50.05 last 76.37) - open/close 4314 (time 1779311903.19 total 60.07 last 74.34) - open/close 5023 (time 1779311913.20 total 70.07 last 70.84) - open/close 5688 (time 1779311923.20 total 80.08 last 66.47) - open/close 6458 (time 1779311933.21 total 90.09 last 76.93) - open/close 7173 (time 1779311943.22 total 100.10 last 71.43) - open/close 7869 (time 1779311953.22 total 110.10 last 69.59) - open/close 8591 (time 1779311963.22 total 120.10 last 72.18) - open/close 9306 (time 1779311973.23 total 130.11 last 71.43) total: 10000 open/close in 139.94 seconds: 71.46 ops/second - unlinked 0 (time 1779311988 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 737 (time 1779312093.55 total 10.02 last 73.57) - open/close 1331 (time 1779312103.56 total 20.02 last 59.35) - open/close 2064 (time 1779312113.57 total 30.03 last 73.23) - open/close 2803 (time 1779312123.58 total 40.04 last 73.85) - open/close 3609 (time 1779312133.61 total 50.07 last 80.37) - open/close 4366 (time 1779312143.61 total 60.08 last 75.65) - open/close 5083 (time 1779312153.62 total 70.08 last 71.66) - open/close 5804 (time 1779312163.64 total 80.10 last 71.94) - open/close 6576 (time 1779312173.65 total 90.11 last 77.17) - open/close 7319 (time 1779312183.67 total 100.14 last 74.10) - open/close 7988 (time 1779312193.70 total 110.17 last 66.69) - open/close 8757 (time 1779312203.71 total 120.18 last 76.84) - open/close 9516 (time 1779312213.72 total 130.18 last 75.87) total: 10000 open/close in 137.66 seconds: 72.64 ops/second - unlinked 0 (time 1779312225 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 647 (time 1779312322.98 total 10.01 last 64.66) - open/close 1387 (time 1779312332.98 total 20.01 last 73.98) - open/close 2123 (time 1779312342.99 total 30.02 last 73.52) - open/close 2703 (time 1779312353.04 total 40.07 last 57.74) - open/close 3114 (time 1779312363.05 total 50.07 last 41.06) - open/close 3668 (time 1779312373.05 total 60.08 last 55.37) - open/close 4346 (time 1779312383.06 total 70.08 last 67.77) - open/close 5137 (time 1779312393.06 total 80.09 last 79.06) - open/close 5893 (time 1779312403.07 total 90.10 last 75.55) - open/close 6655 (time 1779312413.08 total 100.11 last 76.10) - open/close 7453 (time 1779312423.09 total 110.12 last 79.74) - open/close 8240 (time 1779312433.10 total 120.12 last 78.64) - open/close 8973 (time 1779312443.10 total 130.13 last 73.29) - open/close 9745 (time 1779312453.11 total 140.14 last 77.10) total: 10000 open/close in 143.33 seconds: 69.77 ops/second - unlinked 0 (time 1779312460 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.133@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre On OST0, 20520 used inodes rc=0 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.133@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.133@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.133@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.133@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg433-server unloading modules via unload_modules_local on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local oleg433-server: modules unloaded. pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2