== conf-sanity test 69: replace an OST with the same index ========================================================== 04:23:02 (1788510182) start mds service on oleg450-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1933656 Sep 3 13:58 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: kmemleak disabled oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: detected 4 online CPUs by sysfs oleg450-server: MODOPTS_LIBCFS= oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg450-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg450-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg450-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg450-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 592 (time 1788510275.32 total 10.02 last 59.09) - open/close 1221 (time 1788510285.33 total 20.03 last 62.86) - open/close 1876 (time 1788510295.36 total 30.05 last 65.33) - open/close 2457 (time 1788510305.36 total 40.06 last 58.07) - open/close 3148 (time 1788510315.37 total 50.07 last 69.03) - open/close 3759 (time 1788510325.39 total 60.09 last 60.98) - open/close 4350 (time 1788510335.40 total 70.09 last 59.07) - open/close 4944 (time 1788510345.41 total 80.10 last 59.33) - open/close 5567 (time 1788510355.43 total 90.13 last 62.15) - open/close 6261 (time 1788510365.47 total 100.16 last 69.15) - open/close 6915 (time 1788510375.49 total 110.19 last 65.24) - open/close 7524 (time 1788510385.50 total 120.20 last 60.85) - open/close 8199 (time 1788510395.50 total 130.20 last 67.48) - open/close 8833 (time 1788510405.51 total 140.21 last 63.36) - open/close 9424 (time 1788510415.52 total 150.21 last 59.07) total: 10000 open/close in 159.71 seconds: 62.61 ops/second - unlinked 0 (time 1788510430 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 680 (time 1788510548.64 total 10.01 last 67.94) - open/close 1323 (time 1788510558.65 total 20.02 last 64.20) - open/close 1953 (time 1788510568.66 total 30.04 last 62.93) - open/close 2603 (time 1788510578.70 total 40.07 last 64.79) - open/close 3190 (time 1788510588.70 total 50.07 last 58.67) - open/close 3763 (time 1788510598.71 total 60.08 last 57.25) - open/close 4356 (time 1788510608.71 total 70.08 last 59.29) - open/close 4962 (time 1788510618.73 total 80.10 last 60.52) - open/close 5550 (time 1788510628.74 total 90.11 last 58.71) - open/close 6179 (time 1788510638.75 total 100.12 last 62.85) - open/close 6819 (time 1788510648.77 total 110.15 last 63.83) - open/close 7407 (time 1788510658.79 total 120.16 last 58.73) - open/close 8033 (time 1788510668.80 total 130.17 last 62.55) - open/close 8587 (time 1788510678.80 total 140.17 last 55.38) - open/close 9202 (time 1788510688.80 total 150.18 last 61.46) - open/close 9791 (time 1788510698.82 total 160.19 last 58.83) total: 10000 open/close in 163.42 seconds: 61.19 ops/second - unlinked 0 (time 1788510707 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 595 (time 1788510829.73 total 10.01 last 59.45) - open/close 1125 (time 1788510839.76 total 20.04 last 52.84) - open/close 1704 (time 1788510849.78 total 30.06 last 57.77) - open/close 2362 (time 1788510859.82 total 40.10 last 65.57) - open/close 2905 (time 1788510869.84 total 50.12 last 54.19) - open/close 3449 (time 1788510879.84 total 60.12 last 54.39) - open/close 3928 (time 1788510889.85 total 70.13 last 47.85) - open/close 4396 (time 1788510899.85 total 80.13 last 46.79) - open/close 4975 (time 1788510909.86 total 90.14 last 57.88) - open/close 5514 (time 1788510919.87 total 100.15 last 53.80) - open/close 6133 (time 1788510929.88 total 110.16 last 61.86) - open/close 6697 (time 1788510939.88 total 120.16 last 56.40) - open/close 7309 (time 1788510949.92 total 130.20 last 60.99) - open/close 7869 (time 1788510959.92 total 140.20 last 55.98) - open/close 8353 (time 1788510969.93 total 150.21 last 48.35) - open/close 8848 (time 1788510979.97 total 160.25 last 49.28) - open/close 9400 (time 1788510990.00 total 170.28 last 55.04) - open/close 9921 (time 1788511000.02 total 180.30 last 52.03) total: 10000 open/close in 181.63 seconds: 55.06 ops/second - unlinked 0 (time 1788511006 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 545 (time 1788511126.63 total 10.01 last 54.45) - open/close 1123 (time 1788511136.63 total 20.01 last 57.78) - open/close 1706 (time 1788511146.64 total 30.02 last 58.25) - open/close 2367 (time 1788511156.65 total 40.04 last 66.00) - open/close 2993 (time 1788511166.65 total 50.04 last 62.59) - open/close 3649 (time 1788511176.67 total 60.05 last 65.49) - open/close 4319 (time 1788511186.68 total 70.06 last 66.97) - open/close 4989 (time 1788511196.68 total 80.07 last 66.96) - open/close 5685 (time 1788511206.69 total 90.07 last 69.54) - open/close 6278 (time 1788511216.70 total 100.08 last 59.26) - open/close 6875 (time 1788511226.70 total 110.08 last 59.69) - open/close 7556 (time 1788511236.71 total 120.09 last 68.02) - open/close 8160 (time 1788511246.71 total 130.10 last 60.38) - open/close 8765 (time 1788511256.72 total 140.11 last 60.44) - open/close 9357 (time 1788511266.80 total 150.19 last 58.74) - open/close 9893 (time 1788511276.82 total 160.20 last 53.52) total: 10000 open/close in 162.96 seconds: 61.36 ops/second - unlinked 0 (time 1788511285 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 543 (time 1788511400.38 total 10.02 last 54.19) - open/close 1175 (time 1788511410.38 total 20.03 last 63.14) - open/close 1768 (time 1788511420.39 total 30.04 last 59.25) - open/close 2463 (time 1788511430.39 total 40.04 last 69.49) - open/close 3056 (time 1788511440.41 total 50.06 last 59.18) - open/close 3562 (time 1788511450.45 total 60.10 last 50.40) - open/close 4098 (time 1788511460.47 total 70.11 last 53.53) - open/close 4595 (time 1788511470.47 total 80.11 last 49.69) - open/close 5140 (time 1788511480.47 total 90.12 last 54.47) - open/close 5787 (time 1788511490.49 total 100.14 last 64.57) - open/close 6401 (time 1788511500.51 total 110.15 last 61.32) - open/close 7014 (time 1788511510.51 total 120.15 last 61.29) - open/close 7678 (time 1788511520.51 total 130.16 last 66.37) - open/close 8377 (time 1788511530.52 total 140.17 last 69.84) - open/close 9001 (time 1788511540.53 total 150.17 last 62.36) - open/close 9578 (time 1788511550.54 total 160.19 last 57.63) total: 10000 open/close in 167.05 seconds: 59.86 ops/second - unlinked 0 (time 1788511562 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 602 (time 1788511677.59 total 10.01 last 60.16) - open/close 1175 (time 1788511687.59 total 20.02 last 57.25) - open/close 1778 (time 1788511697.60 total 30.02 last 60.28) - open/close 2284 (time 1788511707.61 total 40.03 last 50.56) - open/close 2807 (time 1788511717.62 total 50.04 last 52.23) - open/close 3271 (time 1788511727.66 total 60.08 last 46.22) - open/close 3683 (time 1788511737.66 total 70.08 last 41.20) - open/close 4217 (time 1788511747.66 total 80.08 last 53.38) - open/close 4731 (time 1788511757.66 total 90.08 last 51.40) - open/close 5309 (time 1788511767.66 total 100.09 last 57.78) - open/close 5934 (time 1788511777.68 total 110.10 last 62.41) - open/close 6510 (time 1788511787.68 total 120.10 last 57.58) - open/close 7120 (time 1788511797.69 total 130.11 last 60.94) - open/close 7591 (time 1788511807.71 total 140.13 last 47.02) - open/close 8011 (time 1788511817.73 total 150.16 last 41.89) - open/close 8484 (time 1788511827.77 total 160.19 last 47.14) - open/close 9038 (time 1788511837.77 total 170.19 last 55.39) - open/close 9616 (time 1788511847.77 total 180.20 last 57.78) total: 10000 open/close in 186.78 seconds: 53.54 ops/second - unlinked 0 (time 1788511858 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 605 (time 1788511980.23 total 10.01 last 60.45) - open/close 1181 (time 1788511990.24 total 20.01 last 57.57) - open/close 1851 (time 1788512000.26 total 30.03 last 66.88) - open/close 2464 (time 1788512010.27 total 40.04 last 61.23) - open/close 3118 (time 1788512020.28 total 50.06 last 65.31) - open/close 3812 (time 1788512030.29 total 60.06 last 69.36) - open/close 4439 (time 1788512040.30 total 70.07 last 62.65) - open/close 5018 (time 1788512050.30 total 80.07 last 57.89) - open/close 5619 (time 1788512060.33 total 90.10 last 59.93) - open/close 6224 (time 1788512070.35 total 100.12 last 60.40) - open/close 6895 (time 1788512080.35 total 110.12 last 67.09) - open/close 7517 (time 1788512090.36 total 120.14 last 62.11) - open/close 8131 (time 1788512100.37 total 130.14 last 61.37) - open/close 8749 (time 1788512110.38 total 140.15 last 61.75) - open/close 9408 (time 1788512120.38 total 150.15 last 65.88) total: 10000 open/close in 159.10 seconds: 62.85 ops/second - unlinked 0 (time 1788512133 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 519 (time 1788512247.91 total 10.01 last 51.83) - open/close 1020 (time 1788512257.91 total 20.01 last 50.10) - open/close 1613 (time 1788512267.93 total 30.03 last 59.20) - open/close 2232 (time 1788512277.94 total 40.04 last 61.87) - open/close 2830 (time 1788512287.94 total 50.04 last 59.79) - open/close 3406 (time 1788512297.94 total 60.04 last 57.58) - open/close 3932 (time 1788512307.95 total 70.05 last 52.53) - open/close 4534 (time 1788512317.95 total 80.05 last 60.19) - open/close 5151 (time 1788512327.96 total 90.06 last 61.68) - open/close 5810 (time 1788512337.96 total 100.06 last 65.89) - open/close 6406 (time 1788512347.96 total 110.06 last 59.59) - open/close 6991 (time 1788512357.96 total 120.06 last 58.48) - open/close 7572 (time 1788512367.97 total 130.07 last 58.06) - open/close 8114 (time 1788512377.99 total 140.09 last 54.11) - open/close 8729 (time 1788512388.01 total 150.11 last 61.36) - open/close 9303 (time 1788512398.01 total 160.11 last 57.38) - open/close 9875 (time 1788512408.09 total 170.19 last 56.77) total: 10000 open/close in 172.47 seconds: 57.98 ops/second - unlinked 0 (time 1788512415 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 592 (time 1788512531.37 total 10.03 last 59.04) - open/close 1113 (time 1788512541.38 total 20.03 last 52.07) - open/close 1600 (time 1788512551.38 total 30.03 last 48.70) - open/close 2231 (time 1788512561.39 total 40.05 last 63.02) - open/close 2849 (time 1788512571.40 total 50.05 last 61.75) - open/close 3438 (time 1788512581.41 total 60.06 last 58.87) - open/close 3972 (time 1788512591.41 total 70.07 last 53.36) - open/close 4511 (time 1788512601.44 total 80.09 last 53.75) - open/close 5009 (time 1788512611.44 total 90.10 last 49.79) - open/close 5578 (time 1788512621.45 total 100.10 last 56.85) - open/close 6132 (time 1788512631.46 total 110.12 last 55.33) - open/close 6753 (time 1788512641.48 total 120.13 last 62.02) - open/close 7359 (time 1788512651.48 total 130.14 last 60.55) - open/close 8009 (time 1788512661.49 total 140.14 last 65.00) - open/close 8510 (time 1788512671.50 total 150.16 last 50.02) - open/close 9022 (time 1788512681.51 total 160.16 last 51.15) - open/close 9490 (time 1788512691.52 total 170.17 last 46.77) total: 10000 open/close in 179.62 seconds: 55.67 ops/second - unlinked 0 (time 1788512706 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second - open/close 582 (time 1788512836.77 total 10.01 last 58.12) - open/close 1166 (time 1788512846.80 total 20.04 last 58.25) - open/close 1717 (time 1788512856.80 total 30.05 last 55.07) - open/close 2205 (time 1788512866.81 total 40.05 last 48.77) - open/close 2772 (time 1788512876.82 total 50.06 last 56.64) - open/close 3342 (time 1788512886.83 total 60.07 last 56.94) - open/close 3967 (time 1788512896.84 total 70.08 last 62.48) - open/close 4480 (time 1788512906.84 total 80.08 last 51.27) - open/close 5035 (time 1788512916.88 total 90.12 last 55.30) - open/close 5601 (time 1788512926.89 total 100.14 last 56.50) - open/close 6140 (time 1788512936.91 total 110.15 last 53.83) - open/close 6714 (time 1788512946.91 total 120.16 last 57.37) - open/close 7361 (time 1788512956.92 total 130.16 last 64.66) - open/close 7964 (time 1788512966.92 total 140.16 last 60.29) - open/close 8513 (time 1788512976.94 total 150.18 last 54.82) - open/close 9119 (time 1788512986.95 total 160.19 last 60.52) - open/close 9644 (time 1788512996.95 total 170.20 last 52.48) total: 10000 open/close in 175.91 seconds: 56.85 ops/second - unlinked 0 (time 1788513007 ; total 0 ; last 0) total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg450-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.150@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.150@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg450-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg450-server: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg450-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre On OST0, 16846 used inodes rc=0 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg450-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.150@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.150@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.150@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.150@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.204.150@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg450-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg450-server: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg450-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg450-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server unloading modules via unload_modules_local on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing unload_modules_local oleg450-server: kmemleak disabled modules unloaded. pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2