== conf-sanity test 69: replace an OST with the same index ========================================================== 14:45:09 (1789065909) start mds service on oleg145-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 10 03:22 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: kmemleak disabled oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: MODOPTS_LIBCFS= oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg145-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg145-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg145-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 846 (time 1789066007.40 total 10.00 last 84.60) - open/close 1691 (time 1789066017.44 total 20.03 last 84.23) - open/close 2483 (time 1789066027.44 total 30.03 last 79.18) - open/close 3235 (time 1789066037.44 total 40.04 last 75.16) - open/close 4109 (time 1789066047.45 total 50.04 last 87.37) - open/close 4916 (time 1789066057.45 total 60.04 last 80.70) - open/close 5707 (time 1789066067.46 total 70.05 last 79.02) - open/close 6724 (time 1789066077.46 total 80.06 last 101.68) - open/close 7543 (time 1789066087.46 total 90.06 last 81.85) - open/close 8372 (time 1789066097.48 total 100.07 last 82.79) - open/close 9009 (time 1789066107.48 total 110.08 last 63.68) - open/close 9613 (time 1789066117.49 total 120.08 last 60.37) total: 10000 open/close in 124.71 seconds: 80.19 ops/second - unlinked 0 (time 1789066127 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 679 (time 1789066225.17 total 10.01 last 67.84) - open/close 1395 (time 1789066235.17 total 20.02 last 71.53) - open/close 2092 (time 1789066245.19 total 30.03 last 69.62) - open/close 2944 (time 1789066255.19 total 40.04 last 85.13) - open/close 3760 (time 1789066265.19 total 50.04 last 81.60) - open/close 4476 (time 1789066275.21 total 60.06 last 71.47) - open/close 5192 (time 1789066285.22 total 70.06 last 71.55) - open/close 5870 (time 1789066295.23 total 80.07 last 67.75) - open/close 6273 (time 1789066305.23 total 90.07 last 40.29) - open/close 6848 (time 1789066315.23 total 100.08 last 57.48) - open/close 7552 (time 1789066325.23 total 110.08 last 70.39) - open/close 8293 (time 1789066335.24 total 120.08 last 74.04) - open/close 9107 (time 1789066345.25 total 130.09 last 81.34) - open/close 9809 (time 1789066355.26 total 140.10 last 70.14) total: 10000 open/close in 144.63 seconds: 69.14 ops/second - unlinked 0 (time 1789066365 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 702 (time 1789066463.26 total 10.01 last 70.14) - open/close 1424 (time 1789066473.28 total 20.03 last 72.02) - open/close 1942 (time 1789066483.29 total 30.04 last 51.75) - open/close 2383 (time 1789066493.32 total 40.07 last 43.96) - open/close 2910 (time 1789066503.34 total 50.09 last 52.62) - open/close 3696 (time 1789066513.34 total 60.09 last 78.59) - open/close 4531 (time 1789066523.34 total 70.09 last 83.46) - open/close 5356 (time 1789066533.35 total 80.10 last 82.46) - open/close 6286 (time 1789066543.36 total 90.11 last 92.87) - open/close 6944 (time 1789066553.38 total 100.13 last 65.70) - open/close 7678 (time 1789066563.38 total 110.13 last 73.37) - open/close 8407 (time 1789066573.39 total 120.14 last 72.82) - open/close 9105 (time 1789066583.40 total 130.16 last 69.72) - open/close 9968 (time 1789066593.41 total 140.16 last 86.29) total: 10000 open/close in 140.47 seconds: 71.19 ops/second - unlinked 0 (time 1789066598 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 765 (time 1789066695.09 total 10.01 last 76.46) - open/close 1316 (time 1789066705.10 total 20.01 last 55.07) - open/close 2094 (time 1789066715.11 total 30.03 last 77.66) - open/close 2958 (time 1789066725.13 total 40.05 last 86.25) - open/close 3784 (time 1789066735.15 total 50.06 last 82.50) - open/close 4768 (time 1789066745.15 total 60.07 last 98.33) - open/close 5606 (time 1789066755.16 total 70.07 last 83.76) - open/close 6336 (time 1789066765.17 total 80.08 last 72.93) - open/close 7157 (time 1789066775.18 total 90.09 last 82.02) - open/close 8031 (time 1789066785.18 total 100.10 last 87.34) - open/close 8739 (time 1789066795.19 total 110.10 last 70.77) - open/close 9534 (time 1789066805.19 total 120.10 last 79.49) total: 10000 open/close in 126.73 seconds: 78.91 ops/second - unlinked 0 (time 1789066816 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 725 (time 1789066915.82 total 10.01 last 72.45) - open/close 1434 (time 1789066925.82 total 20.01 last 70.88) - open/close 2198 (time 1789066935.83 total 30.02 last 76.34) - open/close 2981 (time 1789066945.83 total 40.02 last 78.27) - open/close 3781 (time 1789066955.83 total 50.02 last 80.00) - open/close 4520 (time 1789066965.85 total 60.03 last 73.82) - open/close 5313 (time 1789066975.85 total 70.04 last 79.24) - open/close 6143 (time 1789066985.86 total 80.04 last 82.98) - open/close 6865 (time 1789066995.87 total 90.05 last 72.11) - open/close 7656 (time 1789067005.87 total 100.06 last 79.05) - open/close 8393 (time 1789067015.88 total 110.06 last 73.68) - open/close 9190 (time 1789067025.88 total 120.07 last 79.64) - open/close 9969 (time 1789067035.89 total 130.07 last 77.89) total: 10000 open/close in 130.38 seconds: 76.70 ops/second - unlinked 0 (time 1789067041 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 709 (time 1789067133.74 total 10.00 last 70.86) - open/close 1504 (time 1789067143.74 total 20.01 last 79.45) - open/close 2389 (time 1789067153.75 total 30.02 last 88.44) - open/close 3151 (time 1789067163.75 total 40.02 last 76.19) - open/close 3894 (time 1789067173.76 total 50.03 last 74.24) - open/close 4665 (time 1789067183.79 total 60.06 last 76.86) - open/close 5398 (time 1789067193.79 total 70.06 last 73.29) - open/close 6010 (time 1789067203.80 total 80.07 last 61.16) - open/close 6570 (time 1789067213.80 total 90.07 last 55.99) - open/close 7357 (time 1789067223.84 total 100.11 last 78.39) - open/close 8250 (time 1789067233.84 total 110.11 last 89.25) - open/close 9055 (time 1789067243.84 total 120.11 last 80.49) - open/close 9806 (time 1789067253.84 total 130.11 last 75.10) total: 10000 open/close in 132.69 seconds: 75.36 ops/second - unlinked 0 (time 1789067261 ; total 1 ; last 1) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 849 (time 1789067355.40 total 10.01 last 84.80) - open/close 1646 (time 1789067365.41 total 20.02 last 79.62) - open/close 2305 (time 1789067375.42 total 30.03 last 65.83) - open/close 3012 (time 1789067385.43 total 40.04 last 70.64) - open/close 3694 (time 1789067395.43 total 50.04 last 68.18) - open/close 4441 (time 1789067405.44 total 60.06 last 74.60) - open/close 5200 (time 1789067415.46 total 70.07 last 75.77) - open/close 6086 (time 1789067425.47 total 80.08 last 88.54) - open/close 6978 (time 1789067435.47 total 90.09 last 89.13) - open/close 7879 (time 1789067445.48 total 100.10 last 90.04) - open/close 8752 (time 1789067455.49 total 110.11 last 87.20) - open/close 9634 (time 1789067465.51 total 120.12 last 88.05) total: 10000 open/close in 125.00 seconds: 80.00 ops/second - unlinked 0 (time 1789067474 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 885 (time 1789067563.88 total 10.01 last 88.43) - open/close 1664 (time 1789067573.88 total 20.01 last 77.89) - open/close 2365 (time 1789067583.89 total 30.02 last 70.01) - open/close 3100 (time 1789067593.90 total 40.03 last 73.45) - open/close 3937 (time 1789067603.90 total 50.03 last 83.68) - open/close 4834 (time 1789067613.91 total 60.04 last 89.65) - open/close 5604 (time 1789067623.91 total 70.04 last 76.97) - open/close 6303 (time 1789067633.92 total 80.04 last 69.88) - open/close 7223 (time 1789067643.92 total 90.05 last 91.96) - open/close 8191 (time 1789067653.93 total 100.05 last 96.75) - open/close 8964 (time 1789067663.94 total 110.07 last 77.19) - open/close 9689 (time 1789067673.95 total 120.07 last 72.47) total: 10000 open/close in 123.85 seconds: 80.74 ops/second - unlinked 0 (time 1789067681 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 810 (time 1789067779.00 total 10.01 last 80.89) - open/close 1494 (time 1789067789.00 total 20.02 last 68.37) - open/close 2197 (time 1789067799.01 total 30.02 last 70.27) - open/close 2981 (time 1789067809.01 total 40.03 last 78.34) - open/close 3608 (time 1789067819.02 total 50.04 last 62.66) - open/close 4314 (time 1789067829.02 total 60.04 last 70.59) - open/close 5109 (time 1789067839.03 total 70.05 last 79.40) - open/close 6029 (time 1789067849.03 total 80.05 last 92.00) - open/close 6924 (time 1789067859.04 total 90.05 last 89.45) - open/close 7705 (time 1789067869.05 total 100.07 last 78.02) - open/close 8560 (time 1789067879.05 total 110.07 last 85.49) - open/close 9266 (time 1789067889.06 total 120.08 last 70.53) total: 10000 open/close in 129.12 seconds: 77.45 ops/second - unlinked 0 (time 1789067902 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 890 (time 1789067992.00 total 10.01 last 88.88) - open/close 1682 (time 1789068002.01 total 20.02 last 79.11) - open/close 2496 (time 1789068012.02 total 30.03 last 81.36) - open/close 3287 (time 1789068022.02 total 40.03 last 79.09) - open/close 4153 (time 1789068032.03 total 50.04 last 86.48) - open/close 4971 (time 1789068042.03 total 60.04 last 81.80) - open/close 5746 (time 1789068052.04 total 70.05 last 77.46) - open/close 6617 (time 1789068062.04 total 80.05 last 87.07) - open/close 7545 (time 1789068072.05 total 90.06 last 92.75) - open/close 8392 (time 1789068082.06 total 100.07 last 84.62) - open/close 9160 (time 1789068092.07 total 110.08 last 76.68) - open/close 9968 (time 1789068102.09 total 120.10 last 80.65) total: 10000 open/close in 120.51 seconds: 82.98 ops/second - unlinked 0 (time 1789068106 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.145@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg145-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.145@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.145@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.145@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.201.145@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg145-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server unloading modules via unload_modules_local on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local oleg145-server: kmemleak disabled modules unloaded. pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2