== conf-sanity test 69: replace an OST with the same index ========================================================== 19:13:47 (1788390827) start mds service on oleg348-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 2 18:43 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: kmemleak disabled oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg348-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg348-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg348-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 802 (time 1788390896.73 total 10.00 last 80.17) - open/close 1602 (time 1788390906.73 total 20.01 last 79.95) - open/close 2387 (time 1788390916.73 total 30.01 last 78.49) - open/close 3172 (time 1788390926.74 total 40.02 last 78.43) - open/close 4012 (time 1788390936.75 total 50.02 last 83.96) - open/close 4835 (time 1788390946.75 total 60.03 last 82.26) - open/close 5707 (time 1788390956.75 total 70.03 last 87.17) - open/close 6693 (time 1788390966.76 total 80.04 last 98.54) - open/close 7794 (time 1788390976.77 total 90.04 last 110.03) - open/close 8719 (time 1788390986.77 total 100.05 last 92.49) - open/close 9646 (time 1788390996.77 total 110.05 last 92.65) total: 10000 open/close in 113.71 seconds: 87.94 ops/second - unlinked 0 (time 1788391003 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 864 (time 1788391084.63 total 10.01 last 86.31) - open/close 1742 (time 1788391094.63 total 20.01 last 87.79) - open/close 2567 (time 1788391104.65 total 30.03 last 82.35) - open/close 3380 (time 1788391114.65 total 40.04 last 81.25) - open/close 4242 (time 1788391124.66 total 50.04 last 86.17) - open/close 5134 (time 1788391134.66 total 60.05 last 89.15) - open/close 6103 (time 1788391144.66 total 70.05 last 96.87) - open/close 7019 (time 1788391154.66 total 80.05 last 91.60) - open/close 7896 (time 1788391164.67 total 90.05 last 87.65) - open/close 8953 (time 1788391174.68 total 100.07 last 105.56) total: 10000 open/close in 109.51 seconds: 91.31 ops/second - unlinked 0 (time 1788391186 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1128 (time 1788391251.96 total 10.00 last 112.80) - open/close 2276 (time 1788391261.98 total 20.01 last 114.67) - open/close 3394 (time 1788391271.98 total 30.02 last 111.72) - open/close 4461 (time 1788391281.99 total 40.02 last 106.66) - open/close 5680 (time 1788391291.99 total 50.03 last 121.81) - open/close 6975 (time 1788391302.00 total 60.03 last 129.46) - open/close 8050 (time 1788391312.00 total 70.04 last 107.46) - open/close 9091 (time 1788391322.01 total 80.05 last 104.00) total: 10000 open/close in 88.70 seconds: 112.74 ops/second - unlinked 0 (time 1788391333 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1310 (time 1788391393.07 total 10.00 last 130.99) - open/close 2574 (time 1788391403.08 total 20.01 last 126.32) - open/close 3729 (time 1788391413.08 total 30.01 last 115.47) - open/close 5092 (time 1788391423.08 total 40.01 last 136.29) - open/close 6213 (time 1788391433.09 total 50.01 last 112.07) - open/close 7293 (time 1788391443.10 total 60.02 last 107.89) - open/close 8674 (time 1788391453.10 total 70.03 last 138.04) - open/close 9771 (time 1788391463.10 total 80.03 last 109.70) total: 10000 open/close in 82.17 seconds: 121.70 ops/second - unlinked 0 (time 1788391467 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1196 (time 1788391532.91 total 10.00 last 119.56) - open/close 2313 (time 1788391542.91 total 20.00 last 111.68) - open/close 3338 (time 1788391552.91 total 30.01 last 102.45) - open/close 4507 (time 1788391562.92 total 40.01 last 116.86) - open/close 5559 (time 1788391572.92 total 50.02 last 105.15) - open/close 6671 (time 1788391582.92 total 60.02 last 111.19) - open/close 7843 (time 1788391592.92 total 70.02 last 117.19) - open/close 9035 (time 1788391602.92 total 80.02 last 119.18) total: 10000 open/close in 88.44 seconds: 113.07 ops/second - unlinked 0 (time 1788391613 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1105 (time 1788391672.38 total 10.00 last 110.46) - open/close 2360 (time 1788391682.39 total 20.01 last 125.41) - open/close 3531 (time 1788391692.39 total 30.01 last 117.09) - open/close 4640 (time 1788391702.39 total 40.01 last 110.89) - open/close 5701 (time 1788391712.40 total 50.02 last 106.00) - open/close 6692 (time 1788391722.40 total 60.03 last 99.07) - open/close 7737 (time 1788391732.41 total 70.03 last 104.45) - open/close 8983 (time 1788391742.41 total 80.04 last 124.51) total: 10000 open/close in 87.21 seconds: 114.67 ops/second - unlinked 0 (time 1788391751 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1339 (time 1788391809.39 total 10.01 last 133.79) - open/close 2578 (time 1788391819.40 total 20.01 last 123.85) - open/close 3875 (time 1788391829.40 total 30.02 last 129.64) - open/close 5140 (time 1788391839.41 total 40.02 last 126.43) - open/close 6331 (time 1788391849.42 total 50.03 last 119.01) - open/close 7526 (time 1788391859.42 total 60.03 last 119.46) - open/close 8837 (time 1788391869.42 total 70.04 last 131.06) total: 10000 open/close in 78.88 seconds: 126.78 ops/second - unlinked 0 (time 1788391880 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1248 (time 1788391933.81 total 10.00 last 124.79) - open/close 2432 (time 1788391943.82 total 20.01 last 118.33) - open/close 3774 (time 1788391953.82 total 30.01 last 134.16) - open/close 5435 (time 1788391963.82 total 40.01 last 166.06) - open/close 7000 (time 1788391973.84 total 50.02 last 156.33) - open/close 8544 (time 1788391983.84 total 60.03 last 154.29) - open/close 9940 (time 1788391993.85 total 70.04 last 139.50) total: 10000 open/close in 70.53 seconds: 141.79 ops/second - unlinked 0 (time 1788391996 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1410 (time 1788392047.96 total 10.01 last 140.90) - open/close 2755 (time 1788392057.97 total 20.01 last 134.39) - open/close 3946 (time 1788392067.97 total 30.02 last 119.09) - open/close 5171 (time 1788392077.97 total 40.02 last 122.49) - open/close 6484 (time 1788392087.97 total 50.02 last 131.25) - open/close 7733 (time 1788392097.98 total 60.02 last 124.86) - open/close 9263 (time 1788392107.98 total 70.03 last 152.95) total: 10000 open/close in 75.25 seconds: 132.89 ops/second - unlinked 0 (time 1788392115 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1057 (time 1788392166.26 total 10.00 last 105.65) - open/close 2355 (time 1788392176.27 total 20.01 last 129.70) - open/close 4092 (time 1788392186.27 total 30.01 last 173.67) - open/close 5409 (time 1788392196.27 total 40.02 last 131.67) - open/close 6863 (time 1788392206.28 total 50.02 last 145.34) - open/close 8384 (time 1788392216.28 total 60.02 last 152.10) - open/close 9887 (time 1788392226.28 total 70.02 last 150.25) total: 10000 open/close in 70.66 seconds: 141.52 ops/second - unlinked 0 (time 1788392228 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.148@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.148@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg348-server: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg348-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre On OST0, 19462 used inodes rc=0 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.148@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.148@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.148@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.148@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.148@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg348-server: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg348-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server unloading modules via unload_modules_local on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing unload_modules_local oleg348-server: kmemleak disabled modules unloaded. pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2