== conf-sanity test 69: replace an OST with the same index ========================================================== 10:03:26 (1788530606) start mds service on oleg307-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921072 Sep 3 15:55 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: kmemleak disabled oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: detected 4 online CPUs by sysfs oleg307-server: MODOPTS_LIBCFS= oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg307-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg307-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg307-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg307-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg307-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 761 (time 1788530739.08 total 10.01 last 76.01) - open/close 1506 (time 1788530749.08 total 20.02 last 74.43) - open/close 2234 (time 1788530759.09 total 30.03 last 72.74) - open/close 2926 (time 1788530769.10 total 40.03 last 69.17) - open/close 3645 (time 1788530779.12 total 50.05 last 71.76) - open/close 4386 (time 1788530789.13 total 60.06 last 74.02) - open/close 5170 (time 1788530799.13 total 70.07 last 78.36) - open/close 5722 (time 1788530809.16 total 80.10 last 55.04) - open/close 6359 (time 1788530819.16 total 90.10 last 63.69) - open/close 6939 (time 1788530829.17 total 100.10 last 57.98) - open/close 7572 (time 1788530839.17 total 110.11 last 63.28) - open/close 8261 (time 1788530849.18 total 120.11 last 68.87) - open/close 8981 (time 1788530859.18 total 130.11 last 72.00) - open/close 9677 (time 1788530869.19 total 140.12 last 69.54) total: 10000 open/close in 145.51 seconds: 68.72 ops/second - unlinked 0 (time 1788530879 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 720 (time 1788530976.91 total 10.01 last 71.91) - open/close 1473 (time 1788530986.91 total 20.02 last 75.24) - open/close 2247 (time 1788530996.92 total 30.03 last 77.32) - open/close 2942 (time 1788531006.93 total 40.04 last 69.43) - open/close 3628 (time 1788531016.94 total 50.05 last 68.58) - open/close 4363 (time 1788531026.94 total 60.05 last 73.45) - open/close 5057 (time 1788531036.95 total 70.06 last 69.35) - open/close 5688 (time 1788531046.96 total 80.06 last 63.07) - open/close 6244 (time 1788531056.96 total 90.07 last 55.58) - open/close 6845 (time 1788531066.97 total 100.08 last 60.03) - open/close 7488 (time 1788531076.98 total 110.09 last 64.23) - open/close 8162 (time 1788531086.99 total 120.09 last 67.36) - open/close 8849 (time 1788531096.99 total 130.10 last 68.69) - open/close 9488 (time 1788531106.99 total 140.10 last 63.88) total: 10000 open/close in 147.37 seconds: 67.86 ops/second - unlinked 0 (time 1788531118 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 636 (time 1788531211.81 total 10.01 last 63.56) - open/close 1289 (time 1788531221.82 total 20.01 last 65.27) - open/close 1881 (time 1788531231.83 total 30.02 last 59.14) - open/close 2461 (time 1788531241.83 total 40.02 last 58.00) - open/close 3176 (time 1788531251.84 total 50.03 last 71.42) - open/close 3772 (time 1788531261.84 total 60.04 last 59.58) - open/close 4442 (time 1788531271.87 total 70.06 last 66.85) - open/close 5040 (time 1788531281.89 total 80.09 last 59.64) - open/close 5678 (time 1788531291.89 total 90.09 last 63.79) - open/close 6223 (time 1788531301.92 total 100.11 last 54.38) - open/close 6667 (time 1788531311.93 total 110.12 last 44.33) - open/close 7251 (time 1788531321.94 total 120.13 last 58.37) - open/close 7978 (time 1788531331.94 total 130.13 last 72.68) - open/close 8713 (time 1788531341.94 total 140.14 last 73.47) - open/close 9481 (time 1788531351.95 total 150.14 last 76.78) total: 10000 open/close in 157.43 seconds: 63.52 ops/second - unlinked 0 (time 1788531363 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 722 (time 1788531465.35 total 10.00 last 72.20) - open/close 1416 (time 1788531475.36 total 20.02 last 69.28) - open/close 2143 (time 1788531485.37 total 30.02 last 72.69) - open/close 2885 (time 1788531495.37 total 40.02 last 74.18) - open/close 3601 (time 1788531505.37 total 50.03 last 71.56) - open/close 4281 (time 1788531515.39 total 60.04 last 67.92) - open/close 4939 (time 1788531525.39 total 70.04 last 65.79) - open/close 5657 (time 1788531535.39 total 80.04 last 71.79) - open/close 6398 (time 1788531545.39 total 90.05 last 74.06) - open/close 7157 (time 1788531555.40 total 100.05 last 75.85) - open/close 7939 (time 1788531565.40 total 110.05 last 78.19) - open/close 8688 (time 1788531575.41 total 120.06 last 74.87) - open/close 9419 (time 1788531585.42 total 130.07 last 73.02) total: 10000 open/close in 137.85 seconds: 72.54 ops/second - unlinked 0 (time 1788531597 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 715 (time 1788531686.98 total 10.01 last 71.46) - open/close 1517 (time 1788531696.99 total 20.02 last 80.13) - open/close 2260 (time 1788531707.00 total 30.03 last 74.19) - open/close 2954 (time 1788531717.01 total 40.04 last 69.36) - open/close 3789 (time 1788531727.02 total 50.05 last 83.42) - open/close 4595 (time 1788531737.02 total 60.05 last 80.57) - open/close 5375 (time 1788531747.02 total 70.05 last 77.99) - open/close 6109 (time 1788531757.03 total 80.06 last 73.32) - open/close 6869 (time 1788531767.04 total 90.07 last 75.95) - open/close 7545 (time 1788531777.04 total 100.07 last 67.59) - open/close 8306 (time 1788531787.07 total 110.10 last 75.90) - open/close 9077 (time 1788531797.07 total 120.10 last 77.10) - open/close 9728 (time 1788531807.08 total 130.11 last 65.01) total: 10000 open/close in 133.71 seconds: 74.79 ops/second - unlinked 0 (time 1788531815 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 686 (time 1788531914.30 total 10.00 last 68.60) - open/close 1348 (time 1788531924.30 total 20.00 last 66.19) - open/close 2092 (time 1788531934.32 total 30.02 last 74.27) - open/close 2758 (time 1788531944.32 total 40.02 last 66.58) - open/close 3556 (time 1788531954.34 total 50.04 last 79.68) - open/close 4283 (time 1788531964.35 total 60.05 last 72.61) - open/close 5062 (time 1788531974.35 total 70.05 last 77.85) - open/close 5822 (time 1788531984.35 total 80.06 last 75.99) - open/close 6554 (time 1788531994.36 total 90.06 last 73.16) - open/close 7393 (time 1788532004.36 total 100.06 last 83.89) - open/close 8125 (time 1788532014.37 total 110.07 last 73.13) - open/close 8908 (time 1788532024.39 total 120.09 last 78.19) - open/close 9674 (time 1788532034.39 total 130.09 last 76.54) total: 10000 open/close in 134.18 seconds: 74.53 ops/second - unlinked 0 (time 1788532042 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 690 (time 1788532146.07 total 10.01 last 68.96) - open/close 1382 (time 1788532156.09 total 20.02 last 69.10) - open/close 2065 (time 1788532166.10 total 30.04 last 68.20) - open/close 2731 (time 1788532176.12 total 40.05 last 66.48) - open/close 3432 (time 1788532186.14 total 50.07 last 69.98) - open/close 4217 (time 1788532196.14 total 60.08 last 78.46) - open/close 4990 (time 1788532206.15 total 70.09 last 77.23) - open/close 5740 (time 1788532216.16 total 80.09 last 74.96) - open/close 6510 (time 1788532226.16 total 90.10 last 76.95) - open/close 7253 (time 1788532236.17 total 100.10 last 74.25) - open/close 7852 (time 1788532246.20 total 110.13 last 59.74) - open/close 8511 (time 1788532256.21 total 120.15 last 65.81) - open/close 9244 (time 1788532266.21 total 130.15 last 73.29) - open/close 9998 (time 1788532276.22 total 140.15 last 75.34) total: 10000 open/close in 140.16 seconds: 71.35 ops/second - unlinked 0 (time 1788532280 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 689 (time 1788532370.06 total 10.00 last 68.89) - open/close 1425 (time 1788532380.07 total 20.01 last 73.55) - open/close 2251 (time 1788532390.08 total 30.02 last 82.52) - open/close 3024 (time 1788532400.09 total 40.03 last 77.25) - open/close 3823 (time 1788532410.09 total 50.03 last 79.86) - open/close 4544 (time 1788532420.09 total 60.03 last 72.09) - open/close 5348 (time 1788532430.11 total 70.04 last 80.29) - open/close 6189 (time 1788532440.11 total 80.05 last 84.07) - open/close 6918 (time 1788532450.13 total 90.06 last 72.78) - open/close 7574 (time 1788532460.14 total 100.07 last 65.54) - open/close 8227 (time 1788532470.16 total 110.10 last 65.12) - open/close 8963 (time 1788532480.17 total 120.11 last 73.54) - open/close 9644 (time 1788532490.19 total 130.13 last 68.00) total: 10000 open/close in 134.82 seconds: 74.17 ops/second - unlinked 0 (time 1788532499 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 623 (time 1788532588.89 total 10.00 last 62.29) - open/close 1273 (time 1788532598.90 total 20.01 last 64.96) - open/close 1990 (time 1788532608.92 total 30.03 last 71.56) - open/close 2736 (time 1788532618.93 total 40.03 last 74.55) - open/close 3523 (time 1788532628.93 total 50.04 last 78.65) - open/close 4322 (time 1788532638.94 total 60.04 last 79.87) - open/close 5072 (time 1788532648.95 total 70.06 last 74.90) - open/close 5830 (time 1788532658.95 total 80.06 last 75.79) - open/close 6502 (time 1788532668.96 total 90.07 last 67.14) - open/close 7085 (time 1788532678.96 total 100.07 last 58.26) - open/close 7722 (time 1788532688.98 total 110.09 last 63.61) - open/close 8311 (time 1788532698.98 total 120.09 last 58.88) - open/close 8907 (time 1788532709.01 total 130.12 last 59.45) - open/close 9587 (time 1788532719.02 total 140.13 last 67.91) total: 10000 open/close in 145.94 seconds: 68.52 ops/second - unlinked 0 (time 1788532729 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 713 (time 1788532823.31 total 10.00 last 71.29) - open/close 1498 (time 1788532833.31 total 20.00 last 78.49) - open/close 2266 (time 1788532843.32 total 30.01 last 76.73) - open/close 3014 (time 1788532853.34 total 40.03 last 74.70) - open/close 3815 (time 1788532863.34 total 50.03 last 80.03) - open/close 4599 (time 1788532873.36 total 60.05 last 78.30) - open/close 5323 (time 1788532883.36 total 70.05 last 72.39) - open/close 6115 (time 1788532893.36 total 80.05 last 79.17) - open/close 6884 (time 1788532903.36 total 90.05 last 76.89) - open/close 7659 (time 1788532913.38 total 100.06 last 77.41) - open/close 8396 (time 1788532923.38 total 110.07 last 73.65) - open/close 9161 (time 1788532933.39 total 120.07 last 76.48) - open/close 9905 (time 1788532943.39 total 130.08 last 74.34) total: 10000 open/close in 131.65 seconds: 75.96 ops/second - unlinked 0 (time 1788532949 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg307-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg307-server: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg307-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg307-server: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg307-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.107@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.107@tcp sys.timeout=20 start ost1 service on oleg307-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg307-server: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg307-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg307-server: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg307-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server unloading modules via unload_modules_local on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: kmemleak disabled modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2