== conf-sanity test 69: replace an OST with the same index ========================================================== 20:08:44 (1789776524) start mds service on oleg431-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 18 15:30 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: kmemleak disabled oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions oleg431-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg431-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg431-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg431-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg431-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49535 inodes available. Want 99872. rc=0 - open/close 763 (time 1789776632.08 total 10.01 last 76.26) - open/close 1411 (time 1789776642.11 total 20.03 last 64.63) - open/close 2063 (time 1789776652.13 total 30.05 last 65.10) - open/close 2741 (time 1789776662.15 total 40.07 last 67.67) - open/close 3323 (time 1789776672.15 total 50.07 last 58.20) - open/close 3951 (time 1789776682.16 total 60.08 last 62.74) - open/close 4659 (time 1789776692.16 total 70.08 last 70.80) - open/close 5298 (time 1789776702.16 total 80.08 last 63.87) - open/close 5978 (time 1789776712.16 total 90.09 last 67.97) - open/close 6716 (time 1789776722.17 total 100.09 last 73.76) - open/close 7449 (time 1789776732.18 total 110.10 last 73.24) - open/close 8097 (time 1789776742.19 total 120.11 last 64.73) - open/close 8824 (time 1789776752.19 total 130.12 last 72.67) - open/close 9473 (time 1789776762.19 total 140.12 last 64.89) total: 10000 open/close in 147.88 seconds: 67.62 ops/second - unlinked 0 (time 1789776774 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 616 (time 1789776866.74 total 10.00 last 61.58) - open/close 1306 (time 1789776876.76 total 20.02 last 68.88) - open/close 1979 (time 1789776886.76 total 30.02 last 67.29) - open/close 2724 (time 1789776896.78 total 40.04 last 74.34) - open/close 3468 (time 1789776906.79 total 50.04 last 74.39) - open/close 4183 (time 1789776916.79 total 60.05 last 71.46) - open/close 4896 (time 1789776926.80 total 70.06 last 71.22) - open/close 5641 (time 1789776936.82 total 80.08 last 74.34) - open/close 6354 (time 1789776946.83 total 90.09 last 71.26) - open/close 7088 (time 1789776956.84 total 100.10 last 73.30) - open/close 7852 (time 1789776966.85 total 110.11 last 76.34) - open/close 8543 (time 1789776976.85 total 120.11 last 69.08) - open/close 9289 (time 1789776986.86 total 130.12 last 74.52) - open/close 9951 (time 1789776996.87 total 140.13 last 66.15) total: 10000 open/close in 140.82 seconds: 71.01 ops/second - unlinked 0 (time 1789777002 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 618 (time 1789777107.82 total 10.01 last 61.74) - open/close 1342 (time 1789777117.82 total 20.02 last 72.35) - open/close 2028 (time 1789777127.83 total 30.02 last 68.57) - open/close 2760 (time 1789777137.85 total 40.05 last 73.00) - open/close 3508 (time 1789777147.86 total 50.05 last 74.77) - open/close 4178 (time 1789777157.87 total 60.06 last 66.91) - open/close 4852 (time 1789777167.87 total 70.07 last 67.38) - open/close 5583 (time 1789777177.88 total 80.07 last 73.06) - open/close 6298 (time 1789777187.89 total 90.09 last 71.41) - open/close 6953 (time 1789777197.90 total 100.10 last 65.42) - open/close 7637 (time 1789777207.91 total 110.11 last 68.33) - open/close 8230 (time 1789777217.92 total 120.11 last 59.29) - open/close 8890 (time 1789777227.92 total 130.11 last 65.97) - open/close 9612 (time 1789777237.92 total 140.12 last 72.18) total: 10000 open/close in 145.91 seconds: 68.53 ops/second - unlinked 0 (time 1789777248 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 684 (time 1789777342.47 total 10.01 last 68.33) - open/close 1284 (time 1789777352.48 total 20.01 last 59.99) - open/close 1998 (time 1789777362.48 total 30.01 last 71.39) - open/close 2734 (time 1789777372.49 total 40.02 last 73.52) - open/close 3519 (time 1789777382.49 total 50.03 last 78.46) - open/close 4189 (time 1789777392.50 total 60.04 last 66.93) - open/close 4929 (time 1789777402.52 total 70.05 last 73.92) - open/close 5665 (time 1789777412.52 total 80.05 last 73.58) - open/close 6317 (time 1789777422.52 total 90.05 last 65.20) - open/close 7045 (time 1789777432.52 total 100.06 last 72.76) - open/close 7822 (time 1789777442.53 total 110.07 last 77.62) - open/close 8630 (time 1789777452.54 total 120.07 last 80.77) - open/close 9323 (time 1789777462.54 total 130.08 last 69.26) total: 10000 open/close in 139.76 seconds: 71.55 ops/second - unlinked 0 (time 1789777476 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 685 (time 1789777572.71 total 10.01 last 68.46) - open/close 1418 (time 1789777582.72 total 20.01 last 73.25) - open/close 2174 (time 1789777592.72 total 30.01 last 75.58) - open/close 2874 (time 1789777602.73 total 40.03 last 69.92) - open/close 3605 (time 1789777612.74 total 50.04 last 73.03) - open/close 4316 (time 1789777622.75 total 60.05 last 71.03) - open/close 5014 (time 1789777632.76 total 70.05 last 69.76) - open/close 5702 (time 1789777642.76 total 80.06 last 68.77) - open/close 6409 (time 1789777652.77 total 90.07 last 70.62) - open/close 7140 (time 1789777662.78 total 100.07 last 73.08) - open/close 7853 (time 1789777672.79 total 110.08 last 71.23) - open/close 8542 (time 1789777682.79 total 120.08 last 68.87) - open/close 9237 (time 1789777692.80 total 130.09 last 69.43) - open/close 9914 (time 1789777702.82 total 140.11 last 67.58) total: 10000 open/close in 141.36 seconds: 70.74 ops/second - unlinked 0 (time 1789777708 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 643 (time 1789777810.93 total 10.02 last 64.20) - open/close 1226 (time 1789777820.95 total 20.03 last 58.21) - open/close 1823 (time 1789777830.97 total 30.05 last 59.58) - open/close 2483 (time 1789777840.98 total 40.06 last 65.92) - open/close 3116 (time 1789777851.00 total 50.08 last 63.18) - open/close 3860 (time 1789777861.01 total 60.09 last 74.34) - open/close 4482 (time 1789777871.03 total 70.11 last 62.08) - open/close 5133 (time 1789777881.03 total 80.11 last 65.09) - open/close 5806 (time 1789777891.05 total 90.13 last 67.20) - open/close 6547 (time 1789777901.05 total 100.13 last 74.06) - open/close 7231 (time 1789777911.07 total 110.15 last 68.29) - open/close 7962 (time 1789777921.08 total 120.16 last 73.04) - open/close 8687 (time 1789777931.10 total 130.19 last 72.30) - open/close 9391 (time 1789777941.11 total 140.20 last 70.33) total: 10000 open/close in 149.31 seconds: 66.97 ops/second - unlinked 0 (time 1789777954 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 653 (time 1789778049.63 total 10.01 last 65.23) - open/close 1255 (time 1789778059.64 total 20.02 last 60.15) - open/close 1904 (time 1789778069.64 total 30.02 last 64.90) - open/close 2602 (time 1789778079.65 total 40.02 last 69.76) - open/close 3268 (time 1789778089.65 total 50.03 last 66.56) - open/close 3961 (time 1789778099.66 total 60.03 last 69.29) - open/close 4646 (time 1789778109.66 total 70.03 last 68.49) - open/close 5303 (time 1789778119.69 total 80.07 last 65.47) - open/close 6007 (time 1789778129.70 total 90.08 last 70.31) - open/close 6611 (time 1789778139.72 total 100.09 last 60.31) - open/close 7192 (time 1789778149.73 total 110.10 last 58.04) - open/close 7815 (time 1789778159.74 total 120.12 last 62.23) - open/close 8428 (time 1789778169.75 total 130.13 last 61.21) - open/close 9050 (time 1789778179.78 total 140.16 last 62.03) - open/close 9703 (time 1789778189.79 total 150.16 last 65.27) total: 10000 open/close in 154.27 seconds: 64.82 ops/second - unlinked 0 (time 1789778198 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 709 (time 1789778293.20 total 10.00 last 70.89) - open/close 1407 (time 1789778303.22 total 20.02 last 69.70) - open/close 1996 (time 1789778313.23 total 30.03 last 58.82) - open/close 2745 (time 1789778323.23 total 40.03 last 74.87) - open/close 3430 (time 1789778333.24 total 50.04 last 68.46) - open/close 4120 (time 1789778343.27 total 60.07 last 68.79) - open/close 4835 (time 1789778353.28 total 70.08 last 71.45) - open/close 5505 (time 1789778363.30 total 80.10 last 66.88) - open/close 6199 (time 1789778373.30 total 90.10 last 69.40) - open/close 6943 (time 1789778383.30 total 100.10 last 74.36) - open/close 7622 (time 1789778393.31 total 110.11 last 67.86) - open/close 8315 (time 1789778403.31 total 120.11 last 69.28) - open/close 8932 (time 1789778413.32 total 130.11 last 61.68) - open/close 9513 (time 1789778423.32 total 140.12 last 58.09) total: 10000 open/close in 146.38 seconds: 68.31 ops/second - unlinked 0 (time 1789778434 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 515 (time 1789778530.90 total 10.01 last 51.46) - open/close 1250 (time 1789778540.90 total 20.01 last 73.49) - open/close 1957 (time 1789778550.91 total 30.01 last 70.66) - open/close 2690 (time 1789778560.91 total 40.02 last 73.25) - open/close 3388 (time 1789778570.93 total 50.04 last 69.70) - open/close 4141 (time 1789778580.94 total 60.05 last 75.19) - open/close 4872 (time 1789778590.94 total 70.05 last 73.10) - open/close 5596 (time 1789778600.95 total 80.06 last 72.33) - open/close 6228 (time 1789778610.97 total 90.07 last 63.12) - open/close 6918 (time 1789778620.97 total 100.08 last 68.95) - open/close 7711 (time 1789778630.98 total 110.09 last 79.25) - open/close 8419 (time 1789778641.02 total 120.12 last 70.54) - open/close 9232 (time 1789778651.02 total 130.12 last 81.30) - open/close 9961 (time 1789778661.02 total 140.13 last 72.86) total: 10000 open/close in 140.64 seconds: 71.10 ops/second - unlinked 0 (time 1789778665 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 704 (time 1789778765.20 total 10.01 last 70.34) - open/close 1402 (time 1789778775.21 total 20.01 last 69.78) - open/close 2147 (time 1789778785.21 total 30.02 last 74.46) - open/close 2934 (time 1789778795.22 total 40.02 last 78.64) - open/close 3620 (time 1789778805.22 total 50.03 last 68.58) - open/close 4229 (time 1789778815.22 total 60.03 last 60.89) - open/close 4855 (time 1789778825.23 total 70.04 last 62.55) - open/close 5598 (time 1789778835.23 total 80.04 last 74.29) - open/close 6294 (time 1789778845.25 total 90.06 last 69.47) - open/close 6956 (time 1789778855.25 total 100.06 last 66.18) - open/close 7577 (time 1789778865.26 total 110.07 last 62.04) - open/close 8305 (time 1789778875.27 total 120.08 last 72.72) - open/close 9031 (time 1789778885.28 total 130.08 last 72.57) - open/close 9710 (time 1789778895.28 total 140.08 last 67.90) total: 10000 open/close in 144.10 seconds: 69.39 ops/second - unlinked 0 (time 1789778903 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.131@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg431-server: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg431-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg431-server: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg431-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.131@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.131@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.131@tcp sys.timeout=20 start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg431-server: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg431-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg431-server: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg431-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg431-server unloading modules via unload_modules_local on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: kmemleak disabled modules unloaded. pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2