== conf-sanity test 69: replace an OST with the same index ========================================================== 00:34:33 (1788582873) start mds service on oleg352-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1765344 Sep 4 08:26 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: kmemleak disabled oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg352-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg352-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 757 (time 1788582970.50 total 10.01 last 75.61) - open/close 1492 (time 1788582980.51 total 20.02 last 73.42) - open/close 2306 (time 1788582990.52 total 30.03 last 81.31) - open/close 3057 (time 1788583000.52 total 40.04 last 75.07) - open/close 3833 (time 1788583010.53 total 50.05 last 77.53) - open/close 4599 (time 1788583020.53 total 60.05 last 76.58) - open/close 5356 (time 1788583030.54 total 70.06 last 75.66) - open/close 6206 (time 1788583040.54 total 80.06 last 84.97) - open/close 6978 (time 1788583050.55 total 90.07 last 77.13) - open/close 7740 (time 1788583060.56 total 100.07 last 76.18) - open/close 8522 (time 1788583070.57 total 110.08 last 78.11) - open/close 9318 (time 1788583080.59 total 120.11 last 79.43) total: 10000 open/close in 128.62 seconds: 77.75 ops/second - unlinked 0 (time 1788583093 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 777 (time 1788583178.79 total 10.01 last 77.64) - open/close 1546 (time 1788583188.80 total 20.02 last 76.83) - open/close 2242 (time 1788583198.80 total 30.02 last 69.56) - open/close 2816 (time 1788583208.81 total 40.03 last 57.38) - open/close 3552 (time 1788583218.82 total 50.04 last 73.53) - open/close 4275 (time 1788583228.82 total 60.04 last 72.25) - open/close 5014 (time 1788583238.84 total 70.06 last 73.80) - open/close 5787 (time 1788583248.84 total 80.06 last 77.29) - open/close 6563 (time 1788583258.84 total 90.06 last 77.59) - open/close 7349 (time 1788583268.85 total 100.07 last 78.52) - open/close 8130 (time 1788583278.86 total 110.08 last 78.00) - open/close 8864 (time 1788583288.87 total 120.09 last 73.36) - open/close 9612 (time 1788583298.88 total 130.09 last 74.75) total: 10000 open/close in 134.80 seconds: 74.18 ops/second - unlinked 0 (time 1788583307 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 701 (time 1788583394.49 total 10.00 last 70.08) - open/close 1386 (time 1788583404.49 total 20.00 last 68.50) - open/close 2150 (time 1788583414.50 total 30.01 last 76.32) - open/close 2903 (time 1788583424.51 total 40.02 last 75.22) - open/close 3698 (time 1788583434.51 total 50.03 last 79.49) - open/close 4519 (time 1788583444.52 total 60.03 last 82.05) - open/close 5319 (time 1788583454.52 total 70.03 last 79.99) - open/close 6099 (time 1788583464.52 total 80.03 last 77.98) - open/close 6840 (time 1788583474.53 total 90.04 last 74.08) - open/close 7610 (time 1788583484.53 total 100.04 last 76.99) - open/close 8394 (time 1788583494.54 total 110.05 last 78.33) - open/close 9162 (time 1788583504.54 total 120.06 last 76.73) - open/close 9894 (time 1788583514.54 total 130.06 last 73.20) total: 10000 open/close in 131.55 seconds: 76.02 ops/second - unlinked 0 (time 1788583519 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 713 (time 1788583603.06 total 10.01 last 71.23) - open/close 1515 (time 1788583613.07 total 20.02 last 80.12) - open/close 2261 (time 1788583623.08 total 30.02 last 74.57) - open/close 3034 (time 1788583633.09 total 40.03 last 77.25) - open/close 3844 (time 1788583643.09 total 50.04 last 80.95) - open/close 4564 (time 1788583653.10 total 60.04 last 71.98) - open/close 5320 (time 1788583663.10 total 70.05 last 75.55) - open/close 6084 (time 1788583673.11 total 80.06 last 76.34) - open/close 6865 (time 1788583683.11 total 90.06 last 78.07) - open/close 7629 (time 1788583693.12 total 100.06 last 76.38) - open/close 8403 (time 1788583703.13 total 110.07 last 77.32) - open/close 9186 (time 1788583713.14 total 120.08 last 78.23) - open/close 9992 (time 1788583723.15 total 130.09 last 80.51) total: 10000 open/close in 130.17 seconds: 76.82 ops/second - unlinked 0 (time 1788583726 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 809 (time 1788583811.49 total 10.01 last 80.86) - open/close 1651 (time 1788583821.51 total 20.02 last 84.10) - open/close 2399 (time 1788583831.53 total 30.04 last 74.61) - open/close 3157 (time 1788583841.55 total 40.06 last 75.64) - open/close 3975 (time 1788583851.58 total 50.09 last 81.62) - open/close 4716 (time 1788583861.58 total 60.09 last 74.07) - open/close 5471 (time 1788583871.59 total 70.11 last 75.40) - open/close 6189 (time 1788583881.60 total 80.11 last 71.76) - open/close 6909 (time 1788583891.61 total 90.12 last 71.94) - open/close 7649 (time 1788583901.61 total 100.12 last 74.00) - open/close 8421 (time 1788583911.62 total 110.13 last 77.14) - open/close 9185 (time 1788583921.63 total 120.14 last 76.31) - open/close 9902 (time 1788583931.63 total 130.15 last 71.66) total: 10000 open/close in 131.62 seconds: 75.98 ops/second - unlinked 0 (time 1788583937 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 734 (time 1788584032.21 total 10.00 last 73.37) - open/close 1467 (time 1788584042.22 total 20.01 last 73.27) - open/close 2180 (time 1788584052.22 total 30.02 last 71.24) - open/close 2870 (time 1788584062.22 total 40.02 last 69.00) - open/close 3629 (time 1788584072.23 total 50.02 last 75.89) - open/close 4457 (time 1788584082.24 total 60.03 last 82.70) - open/close 5217 (time 1788584092.24 total 70.03 last 76.00) - open/close 5961 (time 1788584102.24 total 80.03 last 74.37) - open/close 6715 (time 1788584112.24 total 90.04 last 75.38) - open/close 7505 (time 1788584122.26 total 100.05 last 78.85) - open/close 8261 (time 1788584132.26 total 110.05 last 75.60) - open/close 8975 (time 1788584142.27 total 120.06 last 71.35) - open/close 9672 (time 1788584152.27 total 130.06 last 69.69) total: 10000 open/close in 134.25 seconds: 74.49 ops/second - unlinked 0 (time 1788584160 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 740 (time 1788584250.64 total 10.01 last 73.96) - open/close 1432 (time 1788584260.65 total 20.02 last 69.11) - open/close 2062 (time 1788584270.66 total 30.02 last 62.97) - open/close 2780 (time 1788584280.66 total 40.02 last 71.79) - open/close 3470 (time 1788584290.66 total 50.03 last 68.96) - open/close 4288 (time 1788584300.68 total 60.04 last 81.69) - open/close 5064 (time 1788584310.68 total 70.04 last 77.60) - open/close 5838 (time 1788584320.69 total 80.05 last 77.33) - open/close 6594 (time 1788584330.69 total 90.06 last 75.57) - open/close 7299 (time 1788584340.69 total 100.06 last 70.50) - open/close 8040 (time 1788584350.69 total 110.06 last 74.06) - open/close 8842 (time 1788584360.70 total 120.07 last 80.15) - open/close 9621 (time 1788584370.71 total 130.07 last 77.86) total: 10000 open/close in 134.39 seconds: 74.41 ops/second - unlinked 0 (time 1788584379 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 566 (time 1788584467.78 total 10.01 last 56.54) - open/close 1279 (time 1788584477.79 total 20.02 last 71.21) - open/close 1985 (time 1788584487.80 total 30.03 last 70.56) - open/close 2788 (time 1788584497.80 total 40.04 last 80.25) - open/close 3586 (time 1788584507.81 total 50.04 last 79.73) - open/close 4359 (time 1788584517.83 total 60.06 last 77.19) - open/close 5148 (time 1788584527.84 total 70.07 last 78.82) - open/close 5908 (time 1788584537.84 total 80.07 last 75.98) - open/close 6705 (time 1788584547.84 total 90.07 last 79.70) - open/close 7474 (time 1788584557.84 total 100.07 last 76.90) - open/close 8263 (time 1788584567.85 total 110.08 last 78.85) - open/close 8993 (time 1788584577.85 total 120.08 last 73.00) - open/close 9684 (time 1788584587.87 total 130.10 last 68.97) total: 10000 open/close in 134.88 seconds: 74.14 ops/second - unlinked 0 (time 1788584597 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 730 (time 1788584686.72 total 10.00 last 73.00) - open/close 1475 (time 1788584696.72 total 20.00 last 74.50) - open/close 2210 (time 1788584706.73 total 30.01 last 73.43) - open/close 3012 (time 1788584716.74 total 40.02 last 80.16) - open/close 3794 (time 1788584726.74 total 50.02 last 78.19) - open/close 4585 (time 1788584736.75 total 60.03 last 78.99) - open/close 5401 (time 1788584746.76 total 70.04 last 81.51) - open/close 6150 (time 1788584756.77 total 80.05 last 74.86) - open/close 6891 (time 1788584766.78 total 90.06 last 74.00) - open/close 7657 (time 1788584776.79 total 100.07 last 76.52) - open/close 8503 (time 1788584786.80 total 110.07 last 84.59) - open/close 9189 (time 1788584796.80 total 120.08 last 68.56) - open/close 9868 (time 1788584806.81 total 130.09 last 67.84) total: 10000 open/close in 131.68 seconds: 75.94 ops/second - unlinked 0 (time 1788584812 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 761 (time 1788584906.99 total 10.01 last 76.02) - open/close 1259 (time 1788584916.99 total 20.01 last 49.79) - open/close 1862 (time 1788584927.01 total 30.03 last 60.18) - open/close 2544 (time 1788584937.03 total 40.06 last 68.04) - open/close 3197 (time 1788584947.06 total 50.08 last 65.15) - open/close 3823 (time 1788584957.07 total 60.09 last 62.52) - open/close 4465 (time 1788584967.07 total 70.10 last 64.17) - open/close 5077 (time 1788584977.08 total 80.10 last 61.18) - open/close 5773 (time 1788584987.08 total 90.10 last 69.58) - open/close 6390 (time 1788584997.08 total 100.11 last 61.66) - open/close 7028 (time 1788585007.10 total 110.12 last 63.70) - open/close 7745 (time 1788585017.12 total 120.14 last 71.58) - open/close 8505 (time 1788585027.13 total 130.15 last 75.93) - open/close 9168 (time 1788585037.14 total 140.17 last 66.20) - open/close 9865 (time 1788585047.15 total 150.17 last 69.65) total: 10000 open/close in 151.69 seconds: 65.93 ops/second - unlinked 0 (time 1788585053 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: kmemleak disabled modules unloaded. pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2