== conf-sanity test 69: replace an OST with the same index ========================================================== 22:19:57 (1789438797) start mds service on oleg643-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 12:06 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg643-server' oleg643-server: oleg643-server.virtnet: executing load_modules_local oleg643-server: kmemleak disabled oleg643-server: Loading modules from /home/green/git/lustre-release/lustre oleg643-server: detected 4 online CPUs by sysfs oleg643-server: MODOPTS_LIBCFS= oleg643-server: Force libcfs to create 2 CPU partitions oleg643-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg643-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg643-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg643-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg643-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg643-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-OST0000 oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg643-client.virtnet: -o user_xattr,flock 192.168.206.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 792 (time 1789438904.06 total 10.00 last 79.17) - open/close 1453 (time 1789438914.06 total 20.01 last 66.05) - open/close 2143 (time 1789438924.07 total 30.02 last 68.95) - open/close 2810 (time 1789438934.07 total 40.02 last 66.68) - open/close 3508 (time 1789438944.08 total 50.02 last 69.79) - open/close 4197 (time 1789438954.08 total 60.03 last 68.87) - open/close 5001 (time 1789438964.09 total 70.03 last 80.36) - open/close 5812 (time 1789438974.09 total 80.03 last 81.10) - open/close 6359 (time 1789438984.09 total 90.03 last 54.69) - open/close 7142 (time 1789438994.09 total 100.04 last 78.27) - open/close 7927 (time 1789439004.10 total 110.04 last 78.47) - open/close 8754 (time 1789439014.11 total 120.05 last 82.62) - open/close 9590 (time 1789439024.11 total 130.06 last 83.54) total: 10000 open/close in 134.61 seconds: 74.29 ops/second - unlinked 0 (time 1789439032 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 793 (time 1789439118.48 total 10.01 last 79.25) - open/close 1572 (time 1789439128.49 total 20.01 last 77.84) - open/close 2309 (time 1789439138.50 total 30.02 last 73.66) - open/close 3137 (time 1789439148.50 total 40.02 last 82.77) - open/close 3691 (time 1789439158.52 total 50.04 last 55.29) - open/close 4182 (time 1789439168.54 total 60.06 last 49.01) - open/close 4845 (time 1789439178.54 total 70.07 last 66.27) - open/close 5492 (time 1789439188.56 total 80.08 last 64.60) - open/close 6239 (time 1789439198.57 total 90.09 last 74.61) - open/close 7032 (time 1789439208.57 total 100.10 last 79.26) - open/close 7817 (time 1789439218.58 total 110.10 last 78.48) - open/close 8532 (time 1789439228.58 total 120.11 last 71.44) - open/close 9250 (time 1789439238.60 total 130.12 last 71.72) - open/close 9958 (time 1789439248.61 total 140.13 last 70.69) total: 10000 open/close in 140.61 seconds: 71.12 ops/second - unlinked 0 (time 1789439254 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 797 (time 1789439339.23 total 10.01 last 79.65) - open/close 1585 (time 1789439349.24 total 20.01 last 78.74) - open/close 2413 (time 1789439359.24 total 30.02 last 82.76) - open/close 3253 (time 1789439369.25 total 40.02 last 83.95) - open/close 4065 (time 1789439379.25 total 50.03 last 81.18) - open/close 4863 (time 1789439389.25 total 60.03 last 79.78) - open/close 5624 (time 1789439399.26 total 70.04 last 76.03) - open/close 6390 (time 1789439409.27 total 80.05 last 76.55) - open/close 7215 (time 1789439419.27 total 90.05 last 82.45) - open/close 8072 (time 1789439429.27 total 100.05 last 85.70) - open/close 8858 (time 1789439439.28 total 110.05 last 78.58) - open/close 9685 (time 1789439449.28 total 120.06 last 82.67) total: 10000 open/close in 124.00 seconds: 80.65 ops/second - unlinked 0 (time 1789439457 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 778 (time 1789439542.27 total 10.00 last 77.79) - open/close 1549 (time 1789439552.29 total 20.02 last 76.97) - open/close 2402 (time 1789439562.31 total 30.04 last 85.12) - open/close 3205 (time 1789439572.31 total 40.04 last 80.26) - open/close 4004 (time 1789439582.32 total 50.06 last 79.80) - open/close 4836 (time 1789439592.33 total 60.06 last 83.17) - open/close 5659 (time 1789439602.33 total 70.06 last 82.28) - open/close 6402 (time 1789439612.33 total 80.07 last 74.26) - open/close 7221 (time 1789439622.34 total 90.08 last 81.82) - open/close 7963 (time 1789439632.35 total 100.08 last 74.18) - open/close 8775 (time 1789439642.35 total 110.09 last 81.16) - open/close 9548 (time 1789439652.36 total 120.09 last 77.24) total: 10000 open/close in 126.28 seconds: 79.19 ops/second - unlinked 0 (time 1789439662 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 876 (time 1789439746.48 total 10.01 last 87.48) - open/close 1732 (time 1789439756.49 total 20.03 last 85.50) - open/close 2574 (time 1789439766.49 total 30.03 last 84.16) - open/close 3377 (time 1789439776.51 total 40.04 last 80.19) - open/close 4189 (time 1789439786.52 total 50.05 last 81.13) - open/close 5075 (time 1789439796.52 total 60.05 last 88.59) - open/close 5835 (time 1789439806.52 total 70.06 last 75.95) - open/close 6623 (time 1789439816.53 total 80.07 last 78.75) - open/close 7452 (time 1789439826.55 total 90.08 last 82.76) - open/close 8256 (time 1789439836.57 total 100.10 last 80.25) - open/close 8963 (time 1789439846.58 total 110.12 last 70.61) - open/close 9758 (time 1789439856.59 total 120.12 last 79.45) total: 10000 open/close in 122.88 seconds: 81.38 ops/second - unlinked 0 (time 1789439863 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 720 (time 1789439945.93 total 10.00 last 71.97) - open/close 1440 (time 1789439955.95 total 20.02 last 71.89) - open/close 2234 (time 1789439965.95 total 30.02 last 79.37) - open/close 3094 (time 1789439975.95 total 40.02 last 85.99) - open/close 3887 (time 1789439985.96 total 50.04 last 79.20) - open/close 4691 (time 1789439995.97 total 60.04 last 80.39) - open/close 5530 (time 1789440005.98 total 70.05 last 83.79) - open/close 6334 (time 1789440015.99 total 80.06 last 80.30) - open/close 7133 (time 1789440026.00 total 90.07 last 79.82) - open/close 7944 (time 1789440036.02 total 100.09 last 80.99) - open/close 8697 (time 1789440046.02 total 110.09 last 75.29) - open/close 9513 (time 1789440056.03 total 120.10 last 81.51) total: 10000 open/close in 125.72 seconds: 79.54 ops/second - unlinked 0 (time 1789440066 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 735 (time 1789440148.98 total 10.01 last 73.42) - open/close 1506 (time 1789440158.99 total 20.02 last 77.01) - open/close 2192 (time 1789440169.00 total 30.04 last 68.51) - open/close 3011 (time 1789440179.01 total 40.04 last 81.85) - open/close 3775 (time 1789440189.01 total 50.04 last 76.39) - open/close 4587 (time 1789440199.01 total 60.04 last 81.18) - open/close 5411 (time 1789440209.02 total 70.05 last 82.36) - open/close 6230 (time 1789440219.02 total 80.05 last 81.90) - open/close 7085 (time 1789440229.02 total 90.06 last 85.44) - open/close 7861 (time 1789440239.03 total 100.07 last 77.53) - open/close 8638 (time 1789440249.04 total 110.07 last 77.66) - open/close 9448 (time 1789440259.05 total 120.08 last 80.93) total: 10000 open/close in 127.05 seconds: 78.71 ops/second - unlinked 0 (time 1789440270 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 835 (time 1789440359.31 total 10.00 last 83.47) - open/close 1603 (time 1789440369.32 total 20.02 last 76.66) - open/close 2288 (time 1789440379.33 total 30.03 last 68.46) - open/close 2988 (time 1789440389.34 total 40.04 last 69.91) - open/close 3729 (time 1789440399.35 total 50.05 last 74.02) - open/close 4484 (time 1789440409.36 total 60.05 last 75.46) - open/close 5327 (time 1789440419.37 total 70.07 last 84.18) - open/close 6110 (time 1789440429.38 total 80.07 last 78.26) - open/close 6927 (time 1789440439.38 total 90.08 last 81.67) - open/close 7740 (time 1789440449.40 total 100.09 last 81.18) - open/close 8582 (time 1789440459.40 total 110.10 last 84.13) - open/close 9369 (time 1789440469.41 total 120.11 last 78.64) total: 10000 open/close in 127.71 seconds: 78.30 ops/second - unlinked 0 (time 1789440481 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 805 (time 1789440566.01 total 10.01 last 80.45) - open/close 1607 (time 1789440576.01 total 20.01 last 80.15) - open/close 2399 (time 1789440586.02 total 30.02 last 79.15) - open/close 3250 (time 1789440596.04 total 40.04 last 84.90) - open/close 4085 (time 1789440606.05 total 50.05 last 83.43) - open/close 4932 (time 1789440616.06 total 60.06 last 84.65) - open/close 5749 (time 1789440626.06 total 70.06 last 81.67) - open/close 6551 (time 1789440636.07 total 80.06 last 80.17) - open/close 7420 (time 1789440646.07 total 90.07 last 86.86) - open/close 8309 (time 1789440656.09 total 100.09 last 88.72) - open/close 9103 (time 1789440666.11 total 110.11 last 79.27) total: 10000 open/close in 119.94 seconds: 83.37 ops/second - unlinked 0 (time 1789440680 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 706 (time 1789440769.62 total 10.00 last 70.60) - open/close 1532 (time 1789440779.63 total 20.01 last 82.49) - open/close 2319 (time 1789440789.64 total 30.02 last 78.64) - open/close 3193 (time 1789440799.65 total 40.03 last 87.32) - open/close 4022 (time 1789440809.65 total 50.03 last 82.90) - open/close 4798 (time 1789440819.65 total 60.03 last 77.59) - open/close 5605 (time 1789440829.66 total 70.04 last 80.61) - open/close 6409 (time 1789440839.68 total 80.06 last 80.24) - open/close 7269 (time 1789440849.69 total 90.07 last 85.98) - open/close 7939 (time 1789440859.69 total 100.07 last 66.96) - open/close 8737 (time 1789440869.70 total 110.08 last 79.77) - open/close 9503 (time 1789440879.70 total 120.08 last 76.59) total: 10000 open/close in 127.56 seconds: 78.40 ops/second - unlinked 0 (time 1789440892 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg643-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg643-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg643-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg643-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-OST0000 oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg643-server: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg643-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg643-server: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg643-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg643-client.virtnet: -o user_xattr,flock 192.168.206.143@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg643-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg643-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg643-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.143@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.143@tcp sys.timeout=20 start ost1 service on oleg643-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-OST0000 oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg643-server: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg643-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg643-server: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg643-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg643-client.virtnet: -o user_xattr,flock 192.168.206.143@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg643-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg643-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg643-server stop mds service on oleg643-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg643-server stop mds service on oleg643-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg643-server unloading modules via unload_modules_local on: 'oleg643-server' oleg643-server: oleg643-server.virtnet: executing unload_modules_local oleg643-server: kmemleak disabled modules unloaded. pdsh@oleg643-client: oleg643-client: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-client: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 2