== conf-sanity test 69: replace an OST with the same index ========================================================== 19:41:15 (1788651675) start mds service on oleg238-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 18:31 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: kmemleak disabled oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: detected 4 online CPUs by sysfs oleg238-server: MODOPTS_LIBCFS= oleg238-server: Force libcfs to create 2 CPU partitions oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg238-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg238-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg238-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 991 (time 1788651790.65 total 10.00 last 99.09) - open/close 1769 (time 1788651800.66 total 20.01 last 77.75) - open/close 2564 (time 1788651810.66 total 30.01 last 79.48) - open/close 3299 (time 1788651820.67 total 40.02 last 73.44) - open/close 4071 (time 1788651830.67 total 50.03 last 77.15) - open/close 4948 (time 1788651840.68 total 60.03 last 87.64) - open/close 5744 (time 1788651850.69 total 70.04 last 79.54) - open/close 6527 (time 1788651860.69 total 80.04 last 78.29) - open/close 7349 (time 1788651870.70 total 90.05 last 82.16) - open/close 8169 (time 1788651880.70 total 100.05 last 81.95) - open/close 8950 (time 1788651890.71 total 110.06 last 78.06) - open/close 9850 (time 1788651900.71 total 120.06 last 89.99) total: 10000 open/close in 121.83 seconds: 82.08 ops/second - unlinked 0 (time 1788651907 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 908 (time 1788651990.26 total 10.01 last 90.70) - open/close 1903 (time 1788652000.27 total 20.02 last 99.45) - open/close 2821 (time 1788652010.27 total 30.02 last 91.77) - open/close 3729 (time 1788652020.30 total 40.05 last 90.54) - open/close 4611 (time 1788652030.31 total 50.05 last 88.16) - open/close 5552 (time 1788652040.32 total 60.06 last 94.01) - open/close 6351 (time 1788652050.32 total 70.07 last 79.82) - open/close 7182 (time 1788652060.33 total 80.08 last 83.06) - open/close 8019 (time 1788652070.33 total 90.08 last 83.68) - open/close 8906 (time 1788652080.34 total 100.08 last 88.66) - open/close 9859 (time 1788652090.35 total 110.09 last 95.20) total: 10000 open/close in 111.54 seconds: 89.66 ops/second - unlinked 0 (time 1788652096 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 802 (time 1788652180.85 total 10.01 last 80.12) - open/close 1517 (time 1788652190.85 total 20.01 last 71.48) - open/close 2341 (time 1788652200.85 total 30.01 last 82.40) - open/close 3139 (time 1788652210.87 total 40.03 last 79.63) - open/close 3935 (time 1788652220.87 total 50.04 last 79.59) - open/close 4860 (time 1788652230.88 total 60.04 last 92.44) - open/close 5663 (time 1788652240.88 total 70.05 last 80.25) - open/close 6406 (time 1788652250.88 total 80.05 last 74.30) - open/close 7296 (time 1788652260.90 total 90.06 last 88.91) - open/close 8105 (time 1788652270.90 total 100.07 last 80.82) - open/close 8937 (time 1788652280.90 total 110.07 last 83.20) - open/close 9683 (time 1788652290.91 total 120.08 last 74.54) total: 10000 open/close in 124.13 seconds: 80.56 ops/second - unlinked 0 (time 1788652300 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 847 (time 1788652383.16 total 10.01 last 84.58) - open/close 1736 (time 1788652393.17 total 20.01 last 88.90) - open/close 2524 (time 1788652403.17 total 30.02 last 78.76) - open/close 3314 (time 1788652413.18 total 40.03 last 78.93) - open/close 4146 (time 1788652423.19 total 50.04 last 83.08) - open/close 4814 (time 1788652433.20 total 60.04 last 66.78) - open/close 5590 (time 1788652443.20 total 70.05 last 77.53) - open/close 6304 (time 1788652453.21 total 80.06 last 71.36) - open/close 7113 (time 1788652463.22 total 90.07 last 80.84) - open/close 7968 (time 1788652473.23 total 100.08 last 85.41) - open/close 8838 (time 1788652483.23 total 110.08 last 86.94) - open/close 9680 (time 1788652493.24 total 120.09 last 84.19) total: 10000 open/close in 123.99 seconds: 80.65 ops/second - unlinked 0 (time 1788652502 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 800 (time 1788652585.34 total 10.00 last 79.99) - open/close 1598 (time 1788652595.35 total 20.01 last 79.72) - open/close 2364 (time 1788652605.36 total 30.02 last 76.53) - open/close 3197 (time 1788652615.37 total 40.02 last 83.27) - open/close 3954 (time 1788652625.37 total 50.03 last 75.66) - open/close 4707 (time 1788652635.38 total 60.04 last 75.21) - open/close 5486 (time 1788652645.39 total 70.04 last 77.89) - open/close 6269 (time 1788652655.39 total 80.05 last 78.27) - open/close 7153 (time 1788652665.39 total 90.05 last 88.35) - open/close 8060 (time 1788652675.40 total 100.06 last 90.66) - open/close 8824 (time 1788652685.40 total 110.06 last 76.40) - open/close 9718 (time 1788652695.40 total 120.06 last 89.39) total: 10000 open/close in 123.04 seconds: 81.28 ops/second - unlinked 0 (time 1788652703 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 814 (time 1788652793.43 total 10.00 last 81.38) - open/close 1785 (time 1788652803.44 total 20.01 last 97.02) - open/close 2691 (time 1788652813.44 total 30.01 last 90.59) - open/close 3442 (time 1788652823.46 total 40.03 last 74.95) - open/close 4239 (time 1788652833.47 total 50.03 last 79.68) - open/close 5001 (time 1788652843.47 total 60.04 last 76.15) - open/close 5776 (time 1788652853.48 total 70.05 last 77.43) - open/close 6460 (time 1788652863.49 total 80.06 last 68.32) - open/close 7137 (time 1788652873.50 total 90.07 last 67.68) - open/close 7866 (time 1788652883.51 total 100.08 last 72.83) - open/close 8539 (time 1788652893.52 total 110.09 last 67.23) - open/close 9147 (time 1788652903.52 total 120.09 last 60.78) - open/close 9945 (time 1788652913.52 total 130.09 last 79.79) total: 10000 open/close in 131.22 seconds: 76.21 ops/second - unlinked 0 (time 1788652919 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 743 (time 1788653002.14 total 10.00 last 74.29) - open/close 1255 (time 1788653012.15 total 20.01 last 51.13) - open/close 1720 (time 1788653022.16 total 30.02 last 46.47) - open/close 2380 (time 1788653032.17 total 40.03 last 65.94) - open/close 3006 (time 1788653042.20 total 50.06 last 62.42) - open/close 3793 (time 1788653052.20 total 60.06 last 78.67) - open/close 4661 (time 1788653062.25 total 70.11 last 86.43) - open/close 5518 (time 1788653072.25 total 80.11 last 85.65) - open/close 6221 (time 1788653082.26 total 90.12 last 70.22) - open/close 7020 (time 1788653092.27 total 100.13 last 79.86) - open/close 7838 (time 1788653102.27 total 110.13 last 81.78) - open/close 8713 (time 1788653112.27 total 120.13 last 87.47) - open/close 9511 (time 1788653122.28 total 130.14 last 79.77) total: 10000 open/close in 136.66 seconds: 73.18 ops/second - unlinked 0 (time 1788653133 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 669 (time 1788653217.22 total 10.00 last 66.90) - open/close 1571 (time 1788653227.22 total 20.00 last 90.16) - open/close 2464 (time 1788653237.22 total 30.01 last 89.28) - open/close 3389 (time 1788653247.23 total 40.01 last 92.47) - open/close 4347 (time 1788653257.23 total 50.01 last 95.76) - open/close 5119 (time 1788653267.25 total 60.03 last 77.08) - open/close 5855 (time 1788653277.25 total 70.03 last 73.59) - open/close 6524 (time 1788653287.25 total 80.03 last 66.89) - open/close 7338 (time 1788653297.25 total 90.04 last 81.35) - open/close 8134 (time 1788653307.27 total 100.05 last 79.52) - open/close 8867 (time 1788653317.28 total 110.06 last 73.20) - open/close 9687 (time 1788653327.30 total 120.08 last 81.84) total: 10000 open/close in 125.30 seconds: 79.81 ops/second - unlinked 0 (time 1788653337 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 753 (time 1788653419.99 total 10.00 last 75.30) - open/close 1602 (time 1788653429.99 total 20.00 last 84.87) - open/close 2390 (time 1788653440.00 total 30.01 last 78.77) - open/close 3119 (time 1788653450.01 total 40.02 last 72.83) - open/close 3846 (time 1788653460.03 total 50.04 last 72.52) - open/close 4711 (time 1788653470.03 total 60.04 last 86.48) - open/close 5642 (time 1788653480.03 total 70.05 last 93.09) - open/close 6408 (time 1788653490.05 total 80.06 last 76.47) - open/close 7134 (time 1788653500.05 total 90.07 last 72.57) - open/close 7994 (time 1788653510.06 total 100.07 last 85.99) - open/close 8751 (time 1788653520.06 total 110.07 last 75.69) - open/close 9335 (time 1788653530.07 total 120.09 last 58.30) - open/close 9944 (time 1788653540.08 total 130.09 last 60.88) total: 10000 open/close in 130.94 seconds: 76.37 ops/second - unlinked 0 (time 1788653545 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 903 (time 1788653635.58 total 10.00 last 90.29) - open/close 1759 (time 1788653645.59 total 20.01 last 85.54) - open/close 2680 (time 1788653655.60 total 30.02 last 92.04) - open/close 3611 (time 1788653665.61 total 40.03 last 93.00) - open/close 4514 (time 1788653675.61 total 50.03 last 90.26) - open/close 5472 (time 1788653685.63 total 60.05 last 95.66) - open/close 6441 (time 1788653695.64 total 70.06 last 96.78) - open/close 7239 (time 1788653705.65 total 80.07 last 79.74) - open/close 8003 (time 1788653715.65 total 90.07 last 76.36) - open/close 8802 (time 1788653725.65 total 100.07 last 79.89) - open/close 9683 (time 1788653735.66 total 110.07 last 88.09) total: 10000 open/close in 113.95 seconds: 87.75 ops/second - unlinked 0 (time 1788653743 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg238-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server unloading modules via unload_modules_local on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: kmemleak disabled modules unloaded. pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2