== conf-sanity test 69: replace an OST with the same index ========================================================== 08:49:40 (1790945380) start mds service on oleg643-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Oct 2 03:41 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg643-server' oleg643-server: oleg643-server.virtnet: executing load_modules_local oleg643-server: kmemleak disabled oleg643-server: Loading modules from /home/green/git/lustre-release/lustre oleg643-server: detected 4 online CPUs by sysfs oleg643-server: MODOPTS_LIBCFS= oleg643-server: Force libcfs to create 2 CPU partitions oleg643-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg643-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg643-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg643-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg643-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg643-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-OST0000 oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg643-client.virtnet: -o user_xattr,flock 192.168.206.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 679 (time 1790945491.67 total 10.01 last 67.84) - open/close 1360 (time 1790945501.68 total 20.02 last 68.01) - open/close 2114 (time 1790945511.70 total 30.04 last 75.26) - open/close 2814 (time 1790945521.70 total 40.04 last 69.99) - open/close 3451 (time 1790945531.72 total 50.06 last 63.57) - open/close 4227 (time 1790945541.73 total 60.06 last 77.60) - open/close 4964 (time 1790945551.73 total 70.07 last 73.68) - open/close 5701 (time 1790945561.73 total 80.07 last 73.65) - open/close 6493 (time 1790945571.75 total 90.09 last 79.09) - open/close 7096 (time 1790945581.76 total 100.10 last 60.20) - open/close 7739 (time 1790945591.77 total 110.11 last 64.29) - open/close 8305 (time 1790945601.77 total 120.11 last 56.58) - open/close 9017 (time 1790945611.77 total 130.11 last 71.20) - open/close 9758 (time 1790945621.77 total 140.11 last 74.08) total: 10000 open/close in 142.81 seconds: 70.02 ops/second - unlinked 0 (time 1790945629 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 694 (time 1790945715.53 total 10.01 last 69.36) - open/close 1488 (time 1790945725.53 total 20.01 last 79.38) - open/close 2297 (time 1790945735.53 total 30.01 last 80.89) - open/close 3007 (time 1790945745.54 total 40.02 last 70.95) - open/close 3727 (time 1790945755.55 total 50.03 last 71.94) - open/close 4178 (time 1790945765.55 total 60.03 last 45.07) - open/close 4715 (time 1790945775.57 total 70.05 last 53.62) - open/close 5340 (time 1790945785.57 total 80.05 last 62.48) - open/close 6070 (time 1790945795.58 total 90.06 last 72.94) - open/close 6830 (time 1790945805.58 total 100.06 last 75.97) - open/close 7625 (time 1790945815.59 total 110.06 last 79.48) - open/close 8242 (time 1790945825.59 total 120.07 last 61.68) - open/close 8896 (time 1790945835.60 total 130.07 last 65.36) - open/close 9549 (time 1790945845.60 total 140.08 last 65.24) total: 10000 open/close in 146.46 seconds: 68.28 ops/second - unlinked 0 (time 1790945856 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 801 (time 1790945940.42 total 10.01 last 80.05) - open/close 1627 (time 1790945950.42 total 20.01 last 82.55) - open/close 2364 (time 1790945960.43 total 30.02 last 73.66) - open/close 3123 (time 1790945970.43 total 40.02 last 75.89) - open/close 3986 (time 1790945980.43 total 50.02 last 86.27) - open/close 4759 (time 1790945990.44 total 60.03 last 77.24) - open/close 5567 (time 1790946000.46 total 70.05 last 80.65) - open/close 6446 (time 1790946010.47 total 80.06 last 87.79) - open/close 7290 (time 1790946020.48 total 90.07 last 84.30) - open/close 8017 (time 1790946030.49 total 100.08 last 72.62) - open/close 8726 (time 1790946040.50 total 110.09 last 70.88) - open/close 9572 (time 1790946050.50 total 120.09 last 84.59) total: 10000 open/close in 126.14 seconds: 79.28 ops/second - unlinked 0 (time 1790946060 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 834 (time 1790946144.44 total 10.00 last 83.36) - open/close 1680 (time 1790946154.45 total 20.02 last 84.48) - open/close 2521 (time 1790946164.45 total 30.02 last 84.10) - open/close 3304 (time 1790946174.45 total 40.02 last 78.29) - open/close 4031 (time 1790946184.46 total 50.02 last 72.66) - open/close 4834 (time 1790946194.47 total 60.03 last 80.24) - open/close 5647 (time 1790946204.47 total 70.04 last 81.28) - open/close 6500 (time 1790946214.47 total 80.04 last 85.29) - open/close 7272 (time 1790946224.47 total 90.04 last 77.18) - open/close 8026 (time 1790946234.49 total 100.05 last 75.30) - open/close 8833 (time 1790946244.49 total 110.06 last 80.67) - open/close 9639 (time 1790946254.50 total 120.06 last 80.55) total: 10000 open/close in 124.36 seconds: 80.41 ops/second - unlinked 0 (time 1790946262 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 801 (time 1790946347.09 total 10.01 last 80.03) - open/close 1528 (time 1790946357.10 total 20.02 last 72.62) - open/close 2321 (time 1790946367.11 total 30.02 last 79.27) - open/close 3141 (time 1790946377.11 total 40.03 last 81.94) - open/close 3921 (time 1790946387.12 total 50.04 last 77.94) - open/close 4744 (time 1790946397.12 total 60.04 last 82.28) - open/close 5531 (time 1790946407.13 total 70.05 last 78.65) - open/close 6393 (time 1790946417.14 total 80.06 last 86.14) - open/close 7154 (time 1790946427.14 total 90.06 last 76.05) - open/close 7879 (time 1790946437.15 total 100.07 last 72.48) - open/close 8472 (time 1790946447.15 total 110.07 last 59.27) - open/close 9059 (time 1790946457.16 total 120.08 last 58.64) - open/close 9629 (time 1790946467.18 total 130.10 last 56.90) total: 10000 open/close in 134.40 seconds: 74.40 ops/second - unlinked 0 (time 1790946476 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 828 (time 1790946560.78 total 10.01 last 82.70) - open/close 1677 (time 1790946570.80 total 20.03 last 84.74) - open/close 2461 (time 1790946580.80 total 30.03 last 78.39) - open/close 3236 (time 1790946590.81 total 40.05 last 77.40) - open/close 3882 (time 1790946600.81 total 50.05 last 64.59) - open/close 4445 (time 1790946610.82 total 60.06 last 56.25) - open/close 5134 (time 1790946620.83 total 70.07 last 68.83) - open/close 5800 (time 1790946630.84 total 80.07 last 66.56) - open/close 6519 (time 1790946640.84 total 90.08 last 71.88) - open/close 7303 (time 1790946650.86 total 100.09 last 78.26) - open/close 8077 (time 1790946660.86 total 110.10 last 77.37) - open/close 8903 (time 1790946670.87 total 120.10 last 82.54) - open/close 9684 (time 1790946680.88 total 130.12 last 78.01) total: 10000 open/close in 134.16 seconds: 74.54 ops/second - unlinked 0 (time 1790946688 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 740 (time 1790946776.93 total 10.01 last 73.95) - open/close 1467 (time 1790946786.94 total 20.02 last 72.61) - open/close 2267 (time 1790946796.94 total 30.02 last 79.95) - open/close 3095 (time 1790946806.95 total 40.03 last 82.76) - open/close 3887 (time 1790946816.96 total 50.04 last 79.11) - open/close 4671 (time 1790946826.96 total 60.04 last 78.39) - open/close 5460 (time 1790946836.97 total 70.05 last 78.80) - open/close 6204 (time 1790946846.98 total 80.06 last 74.37) - open/close 6970 (time 1790946856.99 total 90.07 last 76.49) - open/close 7765 (time 1790946866.99 total 100.07 last 79.49) - open/close 8517 (time 1790946877.00 total 110.08 last 75.14) - open/close 9255 (time 1790946887.02 total 120.10 last 73.70) - open/close 9937 (time 1790946897.02 total 130.10 last 68.19) total: 10000 open/close in 130.98 seconds: 76.35 ops/second - unlinked 0 (time 1790946902 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 785 (time 1790946991.78 total 10.02 last 78.37) - open/close 1591 (time 1790947001.79 total 20.02 last 80.53) - open/close 2424 (time 1790947011.79 total 30.03 last 83.27) - open/close 3219 (time 1790947021.80 total 40.03 last 79.46) - open/close 4047 (time 1790947031.80 total 50.04 last 82.75) - open/close 4805 (time 1790947041.82 total 60.06 last 75.67) - open/close 5536 (time 1790947051.84 total 70.08 last 72.93) - open/close 6292 (time 1790947061.86 total 80.09 last 75.51) - open/close 7096 (time 1790947071.87 total 90.10 last 80.32) - open/close 7858 (time 1790947081.87 total 100.10 last 76.18) - open/close 8593 (time 1790947091.87 total 110.10 last 73.49) - open/close 9376 (time 1790947101.88 total 120.11 last 78.23) total: 10000 open/close in 127.67 seconds: 78.33 ops/second - unlinked 0 (time 1790947113 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 763 (time 1790947200.18 total 10.01 last 76.25) - open/close 1520 (time 1790947210.19 total 20.02 last 75.63) - open/close 2257 (time 1790947220.21 total 30.03 last 73.59) - open/close 2995 (time 1790947230.22 total 40.04 last 73.71) - open/close 3778 (time 1790947240.23 total 50.05 last 78.24) - open/close 4569 (time 1790947250.23 total 60.06 last 79.05) - open/close 5345 (time 1790947260.24 total 70.07 last 77.53) - open/close 6118 (time 1790947270.26 total 80.08 last 77.16) - open/close 6839 (time 1790947280.26 total 90.09 last 72.09) - open/close 7662 (time 1790947290.26 total 100.09 last 82.29) - open/close 8461 (time 1790947300.27 total 110.10 last 79.83) - open/close 9212 (time 1790947310.28 total 120.10 last 75.07) total: 10000 open/close in 129.55 seconds: 77.19 ops/second - unlinked 0 (time 1790947324 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 870 (time 1790947406.43 total 10.00 last 86.97) - open/close 1685 (time 1790947416.44 total 20.01 last 81.42) - open/close 2509 (time 1790947426.45 total 30.03 last 82.28) - open/close 3384 (time 1790947436.46 total 40.03 last 87.47) - open/close 4200 (time 1790947446.47 total 50.04 last 81.53) - open/close 5020 (time 1790947456.48 total 60.05 last 81.88) - open/close 5849 (time 1790947466.48 total 70.06 last 82.88) - open/close 6645 (time 1790947476.49 total 80.06 last 79.54) - open/close 7457 (time 1790947486.49 total 90.07 last 81.17) - open/close 8262 (time 1790947496.50 total 100.07 last 80.47) - open/close 9104 (time 1790947506.50 total 110.08 last 84.15) - open/close 9831 (time 1790947516.51 total 120.08 last 72.69) total: 10000 open/close in 122.08 seconds: 81.91 ops/second - unlinked 0 (time 1790947522 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg643-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg643-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg643-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg643-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-OST0000 oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg643-server: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg643-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg643-server: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg643-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg643-client.virtnet: -o user_xattr,flock 192.168.206.143@tcp:/lustre /mnt/lustre On OST0, 19714 used inodes rc=0 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg643-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg643-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg643-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.143@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.143@tcp sys.timeout=20 start ost1 service on oleg643-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-OST0000 oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg643-server: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg643-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg643-server: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg643-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg643-client.virtnet: -o user_xattr,flock 192.168.206.143@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg643-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg643-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg643-server stop mds service on oleg643-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg643-server stop mds service on oleg643-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg643-server unloading modules via unload_modules_local on: 'oleg643-server' oleg643-server: oleg643-server.virtnet: executing unload_modules_local oleg643-server: kmemleak disabled modules unloaded. pdsh@oleg643-client: oleg643-client: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-client: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 2