== conf-sanity test 69: replace an OST with the same index ========================================================== 10:52:41 (1788619961) start mds service on oleg225-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 17:39 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: kmemleak disabled oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg225-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg225-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg225-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 542 (time 1788620090.90 total 10.05 last 53.93) - open/close 1126 (time 1788620100.90 total 20.05 last 58.40) - open/close 1844 (time 1788620110.90 total 30.05 last 71.79) - open/close 2668 (time 1788620120.90 total 40.05 last 82.38) - open/close 3476 (time 1788620130.91 total 50.06 last 80.72) - open/close 4238 (time 1788620140.92 total 60.07 last 76.14) - open/close 4994 (time 1788620150.94 total 70.09 last 75.47) - open/close 5773 (time 1788620160.94 total 80.09 last 77.88) - open/close 6658 (time 1788620170.94 total 90.09 last 88.48) - open/close 7543 (time 1788620180.95 total 100.10 last 88.44) - open/close 8345 (time 1788620190.96 total 110.11 last 80.11) - open/close 9178 (time 1788620200.96 total 120.11 last 83.28) - open/close 9955 (time 1788620210.97 total 130.12 last 77.63) total: 10000 open/close in 130.68 seconds: 76.52 ops/second - unlinked 0 (time 1788620216 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 792 (time 1788620299.14 total 10.00 last 79.17) - open/close 1620 (time 1788620309.15 total 20.01 last 82.72) - open/close 2559 (time 1788620319.15 total 30.02 last 93.86) - open/close 3381 (time 1788620329.15 total 40.02 last 82.20) - open/close 4198 (time 1788620339.16 total 50.02 last 81.64) - open/close 5079 (time 1788620349.16 total 60.03 last 88.08) - open/close 5887 (time 1788620359.17 total 70.04 last 80.73) - open/close 6711 (time 1788620369.18 total 80.04 last 82.37) - open/close 7542 (time 1788620379.19 total 90.05 last 83.01) - open/close 8399 (time 1788620389.20 total 100.06 last 85.62) - open/close 9267 (time 1788620399.20 total 110.06 last 86.77) total: 10000 open/close in 117.82 seconds: 84.87 ops/second - unlinked 0 (time 1788620411 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 840 (time 1788620493.92 total 10.03 last 83.71) - open/close 1624 (time 1788620503.92 total 20.04 last 78.34) - open/close 2375 (time 1788620513.93 total 30.04 last 75.08) - open/close 3112 (time 1788620523.93 total 40.05 last 73.64) - open/close 3881 (time 1788620533.95 total 50.07 last 76.75) - open/close 4794 (time 1788620543.96 total 60.08 last 91.23) - open/close 5527 (time 1788620553.97 total 70.09 last 73.23) - open/close 6304 (time 1788620563.98 total 80.10 last 77.61) - open/close 7136 (time 1788620573.99 total 90.11 last 83.13) - open/close 8162 (time 1788620584.00 total 100.11 last 102.56) - open/close 9024 (time 1788620594.02 total 110.13 last 86.03) - open/close 9817 (time 1788620604.02 total 120.14 last 79.24) total: 10000 open/close in 122.29 seconds: 81.77 ops/second - unlinked 0 (time 1788620609 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 698 (time 1788620696.70 total 10.01 last 69.75) - open/close 1540 (time 1788620706.70 total 20.01 last 84.18) - open/close 2458 (time 1788620716.70 total 30.01 last 91.78) - open/close 3323 (time 1788620726.71 total 40.02 last 86.45) - open/close 4127 (time 1788620736.71 total 50.02 last 80.37) - open/close 5096 (time 1788620746.72 total 60.02 last 96.86) - open/close 6002 (time 1788620756.72 total 70.03 last 90.57) - open/close 6888 (time 1788620766.73 total 80.03 last 88.53) - open/close 7682 (time 1788620776.73 total 90.04 last 79.39) - open/close 8522 (time 1788620786.74 total 100.04 last 83.94) - open/close 9395 (time 1788620796.74 total 110.05 last 87.23) total: 10000 open/close in 116.90 seconds: 85.54 ops/second - unlinked 0 (time 1788620807 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 826 (time 1788620887.79 total 10.00 last 82.59) - open/close 1780 (time 1788620897.80 total 20.01 last 95.28) - open/close 2722 (time 1788620907.80 total 30.02 last 94.17) - open/close 3438 (time 1788620917.82 total 40.03 last 71.49) - open/close 4294 (time 1788620927.82 total 50.04 last 85.54) - open/close 4981 (time 1788620937.84 total 60.06 last 68.57) - open/close 5643 (time 1788620947.85 total 70.06 last 66.15) - open/close 6289 (time 1788620957.85 total 80.07 last 64.57) - open/close 6960 (time 1788620967.86 total 90.07 last 67.09) - open/close 7745 (time 1788620977.86 total 100.07 last 78.46) - open/close 8648 (time 1788620987.86 total 110.08 last 90.26) - open/close 9576 (time 1788620997.87 total 120.08 last 92.78) total: 10000 open/close in 125.52 seconds: 79.67 ops/second - unlinked 0 (time 1788621007 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 790 (time 1788621087.83 total 10.01 last 78.89) - open/close 1732 (time 1788621097.83 total 20.02 last 94.15) - open/close 2586 (time 1788621107.84 total 30.02 last 85.38) - open/close 3421 (time 1788621117.84 total 40.03 last 83.47) - open/close 4288 (time 1788621127.85 total 50.04 last 86.59) - open/close 5345 (time 1788621137.86 total 60.05 last 105.58) - open/close 6303 (time 1788621147.88 total 70.07 last 95.64) - open/close 7033 (time 1788621157.88 total 80.07 last 72.99) - open/close 8167 (time 1788621167.89 total 90.08 last 113.29) - open/close 9152 (time 1788621177.91 total 100.09 last 98.33) - open/close 9997 (time 1788621187.91 total 110.10 last 84.49) total: 10000 open/close in 110.11 seconds: 90.82 ops/second - unlinked 0 (time 1788621191 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 726 (time 1788621269.46 total 10.01 last 72.52) - open/close 1494 (time 1788621279.46 total 20.01 last 76.78) - open/close 2227 (time 1788621289.48 total 30.04 last 73.13) - open/close 2844 (time 1788621299.49 total 40.04 last 61.66) - open/close 3539 (time 1788621309.50 total 50.05 last 69.42) - open/close 4264 (time 1788621319.51 total 60.06 last 72.45) - open/close 5096 (time 1788621329.51 total 70.06 last 83.17) - open/close 6007 (time 1788621339.53 total 80.08 last 90.93) - open/close 6835 (time 1788621349.53 total 90.09 last 82.76) - open/close 7677 (time 1788621359.54 total 100.09 last 84.14) - open/close 8468 (time 1788621369.57 total 110.12 last 78.88) - open/close 9194 (time 1788621379.62 total 120.17 last 72.25) - open/close 9921 (time 1788621389.62 total 130.17 last 72.67) total: 10000 open/close in 131.32 seconds: 76.15 ops/second - unlinked 0 (time 1788621394 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 519 (time 1788621481.90 total 10.00 last 51.89) - open/close 1285 (time 1788621491.90 total 20.00 last 76.58) - open/close 2039 (time 1788621501.91 total 30.01 last 75.33) - open/close 2951 (time 1788621511.92 total 40.02 last 91.12) - open/close 3780 (time 1788621521.93 total 50.03 last 82.82) - open/close 4694 (time 1788621531.95 total 60.05 last 91.22) - open/close 5486 (time 1788621541.96 total 70.06 last 79.14) - open/close 6300 (time 1788621551.98 total 80.08 last 81.19) - open/close 7102 (time 1788621561.99 total 90.09 last 80.16) - open/close 7929 (time 1788621572.00 total 100.10 last 82.63) - open/close 8672 (time 1788621582.01 total 110.11 last 74.21) - open/close 9422 (time 1788621592.05 total 120.15 last 74.74) total: 10000 open/close in 128.08 seconds: 78.08 ops/second - unlinked 0 (time 1788621605 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 947 (time 1788621695.74 total 10.00 last 94.69) - open/close 1862 (time 1788621705.75 total 20.01 last 91.37) - open/close 2672 (time 1788621715.77 total 30.03 last 80.88) - open/close 3473 (time 1788621725.78 total 40.05 last 79.97) - open/close 4396 (time 1788621735.79 total 50.05 last 92.23) - open/close 5258 (time 1788621745.80 total 60.06 last 86.15) - open/close 6016 (time 1788621755.83 total 70.10 last 75.52) - open/close 6867 (time 1788621765.84 total 80.11 last 85.01) - open/close 7535 (time 1788621775.85 total 90.11 last 66.76) - open/close 8181 (time 1788621785.85 total 100.12 last 64.58) - open/close 8953 (time 1788621795.86 total 110.12 last 77.15) - open/close 9690 (time 1788621805.88 total 120.14 last 73.56) total: 10000 open/close in 124.68 seconds: 80.21 ops/second - unlinked 0 (time 1788621815 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 488 (time 1788621904.85 total 10.01 last 48.76) - open/close 1258 (time 1788621914.87 total 20.03 last 76.85) - open/close 1982 (time 1788621924.88 total 30.04 last 72.32) - open/close 2738 (time 1788621934.88 total 40.04 last 75.60) - open/close 3222 (time 1788621944.89 total 50.05 last 48.35) - open/close 3748 (time 1788621954.90 total 60.05 last 52.58) - open/close 4294 (time 1788621964.90 total 70.06 last 54.57) - open/close 5127 (time 1788621974.91 total 80.07 last 83.21) - open/close 5974 (time 1788621984.93 total 90.08 last 84.57) - open/close 6861 (time 1788621994.94 total 100.09 last 88.63) - open/close 7706 (time 1788622004.94 total 110.09 last 84.48) - open/close 8520 (time 1788622014.94 total 120.10 last 81.37) - open/close 9316 (time 1788622024.95 total 130.11 last 79.51) total: 10000 open/close in 137.88 seconds: 72.53 ops/second - unlinked 0 (time 1788622036 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg225-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg225-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.125@tcp sys.timeout=20 start ost1 service on oleg225-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg225-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg225-server: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg225-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local oleg225-server: kmemleak disabled modules unloaded. pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2