== conf-sanity test 69: replace an OST with the same index ========================================================== 03:44:42 (1788594282) start mds service on oleg440-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1924424 Sep 4 13:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: kmemleak disabled oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg440-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg440-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg440-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 733 (time 1788594404.30 total 10.01 last 73.26) - open/close 1498 (time 1788594414.32 total 20.02 last 76.35) - open/close 2256 (time 1788594424.33 total 30.03 last 75.78) - open/close 2958 (time 1788594434.34 total 40.04 last 70.13) - open/close 3731 (time 1788594444.35 total 50.05 last 77.22) - open/close 4462 (time 1788594454.37 total 60.07 last 72.96) - open/close 5186 (time 1788594464.39 total 70.09 last 72.21) - open/close 5898 (time 1788594474.39 total 80.09 last 71.20) - open/close 6659 (time 1788594484.40 total 90.10 last 76.09) - open/close 7466 (time 1788594494.41 total 100.11 last 80.59) - open/close 8173 (time 1788594504.44 total 110.14 last 70.47) - open/close 8917 (time 1788594514.44 total 120.14 last 74.39) - open/close 9668 (time 1788594524.45 total 130.15 last 75.05) total: 10000 open/close in 134.56 seconds: 74.32 ops/second - unlinked 0 (time 1788594533 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 678 (time 1788594631.50 total 10.02 last 67.69) - open/close 1485 (time 1788594641.52 total 20.03 last 80.56) - open/close 2190 (time 1788594651.52 total 30.03 last 70.49) - open/close 2945 (time 1788594661.52 total 40.04 last 75.47) - open/close 3605 (time 1788594671.52 total 50.04 last 66.00) - open/close 4210 (time 1788594681.53 total 60.05 last 60.45) - open/close 4862 (time 1788594691.55 total 70.07 last 65.07) - open/close 5470 (time 1788594701.56 total 80.08 last 60.73) - open/close 6071 (time 1788594711.56 total 90.08 last 60.08) - open/close 6726 (time 1788594721.56 total 100.08 last 65.49) - open/close 7376 (time 1788594731.57 total 110.08 last 64.97) - open/close 8094 (time 1788594741.57 total 120.09 last 71.78) - open/close 8862 (time 1788594751.59 total 130.11 last 76.64) - open/close 9418 (time 1788594761.59 total 140.11 last 55.59) total: 10000 open/close in 150.05 seconds: 66.64 ops/second - unlinked 0 (time 1788594777 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 755 (time 1788594869.84 total 10.01 last 75.44) - open/close 1676 (time 1788594879.84 total 20.01 last 92.10) - open/close 2509 (time 1788594889.84 total 30.01 last 83.29) - open/close 3387 (time 1788594899.85 total 40.02 last 87.73) - open/close 4147 (time 1788594909.85 total 50.02 last 75.99) - open/close 4992 (time 1788594919.86 total 60.02 last 84.46) - open/close 5670 (time 1788594929.86 total 70.03 last 67.75) - open/close 6380 (time 1788594939.87 total 80.03 last 70.97) - open/close 7126 (time 1788594949.87 total 90.04 last 74.57) - open/close 7938 (time 1788594959.87 total 100.04 last 81.20) - open/close 8802 (time 1788594969.88 total 110.05 last 86.29) - open/close 9644 (time 1788594979.90 total 120.06 last 84.10) total: 10000 open/close in 124.80 seconds: 80.13 ops/second - unlinked 0 (time 1788594989 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 776 (time 1788595074.98 total 10.02 last 77.45) - open/close 1622 (time 1788595085.00 total 20.03 last 84.48) - open/close 2357 (time 1788595095.00 total 30.04 last 73.49) - open/close 3204 (time 1788595105.00 total 40.04 last 84.67) - open/close 3961 (time 1788595115.02 total 50.05 last 75.58) - open/close 4770 (time 1788595125.02 total 60.06 last 80.85) - open/close 5453 (time 1788595135.03 total 70.07 last 68.26) - open/close 6214 (time 1788595145.03 total 80.07 last 76.09) - open/close 6987 (time 1788595155.05 total 90.08 last 77.20) - open/close 7808 (time 1788595165.05 total 100.09 last 82.07) - open/close 8623 (time 1788595175.06 total 110.10 last 81.37) - open/close 9476 (time 1788595185.07 total 120.11 last 85.24) total: 10000 open/close in 126.11 seconds: 79.29 ops/second - unlinked 0 (time 1788595195 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 844 (time 1788595279.90 total 10.02 last 84.27) - open/close 1660 (time 1788595289.92 total 20.03 last 81.47) - open/close 2528 (time 1788595299.92 total 30.03 last 86.78) - open/close 3360 (time 1788595309.93 total 40.04 last 83.12) - open/close 4222 (time 1788595319.93 total 50.04 last 86.20) - open/close 5007 (time 1788595329.94 total 60.05 last 78.45) - open/close 5788 (time 1788595339.94 total 70.06 last 78.04) - open/close 6657 (time 1788595349.95 total 80.06 last 86.86) - open/close 7562 (time 1788595359.95 total 90.06 last 90.49) - open/close 8385 (time 1788595369.95 total 100.07 last 82.26) - open/close 9204 (time 1788595379.96 total 110.07 last 81.86) - open/close 9990 (time 1788595389.96 total 120.07 last 78.60) total: 10000 open/close in 120.16 seconds: 83.22 ops/second - unlinked 0 (time 1788595394 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 861 (time 1788595472.82 total 10.01 last 86.02) - open/close 1687 (time 1788595482.82 total 20.02 last 82.55) - open/close 2490 (time 1788595492.83 total 30.02 last 80.27) - open/close 3222 (time 1788595502.83 total 40.02 last 73.20) - open/close 4010 (time 1788595512.83 total 50.02 last 78.79) - open/close 4873 (time 1788595522.83 total 60.02 last 86.28) - open/close 5668 (time 1788595532.84 total 70.04 last 79.41) - open/close 6421 (time 1788595542.85 total 80.04 last 75.26) - open/close 7239 (time 1788595552.85 total 90.04 last 81.77) - open/close 8096 (time 1788595562.85 total 100.04 last 85.69) - open/close 8978 (time 1788595572.88 total 110.07 last 87.96) - open/close 9796 (time 1788595582.88 total 120.07 last 81.79) total: 10000 open/close in 123.04 seconds: 81.27 ops/second - unlinked 0 (time 1788595590 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 783 (time 1788595674.18 total 10.00 last 78.30) - open/close 1604 (time 1788595684.18 total 20.00 last 82.07) - open/close 2488 (time 1788595694.19 total 30.01 last 88.36) - open/close 3362 (time 1788595704.19 total 40.01 last 87.40) - open/close 4210 (time 1788595714.19 total 50.01 last 84.75) - open/close 5034 (time 1788595724.20 total 60.02 last 82.33) - open/close 5779 (time 1788595734.21 total 70.03 last 74.46) - open/close 6497 (time 1788595744.21 total 80.04 last 71.75) - open/close 7172 (time 1788595754.22 total 90.04 last 67.48) - open/close 7815 (time 1788595764.22 total 100.04 last 64.27) - open/close 8465 (time 1788595774.23 total 110.05 last 64.99) - open/close 9067 (time 1788595784.24 total 120.06 last 60.10) - open/close 9701 (time 1788595794.25 total 130.07 last 63.36) total: 10000 open/close in 133.52 seconds: 74.89 ops/second - unlinked 0 (time 1788595804 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 808 (time 1788595885.38 total 10.01 last 80.75) - open/close 1590 (time 1788595895.38 total 20.01 last 78.17) - open/close 2405 (time 1788595905.38 total 30.01 last 81.49) - open/close 3245 (time 1788595915.39 total 40.02 last 83.94) - open/close 4062 (time 1788595925.40 total 50.03 last 81.60) - open/close 4795 (time 1788595935.41 total 60.04 last 73.23) - open/close 5687 (time 1788595945.42 total 70.04 last 89.16) - open/close 6534 (time 1788595955.42 total 80.05 last 84.67) - open/close 7434 (time 1788595965.42 total 90.05 last 89.97) - open/close 8256 (time 1788595975.44 total 100.07 last 82.08) - open/close 9121 (time 1788595985.44 total 110.07 last 86.50) total: 10000 open/close in 119.94 seconds: 83.38 ops/second - unlinked 0 (time 1788595999 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 858 (time 1788596084.59 total 10.00 last 85.77) - open/close 1724 (time 1788596094.59 total 20.01 last 86.56) - open/close 2622 (time 1788596104.62 total 30.04 last 89.54) - open/close 3476 (time 1788596114.63 total 40.04 last 85.35) - open/close 4389 (time 1788596124.64 total 50.05 last 91.19) - open/close 5214 (time 1788596134.64 total 60.06 last 82.48) - open/close 6085 (time 1788596144.64 total 70.06 last 87.07) - open/close 7025 (time 1788596154.65 total 80.06 last 93.96) - open/close 7871 (time 1788596164.65 total 90.07 last 84.57) - open/close 8706 (time 1788596174.65 total 100.07 last 83.47) - open/close 9564 (time 1788596184.66 total 110.08 last 85.74) total: 10000 open/close in 114.95 seconds: 87.00 ops/second - unlinked 0 (time 1788596193 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 942 (time 1788596276.22 total 10.00 last 94.20) - open/close 1888 (time 1788596286.22 total 20.00 last 94.60) - open/close 2777 (time 1788596296.22 total 30.00 last 88.87) - open/close 3610 (time 1788596306.23 total 40.01 last 83.24) - open/close 4465 (time 1788596316.23 total 50.02 last 85.45) - open/close 5328 (time 1788596326.24 total 60.02 last 86.26) - open/close 6146 (time 1788596336.24 total 70.02 last 81.77) - open/close 6886 (time 1788596346.26 total 80.04 last 73.87) - open/close 7642 (time 1788596356.26 total 90.05 last 75.58) - open/close 8440 (time 1788596366.27 total 100.05 last 79.78) - open/close 9123 (time 1788596376.28 total 110.06 last 68.19) - open/close 9974 (time 1788596386.29 total 120.08 last 85.00) total: 10000 open/close in 120.52 seconds: 82.97 ops/second - unlinked 0 (time 1788596391 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.140@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg440-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.140@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.140@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.140@tcp sys.timeout=20 start ost1 service on oleg440-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: kmemleak disabled modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2