== conf-sanity test 69: replace an OST with the same index ========================================================== 00:10:01 (1789186201) start mds service on oleg150-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 11 13:50 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: kmemleak disabled oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: MODOPTS_LIBCFS= oleg150-server: Force libcfs to create 2 CPU partitions oleg150-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg150-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg150-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg150-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg150-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg150-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 861 (time 1789186307.03 total 10.00 last 86.07) - open/close 1851 (time 1789186317.04 total 20.01 last 98.98) - open/close 2902 (time 1789186327.04 total 30.01 last 105.08) - open/close 3906 (time 1789186337.04 total 40.01 last 100.34) - open/close 4818 (time 1789186347.05 total 50.01 last 91.19) - open/close 5873 (time 1789186357.05 total 60.02 last 105.43) - open/close 6949 (time 1789186367.05 total 70.02 last 107.58) - open/close 7979 (time 1789186377.05 total 80.02 last 102.99) - open/close 9084 (time 1789186387.05 total 90.02 last 110.49) total: 10000 open/close in 98.18 seconds: 101.85 ops/second - unlinked 0 (time 1789186399 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1109 (time 1789186469.02 total 10.01 last 110.81) - open/close 2308 (time 1789186479.02 total 20.01 last 119.87) - open/close 3428 (time 1789186489.02 total 30.01 last 111.99) - open/close 4447 (time 1789186499.03 total 40.02 last 101.81) - open/close 5544 (time 1789186509.04 total 50.03 last 109.61) - open/close 6761 (time 1789186519.05 total 60.04 last 121.58) - open/close 7888 (time 1789186529.05 total 70.04 last 112.63) - open/close 9119 (time 1789186539.06 total 80.05 last 123.05) total: 10000 open/close in 87.46 seconds: 114.34 ops/second - unlinked 0 (time 1789186549 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1258 (time 1789186610.42 total 10.01 last 125.66) - open/close 2391 (time 1789186620.43 total 20.02 last 113.19) - open/close 3548 (time 1789186630.43 total 30.02 last 115.66) - open/close 4674 (time 1789186640.44 total 40.03 last 112.57) - open/close 5877 (time 1789186650.44 total 50.03 last 120.26) - open/close 7128 (time 1789186660.45 total 60.04 last 125.03) - open/close 8307 (time 1789186670.45 total 70.04 last 117.88) - open/close 9442 (time 1789186680.46 total 80.05 last 113.41) total: 10000 open/close in 84.70 seconds: 118.06 ops/second - unlinked 0 (time 1789186688 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1136 (time 1789186749.53 total 10.00 last 113.59) - open/close 2385 (time 1789186759.54 total 20.01 last 124.80) - open/close 3667 (time 1789186769.55 total 30.02 last 128.10) - open/close 4855 (time 1789186779.55 total 40.02 last 118.72) - open/close 6052 (time 1789186789.55 total 50.02 last 119.69) - open/close 7221 (time 1789186799.55 total 60.03 last 116.88) - open/close 8459 (time 1789186809.56 total 70.03 last 123.76) - open/close 9640 (time 1789186819.57 total 80.04 last 117.93) total: 10000 open/close in 82.91 seconds: 120.61 ops/second - unlinked 0 (time 1789186825 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1311 (time 1789186883.35 total 10.00 last 131.08) - open/close 2593 (time 1789186893.36 total 20.00 last 128.17) - open/close 3842 (time 1789186903.36 total 30.01 last 124.87) - open/close 5079 (time 1789186913.36 total 40.01 last 123.65) - open/close 6311 (time 1789186923.37 total 50.02 last 123.11) - open/close 7684 (time 1789186933.37 total 60.02 last 137.26) - open/close 9028 (time 1789186943.37 total 70.02 last 134.40) total: 10000 open/close in 76.93 seconds: 129.98 ops/second - unlinked 0 (time 1789186952 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1147 (time 1789187007.98 total 10.01 last 114.60) - open/close 2317 (time 1789187017.99 total 20.01 last 116.95) - open/close 3450 (time 1789187027.99 total 30.02 last 113.28) - open/close 4590 (time 1789187037.99 total 40.02 last 113.95) - open/close 5858 (time 1789187048.00 total 50.03 last 126.72) - open/close 7073 (time 1789187058.00 total 60.03 last 121.48) - open/close 8279 (time 1789187068.01 total 70.04 last 120.50) - open/close 9297 (time 1789187078.01 total 80.04 last 101.75) total: 10000 open/close in 87.15 seconds: 114.74 ops/second - unlinked 0 (time 1789187088 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1327 (time 1789187149.58 total 10.00 last 132.68) - open/close 2612 (time 1789187159.58 total 20.01 last 128.42) - open/close 3946 (time 1789187169.59 total 30.01 last 133.32) - open/close 5245 (time 1789187179.59 total 40.02 last 129.86) - open/close 6603 (time 1789187189.60 total 50.02 last 135.69) - open/close 7962 (time 1789187199.60 total 60.03 last 135.89) - open/close 9329 (time 1789187209.60 total 70.03 last 136.67) total: 10000 open/close in 75.13 seconds: 133.10 ops/second - unlinked 0 (time 1789187217 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1213 (time 1789187269.50 total 10.01 last 121.22) - open/close 2515 (time 1789187279.50 total 20.01 last 130.20) - open/close 3869 (time 1789187289.50 total 30.01 last 135.38) - open/close 5191 (time 1789187299.50 total 40.01 last 132.15) - open/close 6449 (time 1789187309.50 total 50.01 last 125.80) - open/close 7757 (time 1789187319.51 total 60.02 last 130.72) - open/close 9067 (time 1789187329.52 total 70.03 last 130.90) total: 10000 open/close in 76.33 seconds: 131.01 ops/second - unlinked 0 (time 1789187338 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1367 (time 1789187390.30 total 10.00 last 136.69) - open/close 2665 (time 1789187400.30 total 20.01 last 129.72) - open/close 3922 (time 1789187410.30 total 30.01 last 125.68) - open/close 5190 (time 1789187420.32 total 40.02 last 126.61) - open/close 6479 (time 1789187430.33 total 50.03 last 128.80) - open/close 7895 (time 1789187440.34 total 60.04 last 141.46) - open/close 9222 (time 1789187450.34 total 70.04 last 132.69) total: 10000 open/close in 75.77 seconds: 131.98 ops/second - unlinked 0 (time 1789187458 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1392 (time 1789187509.57 total 10.02 last 138.91) - open/close 2793 (time 1789187519.57 total 20.03 last 140.02) - open/close 4048 (time 1789187529.58 total 30.03 last 125.45) - open/close 5399 (time 1789187539.58 total 40.04 last 135.03) - open/close 6749 (time 1789187549.58 total 50.04 last 134.99) - open/close 8093 (time 1789187559.60 total 60.05 last 134.25) - open/close 9284 (time 1789187569.60 total 70.05 last 119.01) total: 10000 open/close in 75.17 seconds: 133.04 ops/second - unlinked 0 (time 1789187577 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg150-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre On OST0, 16270 used inodes rc=0 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.150@tcp sys.timeout=20 start ost1 service on oleg150-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg150-server unloading modules via unload_modules_local on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local oleg150-server: kmemleak disabled modules unloaded. pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2