== conf-sanity test 69: replace an OST with the same index ========================================================== 23:59:13 (1788580753) start mds service on oleg457-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 4 08:27 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: kmemleak disabled oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: MODOPTS_LIBCFS= oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg457-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg457-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg457-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg457-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg457-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 833 (time 1788580861.15 total 10.01 last 83.20) - open/close 1594 (time 1788580871.16 total 20.02 last 76.06) - open/close 2442 (time 1788580881.17 total 30.03 last 84.71) - open/close 3352 (time 1788580891.18 total 40.04 last 90.88) - open/close 4216 (time 1788580901.19 total 50.05 last 86.34) - open/close 5081 (time 1788580911.19 total 60.05 last 86.47) - open/close 5898 (time 1788580921.19 total 70.05 last 81.69) - open/close 6588 (time 1788580931.19 total 80.05 last 69.00) - open/close 7322 (time 1788580941.19 total 90.05 last 73.39) - open/close 8119 (time 1788580951.20 total 100.06 last 79.64) - open/close 8956 (time 1788580961.21 total 110.07 last 83.61) - open/close 9840 (time 1788580971.22 total 120.07 last 88.38) total: 10000 open/close in 121.87 seconds: 82.06 ops/second - unlinked 0 (time 1788580977 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 796 (time 1788581062.82 total 10.01 last 79.49) - open/close 1613 (time 1788581072.82 total 20.02 last 81.66) - open/close 2474 (time 1788581082.83 total 30.02 last 86.06) - open/close 3291 (time 1788581092.84 total 40.04 last 81.61) - open/close 3912 (time 1788581102.87 total 50.07 last 61.91) - open/close 4670 (time 1788581112.87 total 60.07 last 75.80) - open/close 5547 (time 1788581122.87 total 70.07 last 87.67) - open/close 6370 (time 1788581132.88 total 80.08 last 82.25) - open/close 7261 (time 1788581142.89 total 90.08 last 89.04) - open/close 8129 (time 1788581152.90 total 100.10 last 86.66) - open/close 8940 (time 1788581162.92 total 110.12 last 80.94) - open/close 9573 (time 1788581172.93 total 120.13 last 63.24) total: 10000 open/close in 127.26 seconds: 78.58 ops/second - unlinked 0 (time 1788581184 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 821 (time 1788581269.76 total 10.00 last 82.06) - open/close 1704 (time 1788581279.77 total 20.01 last 88.21) - open/close 2545 (time 1788581289.78 total 30.02 last 84.04) - open/close 3363 (time 1788581299.78 total 40.02 last 81.79) - open/close 4074 (time 1788581309.79 total 50.03 last 71.03) - open/close 4808 (time 1788581319.79 total 60.03 last 73.40) - open/close 5639 (time 1788581329.79 total 70.03 last 83.09) - open/close 6474 (time 1788581339.80 total 80.04 last 83.46) - open/close 7199 (time 1788581349.81 total 90.05 last 72.41) - open/close 7903 (time 1788581359.82 total 100.06 last 70.33) - open/close 8716 (time 1788581369.82 total 110.06 last 81.29) - open/close 9451 (time 1788581379.82 total 120.07 last 73.47) total: 10000 open/close in 127.99 seconds: 78.13 ops/second - unlinked 0 (time 1788581391 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 771 (time 1788581476.42 total 10.00 last 77.09) - open/close 1589 (time 1788581486.44 total 20.02 last 81.68) - open/close 2436 (time 1788581496.44 total 30.02 last 84.66) - open/close 3219 (time 1788581506.45 total 40.03 last 78.22) - open/close 3937 (time 1788581516.46 total 50.04 last 71.75) - open/close 4752 (time 1788581526.47 total 60.05 last 81.45) - open/close 5467 (time 1788581536.47 total 70.05 last 71.49) - open/close 6250 (time 1788581546.47 total 80.05 last 78.30) - open/close 7114 (time 1788581556.48 total 90.05 last 86.34) - open/close 7821 (time 1788581566.48 total 100.06 last 70.67) - open/close 8614 (time 1788581576.50 total 110.08 last 79.11) - open/close 9418 (time 1788581586.52 total 120.10 last 80.24) total: 10000 open/close in 128.12 seconds: 78.05 ops/second - unlinked 0 (time 1788581599 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 755 (time 1788581687.90 total 10.02 last 75.37) - open/close 1446 (time 1788581697.91 total 20.03 last 69.03) - open/close 2216 (time 1788581707.91 total 30.03 last 76.98) - open/close 3066 (time 1788581717.92 total 40.04 last 84.95) - open/close 3840 (time 1788581727.92 total 50.04 last 77.39) - open/close 4659 (time 1788581737.92 total 60.04 last 81.88) - open/close 5461 (time 1788581747.93 total 70.05 last 80.16) - open/close 6300 (time 1788581757.94 total 80.06 last 83.81) - open/close 7064 (time 1788581768.00 total 90.12 last 75.93) - open/close 7896 (time 1788581778.01 total 100.12 last 83.16) - open/close 8789 (time 1788581788.01 total 110.12 last 89.29) - open/close 9643 (time 1788581798.01 total 120.13 last 85.34) total: 10000 open/close in 124.59 seconds: 80.26 ops/second - unlinked 0 (time 1788581807 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 873 (time 1788581893.56 total 10.01 last 87.22) - open/close 1675 (time 1788581903.56 total 20.02 last 80.16) - open/close 2507 (time 1788581913.57 total 30.02 last 83.18) - open/close 3296 (time 1788581923.57 total 40.03 last 78.84) - open/close 4178 (time 1788581933.58 total 50.03 last 88.15) - open/close 4992 (time 1788581943.58 total 60.03 last 81.39) - open/close 5769 (time 1788581953.60 total 70.05 last 77.54) - open/close 6592 (time 1788581963.60 total 80.05 last 82.29) - open/close 7451 (time 1788581973.61 total 90.06 last 85.86) - open/close 8326 (time 1788581983.61 total 100.06 last 87.46) - open/close 9126 (time 1788581993.62 total 110.07 last 79.92) - open/close 9983 (time 1788582003.62 total 120.08 last 85.68) total: 10000 open/close in 120.20 seconds: 83.19 ops/second - unlinked 0 (time 1788582008 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 852 (time 1788582097.93 total 10.01 last 85.15) - open/close 1689 (time 1788582107.93 total 20.01 last 83.68) - open/close 2568 (time 1788582117.95 total 30.02 last 87.78) - open/close 3394 (time 1788582127.95 total 40.03 last 82.56) - open/close 4098 (time 1788582137.96 total 50.03 last 70.36) - open/close 4909 (time 1788582147.97 total 60.05 last 80.98) - open/close 5666 (time 1788582157.97 total 70.05 last 75.67) - open/close 6401 (time 1788582167.98 total 80.06 last 73.45) - open/close 7075 (time 1788582177.99 total 90.07 last 67.32) - open/close 7640 (time 1788582188.00 total 100.08 last 56.46) - open/close 8310 (time 1788582198.01 total 110.08 last 66.95) - open/close 9111 (time 1788582208.01 total 120.09 last 80.04) - open/close 9984 (time 1788582218.02 total 130.10 last 87.26) total: 10000 open/close in 130.23 seconds: 76.79 ops/second - unlinked 0 (time 1788582222 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 835 (time 1788582303.89 total 10.02 last 83.35) - open/close 1671 (time 1788582313.90 total 20.03 last 83.52) - open/close 2516 (time 1788582323.90 total 30.03 last 84.49) - open/close 3395 (time 1788582333.91 total 40.03 last 87.89) - open/close 4234 (time 1788582343.92 total 50.05 last 83.77) - open/close 5118 (time 1788582353.94 total 60.06 last 88.23) - open/close 6072 (time 1788582363.96 total 70.09 last 95.18) - open/close 6920 (time 1788582373.97 total 80.09 last 84.78) - open/close 7836 (time 1788582383.97 total 90.10 last 91.54) - open/close 8642 (time 1788582393.98 total 100.10 last 80.55) - open/close 9564 (time 1788582403.98 total 110.11 last 92.17) total: 10000 open/close in 115.20 seconds: 86.81 ops/second - unlinked 0 (time 1788582413 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 821 (time 1788582492.96 total 10.01 last 82.04) - open/close 1652 (time 1788582502.97 total 20.02 last 83.01) - open/close 2545 (time 1788582512.99 total 30.03 last 89.15) - open/close 3419 (time 1788582522.99 total 40.04 last 87.37) - open/close 4349 (time 1788582533.00 total 50.04 last 92.93) - open/close 5225 (time 1788582543.01 total 60.05 last 87.52) - open/close 6088 (time 1788582553.01 total 70.05 last 86.29) - open/close 6963 (time 1788582563.03 total 80.07 last 87.34) - open/close 7781 (time 1788582573.04 total 90.08 last 81.74) - open/close 8521 (time 1788582583.04 total 100.09 last 73.96) - open/close 9302 (time 1788582593.05 total 110.09 last 78.05) total: 10000 open/close in 117.98 seconds: 84.76 ops/second - unlinked 0 (time 1788582604 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 755 (time 1788582684.85 total 10.01 last 75.41) - open/close 1537 (time 1788582694.86 total 20.02 last 78.12) - open/close 2389 (time 1788582704.87 total 30.03 last 85.11) - open/close 3233 (time 1788582714.87 total 40.04 last 84.37) - open/close 4143 (time 1788582724.89 total 50.05 last 90.85) - open/close 4933 (time 1788582734.89 total 60.06 last 78.95) - open/close 5802 (time 1788582744.90 total 70.07 last 86.83) - open/close 6515 (time 1788582754.91 total 80.07 last 71.24) - open/close 7390 (time 1788582764.92 total 90.08 last 87.42) - open/close 8169 (time 1788582774.92 total 100.09 last 77.87) - open/close 9002 (time 1788582784.93 total 110.10 last 83.20) - open/close 9862 (time 1788582794.94 total 120.10 last 85.99) total: 10000 open/close in 121.41 seconds: 82.37 ops/second - unlinked 0 (time 1788582800 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg457-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg457-server: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg457-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg457-server: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg457-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 start ost1 service on oleg457-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg457-server: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg457-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg457-server: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg457-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg457-server unloading modules via unload_modules_local on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local oleg457-server: kmemleak disabled modules unloaded. pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2