== conf-sanity test 69: replace an OST with the same index ========================================================== 12:46:17 (1789404377) start mds service on oleg133-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 05:02 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: kmemleak disabled oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg133-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg133-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg133-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg133-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg133-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 950 (time 1789404492.04 total 10.02 last 94.84) - open/close 1908 (time 1789404502.04 total 20.02 last 95.78) - open/close 2862 (time 1789404512.05 total 30.03 last 95.32) - open/close 3800 (time 1789404522.05 total 40.03 last 93.77) - open/close 4785 (time 1789404532.05 total 50.03 last 98.46) - open/close 5771 (time 1789404542.06 total 60.04 last 98.53) - open/close 6776 (time 1789404552.06 total 70.04 last 100.48) - open/close 7461 (time 1789404562.07 total 80.05 last 68.45) - open/close 7969 (time 1789404572.08 total 90.06 last 50.74) - open/close 8620 (time 1789404582.09 total 100.07 last 65.04) - open/close 9472 (time 1789404592.09 total 110.07 last 85.19) total: 10000 open/close in 115.36 seconds: 86.68 ops/second - unlinked 0 (time 1789404602 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1007 (time 1789404681.79 total 10.02 last 100.45) - open/close 1893 (time 1789404691.79 total 20.03 last 88.59) - open/close 3005 (time 1789404701.79 total 30.03 last 111.19) - open/close 3936 (time 1789404711.80 total 40.03 last 93.04) - open/close 4782 (time 1789404721.81 total 50.05 last 84.49) - open/close 5733 (time 1789404731.81 total 60.05 last 95.08) - open/close 6703 (time 1789404741.82 total 70.05 last 96.96) - open/close 7725 (time 1789404751.83 total 80.06 last 102.10) - open/close 8566 (time 1789404761.83 total 90.06 last 84.10) - open/close 9479 (time 1789404771.85 total 100.08 last 91.13) total: 10000 open/close in 107.87 seconds: 92.70 ops/second - unlinked 0 (time 1789404784 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 568 (time 1789404874.20 total 10.01 last 56.76) - open/close 1247 (time 1789404884.21 total 20.02 last 67.82) - open/close 2169 (time 1789404894.22 total 30.02 last 92.14) - open/close 2932 (time 1789404904.24 total 40.04 last 76.16) - open/close 3804 (time 1789404914.25 total 50.05 last 87.12) - open/close 4534 (time 1789404924.26 total 60.06 last 72.90) - open/close 5405 (time 1789404934.26 total 70.07 last 87.08) - open/close 6261 (time 1789404944.27 total 80.07 last 85.54) - open/close 7175 (time 1789404954.27 total 90.08 last 91.37) - open/close 8166 (time 1789404964.28 total 100.08 last 99.07) - open/close 9013 (time 1789404974.28 total 110.09 last 84.63) - open/close 9908 (time 1789404984.29 total 120.09 last 89.46) total: 10000 open/close in 121.12 seconds: 82.56 ops/second - unlinked 0 (time 1789404990 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 781 (time 1789405069.77 total 10.00 last 78.10) - open/close 1550 (time 1789405079.77 total 20.00 last 76.89) - open/close 2460 (time 1789405089.78 total 30.00 last 90.97) - open/close 3340 (time 1789405099.79 total 40.02 last 87.88) - open/close 4321 (time 1789405109.80 total 50.03 last 97.99) - open/close 5243 (time 1789405119.81 total 60.04 last 92.09) - open/close 6195 (time 1789405129.82 total 70.04 last 95.16) - open/close 7122 (time 1789405139.83 total 80.05 last 92.60) - open/close 7968 (time 1789405149.83 total 90.06 last 84.59) - open/close 8865 (time 1789405159.83 total 100.06 last 89.70) - open/close 9931 (time 1789405169.85 total 110.07 last 106.42) total: 10000 open/close in 110.76 seconds: 90.28 ops/second - unlinked 0 (time 1789405175 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 846 (time 1789405250.99 total 10.01 last 84.52) - open/close 1667 (time 1789405261.00 total 20.02 last 82.02) - open/close 2573 (time 1789405271.01 total 30.03 last 90.53) - open/close 3489 (time 1789405281.01 total 40.03 last 91.59) - open/close 4411 (time 1789405291.01 total 50.03 last 92.19) - open/close 5311 (time 1789405301.02 total 60.04 last 89.89) - open/close 6082 (time 1789405311.03 total 70.05 last 77.03) - open/close 6870 (time 1789405321.04 total 80.06 last 78.76) - open/close 7777 (time 1789405331.04 total 90.06 last 90.69) - open/close 8794 (time 1789405341.04 total 100.06 last 101.62) - open/close 9746 (time 1789405351.04 total 110.06 last 95.19) total: 10000 open/close in 113.05 seconds: 88.46 ops/second - unlinked 0 (time 1789405358 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 886 (time 1789405437.28 total 10.01 last 88.54) - open/close 1756 (time 1789405447.29 total 20.01 last 86.94) - open/close 2577 (time 1789405457.30 total 30.02 last 82.04) - open/close 3480 (time 1789405467.30 total 40.03 last 90.24) - open/close 4325 (time 1789405477.32 total 50.04 last 84.38) - open/close 5134 (time 1789405487.32 total 60.04 last 80.89) - open/close 6048 (time 1789405497.33 total 70.06 last 91.29) - open/close 6893 (time 1789405507.33 total 80.06 last 84.47) - open/close 7737 (time 1789405517.34 total 90.07 last 84.33) - open/close 8685 (time 1789405527.35 total 100.07 last 94.74) - open/close 9615 (time 1789405537.35 total 110.08 last 92.94) total: 10000 open/close in 114.39 seconds: 87.42 ops/second - unlinked 0 (time 1789405546 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 824 (time 1789405622.14 total 10.01 last 82.35) - open/close 1727 (time 1789405632.15 total 20.01 last 90.25) - open/close 2547 (time 1789405642.15 total 30.02 last 81.95) - open/close 3360 (time 1789405652.16 total 40.02 last 81.29) - open/close 4187 (time 1789405662.17 total 50.03 last 82.62) - open/close 5058 (time 1789405672.17 total 60.03 last 87.10) - open/close 5949 (time 1789405682.17 total 70.03 last 89.10) - open/close 6805 (time 1789405692.18 total 80.04 last 85.47) - open/close 7580 (time 1789405702.19 total 90.05 last 77.48) - open/close 8465 (time 1789405712.19 total 100.05 last 88.48) - open/close 9443 (time 1789405722.20 total 110.06 last 97.68) total: 10000 open/close in 116.80 seconds: 85.62 ops/second - unlinked 0 (time 1789405733 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 914 (time 1789405809.30 total 10.01 last 91.35) - open/close 1830 (time 1789405819.30 total 20.01 last 91.58) - open/close 2825 (time 1789405829.32 total 30.02 last 99.34) - open/close 3807 (time 1789405839.32 total 40.03 last 98.16) - open/close 4672 (time 1789405849.33 total 50.04 last 86.41) - open/close 5482 (time 1789405859.34 total 60.05 last 80.94) - open/close 6374 (time 1789405869.34 total 70.05 last 89.18) - open/close 7272 (time 1789405879.36 total 80.06 last 89.67) - open/close 8260 (time 1789405889.37 total 90.07 last 98.69) - open/close 9152 (time 1789405899.38 total 100.08 last 89.12) total: 10000 open/close in 108.92 seconds: 91.81 ops/second - unlinked 0 (time 1789405912 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 727 (time 1789405987.50 total 10.01 last 72.65) - open/close 1575 (time 1789405997.51 total 20.01 last 84.77) - open/close 2589 (time 1789406007.51 total 30.01 last 101.37) - open/close 3541 (time 1789406017.53 total 40.03 last 95.06) - open/close 4275 (time 1789406027.53 total 50.03 last 73.39) - open/close 4925 (time 1789406037.53 total 60.03 last 64.99) - open/close 5664 (time 1789406047.53 total 70.03 last 73.89) - open/close 6460 (time 1789406057.53 total 80.03 last 79.58) - open/close 7250 (time 1789406067.56 total 90.07 last 78.75) - open/close 8185 (time 1789406077.57 total 100.07 last 93.49) - open/close 9010 (time 1789406087.58 total 110.08 last 82.41) - open/close 9784 (time 1789406097.58 total 120.08 last 77.36) total: 10000 open/close in 123.35 seconds: 81.07 ops/second - unlinked 0 (time 1789406106 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 987 (time 1789406186.59 total 10.01 last 98.64) - open/close 1846 (time 1789406196.60 total 20.01 last 85.86) - open/close 2812 (time 1789406206.60 total 30.01 last 96.57) - open/close 3845 (time 1789406216.60 total 40.02 last 103.26) - open/close 4860 (time 1789406226.60 total 50.02 last 101.50) - open/close 5763 (time 1789406236.61 total 60.03 last 90.20) - open/close 6724 (time 1789406246.62 total 70.03 last 96.05) - open/close 7538 (time 1789406256.62 total 80.04 last 81.35) - open/close 8352 (time 1789406266.66 total 90.07 last 81.14) - open/close 9003 (time 1789406276.66 total 100.08 last 65.08) - open/close 9733 (time 1789406286.67 total 110.09 last 72.91) total: 10000 open/close in 114.78 seconds: 87.12 ops/second - unlinked 0 (time 1789406297 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg133-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 start ost1 service on oleg133-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server unloading modules via unload_modules_local on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: kmemleak disabled modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2