== conf-sanity test 69: replace an OST with the same index ========================================================== 21:29:28 (1788830968) start mds service on oleg420-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: kmemleak disabled oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg420-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg420-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg420-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg420-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 750 (time 1788831075.08 total 10.00 last 74.98) - open/close 1429 (time 1788831085.09 total 20.01 last 67.82) - open/close 2158 (time 1788831095.10 total 30.03 last 72.77) - open/close 2846 (time 1788831105.11 total 40.04 last 68.76) - open/close 3577 (time 1788831115.11 total 50.04 last 73.07) - open/close 4294 (time 1788831125.12 total 60.05 last 71.65) - open/close 5008 (time 1788831135.13 total 70.06 last 71.32) - open/close 5680 (time 1788831145.13 total 80.06 last 67.19) - open/close 6350 (time 1788831155.16 total 90.08 last 66.85) - open/close 7032 (time 1788831165.16 total 100.09 last 68.15) - open/close 7707 (time 1788831175.17 total 110.10 last 67.46) - open/close 8334 (time 1788831185.18 total 120.11 last 62.62) - open/close 8962 (time 1788831195.19 total 130.12 last 62.73) - open/close 9622 (time 1788831205.20 total 140.12 last 65.98) total: 10000 open/close in 146.09 seconds: 68.45 ops/second - unlinked 0 (time 1788831215 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 721 (time 1788831310.85 total 10.01 last 72.05) - open/close 1420 (time 1788831320.86 total 20.02 last 69.80) - open/close 2131 (time 1788831330.88 total 30.04 last 70.99) - open/close 2777 (time 1788831340.88 total 40.04 last 64.59) - open/close 3525 (time 1788831350.88 total 50.04 last 74.78) - open/close 4207 (time 1788831360.90 total 60.06 last 68.08) - open/close 4914 (time 1788831370.90 total 70.06 last 70.69) - open/close 5655 (time 1788831380.90 total 80.06 last 74.09) - open/close 6358 (time 1788831390.90 total 90.06 last 70.27) - open/close 7140 (time 1788831400.91 total 100.07 last 78.18) - open/close 7872 (time 1788831410.92 total 110.08 last 73.11) - open/close 8574 (time 1788831420.93 total 120.09 last 70.09) - open/close 9307 (time 1788831430.94 total 130.10 last 73.30) total: 10000 open/close in 139.95 seconds: 71.46 ops/second - unlinked 0 (time 1788831445 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 659 (time 1788831541.51 total 10.00 last 65.88) - open/close 1372 (time 1788831551.52 total 20.01 last 71.26) - open/close 1986 (time 1788831561.52 total 30.01 last 61.39) - open/close 2755 (time 1788831571.53 total 40.02 last 76.83) - open/close 3520 (time 1788831581.54 total 50.04 last 76.37) - open/close 4251 (time 1788831591.56 total 60.05 last 73.01) - open/close 4989 (time 1788831601.59 total 70.08 last 73.56) - open/close 5717 (time 1788831611.60 total 80.09 last 72.70) - open/close 6422 (time 1788831621.61 total 90.10 last 70.44) - open/close 7061 (time 1788831631.62 total 100.11 last 63.86) - open/close 7705 (time 1788831641.63 total 110.12 last 64.35) - open/close 8366 (time 1788831651.65 total 120.15 last 65.91) - open/close 8940 (time 1788831661.67 total 130.16 last 57.34) - open/close 9477 (time 1788831671.67 total 140.16 last 53.69) total: 10000 open/close in 149.47 seconds: 66.90 ops/second - unlinked 0 (time 1788831685 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 685 (time 1788831783.61 total 10.00 last 68.48) - open/close 1354 (time 1788831793.61 total 20.01 last 66.86) - open/close 1984 (time 1788831803.63 total 30.02 last 62.90) - open/close 2691 (time 1788831813.64 total 40.03 last 70.65) - open/close 3339 (time 1788831823.64 total 50.04 last 64.74) - open/close 3975 (time 1788831833.66 total 60.05 last 63.53) - open/close 4717 (time 1788831843.66 total 70.05 last 74.19) - open/close 5441 (time 1788831853.67 total 80.07 last 72.30) - open/close 6074 (time 1788831863.68 total 90.07 last 63.25) - open/close 6727 (time 1788831873.68 total 100.08 last 65.29) - open/close 7403 (time 1788831883.69 total 110.08 last 67.54) - open/close 8066 (time 1788831893.69 total 120.09 last 66.26) - open/close 8711 (time 1788831903.70 total 130.10 last 64.45) - open/close 9246 (time 1788831913.71 total 140.10 last 53.46) - open/close 9853 (time 1788831923.73 total 150.13 last 60.58) total: 10000 open/close in 152.19 seconds: 65.71 ops/second - unlinked 0 (time 1788831930 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 526 (time 1788832029.18 total 10.01 last 52.54) - open/close 1090 (time 1788832039.19 total 20.01 last 56.39) - open/close 1683 (time 1788832049.19 total 30.01 last 59.30) - open/close 2418 (time 1788832059.20 total 40.03 last 73.40) - open/close 3061 (time 1788832069.20 total 50.03 last 64.26) - open/close 3746 (time 1788832079.22 total 60.04 last 68.42) - open/close 4381 (time 1788832089.22 total 70.05 last 63.48) - open/close 5023 (time 1788832099.22 total 80.05 last 64.20) - open/close 5742 (time 1788832109.23 total 90.06 last 71.84) - open/close 6403 (time 1788832119.24 total 100.07 last 66.02) - open/close 7096 (time 1788832129.25 total 110.08 last 69.21) - open/close 7683 (time 1788832139.26 total 120.09 last 58.66) - open/close 8294 (time 1788832149.27 total 130.09 last 61.05) - open/close 8966 (time 1788832159.30 total 140.12 last 67.00) - open/close 9607 (time 1788832169.30 total 150.13 last 64.09) total: 10000 open/close in 155.13 seconds: 64.46 ops/second - unlinked 0 (time 1788832179 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 650 (time 1788832270.33 total 10.01 last 64.95) - open/close 1344 (time 1788832280.34 total 20.01 last 69.37) - open/close 1983 (time 1788832290.35 total 30.02 last 63.82) - open/close 2683 (time 1788832300.35 total 40.03 last 69.97) - open/close 3446 (time 1788832310.37 total 50.05 last 76.16) - open/close 4118 (time 1788832320.39 total 60.06 last 67.09) - open/close 4821 (time 1788832330.39 total 70.07 last 70.27) - open/close 5393 (time 1788832340.41 total 80.08 last 57.11) - open/close 6061 (time 1788832350.41 total 90.09 last 66.78) - open/close 6746 (time 1788832360.42 total 100.10 last 68.39) - open/close 7414 (time 1788832370.42 total 110.10 last 66.80) - open/close 7952 (time 1788832380.43 total 120.10 last 53.79) - open/close 8565 (time 1788832390.43 total 130.11 last 61.25) - open/close 9190 (time 1788832400.45 total 140.13 last 62.40) - open/close 9799 (time 1788832410.46 total 150.14 last 60.82) total: 10000 open/close in 153.54 seconds: 65.13 ops/second - unlinked 0 (time 1788832419 ; total 1 ; last 1) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 744 (time 1788832510.13 total 10.00 last 74.36) - open/close 1424 (time 1788832520.14 total 20.02 last 67.92) - open/close 2141 (time 1788832530.15 total 30.03 last 71.61) - open/close 2856 (time 1788832540.16 total 40.04 last 71.42) - open/close 3488 (time 1788832550.16 total 50.04 last 63.20) - open/close 4216 (time 1788832560.17 total 60.05 last 72.72) - open/close 4944 (time 1788832570.18 total 70.05 last 72.77) - open/close 5635 (time 1788832580.18 total 80.06 last 69.06) - open/close 6400 (time 1788832590.20 total 90.07 last 76.41) - open/close 6997 (time 1788832600.21 total 100.08 last 59.64) - open/close 7592 (time 1788832610.22 total 110.09 last 59.42) - open/close 8259 (time 1788832620.23 total 120.10 last 66.63) - open/close 8803 (time 1788832630.25 total 130.12 last 54.29) - open/close 9482 (time 1788832640.25 total 140.13 last 67.88) total: 10000 open/close in 148.97 seconds: 67.13 ops/second - unlinked 0 (time 1788832653 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 664 (time 1788832752.31 total 10.02 last 66.29) - open/close 1333 (time 1788832762.32 total 20.02 last 66.85) - open/close 2086 (time 1788832772.33 total 30.03 last 75.24) - open/close 2807 (time 1788832782.33 total 40.03 last 72.09) - open/close 3502 (time 1788832792.35 total 50.05 last 69.36) - open/close 4255 (time 1788832802.36 total 60.06 last 75.24) - open/close 4980 (time 1788832812.36 total 70.06 last 72.48) - open/close 5687 (time 1788832822.37 total 80.07 last 70.66) - open/close 6461 (time 1788832832.38 total 90.08 last 77.33) - open/close 7134 (time 1788832842.39 total 100.09 last 67.24) - open/close 7923 (time 1788832852.39 total 110.09 last 78.88) - open/close 8648 (time 1788832862.40 total 120.10 last 72.46) - open/close 9375 (time 1788832872.40 total 130.10 last 72.69) total: 10000 open/close in 138.53 seconds: 72.19 ops/second - unlinked 0 (time 1788832884 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 714 (time 1788832978.03 total 10.01 last 71.35) - open/close 1391 (time 1788832988.04 total 20.02 last 67.65) - open/close 2224 (time 1788832998.04 total 30.02 last 83.27) - open/close 2992 (time 1788833008.04 total 40.02 last 76.78) - open/close 3723 (time 1788833018.05 total 50.03 last 73.03) - open/close 4452 (time 1788833028.06 total 60.04 last 72.82) - open/close 5220 (time 1788833038.08 total 70.06 last 76.66) - open/close 6036 (time 1788833048.08 total 80.06 last 81.58) - open/close 6788 (time 1788833058.08 total 90.06 last 75.19) - open/close 7565 (time 1788833068.12 total 100.10 last 77.44) - open/close 8243 (time 1788833078.13 total 110.11 last 67.75) - open/close 8943 (time 1788833088.14 total 120.11 last 69.93) - open/close 9714 (time 1788833098.16 total 130.14 last 76.94) total: 10000 open/close in 133.90 seconds: 74.68 ops/second - unlinked 0 (time 1788833106 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 592 (time 1788833200.36 total 10.01 last 59.16) - open/close 1321 (time 1788833210.40 total 20.04 last 72.64) - open/close 2073 (time 1788833220.41 total 30.05 last 75.15) - open/close 2748 (time 1788833230.41 total 40.05 last 67.49) - open/close 3478 (time 1788833240.42 total 50.06 last 72.90) - open/close 4141 (time 1788833250.43 total 60.08 last 66.22) - open/close 4792 (time 1788833260.44 total 70.09 last 65.04) - open/close 5479 (time 1788833270.45 total 80.10 last 68.63) - open/close 6176 (time 1788833280.46 total 90.11 last 69.64) - open/close 6858 (time 1788833290.47 total 100.12 last 68.12) - open/close 7590 (time 1788833300.48 total 110.13 last 73.13) - open/close 8383 (time 1788833310.49 total 120.14 last 79.23) - open/close 9160 (time 1788833320.50 total 130.14 last 77.65) - open/close 9849 (time 1788833330.51 total 140.15 last 68.83) total: 10000 open/close in 142.12 seconds: 70.36 ops/second - unlinked 0 (time 1788833336 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server unloading modules via unload_modules_local on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local oleg420-server: kmemleak disabled modules unloaded. pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2