== conf-sanity test 69: replace an OST with the same index ========================================================== 02:22:19 (1788330139) start mds service on oleg216-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 2 00:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: kmemleak disabled oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: detected 4 online CPUs by sysfs oleg216-server: MODOPTS_LIBCFS= oleg216-server: Force libcfs to create 2 CPU partitions oleg216-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg216-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg216-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg216-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg216-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg216-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 830 (time 1788330255.31 total 10.04 last 82.69) - open/close 1678 (time 1788330265.32 total 20.04 last 84.74) - open/close 2650 (time 1788330275.32 total 30.05 last 97.18) - open/close 3633 (time 1788330285.34 total 40.06 last 98.14) - open/close 4461 (time 1788330295.35 total 50.07 last 82.72) - open/close 5365 (time 1788330305.35 total 60.07 last 90.39) - open/close 6217 (time 1788330315.37 total 70.10 last 84.99) - open/close 7080 (time 1788330325.38 total 80.10 last 86.27) - open/close 7949 (time 1788330335.38 total 90.10 last 86.88) - open/close 8618 (time 1788330345.39 total 100.11 last 66.84) - open/close 9224 (time 1788330355.40 total 110.12 last 60.56) - open/close 9974 (time 1788330365.41 total 120.13 last 74.92) total: 10000 open/close in 120.30 seconds: 83.13 ops/second - unlinked 0 (time 1788330369 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 791 (time 1788330443.68 total 10.00 last 79.07) - open/close 1674 (time 1788330453.71 total 20.03 last 88.03) - open/close 2591 (time 1788330463.72 total 30.04 last 91.62) - open/close 3693 (time 1788330473.72 total 40.05 last 110.16) - open/close 4726 (time 1788330483.72 total 50.05 last 103.25) - open/close 5628 (time 1788330493.74 total 60.07 last 90.06) - open/close 6525 (time 1788330503.75 total 70.07 last 89.64) - open/close 7480 (time 1788330513.75 total 80.08 last 95.44) - open/close 8492 (time 1788330523.75 total 90.08 last 101.17) - open/close 9382 (time 1788330533.76 total 100.09 last 88.95) total: 10000 open/close in 106.94 seconds: 93.51 ops/second - unlinked 0 (time 1788330546 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 604 (time 1788330625.40 total 10.01 last 60.35) - open/close 1565 (time 1788330635.40 total 20.01 last 96.09) - open/close 2568 (time 1788330645.41 total 30.02 last 100.20) - open/close 3450 (time 1788330655.43 total 40.04 last 88.01) - open/close 4355 (time 1788330665.44 total 50.05 last 90.39) - open/close 5207 (time 1788330675.45 total 60.06 last 85.15) - open/close 6194 (time 1788330685.46 total 70.07 last 98.63) - open/close 7368 (time 1788330695.46 total 80.07 last 117.36) - open/close 8435 (time 1788330705.46 total 90.07 last 106.66) - open/close 9348 (time 1788330715.47 total 100.08 last 91.28) total: 10000 open/close in 106.69 seconds: 93.73 ops/second - unlinked 0 (time 1788330726 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 878 (time 1788330799.04 total 10.01 last 87.71) - open/close 1617 (time 1788330809.05 total 20.02 last 73.80) - open/close 2351 (time 1788330819.05 total 30.03 last 73.39) - open/close 3160 (time 1788330829.06 total 40.03 last 80.87) - open/close 3906 (time 1788330839.06 total 50.03 last 74.59) - open/close 4668 (time 1788330849.07 total 60.04 last 76.13) - open/close 5294 (time 1788330859.08 total 70.05 last 62.54) - open/close 6105 (time 1788330869.08 total 80.05 last 81.10) - open/close 7061 (time 1788330879.09 total 90.07 last 95.44) - open/close 7813 (time 1788330889.09 total 100.07 last 75.19) - open/close 8587 (time 1788330899.10 total 110.07 last 77.39) - open/close 9355 (time 1788330909.10 total 120.07 last 76.78) total: 10000 open/close in 127.63 seconds: 78.35 ops/second - unlinked 0 (time 1788330921 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 832 (time 1788330995.00 total 10.01 last 83.14) - open/close 1760 (time 1788331005.01 total 20.01 last 92.73) - open/close 2695 (time 1788331015.02 total 30.03 last 93.38) - open/close 3541 (time 1788331025.02 total 40.03 last 84.55) - open/close 4359 (time 1788331035.03 total 50.04 last 81.77) - open/close 5133 (time 1788331045.04 total 60.05 last 77.34) - open/close 6251 (time 1788331055.04 total 70.05 last 111.76) - open/close 7325 (time 1788331065.06 total 80.07 last 107.22) - open/close 8135 (time 1788331075.06 total 90.07 last 80.94) - open/close 9038 (time 1788331085.06 total 100.07 last 90.30) - open/close 9914 (time 1788331095.07 total 110.08 last 87.55) total: 10000 open/close in 111.23 seconds: 89.90 ops/second - unlinked 0 (time 1788331100 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 887 (time 1788331174.33 total 10.00 last 88.69) - open/close 1672 (time 1788331184.35 total 20.03 last 78.29) - open/close 2604 (time 1788331194.35 total 30.03 last 93.18) - open/close 3496 (time 1788331204.36 total 40.04 last 89.15) - open/close 4474 (time 1788331214.37 total 50.05 last 97.70) - open/close 5459 (time 1788331224.38 total 60.06 last 98.38) - open/close 6216 (time 1788331234.38 total 70.06 last 75.68) - open/close 7003 (time 1788331244.39 total 80.06 last 78.68) - open/close 7868 (time 1788331254.42 total 90.09 last 86.26) - open/close 8705 (time 1788331264.42 total 100.10 last 83.63) - open/close 9606 (time 1788331274.43 total 110.10 last 90.05) total: 10000 open/close in 114.58 seconds: 87.28 ops/second - unlinked 0 (time 1788331283 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 1003 (time 1788331367.67 total 10.01 last 100.23) - open/close 1819 (time 1788331377.67 total 20.01 last 81.58) - open/close 2572 (time 1788331387.68 total 30.02 last 75.20) - open/close 3189 (time 1788331397.69 total 40.03 last 61.63) - open/close 4030 (time 1788331407.70 total 50.04 last 84.03) - open/close 4820 (time 1788331417.72 total 60.06 last 78.86) - open/close 5655 (time 1788331427.73 total 70.07 last 83.40) - open/close 6553 (time 1788331437.74 total 80.08 last 89.77) - open/close 7345 (time 1788331447.74 total 90.08 last 79.15) - open/close 8194 (time 1788331457.75 total 100.09 last 84.85) - open/close 9309 (time 1788331467.76 total 110.10 last 111.35) total: 10000 open/close in 118.31 seconds: 84.53 ops/second - unlinked 0 (time 1788331480 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 962 (time 1788331559.52 total 10.01 last 96.14) - open/close 1890 (time 1788331569.52 total 20.01 last 92.74) - open/close 2979 (time 1788331579.53 total 30.02 last 108.82) - open/close 4100 (time 1788331589.54 total 40.03 last 111.98) - open/close 5198 (time 1788331599.55 total 50.03 last 109.78) - open/close 6059 (time 1788331609.56 total 60.04 last 86.00) - open/close 6939 (time 1788331619.56 total 70.05 last 87.94) - open/close 7854 (time 1788331629.58 total 80.07 last 91.36) - open/close 8587 (time 1788331639.60 total 90.09 last 73.12) - open/close 9404 (time 1788331649.61 total 100.10 last 81.61) total: 10000 open/close in 105.75 seconds: 94.56 ops/second - unlinked 0 (time 1788331659 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 929 (time 1788331740.60 total 10.00 last 92.89) - open/close 1921 (time 1788331750.61 total 20.01 last 99.15) - open/close 3025 (time 1788331760.61 total 30.01 last 110.34) - open/close 3983 (time 1788331770.61 total 40.01 last 95.79) - open/close 4729 (time 1788331780.62 total 50.01 last 74.58) - open/close 5430 (time 1788331790.62 total 60.02 last 70.09) - open/close 6197 (time 1788331800.65 total 70.04 last 76.50) - open/close 7182 (time 1788331810.65 total 80.05 last 98.48) - open/close 8074 (time 1788331820.65 total 90.05 last 89.16) - open/close 8971 (time 1788331830.65 total 100.05 last 89.68) - open/close 9935 (time 1788331840.66 total 110.06 last 96.35) total: 10000 open/close in 110.61 seconds: 90.41 ops/second - unlinked 0 (time 1788331845 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 851 (time 1788331920.81 total 10.02 last 84.94) - open/close 1361 (time 1788331930.81 total 20.03 last 50.95) - open/close 2065 (time 1788331940.83 total 30.05 last 70.27) - open/close 2833 (time 1788331950.84 total 40.05 last 76.78) - open/close 3817 (time 1788331960.84 total 50.06 last 98.33) - open/close 4796 (time 1788331970.85 total 60.07 last 97.82) - open/close 5684 (time 1788331980.87 total 70.08 last 88.65) - open/close 6417 (time 1788331990.87 total 80.08 last 73.30) - open/close 7309 (time 1788332000.88 total 90.09 last 89.13) - open/close 8008 (time 1788332010.89 total 100.10 last 69.83) - open/close 8921 (time 1788332020.89 total 110.11 last 91.24) - open/close 9712 (time 1788332030.90 total 120.11 last 79.07) total: 10000 open/close in 123.68 seconds: 80.85 ops/second - unlinked 0 (time 1788332041 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.116@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg216-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg216-server: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg216-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg216-server: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg216-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.116@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.116@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.116@tcp sys.timeout=20 start ost1 service on oleg216-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg216-server: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg216-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg216-server: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg216-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server unloading modules via unload_modules_local on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing unload_modules_local oleg216-server: kmemleak disabled modules unloaded. pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2