== conf-sanity test 69: replace an OST with the same index ========================================================== 14:02:38 (1788199358) start mds service on oleg317-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Aug 31 03:29 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: kmemleak disabled oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg317-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg317-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg317-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 944 (time 1788199464.53 total 10.00 last 94.39) - open/close 1793 (time 1788199474.53 total 20.00 last 84.88) - open/close 2759 (time 1788199484.54 total 30.01 last 96.55) - open/close 3733 (time 1788199494.54 total 40.01 last 97.40) - open/close 4620 (time 1788199504.54 total 50.01 last 88.68) - open/close 5476 (time 1788199514.55 total 60.03 last 85.47) - open/close 6348 (time 1788199524.56 total 70.04 last 87.10) - open/close 7290 (time 1788199534.57 total 80.04 last 94.17) - open/close 8175 (time 1788199544.59 total 90.06 last 88.34) - open/close 9124 (time 1788199554.59 total 100.06 last 94.87) total: 10000 open/close in 109.62 seconds: 91.23 ops/second - unlinked 0 (time 1788199568 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 917 (time 1788199647.56 total 10.01 last 91.64) - open/close 1815 (time 1788199657.56 total 20.01 last 89.76) - open/close 2684 (time 1788199667.58 total 30.02 last 86.80) - open/close 3643 (time 1788199677.58 total 40.03 last 95.87) - open/close 4602 (time 1788199687.58 total 50.03 last 95.88) - open/close 5556 (time 1788199697.59 total 60.03 last 95.34) - open/close 6466 (time 1788199707.60 total 70.04 last 90.90) - open/close 7388 (time 1788199717.60 total 80.05 last 92.15) - open/close 8436 (time 1788199727.60 total 90.05 last 104.78) - open/close 9397 (time 1788199737.61 total 100.06 last 96.06) total: 10000 open/close in 106.09 seconds: 94.26 ops/second - unlinked 0 (time 1788199747 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 931 (time 1788199822.15 total 10.00 last 93.08) - open/close 1760 (time 1788199832.15 total 20.01 last 82.85) - open/close 2657 (time 1788199842.16 total 30.01 last 89.68) - open/close 3656 (time 1788199852.16 total 40.01 last 99.90) - open/close 4617 (time 1788199862.16 total 50.01 last 96.05) - open/close 5540 (time 1788199872.17 total 60.02 last 92.24) - open/close 6519 (time 1788199882.17 total 70.03 last 97.86) - open/close 7451 (time 1788199892.18 total 80.03 last 93.16) - open/close 8349 (time 1788199902.18 total 90.03 last 89.78) - open/close 9276 (time 1788199912.19 total 100.05 last 92.58) total: 10000 open/close in 107.31 seconds: 93.19 ops/second - unlinked 0 (time 1788199922 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 898 (time 1788199998.61 total 10.01 last 89.71) - open/close 1873 (time 1788200008.63 total 20.03 last 97.33) - open/close 2821 (time 1788200018.64 total 30.04 last 94.73) - open/close 3706 (time 1788200028.65 total 40.05 last 88.39) - open/close 4585 (time 1788200038.66 total 50.06 last 87.83) - open/close 5456 (time 1788200048.66 total 60.06 last 87.05) - open/close 6338 (time 1788200058.67 total 70.07 last 88.13) - open/close 7212 (time 1788200068.68 total 80.08 last 87.34) - open/close 8046 (time 1788200078.68 total 90.08 last 83.40) - open/close 8988 (time 1788200088.68 total 100.08 last 94.16) - open/close 9779 (time 1788200098.69 total 110.09 last 79.03) total: 10000 open/close in 112.75 seconds: 88.69 ops/second - unlinked 0 (time 1788200105 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 899 (time 1788200177.57 total 10.01 last 89.85) - open/close 1852 (time 1788200187.57 total 20.01 last 95.25) - open/close 2816 (time 1788200197.58 total 30.02 last 96.34) - open/close 3824 (time 1788200207.59 total 40.03 last 100.71) - open/close 4720 (time 1788200217.60 total 50.04 last 89.47) - open/close 5658 (time 1788200227.61 total 60.05 last 93.75) - open/close 6508 (time 1788200237.62 total 70.06 last 84.87) - open/close 7366 (time 1788200247.62 total 80.06 last 85.80) - open/close 8239 (time 1788200257.62 total 90.06 last 87.29) - open/close 9211 (time 1788200267.63 total 100.07 last 97.14) total: 10000 open/close in 108.45 seconds: 92.21 ops/second - unlinked 0 (time 1788200280 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1026 (time 1788200355.00 total 10.00 last 102.56) - open/close 1868 (time 1788200365.01 total 20.01 last 84.18) - open/close 2767 (time 1788200375.01 total 30.01 last 89.86) - open/close 3705 (time 1788200385.02 total 40.02 last 93.73) - open/close 4676 (time 1788200395.03 total 50.03 last 96.97) - open/close 5528 (time 1788200405.04 total 60.04 last 85.16) - open/close 6518 (time 1788200415.05 total 70.05 last 98.91) - open/close 7494 (time 1788200425.05 total 80.05 last 97.59) - open/close 8462 (time 1788200435.05 total 90.05 last 96.76) - open/close 9396 (time 1788200445.06 total 100.06 last 93.35) total: 10000 open/close in 106.62 seconds: 93.79 ops/second - unlinked 0 (time 1788200455 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 963 (time 1788200536.82 total 10.01 last 96.24) - open/close 1878 (time 1788200546.82 total 20.01 last 91.45) - open/close 2749 (time 1788200556.83 total 30.02 last 87.01) - open/close 3657 (time 1788200566.84 total 40.03 last 90.76) - open/close 4527 (time 1788200576.84 total 50.03 last 86.95) - open/close 5407 (time 1788200586.85 total 60.04 last 87.94) - open/close 6366 (time 1788200596.86 total 70.05 last 95.84) - open/close 7312 (time 1788200606.86 total 80.05 last 94.60) - open/close 8234 (time 1788200616.86 total 90.05 last 92.14) - open/close 9209 (time 1788200626.87 total 100.06 last 97.45) total: 10000 open/close in 108.00 seconds: 92.60 ops/second - unlinked 0 (time 1788200638 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 849 (time 1788200710.04 total 10.01 last 84.79) - open/close 1810 (time 1788200720.04 total 20.02 last 96.07) - open/close 2805 (time 1788200730.04 total 30.02 last 99.46) - open/close 3847 (time 1788200740.05 total 40.03 last 104.10) - open/close 4804 (time 1788200750.05 total 50.03 last 95.70) - open/close 5759 (time 1788200760.06 total 60.03 last 95.46) - open/close 6734 (time 1788200770.06 total 70.04 last 97.43) - open/close 7748 (time 1788200780.07 total 80.04 last 101.36) - open/close 8718 (time 1788200790.07 total 90.04 last 96.99) - open/close 9703 (time 1788200800.07 total 100.04 last 98.50) total: 10000 open/close in 102.94 seconds: 97.14 ops/second - unlinked 0 (time 1788200807 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 920 (time 1788200885.07 total 10.00 last 91.99) - open/close 2041 (time 1788200895.07 total 20.01 last 112.05) - open/close 3071 (time 1788200905.08 total 30.01 last 102.96) - open/close 4048 (time 1788200915.08 total 40.01 last 97.67) - open/close 4853 (time 1788200925.10 total 50.03 last 80.32) - open/close 5685 (time 1788200935.11 total 60.04 last 83.16) - open/close 6371 (time 1788200945.12 total 70.05 last 68.54) - open/close 7094 (time 1788200955.12 total 80.05 last 72.29) - open/close 7963 (time 1788200965.12 total 90.05 last 86.85) - open/close 8726 (time 1788200975.13 total 100.06 last 76.28) - open/close 9613 (time 1788200985.14 total 110.07 last 88.57) total: 10000 open/close in 114.39 seconds: 87.42 ops/second - unlinked 0 (time 1788200993 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 959 (time 1788201069.95 total 10.00 last 95.90) - open/close 1922 (time 1788201079.96 total 20.01 last 96.23) - open/close 2850 (time 1788201089.97 total 30.02 last 92.67) - open/close 3823 (time 1788201099.98 total 40.03 last 97.21) - open/close 4787 (time 1788201109.99 total 50.04 last 96.35) - open/close 5694 (time 1788201120.00 total 60.04 last 90.63) - open/close 6580 (time 1788201130.00 total 70.05 last 88.55) - open/close 7452 (time 1788201140.02 total 80.07 last 87.06) - open/close 8250 (time 1788201150.04 total 90.09 last 79.62) - open/close 9183 (time 1788201160.04 total 100.09 last 93.27) total: 10000 open/close in 108.65 seconds: 92.04 ops/second - unlinked 0 (time 1788201172 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.117@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg317-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.117@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.117@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.117@tcp sys.timeout=20 start ost1 service on oleg317-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg317-server: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg317-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local oleg317-server: kmemleak disabled modules unloaded. pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2