== conf-sanity test 69: replace an OST with the same index ========================================================== 02:29:08 (1789367348) start mds service on oleg133-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 13 23:22 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: kmemleak disabled oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg133-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg133-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg133-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg133-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg133-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 875 (time 1789367443.41 total 10.01 last 87.42) - open/close 1791 (time 1789367453.42 total 20.01 last 91.57) - open/close 2693 (time 1789367463.43 total 30.02 last 90.11) - open/close 3601 (time 1789367473.43 total 40.03 last 90.74) - open/close 4563 (time 1789367483.44 total 50.04 last 96.12) - open/close 5420 (time 1789367493.45 total 60.04 last 85.67) - open/close 6400 (time 1789367503.45 total 70.05 last 97.94) - open/close 7292 (time 1789367513.46 total 80.06 last 89.13) - open/close 8152 (time 1789367523.46 total 90.06 last 85.96) - open/close 9103 (time 1789367533.47 total 100.06 last 95.10) - open/close 9941 (time 1789367543.47 total 110.07 last 83.76) total: 10000 open/close in 110.79 seconds: 90.26 ops/second - unlinked 0 (time 1789367548 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1098 (time 1789367621.44 total 10.01 last 109.74) - open/close 2125 (time 1789367631.45 total 20.01 last 102.63) - open/close 3149 (time 1789367641.45 total 30.02 last 102.35) - open/close 4088 (time 1789367651.46 total 40.02 last 93.86) - open/close 5013 (time 1789367661.46 total 50.03 last 92.45) - open/close 6047 (time 1789367671.46 total 60.03 last 103.38) - open/close 7072 (time 1789367681.47 total 70.03 last 102.46) - open/close 8115 (time 1789367691.47 total 80.03 last 104.29) - open/close 9194 (time 1789367701.47 total 90.04 last 107.83) total: 10000 open/close in 97.33 seconds: 102.75 ops/second - unlinked 0 (time 1789367711 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 962 (time 1789367781.24 total 10.00 last 96.18) - open/close 2069 (time 1789367791.24 total 20.01 last 110.64) - open/close 3159 (time 1789367801.25 total 30.02 last 108.89) - open/close 4164 (time 1789367811.25 total 40.02 last 100.49) - open/close 5227 (time 1789367821.26 total 50.03 last 106.21) - open/close 6283 (time 1789367831.27 total 60.03 last 105.55) - open/close 7484 (time 1789367841.27 total 70.03 last 120.08) - open/close 8615 (time 1789367851.27 total 80.04 last 113.02) - open/close 9657 (time 1789367861.28 total 90.05 last 104.10) total: 10000 open/close in 93.23 seconds: 107.26 ops/second - unlinked 0 (time 1789367867 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1252 (time 1789367928.28 total 10.00 last 125.19) - open/close 2721 (time 1789367938.28 total 20.00 last 146.90) - open/close 4152 (time 1789367948.28 total 30.00 last 143.08) - open/close 5220 (time 1789367958.28 total 40.00 last 106.77) - open/close 6391 (time 1789367968.29 total 50.01 last 117.08) - open/close 7724 (time 1789367978.29 total 60.01 last 133.28) - open/close 8938 (time 1789367988.29 total 70.01 last 121.37) total: 10000 open/close in 79.92 seconds: 125.12 ops/second - unlinked 0 (time 1789368000 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1058 (time 1789368060.40 total 10.00 last 105.77) - open/close 2099 (time 1789368070.41 total 20.01 last 104.04) - open/close 3126 (time 1789368080.41 total 30.02 last 102.62) - open/close 4100 (time 1789368090.42 total 40.02 last 97.39) - open/close 5055 (time 1789368100.43 total 50.03 last 95.39) - open/close 6052 (time 1789368110.44 total 60.04 last 99.58) - open/close 7071 (time 1789368120.45 total 70.05 last 101.76) - open/close 8156 (time 1789368130.45 total 80.05 last 108.50) - open/close 9290 (time 1789368140.46 total 90.06 last 113.34) total: 10000 open/close in 96.56 seconds: 103.56 ops/second - unlinked 0 (time 1789368149 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1162 (time 1789368209.52 total 10.01 last 116.06) - open/close 2247 (time 1789368219.52 total 20.02 last 108.46) - open/close 3352 (time 1789368229.53 total 30.02 last 110.41) - open/close 4535 (time 1789368239.54 total 40.03 last 118.26) - open/close 5591 (time 1789368249.54 total 50.03 last 105.57) - open/close 6740 (time 1789368259.54 total 60.03 last 114.87) - open/close 7902 (time 1789368269.55 total 70.04 last 116.14) - open/close 9052 (time 1789368279.55 total 80.04 last 114.96) total: 10000 open/close in 87.92 seconds: 113.74 ops/second - unlinked 0 (time 1789368289 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1322 (time 1789368348.15 total 10.00 last 132.17) - open/close 2633 (time 1789368358.15 total 20.01 last 131.04) - open/close 3797 (time 1789368368.16 total 30.01 last 116.31) - open/close 4866 (time 1789368378.16 total 40.02 last 106.85) - open/close 5885 (time 1789368388.18 total 50.03 last 101.74) - open/close 7055 (time 1789368398.18 total 60.03 last 117.00) - open/close 8244 (time 1789368408.18 total 70.04 last 118.86) - open/close 9459 (time 1789368418.20 total 80.05 last 121.34) total: 10000 open/close in 85.20 seconds: 117.37 ops/second - unlinked 0 (time 1789368426 ; total 1 ; last 1) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1330 (time 1789368480.11 total 10.00 last 133.00) - open/close 2634 (time 1789368490.12 total 20.00 last 130.37) - open/close 3784 (time 1789368500.12 total 30.00 last 114.99) - open/close 5009 (time 1789368510.12 total 40.01 last 122.46) - open/close 6323 (time 1789368520.12 total 50.01 last 131.35) - open/close 7782 (time 1789368530.12 total 60.01 last 145.88) - open/close 9652 (time 1789368540.13 total 70.02 last 186.91) total: 10000 open/close in 71.77 seconds: 139.34 ops/second - unlinked 0 (time 1789368543 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 2218 (time 1789368587.33 total 10.00 last 221.78) - open/close 4405 (time 1789368597.33 total 20.00 last 218.64) - open/close 6137 (time 1789368607.34 total 30.01 last 173.17) - open/close 8152 (time 1789368617.34 total 40.01 last 201.45) total: 10000 open/close in 49.28 seconds: 202.93 ops/second - unlinked 0 (time 1789368628 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1258 (time 1789368674.67 total 10.00 last 125.74) - open/close 2643 (time 1789368684.67 total 20.01 last 138.49) - open/close 4038 (time 1789368694.68 total 30.01 last 139.48) - open/close 5333 (time 1789368704.68 total 40.01 last 129.49) - open/close 6971 (time 1789368714.68 total 50.01 last 163.72) - open/close 8490 (time 1789368724.69 total 60.02 last 151.79) - open/close 9838 (time 1789368734.69 total 70.02 last 134.76) total: 10000 open/close in 71.25 seconds: 140.34 ops/second - unlinked 0 (time 1789368738 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg133-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 start ost1 service on oleg133-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server unloading modules via unload_modules_local on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: kmemleak disabled modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2