== conf-sanity test 69: replace an OST with the same index ========================================================== 19:14:53 (1788909293) start mds service on oleg118-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1757936 Sep 8 18:39 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: kmemleak disabled oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg118-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg118-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg118-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 996 (time 1788909365.68 total 10.01 last 99.52) - open/close 1854 (time 1788909375.69 total 20.02 last 85.71) - open/close 2943 (time 1788909385.71 total 30.04 last 108.69) - open/close 3906 (time 1788909395.71 total 40.04 last 96.27) - open/close 4832 (time 1788909405.71 total 50.05 last 92.57) - open/close 5861 (time 1788909415.72 total 60.05 last 102.81) - open/close 6971 (time 1788909425.72 total 70.06 last 110.99) - open/close 8015 (time 1788909435.73 total 80.06 last 104.35) - open/close 9021 (time 1788909445.74 total 90.07 last 100.52) - open/close 9989 (time 1788909455.76 total 100.09 last 96.61) total: 10000 open/close in 100.19 seconds: 99.81 ops/second - unlinked 0 (time 1788909458 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1069 (time 1788909518.75 total 10.00 last 106.86) - open/close 2337 (time 1788909528.75 total 20.01 last 126.76) - open/close 3379 (time 1788909538.76 total 30.01 last 104.13) - open/close 4391 (time 1788909548.76 total 40.02 last 101.15) - open/close 5591 (time 1788909558.77 total 50.02 last 119.94) - open/close 6792 (time 1788909568.77 total 60.03 last 120.05) - open/close 7948 (time 1788909578.78 total 70.03 last 115.54) - open/close 9080 (time 1788909588.78 total 80.04 last 113.17) total: 10000 open/close in 87.61 seconds: 114.14 ops/second - unlinked 0 (time 1788909598 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1313 (time 1788909655.19 total 10.00 last 131.25) - open/close 2576 (time 1788909665.19 total 20.00 last 126.28) - open/close 3970 (time 1788909675.20 total 30.01 last 139.34) - open/close 5343 (time 1788909685.20 total 40.01 last 137.23) - open/close 6857 (time 1788909695.20 total 50.02 last 151.37) - open/close 8168 (time 1788909705.21 total 60.02 last 131.00) - open/close 9600 (time 1788909715.22 total 70.03 last 143.12) total: 10000 open/close in 73.69 seconds: 135.71 ops/second - unlinked 0 (time 1788909720 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1389 (time 1788909770.93 total 10.00 last 138.85) - open/close 2672 (time 1788909780.93 total 20.01 last 128.28) - open/close 3998 (time 1788909790.94 total 30.01 last 132.52) - open/close 5403 (time 1788909800.94 total 40.01 last 140.46) - open/close 6772 (time 1788909810.94 total 50.02 last 136.87) - open/close 8143 (time 1788909820.95 total 60.02 last 137.05) - open/close 9827 (time 1788909830.95 total 70.03 last 168.30) total: 10000 open/close in 71.08 seconds: 140.69 ops/second - unlinked 0 (time 1788909834 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1574 (time 1788909881.47 total 10.00 last 157.32) - open/close 3068 (time 1788909891.48 total 20.01 last 149.36) - open/close 4981 (time 1788909901.48 total 30.01 last 191.30) - open/close 6634 (time 1788909911.48 total 40.01 last 165.29) - open/close 8202 (time 1788909921.49 total 50.02 last 156.64) total: 10000 open/close in 59.76 seconds: 167.33 ops/second - unlinked 0 (time 1788909933 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1549 (time 1788909973.39 total 10.01 last 154.82) - open/close 2952 (time 1788909983.39 total 20.01 last 140.20) - open/close 4332 (time 1788909993.40 total 30.02 last 137.95) - open/close 5811 (time 1788910003.40 total 40.02 last 147.89) - open/close 7243 (time 1788910013.41 total 50.03 last 143.03) - open/close 8759 (time 1788910023.42 total 60.04 last 151.50) total: 10000 open/close in 67.54 seconds: 148.05 ops/second - unlinked 0 (time 1788910032 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1379 (time 1788910081.36 total 10.01 last 137.80) - open/close 2935 (time 1788910091.37 total 20.01 last 155.48) - open/close 4816 (time 1788910101.37 total 30.02 last 188.08) - open/close 7287 (time 1788910111.37 total 40.02 last 247.06) - open/close 8909 (time 1788910121.38 total 50.03 last 161.99) total: 10000 open/close in 56.71 seconds: 176.32 ops/second - unlinked 0 (time 1788910130 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1622 (time 1788910174.44 total 10.00 last 162.20) - open/close 3390 (time 1788910184.44 total 20.01 last 176.68) - open/close 5457 (time 1788910194.45 total 30.01 last 206.66) - open/close 7222 (time 1788910204.45 total 40.02 last 176.37) - open/close 9218 (time 1788910214.46 total 50.02 last 199.57) total: 10000 open/close in 53.43 seconds: 187.16 ops/second - unlinked 0 (time 1788910219 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 1711 (time 1788910255.91 total 10.00 last 171.04) - open/close 4164 (time 1788910265.91 total 20.01 last 245.25) - open/close 6040 (time 1788910275.91 total 30.01 last 187.46) - open/close 8387 (time 1788910285.92 total 40.02 last 234.61) total: 10000 open/close in 49.06 seconds: 203.84 ops/second - unlinked 0 (time 1788910296 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2359 (time 1788910331.31 total 10.01 last 235.75) - open/close 4347 (time 1788910341.31 total 20.01 last 198.77) - open/close 6322 (time 1788910351.32 total 30.01 last 197.44) - open/close 8242 (time 1788910361.32 total 40.01 last 191.96) total: 10000 open/close in 48.85 seconds: 204.73 ops/second - unlinked 0 (time 1788910372 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg118-server: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg118-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 15 sec oleg118-server: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg118-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre On OST0, 17262 used inodes rc=0 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.118@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.118@tcp sys.timeout=20 start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg118-server: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg118-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec oleg118-server: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg118-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: kmemleak disabled modules unloaded. pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2