== conf-sanity test 69: replace an OST with the same index ========================================================== 22:19:46 (1788833986) start mds service on oleg408-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:47 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: kmemleak disabled oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg408-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 736 (time 1788834101.25 total 10.01 last 73.50) - open/close 1434 (time 1788834111.29 total 20.06 last 69.48) - open/close 2224 (time 1788834121.31 total 30.07 last 78.90) - open/close 2861 (time 1788834131.34 total 40.10 last 63.50) - open/close 3395 (time 1788834141.34 total 50.11 last 53.38) - open/close 3997 (time 1788834151.34 total 60.11 last 60.17) - open/close 4592 (time 1788834161.35 total 70.12 last 59.45) - open/close 5289 (time 1788834171.36 total 80.12 last 69.67) - open/close 5994 (time 1788834181.37 total 90.14 last 70.40) - open/close 6771 (time 1788834191.38 total 100.14 last 77.65) - open/close 7513 (time 1788834201.39 total 110.16 last 74.12) - open/close 8258 (time 1788834211.39 total 120.16 last 74.47) - open/close 8964 (time 1788834221.40 total 130.16 last 70.58) - open/close 9716 (time 1788834231.40 total 140.17 last 75.15) total: 10000 open/close in 143.89 seconds: 69.50 ops/second - unlinked 0 (time 1788834238 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 692 (time 1788834325.67 total 10.01 last 69.15) - open/close 1293 (time 1788834335.68 total 20.02 last 60.04) - open/close 1966 (time 1788834345.68 total 30.02 last 67.29) - open/close 2624 (time 1788834355.69 total 40.03 last 65.71) - open/close 3370 (time 1788834365.69 total 50.03 last 74.59) - open/close 4057 (time 1788834375.70 total 60.04 last 68.69) - open/close 4723 (time 1788834385.73 total 70.07 last 66.38) - open/close 5396 (time 1788834395.74 total 80.08 last 67.23) - open/close 6151 (time 1788834405.75 total 90.09 last 75.44) - open/close 6929 (time 1788834415.75 total 100.09 last 77.78) - open/close 7678 (time 1788834425.76 total 110.10 last 74.85) - open/close 8440 (time 1788834435.77 total 120.11 last 76.13) - open/close 9158 (time 1788834445.78 total 130.12 last 71.72) - open/close 9885 (time 1788834455.78 total 140.12 last 72.70) total: 10000 open/close in 141.93 seconds: 70.46 ops/second - unlinked 0 (time 1788834461 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 697 (time 1788834556.19 total 10.01 last 69.61) - open/close 1402 (time 1788834566.21 total 20.04 last 70.32) - open/close 2144 (time 1788834576.22 total 30.05 last 74.15) - open/close 2786 (time 1788834586.26 total 40.08 last 63.98) - open/close 3425 (time 1788834596.26 total 50.09 last 63.86) - open/close 4141 (time 1788834606.27 total 60.09 last 71.58) - open/close 4813 (time 1788834616.27 total 70.09 last 67.18) - open/close 5454 (time 1788834626.28 total 80.11 last 64.03) - open/close 6152 (time 1788834636.28 total 90.11 last 69.78) - open/close 6838 (time 1788834646.30 total 100.12 last 68.50) - open/close 7551 (time 1788834656.30 total 110.13 last 71.26) - open/close 8087 (time 1788834666.32 total 120.14 last 53.52) - open/close 8694 (time 1788834676.33 total 130.15 last 60.64) - open/close 9401 (time 1788834686.33 total 140.16 last 70.67) total: 10000 open/close in 149.89 seconds: 66.71 ops/second - unlinked 0 (time 1788834700 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 640 (time 1788834791.14 total 10.01 last 63.93) - open/close 1307 (time 1788834801.14 total 20.01 last 66.69) - open/close 1939 (time 1788834811.15 total 30.02 last 63.16) - open/close 2721 (time 1788834821.15 total 40.02 last 78.15) - open/close 3408 (time 1788834831.16 total 50.03 last 68.65) - open/close 4092 (time 1788834841.17 total 60.04 last 68.35) - open/close 4719 (time 1788834851.18 total 70.05 last 62.66) - open/close 5400 (time 1788834861.18 total 80.05 last 68.05) - open/close 6058 (time 1788834871.20 total 90.07 last 65.67) - open/close 6679 (time 1788834881.21 total 100.08 last 62.03) - open/close 7406 (time 1788834891.22 total 110.09 last 72.67) - open/close 8088 (time 1788834901.25 total 120.12 last 67.97) - open/close 8742 (time 1788834911.26 total 130.13 last 65.35) - open/close 9486 (time 1788834921.26 total 140.13 last 74.37) total: 10000 open/close in 147.47 seconds: 67.81 ops/second - unlinked 0 (time 1788834932 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 712 (time 1788835019.50 total 10.01 last 71.15) - open/close 1425 (time 1788835029.50 total 20.01 last 71.29) - open/close 2160 (time 1788835039.50 total 30.01 last 73.47) - open/close 2913 (time 1788835049.51 total 40.02 last 75.24) - open/close 3687 (time 1788835059.51 total 50.02 last 77.40) - open/close 4448 (time 1788835069.52 total 60.03 last 76.01) - open/close 5137 (time 1788835079.52 total 70.03 last 68.90) - open/close 5888 (time 1788835089.53 total 80.04 last 75.08) - open/close 6656 (time 1788835099.53 total 90.04 last 76.77) - open/close 7384 (time 1788835109.54 total 100.05 last 72.74) - open/close 8078 (time 1788835119.55 total 110.06 last 69.29) - open/close 8815 (time 1788835129.56 total 120.07 last 73.69) - open/close 9574 (time 1788835139.57 total 130.08 last 75.77) total: 10000 open/close in 135.65 seconds: 73.72 ops/second - unlinked 0 (time 1788835149 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 738 (time 1788835239.50 total 10.01 last 73.76) - open/close 1498 (time 1788835249.51 total 20.01 last 75.93) - open/close 2282 (time 1788835259.51 total 30.02 last 78.38) - open/close 3007 (time 1788835269.53 total 40.04 last 72.36) - open/close 3671 (time 1788835279.54 total 50.04 last 66.37) - open/close 4316 (time 1788835289.54 total 60.05 last 64.46) - open/close 4866 (time 1788835299.55 total 70.05 last 54.99) - open/close 5508 (time 1788835309.56 total 80.06 last 64.12) - open/close 6297 (time 1788835319.58 total 90.08 last 78.77) - open/close 7008 (time 1788835329.59 total 100.09 last 71.00) - open/close 7796 (time 1788835339.60 total 110.10 last 78.74) - open/close 8585 (time 1788835349.61 total 120.11 last 78.82) - open/close 9291 (time 1788835359.61 total 130.11 last 70.57) - open/close 9987 (time 1788835369.62 total 140.13 last 69.51) total: 10000 open/close in 140.33 seconds: 71.26 ops/second - unlinked 0 (time 1788835374 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 760 (time 1788835469.54 total 10.01 last 75.95) - open/close 1542 (time 1788835479.55 total 20.01 last 78.16) - open/close 2271 (time 1788835489.55 total 30.02 last 72.88) - open/close 2931 (time 1788835499.57 total 40.03 last 65.90) - open/close 3641 (time 1788835509.57 total 50.03 last 70.98) - open/close 4375 (time 1788835519.58 total 60.05 last 73.30) - open/close 5096 (time 1788835529.59 total 70.05 last 72.05) - open/close 5854 (time 1788835539.59 total 80.06 last 75.77) - open/close 6622 (time 1788835549.62 total 90.08 last 76.62) - open/close 7293 (time 1788835559.62 total 100.08 last 67.08) - open/close 7914 (time 1788835569.63 total 110.09 last 62.03) - open/close 8653 (time 1788835579.64 total 120.10 last 73.86) - open/close 9380 (time 1788835589.64 total 130.11 last 72.66) total: 10000 open/close in 138.98 seconds: 71.95 ops/second - unlinked 0 (time 1788835602 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 730 (time 1788835699.93 total 10.00 last 73.00) - open/close 1421 (time 1788835709.93 total 20.00 last 69.10) - open/close 2170 (time 1788835719.94 total 30.01 last 74.83) - open/close 2910 (time 1788835729.95 total 40.03 last 73.89) - open/close 3638 (time 1788835739.96 total 50.03 last 72.78) - open/close 4382 (time 1788835749.96 total 60.03 last 74.37) - open/close 5151 (time 1788835759.96 total 70.03 last 76.88) - open/close 5886 (time 1788835769.97 total 80.04 last 73.43) - open/close 6640 (time 1788835779.99 total 90.06 last 75.31) - open/close 7334 (time 1788835789.99 total 100.06 last 69.36) - open/close 8075 (time 1788835800.00 total 110.07 last 74.02) - open/close 8805 (time 1788835810.01 total 120.08 last 72.93) - open/close 9499 (time 1788835820.01 total 130.09 last 69.38) total: 10000 open/close in 138.06 seconds: 72.43 ops/second - unlinked 0 (time 1788835834 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 844 (time 1788835926.97 total 10.01 last 84.32) - open/close 1523 (time 1788835936.97 total 20.01 last 67.88) - open/close 2352 (time 1788835946.99 total 30.03 last 82.79) - open/close 3233 (time 1788835957.00 total 40.04 last 88.00) - open/close 4079 (time 1788835967.00 total 50.04 last 84.54) - open/close 4967 (time 1788835977.01 total 60.05 last 88.78) - open/close 5815 (time 1788835987.01 total 70.05 last 84.74) - open/close 6683 (time 1788835997.02 total 80.06 last 86.77) - open/close 7531 (time 1788836007.02 total 90.06 last 84.80) - open/close 8398 (time 1788836017.02 total 100.06 last 86.70) - open/close 9269 (time 1788836027.02 total 110.06 last 87.09) total: 10000 open/close in 118.22 seconds: 84.59 ops/second - unlinked 0 (time 1788836038 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 625 (time 1788836129.94 total 10.03 last 62.32) - open/close 1347 (time 1788836139.95 total 20.03 last 72.18) - open/close 2169 (time 1788836149.95 total 30.04 last 82.14) - open/close 2962 (time 1788836159.96 total 40.05 last 79.23) - open/close 3825 (time 1788836169.96 total 50.05 last 86.28) - open/close 4731 (time 1788836179.96 total 60.05 last 90.60) - open/close 5450 (time 1788836189.98 total 70.06 last 71.82) - open/close 6206 (time 1788836199.98 total 80.07 last 75.54) - open/close 7167 (time 1788836209.99 total 90.07 last 96.09) - open/close 8044 (time 1788836219.99 total 100.07 last 87.69) - open/close 8844 (time 1788836229.99 total 110.07 last 79.98) - open/close 9756 (time 1788836239.99 total 120.08 last 91.18) total: 10000 open/close in 124.03 seconds: 80.63 ops/second - unlinked 0 (time 1788836248 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: kmemleak disabled modules unloaded. pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2