== conf-sanity test 69: replace an OST with the same index ========================================================== 23:47:34 (1788666454) start mds service on oleg226-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1922328 Sep 5 15:39 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing load_modules_local oleg226-server: kmemleak disabled oleg226-server: Loading modules from /home/green/git/lustre-release/lustre oleg226-server: detected 4 online CPUs by sysfs oleg226-server: MODOPTS_LIBCFS= oleg226-server: Force libcfs to create 2 CPU partitions oleg226-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg226-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg226-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg226-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg226-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 740 (time 1788666563.66 total 10.01 last 73.91) - open/close 1383 (time 1788666573.67 total 20.02 last 64.23) - open/close 2173 (time 1788666583.68 total 30.04 last 78.90) - open/close 3026 (time 1788666593.69 total 40.04 last 85.24) - open/close 3761 (time 1788666603.69 total 50.05 last 73.46) - open/close 4572 (time 1788666613.70 total 60.06 last 81.03) - open/close 5330 (time 1788666623.71 total 70.06 last 75.76) - open/close 6063 (time 1788666633.71 total 80.07 last 73.26) - open/close 6884 (time 1788666643.71 total 90.07 last 82.08) - open/close 7586 (time 1788666653.72 total 100.07 last 70.18) - open/close 8197 (time 1788666663.72 total 110.08 last 61.07) - open/close 8883 (time 1788666673.74 total 120.10 last 68.47) - open/close 9579 (time 1788666683.75 total 130.11 last 69.52) total: 10000 open/close in 135.67 seconds: 73.71 ops/second - unlinked 0 (time 1788666693 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 773 (time 1788666780.68 total 10.00 last 77.29) - open/close 1478 (time 1788666790.68 total 20.01 last 70.47) - open/close 2196 (time 1788666800.69 total 30.01 last 71.77) - open/close 2980 (time 1788666810.69 total 40.01 last 78.40) - open/close 3893 (time 1788666820.69 total 50.01 last 91.28) - open/close 4731 (time 1788666830.70 total 60.02 last 83.72) - open/close 5468 (time 1788666840.70 total 70.02 last 73.70) - open/close 6101 (time 1788666850.71 total 80.04 last 63.22) - open/close 6722 (time 1788666860.71 total 90.04 last 62.08) - open/close 7437 (time 1788666870.73 total 100.05 last 71.39) - open/close 8179 (time 1788666880.75 total 110.07 last 74.05) - open/close 8903 (time 1788666890.75 total 120.08 last 72.37) - open/close 9713 (time 1788666900.77 total 130.09 last 80.91) total: 10000 open/close in 134.22 seconds: 74.50 ops/second - unlinked 0 (time 1788666909 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 841 (time 1788666993.33 total 10.01 last 84.03) - open/close 1669 (time 1788667003.34 total 20.02 last 82.74) - open/close 2475 (time 1788667013.36 total 30.03 last 80.48) - open/close 3302 (time 1788667023.37 total 40.04 last 82.63) - open/close 4100 (time 1788667033.37 total 50.04 last 79.77) - open/close 5038 (time 1788667043.38 total 60.05 last 93.73) - open/close 6050 (time 1788667053.38 total 70.05 last 101.20) - open/close 6979 (time 1788667063.38 total 80.05 last 92.88) - open/close 7704 (time 1788667073.38 total 90.05 last 72.49) - open/close 8563 (time 1788667083.39 total 100.07 last 85.81) - open/close 9427 (time 1788667093.40 total 110.07 last 86.35) total: 10000 open/close in 115.74 seconds: 86.40 ops/second - unlinked 0 (time 1788667103 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 787 (time 1788667187.33 total 10.02 last 78.51) - open/close 1536 (time 1788667197.34 total 20.04 last 74.81) - open/close 2400 (time 1788667207.35 total 30.04 last 86.33) - open/close 3333 (time 1788667217.35 total 40.05 last 93.30) - open/close 4238 (time 1788667227.35 total 50.05 last 90.43) - open/close 5150 (time 1788667237.36 total 60.06 last 91.17) - open/close 6040 (time 1788667247.36 total 70.06 last 88.98) - open/close 6811 (time 1788667257.37 total 80.07 last 77.05) - open/close 7591 (time 1788667267.38 total 90.07 last 77.93) - open/close 8461 (time 1788667277.38 total 100.08 last 86.93) - open/close 9312 (time 1788667287.39 total 110.09 last 85.07) total: 10000 open/close in 119.67 seconds: 83.56 ops/second - unlinked 0 (time 1788667301 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 586 (time 1788667388.42 total 10.02 last 58.51) - open/close 1363 (time 1788667398.43 total 20.02 last 77.64) - open/close 2267 (time 1788667408.43 total 30.03 last 90.32) - open/close 3088 (time 1788667418.44 total 40.03 last 82.08) - open/close 3917 (time 1788667428.44 total 50.04 last 82.82) - open/close 4849 (time 1788667438.45 total 60.05 last 93.17) - open/close 5729 (time 1788667448.46 total 70.06 last 87.91) - open/close 6566 (time 1788667458.48 total 80.07 last 83.55) - open/close 7382 (time 1788667468.48 total 90.08 last 81.58) - open/close 8145 (time 1788667478.48 total 100.08 last 76.27) - open/close 8984 (time 1788667488.49 total 110.09 last 83.85) - open/close 9624 (time 1788667498.50 total 120.10 last 63.94) total: 10000 open/close in 125.43 seconds: 79.72 ops/second - unlinked 0 (time 1788667509 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 878 (time 1788667588.73 total 10.00 last 87.80) - open/close 1575 (time 1788667598.74 total 20.01 last 69.66) - open/close 2385 (time 1788667608.74 total 30.01 last 80.95) - open/close 3108 (time 1788667618.75 total 40.02 last 72.28) - open/close 3705 (time 1788667628.77 total 50.04 last 59.57) - open/close 4491 (time 1788667638.77 total 60.04 last 78.57) - open/close 5203 (time 1788667648.78 total 70.05 last 71.14) - open/close 5862 (time 1788667658.79 total 80.05 last 65.88) - open/close 6460 (time 1788667668.81 total 90.08 last 59.66) - open/close 7299 (time 1788667678.81 total 100.08 last 83.89) - open/close 8034 (time 1788667688.81 total 110.08 last 73.49) - open/close 8747 (time 1788667698.83 total 120.10 last 71.13) - open/close 9472 (time 1788667708.84 total 130.11 last 72.44) total: 10000 open/close in 136.11 seconds: 73.47 ops/second - unlinked 0 (time 1788667718 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 653 (time 1788667805.55 total 10.01 last 65.25) - open/close 1220 (time 1788667815.56 total 20.01 last 56.66) - open/close 1924 (time 1788667825.56 total 30.02 last 70.36) - open/close 2709 (time 1788667835.57 total 40.03 last 78.40) - open/close 3481 (time 1788667845.58 total 50.04 last 77.14) - open/close 4152 (time 1788667855.59 total 60.05 last 67.06) - open/close 4897 (time 1788667865.59 total 70.05 last 74.47) - open/close 5693 (time 1788667875.60 total 80.06 last 79.52) - open/close 6378 (time 1788667885.60 total 90.06 last 68.49) - open/close 7085 (time 1788667895.60 total 100.06 last 70.70) - open/close 7864 (time 1788667905.61 total 110.07 last 77.85) - open/close 8652 (time 1788667915.61 total 120.07 last 78.77) - open/close 9373 (time 1788667925.61 total 130.07 last 72.08) total: 10000 open/close in 137.86 seconds: 72.54 ops/second - unlinked 0 (time 1788667938 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 697 (time 1788668026.32 total 10.01 last 69.65) - open/close 1411 (time 1788668036.32 total 20.01 last 71.36) - open/close 2159 (time 1788668046.33 total 30.02 last 74.77) - open/close 2738 (time 1788668056.35 total 40.04 last 57.74) - open/close 3308 (time 1788668066.37 total 50.06 last 56.93) - open/close 3862 (time 1788668076.38 total 60.07 last 55.31) - open/close 4433 (time 1788668086.39 total 70.08 last 57.06) - open/close 5108 (time 1788668096.40 total 80.08 last 67.47) - open/close 5815 (time 1788668106.40 total 90.09 last 70.68) - open/close 6503 (time 1788668116.40 total 100.09 last 68.77) - open/close 7044 (time 1788668126.41 total 110.10 last 54.04) - open/close 7725 (time 1788668136.43 total 120.12 last 68.02) - open/close 8488 (time 1788668146.43 total 130.12 last 76.26) - open/close 9289 (time 1788668156.44 total 140.13 last 80.06) - open/close 9991 (time 1788668166.44 total 150.13 last 70.16) total: 10000 open/close in 150.20 seconds: 66.58 ops/second - unlinked 0 (time 1788668171 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 595 (time 1788668275.78 total 10.01 last 59.42) - open/close 1144 (time 1788668285.79 total 20.01 last 54.90) - open/close 1656 (time 1788668295.79 total 30.02 last 51.17) - open/close 2208 (time 1788668305.81 total 40.04 last 55.11) - open/close 2789 (time 1788668315.82 total 50.05 last 58.04) - open/close 3347 (time 1788668325.83 total 60.06 last 55.76) - open/close 4035 (time 1788668335.83 total 70.06 last 68.75) - open/close 4787 (time 1788668345.84 total 80.06 last 75.19) - open/close 5484 (time 1788668355.84 total 90.07 last 69.67) - open/close 6252 (time 1788668365.84 total 100.07 last 76.77) - open/close 6888 (time 1788668375.85 total 110.08 last 63.58) - open/close 7544 (time 1788668385.86 total 120.08 last 65.54) - open/close 8185 (time 1788668395.86 total 130.09 last 64.07) - open/close 8921 (time 1788668405.87 total 140.10 last 73.56) - open/close 9644 (time 1788668415.88 total 150.11 last 72.22) total: 10000 open/close in 154.29 seconds: 64.81 ops/second - unlinked 0 (time 1788668425 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 783 (time 1788668510.42 total 10.01 last 78.24) - open/close 1536 (time 1788668520.43 total 20.01 last 75.26) - open/close 2353 (time 1788668530.44 total 30.03 last 81.60) - open/close 3118 (time 1788668540.44 total 40.03 last 76.47) - open/close 3834 (time 1788668550.45 total 50.04 last 71.54) - open/close 4652 (time 1788668560.46 total 60.04 last 81.76) - open/close 5513 (time 1788668570.46 total 70.05 last 86.08) - open/close 6252 (time 1788668580.46 total 80.05 last 73.90) - open/close 6997 (time 1788668590.47 total 90.05 last 74.44) - open/close 7770 (time 1788668600.47 total 100.05 last 77.29) - open/close 8687 (time 1788668610.47 total 110.06 last 91.69) - open/close 9653 (time 1788668620.47 total 120.06 last 96.55) total: 10000 open/close in 124.80 seconds: 80.13 ops/second - unlinked 0 (time 1788668629 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg226-server unloading modules via unload_modules_local on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local oleg226-server: kmemleak disabled modules unloaded. pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2