== conf-sanity test 69: replace an OST with the same index ========================================================== 18:27:02 (1790980022) start mds service on oleg151-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1946968 Oct 2 10:36 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: kmemleak disabled oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: MODOPTS_LIBCFS= oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg151-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg151-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg151-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg151-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 820 (time 1790980133.93 total 10.00 last 81.97) - open/close 1770 (time 1790980143.93 total 20.01 last 94.96) - open/close 2594 (time 1790980153.95 total 30.02 last 82.26) - open/close 3399 (time 1790980163.96 total 40.03 last 80.44) - open/close 4382 (time 1790980173.96 total 50.03 last 98.29) - open/close 5270 (time 1790980183.97 total 60.04 last 88.73) - open/close 6123 (time 1790980193.97 total 70.05 last 85.25) - open/close 7073 (time 1790980203.98 total 80.06 last 94.91) - open/close 8142 (time 1790980213.98 total 90.06 last 106.89) - open/close 9130 (time 1790980223.98 total 100.06 last 98.80) total: 10000 open/close in 108.84 seconds: 91.88 ops/second - unlinked 0 (time 1790980236 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1036 (time 1790980315.29 total 10.00 last 103.57) - open/close 1943 (time 1790980325.29 total 20.01 last 90.67) - open/close 2741 (time 1790980335.30 total 30.02 last 79.71) - open/close 3635 (time 1790980345.31 total 40.02 last 89.37) - open/close 4454 (time 1790980355.31 total 50.02 last 81.87) - open/close 5398 (time 1790980365.31 total 60.03 last 94.39) - open/close 6250 (time 1790980375.32 total 70.03 last 85.12) - open/close 7149 (time 1790980385.32 total 80.04 last 89.90) - open/close 8038 (time 1790980395.34 total 90.06 last 88.70) - open/close 8699 (time 1790980405.35 total 100.06 last 66.05) - open/close 9563 (time 1790980415.35 total 110.07 last 86.39) total: 10000 open/close in 114.55 seconds: 87.30 ops/second - unlinked 0 (time 1790980423 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1071 (time 1790980503.29 total 10.00 last 107.06) - open/close 2182 (time 1790980513.29 total 20.01 last 111.07) - open/close 3123 (time 1790980523.30 total 30.01 last 94.07) - open/close 3943 (time 1790980533.30 total 40.02 last 81.95) - open/close 4953 (time 1790980543.30 total 50.02 last 100.98) - open/close 5854 (time 1790980553.32 total 60.03 last 89.99) - open/close 6689 (time 1790980563.32 total 70.04 last 83.44) - open/close 7563 (time 1790980573.33 total 80.05 last 87.32) - open/close 8464 (time 1790980583.36 total 90.07 last 89.91) - open/close 9263 (time 1790980593.36 total 100.08 last 79.83) total: 10000 open/close in 109.97 seconds: 90.94 ops/second - unlinked 0 (time 1790980607 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1022 (time 1790980690.35 total 10.00 last 102.19) - open/close 1904 (time 1790980700.36 total 20.01 last 88.09) - open/close 2830 (time 1790980710.37 total 30.02 last 92.54) - open/close 3916 (time 1790980720.37 total 40.02 last 108.60) - open/close 4980 (time 1790980730.38 total 50.03 last 106.30) - open/close 5903 (time 1790980740.39 total 60.04 last 92.23) - open/close 6476 (time 1790980750.39 total 70.04 last 57.30) - open/close 7162 (time 1790980760.39 total 80.04 last 68.58) - open/close 7826 (time 1790980770.39 total 90.04 last 66.40) - open/close 8552 (time 1790980780.40 total 100.05 last 72.54) - open/close 9230 (time 1790980790.41 total 110.06 last 67.74) - open/close 9815 (time 1790980800.41 total 120.06 last 58.50) total: 10000 open/close in 122.03 seconds: 81.95 ops/second - unlinked 0 (time 1790980808 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 933 (time 1790980886.16 total 10.00 last 93.27) - open/close 1799 (time 1790980896.18 total 20.03 last 86.38) - open/close 2395 (time 1790980906.19 total 30.04 last 59.56) - open/close 3003 (time 1790980916.19 total 40.04 last 60.80) - open/close 3751 (time 1790980926.20 total 50.05 last 74.72) - open/close 4717 (time 1790980936.20 total 60.05 last 96.59) - open/close 5808 (time 1790980946.20 total 70.05 last 109.09) - open/close 6607 (time 1790980956.21 total 80.06 last 79.84) - open/close 7388 (time 1790980966.22 total 90.07 last 78.01) - open/close 8267 (time 1790980976.23 total 100.08 last 87.82) - open/close 9321 (time 1790980986.23 total 110.08 last 105.36) total: 10000 open/close in 116.82 seconds: 85.60 ops/second - unlinked 0 (time 1790980997 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 916 (time 1790981077.87 total 10.01 last 91.54) - open/close 1572 (time 1790981087.88 total 20.01 last 65.55) - open/close 2201 (time 1790981097.88 total 30.02 last 62.89) - open/close 2912 (time 1790981107.89 total 40.03 last 71.02) - open/close 3506 (time 1790981117.90 total 50.03 last 59.37) - open/close 4352 (time 1790981127.91 total 60.05 last 84.49) - open/close 5039 (time 1790981137.92 total 70.06 last 68.61) - open/close 5637 (time 1790981147.93 total 80.06 last 59.79) - open/close 6428 (time 1790981157.93 total 90.06 last 79.08) - open/close 7090 (time 1790981167.94 total 100.08 last 66.10) - open/close 7642 (time 1790981177.95 total 110.08 last 55.16) - open/close 8327 (time 1790981187.96 total 120.10 last 68.40) - open/close 8995 (time 1790981197.96 total 130.10 last 66.80) - open/close 9725 (time 1790981207.98 total 140.11 last 72.90) total: 10000 open/close in 145.03 seconds: 68.95 ops/second - unlinked 0 (time 1790981218 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 884 (time 1790981299.94 total 10.00 last 88.37) - open/close 1750 (time 1790981309.94 total 20.01 last 86.57) - open/close 2566 (time 1790981319.95 total 30.02 last 81.54) - open/close 3534 (time 1790981329.95 total 40.02 last 96.79) - open/close 4376 (time 1790981339.97 total 50.03 last 84.09) - open/close 5339 (time 1790981349.97 total 60.04 last 96.23) - open/close 6162 (time 1790981359.98 total 70.04 last 82.28) - open/close 7095 (time 1790981369.99 total 80.05 last 93.16) - open/close 7994 (time 1790981379.99 total 90.06 last 89.88) - open/close 8936 (time 1790981390.00 total 100.06 last 94.17) - open/close 9791 (time 1790981400.00 total 110.06 last 85.47) total: 10000 open/close in 111.96 seconds: 89.32 ops/second - unlinked 0 (time 1790981405 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 772 (time 1790981485.60 total 10.01 last 77.16) - open/close 1280 (time 1790981495.62 total 20.03 last 50.68) - open/close 2016 (time 1790981505.64 total 30.04 last 73.49) - open/close 2844 (time 1790981515.65 total 40.05 last 82.73) - open/close 3617 (time 1790981525.66 total 50.06 last 77.21) - open/close 4356 (time 1790981535.66 total 60.07 last 73.85) - open/close 5243 (time 1790981545.67 total 70.07 last 88.67) - open/close 6140 (time 1790981555.67 total 80.07 last 89.68) - open/close 6951 (time 1790981565.69 total 90.10 last 80.93) - open/close 7919 (time 1790981575.70 total 100.11 last 96.70) - open/close 8930 (time 1790981585.71 total 110.12 last 100.99) - open/close 9921 (time 1790981595.72 total 120.12 last 99.03) total: 10000 open/close in 121.15 seconds: 82.54 ops/second - unlinked 0 (time 1790981601 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 792 (time 1790981677.23 total 10.00 last 79.19) - open/close 1540 (time 1790981687.24 total 20.01 last 74.76) - open/close 2291 (time 1790981697.24 total 30.01 last 75.07) - open/close 3110 (time 1790981707.25 total 40.02 last 81.85) - open/close 4017 (time 1790981717.25 total 50.02 last 90.66) - open/close 4987 (time 1790981727.26 total 60.03 last 96.95) - open/close 5930 (time 1790981737.27 total 70.04 last 94.21) - open/close 6925 (time 1790981747.27 total 80.04 last 99.46) - open/close 7788 (time 1790981757.28 total 90.05 last 86.25) - open/close 8535 (time 1790981767.29 total 100.06 last 74.62) - open/close 9301 (time 1790981777.30 total 110.06 last 76.55) total: 10000 open/close in 118.58 seconds: 84.33 ops/second - unlinked 0 (time 1790981790 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 630 (time 1790981875.91 total 10.01 last 62.94) - open/close 1367 (time 1790981885.92 total 20.01 last 73.68) - open/close 2240 (time 1790981895.92 total 30.01 last 87.30) - open/close 2900 (time 1790981905.92 total 40.01 last 66.00) - open/close 3649 (time 1790981915.93 total 50.02 last 74.84) - open/close 4462 (time 1790981925.94 total 60.04 last 81.18) - open/close 5141 (time 1790981935.95 total 70.05 last 67.83) - open/close 5839 (time 1790981945.96 total 80.05 last 69.75) - open/close 6632 (time 1790981955.97 total 90.06 last 79.24) - open/close 7383 (time 1790981965.97 total 100.06 last 75.08) - open/close 8243 (time 1790981975.97 total 110.06 last 85.99) - open/close 9090 (time 1790981985.97 total 120.07 last 84.66) total: 10000 open/close in 129.81 seconds: 77.04 ops/second - unlinked 0 (time 1790982000 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg151-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.151@tcp sys.timeout=20 start ost1 service on oleg151-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server unloading modules via unload_modules_local on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local oleg151-server: kmemleak disabled modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2