== conf-sanity test 69: replace an OST with the same index ========================================================== 07:10:08 (1789297808) start mds service on oleg119-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 12 19:13 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: kmemleak disabled oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions oleg119-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg119-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg119-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg119-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg119-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 835 (time 1789297921.05 total 10.00 last 83.49) - open/close 1638 (time 1789297931.06 total 20.01 last 80.20) - open/close 2530 (time 1789297941.06 total 30.01 last 89.19) - open/close 3364 (time 1789297951.06 total 40.02 last 83.38) - open/close 4121 (time 1789297961.07 total 50.02 last 75.64) - open/close 4880 (time 1789297971.10 total 60.05 last 75.67) - open/close 5718 (time 1789297981.11 total 70.06 last 83.75) - open/close 6546 (time 1789297991.11 total 80.06 last 82.78) - open/close 7381 (time 1789298001.11 total 90.07 last 83.46) - open/close 8103 (time 1789298011.11 total 100.07 last 72.18) - open/close 8994 (time 1789298021.13 total 110.08 last 89.00) - open/close 9634 (time 1789298031.13 total 120.08 last 63.99) total: 10000 open/close in 124.25 seconds: 80.49 ops/second - unlinked 0 (time 1789298040 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 791 (time 1789298129.50 total 10.01 last 79.04) - open/close 1530 (time 1789298139.51 total 20.02 last 73.81) - open/close 2306 (time 1789298149.52 total 30.03 last 77.53) - open/close 3098 (time 1789298159.53 total 40.04 last 79.14) - open/close 3959 (time 1789298169.53 total 50.04 last 86.05) - open/close 4852 (time 1789298179.55 total 60.06 last 89.18) - open/close 5668 (time 1789298189.55 total 70.06 last 81.60) - open/close 6397 (time 1789298199.55 total 80.06 last 72.88) - open/close 7210 (time 1789298209.55 total 90.06 last 81.29) - open/close 8018 (time 1789298219.57 total 100.07 last 80.69) - open/close 8855 (time 1789298229.57 total 110.08 last 83.64) - open/close 9712 (time 1789298239.58 total 120.09 last 85.65) total: 10000 open/close in 124.05 seconds: 80.61 ops/second - unlinked 0 (time 1789298247 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 741 (time 1789298333.73 total 10.01 last 74.06) - open/close 1517 (time 1789298343.74 total 20.01 last 77.57) - open/close 2259 (time 1789298353.75 total 30.03 last 74.09) - open/close 2969 (time 1789298363.75 total 40.03 last 70.98) - open/close 3805 (time 1789298373.76 total 50.03 last 83.55) - open/close 4656 (time 1789298383.76 total 60.04 last 85.09) - open/close 5482 (time 1789298393.77 total 70.04 last 82.54) - open/close 6367 (time 1789298403.78 total 80.05 last 88.44) - open/close 7159 (time 1789298413.83 total 90.10 last 78.77) - open/close 7898 (time 1789298423.83 total 100.11 last 73.89) - open/close 8625 (time 1789298433.85 total 110.13 last 72.54) - open/close 9400 (time 1789298443.87 total 120.14 last 77.39) total: 10000 open/close in 127.53 seconds: 78.41 ops/second - unlinked 0 (time 1789298455 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 810 (time 1789298549.65 total 10.01 last 80.90) - open/close 1491 (time 1789298559.66 total 20.02 last 68.05) - open/close 2209 (time 1789298569.67 total 30.03 last 71.74) - open/close 2955 (time 1789298579.67 total 40.03 last 74.57) - open/close 3819 (time 1789298589.67 total 50.04 last 86.36) - open/close 4604 (time 1789298599.69 total 60.05 last 78.36) - open/close 5404 (time 1789298609.70 total 70.06 last 79.95) - open/close 6183 (time 1789298619.70 total 80.07 last 77.85) - open/close 6879 (time 1789298629.71 total 90.07 last 69.57) - open/close 7779 (time 1789298639.71 total 100.07 last 90.00) - open/close 8546 (time 1789298649.72 total 110.08 last 76.62) - open/close 9237 (time 1789298659.75 total 120.11 last 68.92) - open/close 9823 (time 1789298669.75 total 130.12 last 58.55) total: 10000 open/close in 133.72 seconds: 74.78 ops/second - unlinked 0 (time 1789298678 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 856 (time 1789298761.88 total 10.01 last 85.52) - open/close 1725 (time 1789298771.89 total 20.02 last 86.81) - open/close 2698 (time 1789298781.89 total 30.02 last 97.25) - open/close 3729 (time 1789298791.90 total 40.03 last 103.06) - open/close 4678 (time 1789298801.90 total 50.03 last 94.90) - open/close 5502 (time 1789298811.95 total 60.08 last 81.99) - open/close 6081 (time 1789298821.95 total 70.08 last 57.90) - open/close 6809 (time 1789298831.96 total 80.09 last 72.74) - open/close 7706 (time 1789298841.96 total 90.10 last 89.61) - open/close 8569 (time 1789298851.97 total 100.10 last 86.26) - open/close 9506 (time 1789298861.97 total 110.10 last 93.70) total: 10000 open/close in 114.95 seconds: 87.00 ops/second - unlinked 0 (time 1789298871 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 778 (time 1789298965.53 total 10.00 last 77.78) - open/close 1191 (time 1789298975.56 total 20.03 last 41.17) - open/close 1925 (time 1789298985.57 total 30.04 last 73.35) - open/close 2984 (time 1789298995.57 total 40.05 last 105.86) - open/close 3834 (time 1789299005.58 total 50.05 last 84.94) - open/close 4706 (time 1789299015.58 total 60.05 last 87.19) - open/close 5490 (time 1789299025.60 total 70.07 last 78.26) - open/close 6391 (time 1789299035.61 total 80.08 last 90.03) - open/close 7367 (time 1789299045.61 total 90.08 last 97.55) - open/close 8171 (time 1789299055.61 total 100.08 last 80.40) - open/close 8925 (time 1789299065.61 total 110.09 last 75.39) - open/close 9781 (time 1789299075.62 total 120.09 last 85.55) total: 10000 open/close in 122.68 seconds: 81.51 ops/second - unlinked 0 (time 1789299083 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 720 (time 1789299170.36 total 10.00 last 71.99) - open/close 1483 (time 1789299180.37 total 20.01 last 76.25) - open/close 2288 (time 1789299190.37 total 30.01 last 80.47) - open/close 3122 (time 1789299200.38 total 40.02 last 83.31) - open/close 3709 (time 1789299210.39 total 50.03 last 58.65) - open/close 4325 (time 1789299220.40 total 60.04 last 61.57) - open/close 5023 (time 1789299230.41 total 70.05 last 69.72) - open/close 5705 (time 1789299240.41 total 80.05 last 68.15) - open/close 6482 (time 1789299250.43 total 90.07 last 77.60) - open/close 7180 (time 1789299260.44 total 100.08 last 69.69) - open/close 8033 (time 1789299270.45 total 110.09 last 85.21) - open/close 8874 (time 1789299280.46 total 120.10 last 84.04) - open/close 9544 (time 1789299290.47 total 130.11 last 66.94) total: 10000 open/close in 135.72 seconds: 73.68 ops/second - unlinked 0 (time 1789299301 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 792 (time 1789299395.54 total 10.03 last 78.97) - open/close 1617 (time 1789299405.54 total 20.04 last 82.44) - open/close 2361 (time 1789299415.54 total 30.04 last 74.40) - open/close 3321 (time 1789299425.56 total 40.05 last 95.89) - open/close 4181 (time 1789299435.57 total 50.06 last 85.91) - open/close 5051 (time 1789299445.57 total 60.07 last 86.95) - open/close 5978 (time 1789299455.58 total 70.08 last 92.58) - open/close 6820 (time 1789299465.59 total 80.09 last 84.14) - open/close 7610 (time 1789299475.60 total 90.09 last 78.96) - open/close 8359 (time 1789299485.61 total 100.10 last 74.80) - open/close 9198 (time 1789299495.62 total 110.12 last 83.78) total: 10000 open/close in 118.64 seconds: 84.29 ops/second - unlinked 0 (time 1789299508 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 778 (time 1789299591.37 total 10.00 last 77.79) - open/close 1596 (time 1789299601.40 total 20.03 last 81.55) - open/close 2306 (time 1789299611.40 total 30.04 last 70.96) - open/close 3133 (time 1789299621.41 total 40.04 last 82.64) - open/close 3897 (time 1789299631.42 total 50.05 last 76.34) - open/close 4708 (time 1789299641.43 total 60.06 last 80.99) - open/close 5474 (time 1789299651.43 total 70.07 last 76.59) - open/close 6399 (time 1789299661.44 total 80.08 last 92.39) - open/close 7392 (time 1789299671.45 total 90.08 last 99.25) - open/close 8160 (time 1789299681.46 total 100.09 last 76.74) - open/close 9059 (time 1789299691.46 total 110.09 last 89.89) total: 10000 open/close in 119.67 seconds: 83.56 ops/second - unlinked 0 (time 1789299705 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 707 (time 1789299785.95 total 10.01 last 70.62) - open/close 1334 (time 1789299795.95 total 20.02 last 62.66) - open/close 2066 (time 1789299805.97 total 30.04 last 73.04) - open/close 2726 (time 1789299815.98 total 40.04 last 65.97) - open/close 3444 (time 1789299825.99 total 50.05 last 71.74) - open/close 4211 (time 1789299836.00 total 60.06 last 76.62) - open/close 4926 (time 1789299846.02 total 70.08 last 71.36) - open/close 5633 (time 1789299856.02 total 80.09 last 70.68) - open/close 6422 (time 1789299866.02 total 90.09 last 78.90) - open/close 7053 (time 1789299876.04 total 100.11 last 62.97) - open/close 7842 (time 1789299886.05 total 110.11 last 78.83) - open/close 8655 (time 1789299896.06 total 120.13 last 81.21) - open/close 9481 (time 1789299906.06 total 130.13 last 82.58) total: 10000 open/close in 136.36 seconds: 73.34 ops/second - unlinked 0 (time 1789299917 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local oleg119-server: kmemleak disabled modules unloaded. pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2