== conf-sanity test 69: replace an OST with the same index ========================================================== 21:56:04 (1788918964) start mds service on oleg102-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 8 20:20 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing load_modules_local oleg102-server: kmemleak disabled oleg102-server: Loading modules from /home/green/git/lustre-release/lustre oleg102-server: detected 4 online CPUs by sysfs oleg102-server: MODOPTS_LIBCFS= oleg102-server: Force libcfs to create 2 CPU partitions oleg102-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg102-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg102-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg102-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg102-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49665 inodes available. Want 99872. rc=0 - open/close 963 (time 1788919058.22 total 10.01 last 96.17) - open/close 1886 (time 1788919068.22 total 20.01 last 92.30) - open/close 2790 (time 1788919078.22 total 30.01 last 90.40) - open/close 3671 (time 1788919088.23 total 40.02 last 88.03) - open/close 4542 (time 1788919098.23 total 50.03 last 87.05) - open/close 5445 (time 1788919108.24 total 60.04 last 90.24) - open/close 6454 (time 1788919118.25 total 70.05 last 100.80) - open/close 7261 (time 1788919128.26 total 80.05 last 80.64) - open/close 8144 (time 1788919138.26 total 90.05 last 88.29) - open/close 9039 (time 1788919148.26 total 100.05 last 89.49) - open/close 9982 (time 1788919158.26 total 110.06 last 94.27) total: 10000 open/close in 110.22 seconds: 90.72 ops/second - unlinked 0 (time 1788919162 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 958 (time 1788919234.76 total 10.01 last 95.68) - open/close 1935 (time 1788919244.77 total 20.02 last 97.63) - open/close 2933 (time 1788919254.77 total 30.03 last 99.74) - open/close 3824 (time 1788919264.79 total 40.04 last 88.99) - open/close 4887 (time 1788919274.79 total 50.04 last 106.29) - open/close 5922 (time 1788919284.79 total 60.04 last 103.46) - open/close 6882 (time 1788919294.80 total 70.05 last 95.94) - open/close 7926 (time 1788919304.80 total 80.05 last 104.38) - open/close 8913 (time 1788919314.80 total 90.05 last 98.69) - open/close 9968 (time 1788919324.81 total 100.07 last 105.34) total: 10000 open/close in 100.40 seconds: 99.61 ops/second - unlinked 0 (time 1788919328 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1270 (time 1788919386.19 total 10.00 last 126.95) - open/close 2356 (time 1788919396.20 total 20.01 last 108.54) - open/close 3559 (time 1788919406.21 total 30.02 last 120.19) - open/close 4572 (time 1788919416.21 total 40.02 last 101.25) - open/close 5665 (time 1788919426.22 total 50.03 last 109.22) - open/close 6788 (time 1788919436.22 total 60.04 last 112.24) - open/close 7892 (time 1788919446.23 total 70.04 last 110.33) - open/close 9036 (time 1788919456.24 total 80.05 last 114.33) total: 10000 open/close in 89.07 seconds: 112.27 ops/second - unlinked 0 (time 1788919468 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1233 (time 1788919524.83 total 10.00 last 123.25) - open/close 2145 (time 1788919534.83 total 20.01 last 91.17) - open/close 3310 (time 1788919544.84 total 30.01 last 116.47) - open/close 4636 (time 1788919554.84 total 40.01 last 132.55) - open/close 5912 (time 1788919564.84 total 50.02 last 127.58) - open/close 7109 (time 1788919574.86 total 60.03 last 119.51) - open/close 8316 (time 1788919584.87 total 70.04 last 120.61) - open/close 9554 (time 1788919594.87 total 80.04 last 123.75) total: 10000 open/close in 83.38 seconds: 119.93 ops/second - unlinked 0 (time 1788919600 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1228 (time 1788919651.49 total 10.00 last 122.77) - open/close 2377 (time 1788919661.49 total 20.01 last 114.87) - open/close 3648 (time 1788919671.50 total 30.01 last 127.04) - open/close 5049 (time 1788919681.50 total 40.01 last 140.08) - open/close 6197 (time 1788919691.51 total 50.02 last 114.71) - open/close 7629 (time 1788919701.51 total 60.02 last 143.17) - open/close 9095 (time 1788919711.51 total 70.03 last 146.54) total: 10000 open/close in 77.81 seconds: 128.52 ops/second - unlinked 0 (time 1788919722 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1176 (time 1788919773.59 total 10.00 last 117.55) - open/close 2474 (time 1788919783.59 total 20.01 last 129.77) - open/close 3685 (time 1788919793.60 total 30.01 last 121.01) - open/close 5121 (time 1788919803.60 total 40.02 last 143.55) - open/close 6389 (time 1788919813.61 total 50.02 last 126.72) - open/close 7656 (time 1788919823.62 total 60.03 last 126.64) - open/close 9175 (time 1788919833.63 total 70.04 last 151.74) total: 10000 open/close in 75.90 seconds: 131.75 ops/second - unlinked 0 (time 1788919841 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1355 (time 1788919891.39 total 10.01 last 135.43) - open/close 2558 (time 1788919901.40 total 20.02 last 120.18) - open/close 3887 (time 1788919911.41 total 30.02 last 132.89) - open/close 5231 (time 1788919921.41 total 40.02 last 134.32) - open/close 6463 (time 1788919931.41 total 50.03 last 123.17) - open/close 7827 (time 1788919941.42 total 60.03 last 136.30) - open/close 8651 (time 1788919951.44 total 70.05 last 82.27) - open/close 9629 (time 1788919961.44 total 80.05 last 97.78) total: 10000 open/close in 85.83 seconds: 116.51 ops/second - unlinked 0 (time 1788919973 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 909 (time 1788920057.80 total 10.00 last 90.87) - open/close 1853 (time 1788920067.84 total 20.04 last 94.08) - open/close 2752 (time 1788920077.85 total 30.05 last 89.77) - open/close 3667 (time 1788920087.86 total 40.05 last 91.47) - open/close 4537 (time 1788920097.86 total 50.05 last 87.00) - open/close 5438 (time 1788920107.88 total 60.08 last 89.89) - open/close 6361 (time 1788920117.88 total 70.08 last 92.30) - open/close 7185 (time 1788920127.88 total 80.08 last 82.38) - open/close 8146 (time 1788920137.89 total 90.09 last 96.06) - open/close 9095 (time 1788920147.89 total 100.09 last 94.86) total: 10000 open/close in 109.73 seconds: 91.14 ops/second - unlinked 0 (time 1788920162 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1001 (time 1788920241.86 total 10.00 last 100.07) - open/close 1946 (time 1788920251.87 total 20.01 last 94.43) - open/close 2785 (time 1788920261.88 total 30.02 last 83.83) - open/close 3469 (time 1788920271.88 total 40.02 last 68.38) - open/close 4112 (time 1788920281.88 total 50.02 last 64.30) - open/close 4775 (time 1788920291.88 total 60.03 last 66.27) - open/close 5494 (time 1788920301.89 total 70.03 last 71.88) - open/close 6126 (time 1788920311.89 total 80.03 last 63.19) - open/close 6892 (time 1788920321.89 total 90.03 last 76.59) - open/close 7434 (time 1788920331.90 total 100.04 last 54.15) - open/close 8171 (time 1788920341.90 total 110.05 last 73.67) - open/close 8811 (time 1788920351.91 total 120.06 last 63.93) - open/close 9408 (time 1788920361.92 total 130.06 last 59.67) total: 10000 open/close in 137.82 seconds: 72.56 ops/second - unlinked 0 (time 1788920374 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 783 (time 1788920456.23 total 10.01 last 78.26) - open/close 1506 (time 1788920466.26 total 20.03 last 72.09) - open/close 2380 (time 1788920476.26 total 30.03 last 87.40) - open/close 3236 (time 1788920486.28 total 40.05 last 85.48) - open/close 4068 (time 1788920496.29 total 50.06 last 83.08) - open/close 4873 (time 1788920506.30 total 60.07 last 80.46) - open/close 5590 (time 1788920516.31 total 70.09 last 71.57) - open/close 6321 (time 1788920526.32 total 80.09 last 73.04) - open/close 7075 (time 1788920536.33 total 90.10 last 75.35) - open/close 7740 (time 1788920546.34 total 100.12 last 66.39) - open/close 8483 (time 1788920556.36 total 110.13 last 74.19) - open/close 9186 (time 1788920566.36 total 120.13 last 70.29) total: 10000 open/close in 129.44 seconds: 77.26 ops/second - unlinked 0 (time 1788920580 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.102@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg102-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg102-server: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg102-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre On OST0, 20236 used inodes rc=0 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.102@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.102@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.102@tcp sys.timeout=20 start ost1 service on oleg102-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg102-server: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg102-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg102-server unloading modules via unload_modules_local on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing unload_modules_local oleg102-server: kmemleak disabled modules unloaded. pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2