== conf-sanity test 69: replace an OST with the same index ========================================================== 23:55:16 (1788753316) start mds service on oleg360-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 6 20:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: kmemleak disabled oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions oleg360-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg360-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg360-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg360-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg360-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg360-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 952 (time 1788753413.66 total 10.00 last 95.18) - open/close 1952 (time 1788753423.67 total 20.01 last 99.89) - open/close 2944 (time 1788753433.68 total 30.02 last 99.18) - open/close 3838 (time 1788753443.69 total 40.03 last 89.25) - open/close 4768 (time 1788753453.70 total 50.04 last 92.97) - open/close 5680 (time 1788753463.70 total 60.04 last 91.17) - open/close 6593 (time 1788753473.70 total 70.04 last 91.29) - open/close 7536 (time 1788753483.71 total 80.05 last 94.22) - open/close 8495 (time 1788753493.72 total 90.06 last 95.78) - open/close 9370 (time 1788753503.73 total 100.07 last 87.45) total: 10000 open/close in 107.08 seconds: 93.39 ops/second - unlinked 0 (time 1788753514 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 843 (time 1788753589.22 total 10.00 last 84.28) - open/close 1721 (time 1788753599.23 total 20.01 last 87.74) - open/close 2760 (time 1788753609.23 total 30.02 last 103.82) - open/close 3731 (time 1788753619.24 total 40.03 last 96.99) - open/close 4762 (time 1788753629.25 total 50.03 last 103.06) - open/close 5711 (time 1788753639.26 total 60.04 last 94.82) - open/close 6655 (time 1788753649.27 total 70.06 last 94.26) - open/close 7614 (time 1788753659.27 total 80.06 last 95.88) - open/close 8525 (time 1788753669.28 total 90.06 last 91.06) - open/close 9485 (time 1788753679.28 total 100.06 last 95.99) total: 10000 open/close in 105.96 seconds: 94.38 ops/second - unlinked 0 (time 1788753688 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 932 (time 1788753760.65 total 10.01 last 93.13) - open/close 1888 (time 1788753770.67 total 20.02 last 95.46) - open/close 2875 (time 1788753780.68 total 30.03 last 98.59) - open/close 3808 (time 1788753790.68 total 40.04 last 93.26) - open/close 4739 (time 1788753800.70 total 50.05 last 92.94) - open/close 5665 (time 1788753810.70 total 60.06 last 92.57) - open/close 6588 (time 1788753820.71 total 70.06 last 92.25) - open/close 7484 (time 1788753830.72 total 80.07 last 89.52) - open/close 8352 (time 1788753840.72 total 90.07 last 86.78) - open/close 9302 (time 1788753850.73 total 100.08 last 94.92) total: 10000 open/close in 108.11 seconds: 92.50 ops/second - unlinked 0 (time 1788753862 ; total 1 ; last 1) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 969 (time 1788753936.50 total 10.01 last 96.79) - open/close 1892 (time 1788753946.51 total 20.01 last 92.29) - open/close 2877 (time 1788753956.51 total 30.02 last 98.47) - open/close 3832 (time 1788753966.53 total 40.03 last 95.32) - open/close 4674 (time 1788753976.53 total 50.04 last 84.18) - open/close 5659 (time 1788753986.54 total 60.05 last 98.38) - open/close 6639 (time 1788753996.55 total 70.05 last 97.96) - open/close 7647 (time 1788754006.55 total 80.06 last 100.74) - open/close 8583 (time 1788754016.57 total 90.07 last 93.45) - open/close 9475 (time 1788754026.57 total 100.08 last 89.19) total: 10000 open/close in 105.55 seconds: 94.75 ops/second - unlinked 0 (time 1788754035 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 999 (time 1788754103.57 total 10.00 last 99.86) - open/close 1997 (time 1788754113.57 total 20.01 last 99.77) - open/close 2894 (time 1788754123.58 total 30.02 last 89.61) - open/close 3768 (time 1788754133.58 total 40.02 last 87.36) - open/close 4745 (time 1788754143.59 total 50.03 last 97.64) - open/close 5673 (time 1788754153.59 total 60.03 last 92.79) - open/close 6590 (time 1788754163.61 total 70.05 last 91.51) - open/close 7500 (time 1788754173.62 total 80.06 last 90.89) - open/close 8409 (time 1788754183.63 total 90.07 last 90.87) - open/close 9351 (time 1788754193.63 total 100.07 last 94.20) total: 10000 open/close in 107.01 seconds: 93.45 ops/second - unlinked 0 (time 1788754204 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 914 (time 1788754277.68 total 10.01 last 91.27) - open/close 1861 (time 1788754287.69 total 20.02 last 94.60) - open/close 2704 (time 1788754297.70 total 30.03 last 84.22) - open/close 3547 (time 1788754307.71 total 40.04 last 84.22) - open/close 4496 (time 1788754317.72 total 50.05 last 94.85) - open/close 5452 (time 1788754327.73 total 60.06 last 95.47) - open/close 6442 (time 1788754337.74 total 70.07 last 98.90) - open/close 7355 (time 1788754347.74 total 80.07 last 91.29) - open/close 8256 (time 1788754357.75 total 90.08 last 90.02) - open/close 9042 (time 1788754367.76 total 100.09 last 78.56) - open/close 9820 (time 1788754377.76 total 110.09 last 77.78) total: 10000 open/close in 112.20 seconds: 89.13 ops/second - unlinked 0 (time 1788754384 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 919 (time 1788754462.46 total 10.01 last 91.80) - open/close 1898 (time 1788754472.47 total 20.02 last 97.83) - open/close 2816 (time 1788754482.48 total 30.03 last 91.72) - open/close 3746 (time 1788754492.48 total 40.03 last 92.99) - open/close 4704 (time 1788754502.48 total 50.03 last 95.75) - open/close 5589 (time 1788754512.48 total 60.03 last 88.50) - open/close 6516 (time 1788754522.49 total 70.04 last 92.65) - open/close 7394 (time 1788754532.49 total 80.04 last 87.76) - open/close 8374 (time 1788754542.50 total 90.05 last 97.90) - open/close 9286 (time 1788754552.51 total 100.06 last 91.15) total: 10000 open/close in 106.95 seconds: 93.50 ops/second - unlinked 0 (time 1788754562 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 931 (time 1788754633.97 total 10.01 last 93.05) - open/close 1871 (time 1788754643.97 total 20.01 last 93.95) - open/close 2819 (time 1788754653.97 total 30.01 last 94.78) - open/close 3753 (time 1788754663.98 total 40.02 last 93.37) - open/close 4709 (time 1788754673.99 total 50.03 last 95.45) - open/close 5609 (time 1788754684.01 total 60.05 last 89.87) - open/close 6576 (time 1788754694.01 total 70.05 last 96.64) - open/close 7326 (time 1788754704.02 total 80.05 last 74.99) - open/close 8063 (time 1788754714.03 total 90.07 last 73.62) - open/close 8955 (time 1788754724.03 total 100.07 last 89.18) - open/close 9886 (time 1788754734.03 total 110.07 last 93.06) total: 10000 open/close in 111.33 seconds: 89.82 ops/second - unlinked 0 (time 1788754739 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 815 (time 1788754811.92 total 10.01 last 81.43) - open/close 1662 (time 1788754821.93 total 20.02 last 84.62) - open/close 2574 (time 1788754831.94 total 30.02 last 91.15) - open/close 3483 (time 1788754841.95 total 40.03 last 90.84) - open/close 4393 (time 1788754851.95 total 50.04 last 90.95) - open/close 5168 (time 1788754861.96 total 60.04 last 77.46) - open/close 6018 (time 1788754871.96 total 70.05 last 84.93) - open/close 6940 (time 1788754881.97 total 80.05 last 92.17) - open/close 7850 (time 1788754891.97 total 90.06 last 90.97) - open/close 8830 (time 1788754901.97 total 100.06 last 97.96) - open/close 9781 (time 1788754911.98 total 110.06 last 95.07) total: 10000 open/close in 112.52 seconds: 88.87 ops/second - unlinked 0 (time 1788754919 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 919 (time 1788754995.22 total 10.00 last 91.88) - open/close 1733 (time 1788755005.23 total 20.02 last 81.29) - open/close 2638 (time 1788755015.23 total 30.02 last 90.47) - open/close 3500 (time 1788755025.24 total 40.02 last 86.18) - open/close 4427 (time 1788755035.24 total 50.03 last 92.66) - open/close 5345 (time 1788755045.24 total 60.03 last 91.80) - open/close 6293 (time 1788755055.24 total 70.03 last 94.77) - open/close 7209 (time 1788755065.25 total 80.03 last 91.59) - open/close 8204 (time 1788755075.25 total 90.03 last 99.50) - open/close 9144 (time 1788755085.25 total 100.04 last 93.94) total: 10000 open/close in 109.07 seconds: 91.68 ops/second - unlinked 0 (time 1788755098 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.160@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg360-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg360-server: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg360-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg360-server: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg360-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.160@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.160@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.160@tcp sys.timeout=20 start ost1 service on oleg360-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg360-server: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg360-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg360-server: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg360-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server unloading modules via unload_modules_local on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing unload_modules_local oleg360-server: kmemleak disabled modules unloaded. pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2