== conf-sanity test 69: replace an OST with the same index ========================================================== 00:39:32 (1788842372) start mds service on oleg357-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:40 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: kmemleak disabled oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg357-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg357-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 839 (time 1788842481.73 total 10.01 last 83.85) - open/close 1723 (time 1788842491.74 total 20.02 last 88.28) - open/close 2617 (time 1788842501.76 total 30.04 last 89.23) - open/close 3490 (time 1788842511.77 total 40.04 last 87.25) - open/close 4378 (time 1788842521.77 total 50.05 last 88.75) - open/close 5300 (time 1788842531.78 total 60.06 last 92.15) - open/close 6047 (time 1788842541.78 total 70.06 last 74.67) - open/close 6907 (time 1788842551.79 total 80.06 last 85.97) - open/close 7831 (time 1788842561.80 total 90.07 last 92.29) - open/close 8758 (time 1788842571.80 total 100.08 last 92.69) - open/close 9625 (time 1788842581.80 total 110.08 last 86.70) total: 10000 open/close in 114.73 seconds: 87.16 ops/second - unlinked 0 (time 1788842590 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 951 (time 1788842666.39 total 10.00 last 95.09) - open/close 1909 (time 1788842676.42 total 20.03 last 95.50) - open/close 2824 (time 1788842686.42 total 30.03 last 91.47) - open/close 3578 (time 1788842696.44 total 40.05 last 75.31) - open/close 4330 (time 1788842706.45 total 50.06 last 75.11) - open/close 5209 (time 1788842716.45 total 60.06 last 87.89) - open/close 6058 (time 1788842726.46 total 70.07 last 84.81) - open/close 6918 (time 1788842736.47 total 80.08 last 85.92) - open/close 7697 (time 1788842746.47 total 90.08 last 77.86) - open/close 8535 (time 1788842756.48 total 100.09 last 83.75) - open/close 9486 (time 1788842766.49 total 110.10 last 94.95) total: 10000 open/close in 116.37 seconds: 85.93 ops/second - unlinked 0 (time 1788842776 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 814 (time 1788842861.19 total 10.02 last 81.26) - open/close 1570 (time 1788842871.20 total 20.02 last 75.54) - open/close 2240 (time 1788842881.20 total 30.02 last 67.00) - open/close 2907 (time 1788842891.20 total 40.03 last 66.67) - open/close 3730 (time 1788842901.21 total 50.04 last 82.22) - open/close 4514 (time 1788842911.22 total 60.05 last 78.36) - open/close 5283 (time 1788842921.22 total 70.05 last 76.86) - open/close 6061 (time 1788842931.23 total 80.06 last 77.76) - open/close 6874 (time 1788842941.23 total 90.06 last 81.28) - open/close 7695 (time 1788842951.25 total 100.08 last 81.93) - open/close 8564 (time 1788842961.27 total 110.10 last 86.75) - open/close 9339 (time 1788842971.28 total 120.11 last 77.40) total: 10000 open/close in 128.27 seconds: 77.96 ops/second - unlinked 0 (time 1788842983 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 830 (time 1788843072.84 total 10.01 last 82.95) - open/close 1608 (time 1788843082.85 total 20.01 last 77.73) - open/close 2469 (time 1788843092.87 total 30.04 last 85.91) - open/close 3343 (time 1788843102.87 total 40.04 last 87.37) - open/close 4159 (time 1788843112.88 total 50.04 last 81.56) - open/close 5019 (time 1788843122.88 total 60.05 last 85.94) - open/close 5836 (time 1788843132.89 total 70.06 last 81.66) - open/close 6591 (time 1788843142.89 total 80.06 last 75.44) - open/close 7448 (time 1788843152.90 total 90.07 last 85.64) - open/close 8271 (time 1788843162.91 total 100.08 last 82.26) - open/close 9022 (time 1788843172.92 total 110.09 last 74.97) - open/close 9780 (time 1788843182.93 total 120.10 last 75.72) total: 10000 open/close in 122.88 seconds: 81.38 ops/second - unlinked 0 (time 1788843190 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 680 (time 1788843274.73 total 10.00 last 67.99) - open/close 1404 (time 1788843284.75 total 20.03 last 72.23) - open/close 2167 (time 1788843294.76 total 30.03 last 76.25) - open/close 2924 (time 1788843304.76 total 40.04 last 75.66) - open/close 3751 (time 1788843314.78 total 50.05 last 82.59) - open/close 4626 (time 1788843324.78 total 60.05 last 87.50) - open/close 5511 (time 1788843334.78 total 70.05 last 88.49) - open/close 6385 (time 1788843344.78 total 80.06 last 87.35) - open/close 7255 (time 1788843354.79 total 90.07 last 86.92) - open/close 8181 (time 1788843364.79 total 100.07 last 92.58) - open/close 9116 (time 1788843374.80 total 110.07 last 93.44) - open/close 9976 (time 1788843384.80 total 120.08 last 85.97) total: 10000 open/close in 120.37 seconds: 83.08 ops/second - unlinked 0 (time 1788843389 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 879 (time 1788843468.46 total 10.01 last 87.79) - open/close 1760 (time 1788843478.46 total 20.02 last 88.04) - open/close 2651 (time 1788843488.47 total 30.02 last 89.06) - open/close 3547 (time 1788843498.48 total 40.03 last 89.50) - open/close 4343 (time 1788843508.48 total 50.04 last 79.59) - open/close 5095 (time 1788843518.48 total 60.04 last 75.20) - open/close 5984 (time 1788843528.49 total 70.04 last 88.85) - open/close 6924 (time 1788843538.49 total 80.04 last 93.97) - open/close 7832 (time 1788843548.50 total 90.05 last 90.73) - open/close 8680 (time 1788843558.50 total 100.06 last 84.75) - open/close 9587 (time 1788843568.51 total 110.06 last 90.66) total: 10000 open/close in 114.62 seconds: 87.25 ops/second - unlinked 0 (time 1788843577 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 936 (time 1788843655.63 total 10.00 last 93.59) - open/close 1855 (time 1788843665.63 total 20.00 last 91.87) - open/close 2781 (time 1788843675.65 total 30.02 last 92.49) - open/close 3605 (time 1788843685.67 total 40.04 last 82.24) - open/close 4484 (time 1788843695.67 total 50.04 last 87.89) - open/close 5377 (time 1788843705.67 total 60.04 last 89.25) - open/close 6296 (time 1788843715.68 total 70.05 last 91.79) - open/close 7189 (time 1788843725.69 total 80.06 last 89.21) - open/close 8190 (time 1788843735.71 total 90.08 last 99.97) - open/close 9138 (time 1788843745.71 total 100.08 last 94.78) total: 10000 open/close in 109.89 seconds: 91.00 ops/second - unlinked 0 (time 1788843759 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 942 (time 1788843840.09 total 10.01 last 94.13) - open/close 1789 (time 1788843850.09 total 20.01 last 84.69) - open/close 2729 (time 1788843860.10 total 30.02 last 93.92) - open/close 3558 (time 1788843870.10 total 40.02 last 82.85) - open/close 4452 (time 1788843880.11 total 50.03 last 89.35) - open/close 5435 (time 1788843890.11 total 60.03 last 98.25) - open/close 6388 (time 1788843900.11 total 70.04 last 95.28) - open/close 7381 (time 1788843910.12 total 80.04 last 99.24) - open/close 8148 (time 1788843920.13 total 90.05 last 76.64) - open/close 8839 (time 1788843930.13 total 100.06 last 69.06) - open/close 9619 (time 1788843940.14 total 110.06 last 77.97) total: 10000 open/close in 114.12 seconds: 87.63 ops/second - unlinked 0 (time 1788843948 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 871 (time 1788844027.17 total 10.00 last 87.06) - open/close 1700 (time 1788844037.17 total 20.01 last 82.89) - open/close 2488 (time 1788844047.18 total 30.01 last 78.75) - open/close 3392 (time 1788844057.18 total 40.01 last 90.39) - open/close 4255 (time 1788844067.18 total 50.02 last 86.27) - open/close 5168 (time 1788844077.19 total 60.02 last 91.27) - open/close 6218 (time 1788844087.19 total 70.03 last 104.93) - open/close 7121 (time 1788844097.20 total 80.03 last 90.23) - open/close 8088 (time 1788844107.21 total 90.04 last 96.64) - open/close 8997 (time 1788844117.21 total 100.05 last 90.84) total: 10000 open/close in 109.82 seconds: 91.06 ops/second - unlinked 0 (time 1788844130 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 902 (time 1788844209.45 total 10.01 last 90.15) - open/close 1792 (time 1788844219.46 total 20.02 last 88.89) - open/close 2811 (time 1788844229.46 total 30.02 last 101.88) - open/close 3700 (time 1788844239.47 total 40.03 last 88.84) - open/close 4642 (time 1788844249.47 total 50.03 last 94.15) - open/close 5578 (time 1788844259.48 total 60.04 last 93.56) - open/close 6554 (time 1788844269.48 total 70.04 last 97.55) - open/close 7474 (time 1788844279.50 total 80.06 last 91.86) - open/close 8358 (time 1788844289.51 total 90.07 last 88.27) - open/close 9332 (time 1788844299.52 total 100.08 last 97.34) total: 10000 open/close in 107.03 seconds: 93.44 ops/second - unlinked 0 (time 1788844310 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local oleg357-server: kmemleak disabled modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2