== conf-sanity test 69: replace an OST with the same index ========================================================== 12:20:56 (1788970856) start mds service on oleg337-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 9 02:32 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing load_modules_local oleg337-server: kmemleak disabled oleg337-server: Loading modules from /home/green/git/lustre-release/lustre oleg337-server: detected 4 online CPUs by sysfs oleg337-server: MODOPTS_LIBCFS= oleg337-server: Force libcfs to create 2 CPU partitions oleg337-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg337-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg337-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg337-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg337-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg337-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 893 (time 1788970963.18 total 10.00 last 89.29) - open/close 1873 (time 1788970973.18 total 20.00 last 97.99) - open/close 2718 (time 1788970983.18 total 30.01 last 84.47) - open/close 3641 (time 1788970993.19 total 40.02 last 92.17) - open/close 4326 (time 1788971003.20 total 50.02 last 68.48) - open/close 5186 (time 1788971013.21 total 60.03 last 85.93) - open/close 6048 (time 1788971023.21 total 70.03 last 86.17) - open/close 6891 (time 1788971033.22 total 80.04 last 84.24) - open/close 7775 (time 1788971043.22 total 90.04 last 88.37) - open/close 8752 (time 1788971053.22 total 100.05 last 97.68) - open/close 9622 (time 1788971063.23 total 110.05 last 86.96) total: 10000 open/close in 114.14 seconds: 87.61 ops/second - unlinked 0 (time 1788971071 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 864 (time 1788971146.97 total 10.01 last 86.35) - open/close 1845 (time 1788971156.97 total 20.01 last 98.08) - open/close 2700 (time 1788971166.98 total 30.02 last 85.44) - open/close 3557 (time 1788971176.99 total 40.02 last 85.65) - open/close 4454 (time 1788971186.99 total 50.03 last 89.63) - open/close 5272 (time 1788971197.00 total 60.03 last 81.77) - open/close 6285 (time 1788971207.00 total 70.04 last 101.25) - open/close 7186 (time 1788971217.01 total 80.04 last 90.04) - open/close 8055 (time 1788971227.02 total 90.05 last 86.82) - open/close 8768 (time 1788971237.04 total 100.08 last 71.14) - open/close 9668 (time 1788971247.05 total 110.08 last 89.95) total: 10000 open/close in 113.52 seconds: 88.09 ops/second - unlinked 0 (time 1788971254 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 905 (time 1788971332.05 total 10.00 last 90.48) - open/close 1762 (time 1788971342.06 total 20.01 last 85.65) - open/close 2637 (time 1788971352.06 total 30.01 last 87.48) - open/close 3600 (time 1788971362.07 total 40.02 last 96.17) - open/close 4539 (time 1788971372.07 total 50.02 last 93.89) - open/close 5403 (time 1788971382.08 total 60.03 last 86.39) - open/close 6350 (time 1788971392.08 total 70.03 last 94.63) - open/close 7273 (time 1788971402.09 total 80.04 last 92.24) - open/close 8226 (time 1788971412.09 total 90.04 last 95.28) - open/close 9155 (time 1788971422.10 total 100.05 last 92.83) total: 10000 open/close in 108.45 seconds: 92.21 ops/second - unlinked 0 (time 1788971433 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 957 (time 1788971509.10 total 10.01 last 95.64) - open/close 1889 (time 1788971519.10 total 20.01 last 93.16) - open/close 2783 (time 1788971529.11 total 30.02 last 89.35) - open/close 3715 (time 1788971539.11 total 40.02 last 93.19) - open/close 4680 (time 1788971549.11 total 50.02 last 96.50) - open/close 5589 (time 1788971559.12 total 60.02 last 90.83) - open/close 6527 (time 1788971569.12 total 70.03 last 93.76) - open/close 7417 (time 1788971579.13 total 80.04 last 88.88) - open/close 8369 (time 1788971589.14 total 90.05 last 95.17) - open/close 9355 (time 1788971599.14 total 100.05 last 98.57) total: 10000 open/close in 106.88 seconds: 93.56 ops/second - unlinked 0 (time 1788971610 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 940 (time 1788971687.56 total 10.00 last 93.97) - open/close 1911 (time 1788971697.56 total 20.00 last 97.10) - open/close 2758 (time 1788971707.56 total 30.01 last 84.64) - open/close 3717 (time 1788971717.57 total 40.01 last 95.86) - open/close 4631 (time 1788971727.58 total 50.03 last 91.26) - open/close 5489 (time 1788971737.59 total 60.04 last 85.73) - open/close 6330 (time 1788971747.59 total 70.04 last 84.10) - open/close 7276 (time 1788971757.60 total 80.04 last 94.55) - open/close 8165 (time 1788971767.60 total 90.05 last 88.83) - open/close 8932 (time 1788971777.61 total 100.05 last 76.69) - open/close 9727 (time 1788971787.63 total 110.07 last 79.35) total: 10000 open/close in 114.22 seconds: 87.55 ops/second - unlinked 0 (time 1788971796 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 1005 (time 1788971881.10 total 10.01 last 100.44) - open/close 1562 (time 1788971891.11 total 20.01 last 55.65) - open/close 2553 (time 1788971901.11 total 30.02 last 99.04) - open/close 3381 (time 1788971911.11 total 40.02 last 82.79) - open/close 4325 (time 1788971921.12 total 50.03 last 94.35) - open/close 5216 (time 1788971931.13 total 60.03 last 89.03) - open/close 5996 (time 1788971941.14 total 70.05 last 77.90) - open/close 6889 (time 1788971951.14 total 80.05 last 89.28) - open/close 7793 (time 1788971961.15 total 90.06 last 90.32) - open/close 8655 (time 1788971971.15 total 100.06 last 86.17) - open/close 9483 (time 1788971981.17 total 110.08 last 82.68) total: 10000 open/close in 115.14 seconds: 86.85 ops/second - unlinked 0 (time 1788971989 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 869 (time 1788972066.65 total 10.01 last 86.81) - open/close 1709 (time 1788972076.65 total 20.01 last 83.98) - open/close 2574 (time 1788972086.65 total 30.01 last 86.49) - open/close 3401 (time 1788972096.65 total 40.01 last 82.70) - open/close 4295 (time 1788972106.65 total 50.02 last 89.38) - open/close 5233 (time 1788972116.66 total 60.02 last 93.74) - open/close 6188 (time 1788972126.66 total 70.02 last 95.49) - open/close 7091 (time 1788972136.66 total 80.02 last 90.29) - open/close 8007 (time 1788972146.66 total 90.03 last 91.57) - open/close 8939 (time 1788972156.67 total 100.03 last 93.15) - open/close 9838 (time 1788972166.67 total 110.04 last 89.86) total: 10000 open/close in 112.14 seconds: 89.18 ops/second - unlinked 0 (time 1788972172 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 970 (time 1788972246.15 total 10.01 last 96.92) - open/close 1878 (time 1788972256.15 total 20.01 last 90.80) - open/close 2832 (time 1788972266.16 total 30.02 last 95.33) - open/close 3772 (time 1788972276.17 total 40.02 last 93.91) - open/close 4763 (time 1788972286.17 total 50.03 last 99.06) - open/close 5785 (time 1788972296.18 total 60.03 last 102.16) - open/close 6595 (time 1788972306.18 total 70.04 last 80.94) - open/close 7250 (time 1788972316.19 total 80.04 last 65.47) - open/close 7950 (time 1788972326.21 total 90.06 last 69.88) - open/close 8682 (time 1788972336.22 total 100.08 last 73.06) - open/close 9375 (time 1788972346.25 total 110.10 last 69.15) total: 10000 open/close in 117.58 seconds: 85.05 ops/second - unlinked 0 (time 1788972358 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 768 (time 1788972434.72 total 10.01 last 76.69) - open/close 1611 (time 1788972444.73 total 20.03 last 84.18) - open/close 2487 (time 1788972454.74 total 30.04 last 87.52) - open/close 3229 (time 1788972464.75 total 40.04 last 74.16) - open/close 4053 (time 1788972474.75 total 50.05 last 82.39) - open/close 4879 (time 1788972484.75 total 60.05 last 82.58) - open/close 5859 (time 1788972494.76 total 70.06 last 97.89) - open/close 6767 (time 1788972504.77 total 80.07 last 90.73) - open/close 7691 (time 1788972514.78 total 90.08 last 92.27) - open/close 8365 (time 1788972524.79 total 100.09 last 67.33) - open/close 9306 (time 1788972534.80 total 110.10 last 94.06) total: 10000 open/close in 117.66 seconds: 84.99 ops/second - unlinked 0 (time 1788972546 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 919 (time 1788972626.43 total 10.02 last 91.71) - open/close 1782 (time 1788972636.44 total 20.02 last 86.27) - open/close 2661 (time 1788972646.45 total 30.04 last 87.76) - open/close 3621 (time 1788972656.46 total 40.05 last 95.89) - open/close 4564 (time 1788972666.47 total 50.06 last 94.24) - open/close 5430 (time 1788972676.47 total 60.06 last 86.58) - open/close 6260 (time 1788972686.48 total 70.06 last 82.96) - open/close 7151 (time 1788972696.48 total 80.07 last 89.08) - open/close 7872 (time 1788972706.48 total 90.07 last 72.06) - open/close 8636 (time 1788972716.49 total 100.08 last 76.38) - open/close 9597 (time 1788972726.49 total 110.08 last 96.03) total: 10000 open/close in 114.38 seconds: 87.43 ops/second - unlinked 0 (time 1788972735 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.137@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg337-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg337-server: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg337-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg337-server: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg337-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.137@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.137@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.137@tcp sys.timeout=20 start ost1 service on oleg337-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg337-server: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg337-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg337-server: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg337-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg337-server unloading modules via unload_modules_local on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing unload_modules_local oleg337-server: kmemleak disabled modules unloaded. pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2