== conf-sanity test 69: replace an OST with the same index ========================================================== 02:09:25 (1788847765) start mds service on oleg335-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 8 01:49 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: kmemleak disabled oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions oleg335-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg335-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg335-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg335-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg335-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 820 (time 1788847873.58 total 10.01 last 81.90) - open/close 1446 (time 1788847883.59 total 20.02 last 62.55) - open/close 2141 (time 1788847893.60 total 30.03 last 69.41) - open/close 3001 (time 1788847903.61 total 40.04 last 85.94) - open/close 3920 (time 1788847913.61 total 50.04 last 91.88) - open/close 4779 (time 1788847923.63 total 60.06 last 85.73) - open/close 5677 (time 1788847933.65 total 70.08 last 89.64) - open/close 6471 (time 1788847943.67 total 80.10 last 79.22) - open/close 7164 (time 1788847953.68 total 90.11 last 69.26) - open/close 7825 (time 1788847963.68 total 100.11 last 66.06) - open/close 8522 (time 1788847973.69 total 110.12 last 69.68) - open/close 9291 (time 1788847983.70 total 120.13 last 76.79) total: 10000 open/close in 128.86 seconds: 77.60 ops/second - unlinked 0 (time 1788847997 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 871 (time 1788848084.80 total 10.01 last 87.00) - open/close 1657 (time 1788848094.83 total 20.04 last 78.38) - open/close 2473 (time 1788848104.83 total 30.05 last 81.55) - open/close 3306 (time 1788848114.85 total 40.06 last 83.14) - open/close 4110 (time 1788848124.86 total 50.07 last 80.36) - open/close 4932 (time 1788848134.87 total 60.08 last 82.09) - open/close 5791 (time 1788848144.88 total 70.09 last 85.84) - open/close 6647 (time 1788848154.88 total 80.10 last 85.54) - open/close 7522 (time 1788848164.88 total 90.10 last 87.49) - open/close 8381 (time 1788848174.90 total 100.11 last 85.75) - open/close 9091 (time 1788848184.91 total 110.12 last 70.96) - open/close 9903 (time 1788848194.91 total 120.13 last 81.14) total: 10000 open/close in 121.53 seconds: 82.28 ops/second - unlinked 0 (time 1788848200 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 794 (time 1788848282.67 total 10.00 last 79.38) - open/close 1550 (time 1788848292.67 total 20.01 last 75.53) - open/close 2399 (time 1788848302.69 total 30.02 last 84.80) - open/close 3378 (time 1788848312.69 total 40.03 last 97.82) - open/close 4183 (time 1788848322.72 total 50.06 last 80.28) - open/close 4968 (time 1788848332.73 total 60.07 last 78.40) - open/close 5730 (time 1788848342.75 total 70.08 last 76.10) - open/close 6543 (time 1788848352.75 total 80.09 last 81.29) - open/close 7338 (time 1788848362.76 total 90.09 last 79.43) - open/close 8196 (time 1788848372.76 total 100.10 last 85.76) - open/close 9056 (time 1788848382.77 total 110.10 last 85.98) - open/close 9855 (time 1788848392.77 total 120.11 last 79.86) total: 10000 open/close in 121.88 seconds: 82.05 ops/second - unlinked 0 (time 1788848398 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 798 (time 1788848481.58 total 10.00 last 79.78) - open/close 1574 (time 1788848491.58 total 20.00 last 77.59) - open/close 2258 (time 1788848501.58 total 30.01 last 68.38) - open/close 3094 (time 1788848511.59 total 40.01 last 83.57) - open/close 3926 (time 1788848521.60 total 50.02 last 83.11) - open/close 4747 (time 1788848531.61 total 60.03 last 82.00) - open/close 5460 (time 1788848541.62 total 70.05 last 71.20) - open/close 6204 (time 1788848551.63 total 80.05 last 74.35) - open/close 6959 (time 1788848561.64 total 90.06 last 75.46) - open/close 7578 (time 1788848571.64 total 100.06 last 61.90) - open/close 8202 (time 1788848581.66 total 110.08 last 62.26) - open/close 8953 (time 1788848591.66 total 120.08 last 75.07) - open/close 9763 (time 1788848601.68 total 130.10 last 80.84) total: 10000 open/close in 133.39 seconds: 74.97 ops/second - unlinked 0 (time 1788848609 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 830 (time 1788848695.66 total 10.00 last 82.96) - open/close 1628 (time 1788848705.67 total 20.01 last 79.77) - open/close 2371 (time 1788848715.68 total 30.02 last 74.19) - open/close 3148 (time 1788848725.69 total 40.03 last 77.67) - open/close 3851 (time 1788848735.69 total 50.03 last 70.29) - open/close 4654 (time 1788848745.69 total 60.03 last 80.27) - open/close 5385 (time 1788848755.69 total 70.03 last 73.10) - open/close 6166 (time 1788848765.73 total 80.07 last 77.84) - open/close 7054 (time 1788848775.73 total 90.07 last 88.76) - open/close 7990 (time 1788848785.74 total 100.08 last 93.53) - open/close 8863 (time 1788848795.75 total 110.09 last 87.23) - open/close 9755 (time 1788848805.75 total 120.09 last 89.19) total: 10000 open/close in 123.09 seconds: 81.24 ops/second - unlinked 0 (time 1788848813 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 834 (time 1788848897.31 total 10.00 last 83.39) - open/close 1662 (time 1788848907.31 total 20.00 last 82.78) - open/close 2528 (time 1788848917.32 total 30.01 last 86.57) - open/close 3411 (time 1788848927.32 total 40.01 last 88.25) - open/close 4311 (time 1788848937.33 total 50.02 last 89.96) - open/close 5032 (time 1788848947.33 total 60.02 last 72.05) - open/close 5884 (time 1788848957.37 total 70.06 last 84.93) - open/close 6744 (time 1788848967.38 total 80.07 last 85.90) - open/close 7593 (time 1788848977.38 total 90.07 last 84.89) - open/close 8473 (time 1788848987.39 total 100.08 last 87.89) - open/close 9322 (time 1788848997.39 total 110.08 last 84.88) total: 10000 open/close in 117.21 seconds: 85.32 ops/second - unlinked 0 (time 1788849008 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 866 (time 1788849089.08 total 10.01 last 86.53) - open/close 1553 (time 1788849099.09 total 20.02 last 68.63) - open/close 2299 (time 1788849109.10 total 30.02 last 74.56) - open/close 3030 (time 1788849119.10 total 40.03 last 73.06) - open/close 3871 (time 1788849129.10 total 50.03 last 84.10) - open/close 4740 (time 1788849139.11 total 60.03 last 86.87) - open/close 5573 (time 1788849149.13 total 70.06 last 83.10) - open/close 6438 (time 1788849159.13 total 80.06 last 86.47) - open/close 7360 (time 1788849169.14 total 90.07 last 92.10) - open/close 8272 (time 1788849179.15 total 100.08 last 91.12) - open/close 9079 (time 1788849189.15 total 110.08 last 80.68) - open/close 9884 (time 1788849199.20 total 120.12 last 80.17) total: 10000 open/close in 121.38 seconds: 82.39 ops/second - unlinked 0 (time 1788849204 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 624 (time 1788849283.91 total 10.00 last 62.39) - open/close 1557 (time 1788849293.91 total 20.01 last 93.24) - open/close 2441 (time 1788849303.92 total 30.01 last 88.36) - open/close 3247 (time 1788849313.92 total 40.02 last 80.57) - open/close 4171 (time 1788849323.93 total 50.03 last 92.30) - open/close 5101 (time 1788849333.94 total 60.04 last 92.92) - open/close 5978 (time 1788849343.95 total 70.05 last 87.61) - open/close 6866 (time 1788849353.95 total 80.05 last 88.76) - open/close 7774 (time 1788849363.98 total 90.07 last 90.60) - open/close 8716 (time 1788849373.98 total 100.08 last 94.13) - open/close 9666 (time 1788849383.99 total 110.09 last 94.91) total: 10000 open/close in 113.62 seconds: 88.01 ops/second - unlinked 0 (time 1788849391 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 907 (time 1788849465.69 total 10.01 last 90.58) - open/close 1794 (time 1788849475.70 total 20.02 last 88.66) - open/close 2702 (time 1788849485.70 total 30.02 last 90.77) - open/close 3627 (time 1788849495.71 total 40.03 last 92.40) - open/close 4503 (time 1788849505.72 total 50.04 last 87.53) - open/close 5467 (time 1788849515.72 total 60.04 last 96.34) - open/close 6388 (time 1788849525.72 total 70.04 last 92.10) - open/close 7335 (time 1788849535.73 total 80.05 last 94.69) - open/close 8300 (time 1788849545.73 total 90.05 last 96.42) - open/close 9211 (time 1788849555.74 total 100.06 last 91.04) total: 10000 open/close in 108.57 seconds: 92.11 ops/second - unlinked 0 (time 1788849567 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 909 (time 1788849642.01 total 10.01 last 90.78) - open/close 1845 (time 1788849652.02 total 20.02 last 93.52) - open/close 2817 (time 1788849662.03 total 30.03 last 97.11) - open/close 3759 (time 1788849672.04 total 40.05 last 94.03) - open/close 4698 (time 1788849682.06 total 50.07 last 93.71) - open/close 5666 (time 1788849692.06 total 60.07 last 96.80) - open/close 6596 (time 1788849702.06 total 70.07 last 93.00) - open/close 7527 (time 1788849712.07 total 80.07 last 93.06) - open/close 8488 (time 1788849722.07 total 90.08 last 96.08) - open/close 9437 (time 1788849732.08 total 100.09 last 94.80) total: 10000 open/close in 106.06 seconds: 94.29 ops/second - unlinked 0 (time 1788849741 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: kmemleak disabled modules unloaded. pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2