== conf-sanity test 69: replace an OST with the same index ========================================================== 20:23:54 (1789518234) start mds service on oleg335-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 15 17:59 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: kmemleak disabled oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions oleg335-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg335-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg335-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg335-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg335-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 707 (time 1789518353.76 total 10.02 last 70.53) - open/close 1466 (time 1789518363.76 total 20.03 last 75.89) - open/close 2190 (time 1789518373.78 total 30.04 last 72.30) - open/close 2907 (time 1789518383.80 total 40.06 last 71.53) - open/close 3624 (time 1789518393.81 total 50.07 last 71.65) - open/close 4445 (time 1789518403.82 total 60.09 last 81.98) - open/close 5189 (time 1789518413.83 total 70.10 last 74.32) - open/close 5874 (time 1789518423.84 total 80.10 last 68.44) - open/close 6572 (time 1789518433.85 total 90.11 last 69.74) - open/close 7375 (time 1789518443.87 total 100.13 last 80.17) - open/close 8084 (time 1789518453.87 total 110.14 last 70.85) - open/close 8724 (time 1789518463.89 total 120.15 last 63.92) - open/close 9441 (time 1789518473.90 total 130.16 last 71.61) total: 10000 open/close in 138.19 seconds: 72.37 ops/second - unlinked 0 (time 1789518486 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 722 (time 1789518574.55 total 10.00 last 72.20) - open/close 1324 (time 1789518584.57 total 20.01 last 60.13) - open/close 1884 (time 1789518594.58 total 30.02 last 55.95) - open/close 2562 (time 1789518604.58 total 40.02 last 67.79) - open/close 3225 (time 1789518614.59 total 50.04 last 66.19) - open/close 3905 (time 1789518624.60 total 60.04 last 67.99) - open/close 4565 (time 1789518634.60 total 70.04 last 65.99) - open/close 5275 (time 1789518644.60 total 80.05 last 70.95) - open/close 5979 (time 1789518654.61 total 90.05 last 70.37) - open/close 6663 (time 1789518664.62 total 100.06 last 68.33) - open/close 7407 (time 1789518674.62 total 110.07 last 74.36) - open/close 8159 (time 1789518684.63 total 120.08 last 75.11) - open/close 8879 (time 1789518694.63 total 130.08 last 71.99) - open/close 9460 (time 1789518704.64 total 140.08 last 58.09) total: 10000 open/close in 148.95 seconds: 67.14 ops/second - unlinked 0 (time 1789518718 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 702 (time 1789518815.85 total 10.01 last 70.10) - open/close 1417 (time 1789518825.86 total 20.02 last 71.44) - open/close 2171 (time 1789518835.88 total 30.04 last 75.25) - open/close 2962 (time 1789518845.89 total 40.05 last 79.03) - open/close 3741 (time 1789518855.91 total 50.07 last 77.76) - open/close 4475 (time 1789518865.92 total 60.08 last 73.33) - open/close 5243 (time 1789518875.92 total 70.08 last 76.75) - open/close 5987 (time 1789518885.93 total 80.10 last 74.31) - open/close 6643 (time 1789518895.94 total 90.10 last 65.55) - open/close 7372 (time 1789518905.95 total 100.12 last 72.81) - open/close 8065 (time 1789518915.96 total 110.13 last 69.23) - open/close 8818 (time 1789518925.98 total 120.15 last 75.16) - open/close 9648 (time 1789518936.00 total 130.16 last 82.90) total: 10000 open/close in 135.13 seconds: 74.00 ops/second - unlinked 0 (time 1789518945 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 634 (time 1789519028.28 total 10.02 last 63.29) - open/close 1382 (time 1789519038.28 total 20.02 last 74.77) - open/close 2103 (time 1789519048.30 total 30.04 last 71.98) - open/close 2834 (time 1789519058.32 total 40.06 last 72.92) - open/close 3507 (time 1789519068.33 total 50.07 last 67.24) - open/close 4207 (time 1789519078.33 total 60.07 last 69.99) - open/close 4904 (time 1789519088.33 total 70.08 last 69.69) - open/close 5667 (time 1789519098.35 total 80.09 last 76.18) - open/close 6450 (time 1789519108.36 total 90.10 last 78.20) - open/close 7264 (time 1789519118.37 total 100.11 last 81.35) - open/close 8028 (time 1789519128.38 total 110.12 last 76.35) - open/close 8794 (time 1789519138.38 total 120.12 last 76.55) - open/close 9533 (time 1789519148.39 total 130.13 last 73.87) total: 10000 open/close in 137.11 seconds: 72.93 ops/second - unlinked 0 (time 1789519159 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 742 (time 1789519250.91 total 10.01 last 74.15) - open/close 1487 (time 1789519260.92 total 20.02 last 74.42) - open/close 2216 (time 1789519270.93 total 30.02 last 72.87) - open/close 2906 (time 1789519280.93 total 40.02 last 68.99) - open/close 3698 (time 1789519290.93 total 50.03 last 79.17) - open/close 4470 (time 1789519300.93 total 60.03 last 77.19) - open/close 5251 (time 1789519310.94 total 70.04 last 78.04) - open/close 5999 (time 1789519320.94 total 80.04 last 74.79) - open/close 6865 (time 1789519330.96 total 90.06 last 86.41) - open/close 7644 (time 1789519340.97 total 100.06 last 77.85) - open/close 8481 (time 1789519350.97 total 110.07 last 83.66) - open/close 9248 (time 1789519360.97 total 120.07 last 76.68) total: 10000 open/close in 129.46 seconds: 77.24 ops/second - unlinked 0 (time 1789519374 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 861 (time 1789519461.84 total 10.01 last 86.03) - open/close 1688 (time 1789519471.85 total 20.02 last 82.62) - open/close 2513 (time 1789519481.85 total 30.02 last 82.50) - open/close 3236 (time 1789519491.87 total 40.04 last 72.16) - open/close 4107 (time 1789519501.88 total 50.04 last 87.08) - open/close 4816 (time 1789519511.90 total 60.06 last 70.73) - open/close 5579 (time 1789519521.90 total 70.06 last 76.30) - open/close 6407 (time 1789519531.91 total 80.07 last 82.74) - open/close 7149 (time 1789519541.92 total 90.08 last 74.13) - open/close 7947 (time 1789519551.93 total 100.09 last 79.73) - open/close 8778 (time 1789519561.93 total 110.09 last 83.07) - open/close 9568 (time 1789519571.93 total 120.10 last 78.97) total: 10000 open/close in 125.36 seconds: 79.77 ops/second - unlinked 0 (time 1789519581 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 813 (time 1789519673.26 total 10.00 last 81.29) - open/close 1607 (time 1789519683.27 total 20.01 last 79.33) - open/close 2463 (time 1789519693.28 total 30.02 last 85.48) - open/close 3247 (time 1789519703.29 total 40.03 last 78.35) - open/close 4077 (time 1789519713.30 total 50.04 last 82.90) - open/close 4933 (time 1789519723.31 total 60.05 last 85.53) - open/close 5759 (time 1789519733.31 total 70.05 last 82.59) - open/close 6482 (time 1789519743.32 total 80.06 last 72.24) - open/close 7161 (time 1789519753.33 total 90.07 last 67.84) - open/close 7667 (time 1789519763.33 total 100.07 last 50.60) - open/close 8403 (time 1789519773.34 total 110.08 last 73.52) - open/close 9136 (time 1789519783.35 total 120.09 last 73.24) - open/close 9856 (time 1789519793.35 total 130.09 last 71.99) total: 10000 open/close in 132.07 seconds: 75.72 ops/second - unlinked 0 (time 1789519799 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 768 (time 1789519886.65 total 10.00 last 76.77) - open/close 1551 (time 1789519896.65 total 20.01 last 78.27) - open/close 2311 (time 1789519906.66 total 30.02 last 75.94) - open/close 3063 (time 1789519916.67 total 40.02 last 75.14) - open/close 3930 (time 1789519926.68 total 50.03 last 86.64) - open/close 4680 (time 1789519936.68 total 60.04 last 74.96) - open/close 5417 (time 1789519946.68 total 70.04 last 73.68) - open/close 6209 (time 1789519956.69 total 80.05 last 79.15) - open/close 6923 (time 1789519966.70 total 90.05 last 71.34) - open/close 7671 (time 1789519976.70 total 100.06 last 74.75) - open/close 8407 (time 1789519986.71 total 110.06 last 73.59) - open/close 9122 (time 1789519996.71 total 120.07 last 71.46) - open/close 9854 (time 1789520006.72 total 130.07 last 73.16) total: 10000 open/close in 132.12 seconds: 75.69 ops/second - unlinked 0 (time 1789520013 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 766 (time 1789520111.12 total 10.01 last 76.51) - open/close 1506 (time 1789520121.13 total 20.02 last 73.96) - open/close 2324 (time 1789520131.13 total 30.02 last 81.79) - open/close 3076 (time 1789520141.14 total 40.03 last 75.13) - open/close 3839 (time 1789520151.15 total 50.04 last 76.19) - open/close 4597 (time 1789520161.15 total 60.04 last 75.79) - open/close 5336 (time 1789520171.15 total 70.05 last 73.88) - open/close 5995 (time 1789520181.16 total 80.06 last 65.83) - open/close 6757 (time 1789520191.17 total 90.06 last 76.15) - open/close 7420 (time 1789520201.19 total 100.08 last 66.20) - open/close 8053 (time 1789520211.19 total 110.08 last 63.29) - open/close 8798 (time 1789520221.20 total 120.09 last 74.40) - open/close 9456 (time 1789520231.20 total 130.09 last 65.80) total: 10000 open/close in 137.39 seconds: 72.79 ops/second - unlinked 0 (time 1789520242 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 777 (time 1789520332.45 total 10.01 last 77.61) - open/close 1616 (time 1789520342.45 total 20.01 last 83.89) - open/close 2417 (time 1789520352.45 total 30.01 last 80.09) - open/close 3202 (time 1789520362.46 total 40.02 last 78.45) - open/close 4020 (time 1789520372.46 total 50.02 last 81.77) - open/close 4791 (time 1789520382.47 total 60.03 last 77.03) - open/close 5545 (time 1789520392.48 total 70.05 last 75.31) - open/close 6320 (time 1789520402.50 total 80.06 last 77.38) - open/close 7094 (time 1789520412.50 total 90.06 last 77.39) - open/close 7876 (time 1789520422.51 total 100.07 last 78.13) - open/close 8701 (time 1789520432.51 total 110.07 last 82.48) - open/close 9526 (time 1789520442.52 total 120.08 last 82.44) total: 10000 open/close in 126.02 seconds: 79.35 ops/second - unlinked 0 (time 1789520452 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.135@tcp sys.timeout=20 start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: kmemleak disabled modules unloaded. pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2