== conf-sanity test 69: replace an OST with the same index ========================================================== 01:39:03 (1788845943) start mds service on oleg633-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg633-server' oleg633-server: oleg633-server.virtnet: executing load_modules_local oleg633-server: kmemleak disabled oleg633-server: Loading modules from /home/green/git/lustre-release/lustre oleg633-server: detected 4 online CPUs by sysfs oleg633-server: MODOPTS_LIBCFS= oleg633-server: Force libcfs to create 2 CPU partitions oleg633-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg633-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg633-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg633-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg633-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg633-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0000 oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 861 (time 1788846042.21 total 10.02 last 85.91) - open/close 1677 (time 1788846052.22 total 20.03 last 81.56) - open/close 2569 (time 1788846062.24 total 30.05 last 88.99) - open/close 3419 (time 1788846072.24 total 40.05 last 84.98) - open/close 4231 (time 1788846082.25 total 50.06 last 81.16) - open/close 5069 (time 1788846092.25 total 60.06 last 83.77) - open/close 5926 (time 1788846102.25 total 70.07 last 85.66) - open/close 6759 (time 1788846112.26 total 80.07 last 83.29) - open/close 7624 (time 1788846122.26 total 90.08 last 86.44) - open/close 8472 (time 1788846132.27 total 100.08 last 84.77) - open/close 9252 (time 1788846142.28 total 110.09 last 77.88) total: 10000 open/close in 119.63 seconds: 83.59 ops/second - unlinked 0 (time 1788846155 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 840 (time 1788846241.88 total 10.01 last 83.95) - open/close 1624 (time 1788846251.89 total 20.01 last 78.35) - open/close 2430 (time 1788846261.90 total 30.03 last 80.49) - open/close 3199 (time 1788846271.92 total 40.04 last 76.75) - open/close 3924 (time 1788846281.94 total 50.06 last 72.39) - open/close 4588 (time 1788846291.94 total 60.06 last 66.39) - open/close 5434 (time 1788846301.94 total 70.06 last 84.58) - open/close 6265 (time 1788846311.94 total 80.06 last 83.10) - open/close 7034 (time 1788846321.94 total 90.06 last 76.90) - open/close 7796 (time 1788846331.96 total 100.08 last 76.08) - open/close 8666 (time 1788846341.96 total 110.08 last 86.99) - open/close 9514 (time 1788846351.98 total 120.10 last 84.65) total: 10000 open/close in 125.82 seconds: 79.48 ops/second - unlinked 0 (time 1788846361 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 837 (time 1788846442.92 total 10.00 last 83.70) - open/close 1747 (time 1788846452.93 total 20.01 last 90.95) - open/close 2586 (time 1788846462.93 total 30.01 last 83.87) - open/close 3285 (time 1788846472.94 total 40.02 last 69.83) - open/close 4100 (time 1788846482.96 total 50.04 last 81.35) - open/close 4870 (time 1788846492.97 total 60.04 last 76.97) - open/close 5637 (time 1788846502.97 total 70.05 last 76.64) - open/close 6507 (time 1788846512.97 total 80.05 last 87.00) - open/close 7312 (time 1788846522.98 total 90.06 last 80.42) - open/close 8128 (time 1788846532.99 total 100.06 last 81.56) - open/close 8941 (time 1788846543.00 total 110.07 last 81.22) - open/close 9768 (time 1788846553.01 total 120.08 last 82.63) total: 10000 open/close in 122.59 seconds: 81.57 ops/second - unlinked 0 (time 1788846559 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 865 (time 1788846643.71 total 10.00 last 86.48) - open/close 1726 (time 1788846653.72 total 20.01 last 86.02) - open/close 2496 (time 1788846663.72 total 30.02 last 76.95) - open/close 3317 (time 1788846673.74 total 40.03 last 81.98) - open/close 4042 (time 1788846683.75 total 50.04 last 72.42) - open/close 4640 (time 1788846693.77 total 60.07 last 59.67) - open/close 5369 (time 1788846703.78 total 70.07 last 72.84) - open/close 6153 (time 1788846713.79 total 80.08 last 78.34) - open/close 7040 (time 1788846723.79 total 90.09 last 88.64) - open/close 7842 (time 1788846733.80 total 100.10 last 80.13) - open/close 8647 (time 1788846743.81 total 110.11 last 80.41) - open/close 9446 (time 1788846753.81 total 120.11 last 79.88) total: 10000 open/close in 126.77 seconds: 78.88 ops/second - unlinked 0 (time 1788846764 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 867 (time 1788846843.90 total 10.01 last 86.65) - open/close 1680 (time 1788846853.91 total 20.01 last 81.24) - open/close 2544 (time 1788846863.92 total 30.02 last 86.31) - open/close 3410 (time 1788846873.94 total 40.04 last 86.43) - open/close 4230 (time 1788846883.94 total 50.04 last 81.99) - open/close 5007 (time 1788846893.94 total 60.05 last 77.68) - open/close 5830 (time 1788846903.95 total 70.06 last 82.19) - open/close 6654 (time 1788846913.96 total 80.07 last 82.35) - open/close 7524 (time 1788846923.96 total 90.07 last 86.99) - open/close 8360 (time 1788846933.97 total 100.07 last 83.54) - open/close 9199 (time 1788846943.97 total 110.08 last 83.84) total: 10000 open/close in 119.23 seconds: 83.87 ops/second - unlinked 0 (time 1788846957 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 810 (time 1788847037.69 total 10.00 last 80.99) - open/close 1652 (time 1788847047.69 total 20.00 last 84.19) - open/close 2531 (time 1788847057.69 total 30.01 last 87.86) - open/close 3397 (time 1788847067.70 total 40.01 last 86.55) - open/close 4256 (time 1788847077.70 total 50.02 last 85.87) - open/close 5141 (time 1788847087.71 total 60.02 last 88.44) - open/close 6024 (time 1788847097.72 total 70.03 last 88.24) - open/close 6862 (time 1788847107.72 total 80.03 last 83.79) - open/close 7701 (time 1788847117.73 total 90.04 last 83.86) - open/close 8563 (time 1788847127.74 total 100.05 last 86.11) - open/close 9377 (time 1788847137.74 total 110.05 last 81.37) total: 10000 open/close in 118.88 seconds: 84.12 ops/second - unlinked 0 (time 1788847152 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 800 (time 1788847239.33 total 10.00 last 79.98) - open/close 1682 (time 1788847249.33 total 20.01 last 88.18) - open/close 2521 (time 1788847259.34 total 30.01 last 83.87) - open/close 3162 (time 1788847269.34 total 40.01 last 64.06) - open/close 3878 (time 1788847279.34 total 50.01 last 71.60) - open/close 4563 (time 1788847289.35 total 60.02 last 68.45) - open/close 5348 (time 1788847299.36 total 70.03 last 78.46) - open/close 6186 (time 1788847309.37 total 80.04 last 83.69) - open/close 7020 (time 1788847319.37 total 90.04 last 83.40) - open/close 7893 (time 1788847329.37 total 100.04 last 87.28) - open/close 8726 (time 1788847339.37 total 110.05 last 83.28) - open/close 9579 (time 1788847349.39 total 120.06 last 85.21) total: 10000 open/close in 125.44 seconds: 79.72 ops/second - unlinked 0 (time 1788847359 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 862 (time 1788847439.18 total 10.01 last 86.14) - open/close 1563 (time 1788847449.18 total 20.01 last 70.07) - open/close 2393 (time 1788847459.19 total 30.02 last 82.92) - open/close 3162 (time 1788847469.19 total 40.02 last 76.89) - open/close 3913 (time 1788847479.19 total 50.02 last 75.09) - open/close 4769 (time 1788847489.20 total 60.03 last 85.53) - open/close 5554 (time 1788847499.20 total 70.03 last 78.50) - open/close 6360 (time 1788847509.20 total 80.03 last 80.59) - open/close 7234 (time 1788847519.20 total 90.03 last 87.39) - open/close 8027 (time 1788847529.21 total 100.04 last 79.26) - open/close 8891 (time 1788847539.21 total 110.04 last 86.36) - open/close 9726 (time 1788847549.23 total 120.06 last 83.39) total: 10000 open/close in 123.19 seconds: 81.18 ops/second - unlinked 0 (time 1788847556 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 773 (time 1788847641.33 total 10.01 last 77.22) - open/close 1571 (time 1788847651.34 total 20.02 last 79.71) - open/close 2427 (time 1788847661.36 total 30.03 last 85.50) - open/close 3297 (time 1788847671.37 total 40.05 last 86.86) - open/close 4067 (time 1788847681.39 total 50.06 last 76.89) - open/close 4962 (time 1788847691.39 total 60.07 last 89.48) - open/close 5805 (time 1788847701.39 total 70.07 last 84.25) - open/close 6723 (time 1788847711.40 total 80.08 last 91.76) - open/close 7574 (time 1788847721.40 total 90.08 last 85.05) - open/close 8319 (time 1788847731.40 total 100.08 last 74.49) - open/close 9178 (time 1788847741.41 total 110.09 last 85.87) total: 10000 open/close in 119.77 seconds: 83.49 ops/second - unlinked 0 (time 1788847754 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 623 (time 1788847839.08 total 10.01 last 62.27) - open/close 1363 (time 1788847849.09 total 20.01 last 73.94) - open/close 2056 (time 1788847859.11 total 30.04 last 69.15) - open/close 2757 (time 1788847869.12 total 40.04 last 70.07) - open/close 3453 (time 1788847879.14 total 50.06 last 69.44) - open/close 4070 (time 1788847889.14 total 60.07 last 61.68) - open/close 4787 (time 1788847899.15 total 70.07 last 71.67) - open/close 5466 (time 1788847909.15 total 80.08 last 67.86) - open/close 6218 (time 1788847919.15 total 90.08 last 75.18) - open/close 7003 (time 1788847929.15 total 100.08 last 78.48) - open/close 7536 (time 1788847939.17 total 110.10 last 53.21) - open/close 8234 (time 1788847949.18 total 120.10 last 69.75) - open/close 8851 (time 1788847959.19 total 130.11 last 61.66) - open/close 9556 (time 1788847969.19 total 140.11 last 70.48) total: 10000 open/close in 145.29 seconds: 68.83 ops/second - unlinked 0 (time 1788847978 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg633-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg633-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg633-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.133@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg633-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0000 oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg633-server: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg633-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg633-server: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg633-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg633-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg633-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg633-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.133@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.133@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.133@tcp sys.timeout=20 start ost1 service on oleg633-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0000 oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg633-server: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg633-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg633-server: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg633-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg633-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg633-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg633-server stop mds service on oleg633-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg633-server stop mds service on oleg633-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg633-server unloading modules via unload_modules_local on: 'oleg633-server' oleg633-server: oleg633-server.virtnet: executing unload_modules_local oleg633-server: kmemleak disabled modules unloaded. pdsh@oleg633-client: oleg633-client: ssh exited with exit code 2 pdsh@oleg633-client: oleg633-server: ssh exited with exit code 2 pdsh@oleg633-client: oleg633-server: ssh exited with exit code 2 pdsh@oleg633-client: oleg633-client: ssh exited with exit code 2