== conf-sanity test 69: replace an OST with the same index ========================================================== 23:50:31 (1788839431) start mds service on oleg221-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:37 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: kmemleak disabled oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions oleg221-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg221-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg221-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg221-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg221-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49535 inodes available. Want 99872. rc=0 - open/close 722 (time 1788839548.71 total 10.00 last 72.17) - open/close 1436 (time 1788839558.71 total 20.01 last 71.36) - open/close 2139 (time 1788839568.71 total 30.01 last 70.30) - open/close 3014 (time 1788839578.73 total 40.02 last 87.40) - open/close 3718 (time 1788839588.73 total 50.02 last 70.40) - open/close 4444 (time 1788839598.73 total 60.03 last 72.54) - open/close 5134 (time 1788839608.74 total 70.03 last 68.96) - open/close 5955 (time 1788839618.74 total 80.04 last 82.07) - open/close 6756 (time 1788839628.75 total 90.05 last 80.02) - open/close 7571 (time 1788839638.77 total 100.07 last 81.32) - open/close 8379 (time 1788839648.78 total 110.07 last 80.79) - open/close 9353 (time 1788839658.78 total 120.08 last 97.35) total: 10000 open/close in 125.95 seconds: 79.40 ops/second - unlinked 0 (time 1788839669 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 729 (time 1788839754.15 total 10.00 last 72.88) - open/close 1483 (time 1788839764.16 total 20.02 last 75.30) - open/close 2279 (time 1788839774.16 total 30.02 last 79.59) - open/close 3176 (time 1788839784.17 total 40.03 last 89.64) - open/close 4113 (time 1788839794.17 total 50.03 last 93.68) - open/close 5005 (time 1788839804.18 total 60.03 last 89.16) - open/close 5837 (time 1788839814.18 total 70.04 last 83.16) - open/close 6718 (time 1788839824.19 total 80.05 last 88.02) - open/close 7527 (time 1788839834.20 total 90.06 last 80.80) - open/close 8446 (time 1788839844.21 total 100.07 last 91.83) - open/close 9349 (time 1788839854.23 total 110.08 last 90.16) total: 10000 open/close in 119.23 seconds: 83.87 ops/second - unlinked 0 (time 1788839867 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 589 (time 1788839956.52 total 10.00 last 58.89) - open/close 1110 (time 1788839966.54 total 20.03 last 51.96) - open/close 1651 (time 1788839976.54 total 30.03 last 54.09) - open/close 2391 (time 1788839986.56 total 40.04 last 73.90) - open/close 3155 (time 1788839996.57 total 50.06 last 76.31) - open/close 3949 (time 1788840006.57 total 60.06 last 79.36) - open/close 4856 (time 1788840016.58 total 70.06 last 90.69) - open/close 5755 (time 1788840026.58 total 80.07 last 89.86) - open/close 6518 (time 1788840036.60 total 90.09 last 76.13) - open/close 7266 (time 1788840046.61 total 100.09 last 74.78) - open/close 8117 (time 1788840056.61 total 110.10 last 85.07) - open/close 8897 (time 1788840066.62 total 120.11 last 77.92) - open/close 9829 (time 1788840076.62 total 130.11 last 93.15) total: 10000 open/close in 132.20 seconds: 75.64 ops/second - unlinked 0 (time 1788840082 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 846 (time 1788840167.46 total 10.01 last 84.55) - open/close 1587 (time 1788840177.47 total 20.02 last 74.01) - open/close 2261 (time 1788840187.48 total 30.03 last 67.35) - open/close 3095 (time 1788840197.49 total 40.03 last 83.33) - open/close 3916 (time 1788840207.50 total 50.05 last 81.96) - open/close 4657 (time 1788840217.51 total 60.05 last 74.09) - open/close 5444 (time 1788840227.52 total 70.07 last 78.57) - open/close 6285 (time 1788840237.52 total 80.07 last 84.10) - open/close 7123 (time 1788840247.53 total 90.07 last 83.77) - open/close 7942 (time 1788840257.53 total 100.07 last 81.90) - open/close 8818 (time 1788840267.53 total 110.08 last 87.57) - open/close 9576 (time 1788840277.53 total 120.08 last 75.80) total: 10000 open/close in 125.20 seconds: 79.87 ops/second - unlinked 0 (time 1788840286 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 863 (time 1788840366.31 total 10.01 last 86.24) - open/close 1605 (time 1788840376.32 total 20.01 last 74.18) - open/close 2438 (time 1788840386.33 total 30.02 last 83.23) - open/close 3294 (time 1788840396.35 total 40.04 last 85.38) - open/close 4172 (time 1788840406.35 total 50.05 last 87.77) - open/close 5019 (time 1788840416.35 total 60.05 last 84.70) - open/close 5889 (time 1788840426.36 total 70.05 last 86.95) - open/close 6771 (time 1788840436.37 total 80.06 last 88.13) - open/close 7570 (time 1788840446.38 total 90.07 last 79.82) - open/close 8416 (time 1788840456.38 total 100.08 last 84.55) - open/close 9352 (time 1788840466.39 total 110.08 last 93.55) total: 10000 open/close in 118.08 seconds: 84.69 ops/second - unlinked 0 (time 1788840478 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 740 (time 1788840562.38 total 10.01 last 73.92) - open/close 1541 (time 1788840572.38 total 20.01 last 80.08) - open/close 2378 (time 1788840582.39 total 30.01 last 83.68) - open/close 3210 (time 1788840592.40 total 40.02 last 83.12) - open/close 4104 (time 1788840602.40 total 50.03 last 89.32) - open/close 4924 (time 1788840612.41 total 60.04 last 81.97) - open/close 5770 (time 1788840622.42 total 70.05 last 84.52) - open/close 6657 (time 1788840632.43 total 80.05 last 88.64) - open/close 7503 (time 1788840642.43 total 90.06 last 84.57) - open/close 8247 (time 1788840652.43 total 100.06 last 74.36) - open/close 9057 (time 1788840662.44 total 110.07 last 80.97) - open/close 9779 (time 1788840672.45 total 120.08 last 72.14) total: 10000 open/close in 122.91 seconds: 81.36 ops/second - unlinked 0 (time 1788840679 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 802 (time 1788840763.42 total 10.01 last 80.11) - open/close 1675 (time 1788840773.43 total 20.02 last 87.26) - open/close 2574 (time 1788840783.44 total 30.02 last 89.84) - open/close 3267 (time 1788840793.46 total 40.04 last 69.16) - open/close 3991 (time 1788840803.47 total 50.06 last 72.28) - open/close 4772 (time 1788840813.48 total 60.07 last 78.02) - open/close 5541 (time 1788840823.51 total 70.10 last 76.70) - open/close 6229 (time 1788840833.53 total 80.12 last 68.67) - open/close 7001 (time 1788840843.54 total 90.13 last 77.10) - open/close 7931 (time 1788840853.55 total 100.14 last 92.95) - open/close 8868 (time 1788840863.57 total 110.15 last 93.52) - open/close 9744 (time 1788840873.57 total 120.16 last 87.57) total: 10000 open/close in 123.01 seconds: 81.29 ops/second - unlinked 0 (time 1788840880 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 715 (time 1788840968.68 total 10.01 last 71.46) - open/close 1459 (time 1788840978.68 total 20.01 last 74.38) - open/close 2264 (time 1788840988.69 total 30.01 last 80.46) - open/close 3101 (time 1788840998.69 total 40.01 last 83.70) - open/close 3864 (time 1788841008.69 total 50.02 last 76.26) - open/close 4414 (time 1788841018.70 total 60.03 last 54.95) - open/close 5276 (time 1788841028.70 total 70.03 last 86.18) - open/close 6113 (time 1788841038.73 total 80.05 last 83.52) - open/close 6976 (time 1788841048.73 total 90.06 last 86.23) - open/close 7854 (time 1788841058.74 total 100.07 last 87.76) - open/close 8689 (time 1788841068.75 total 110.08 last 83.41) - open/close 9525 (time 1788841078.76 total 120.09 last 83.51) total: 10000 open/close in 127.92 seconds: 78.18 ops/second - unlinked 0 (time 1788841091 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 840 (time 1788841176.05 total 10.00 last 84.00) - open/close 1647 (time 1788841186.05 total 20.00 last 80.69) - open/close 2559 (time 1788841196.05 total 30.00 last 91.18) - open/close 3574 (time 1788841206.05 total 40.00 last 101.49) - open/close 4437 (time 1788841216.05 total 50.00 last 86.29) - open/close 5239 (time 1788841226.06 total 60.01 last 80.13) - open/close 5989 (time 1788841236.07 total 70.02 last 74.94) - open/close 6727 (time 1788841246.07 total 80.02 last 73.78) - open/close 7602 (time 1788841256.07 total 90.03 last 87.47) - open/close 8464 (time 1788841266.08 total 100.03 last 86.18) - open/close 9263 (time 1788841276.08 total 110.04 last 79.85) total: 10000 open/close in 119.12 seconds: 83.95 ops/second - unlinked 0 (time 1788841290 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 791 (time 1788841372.42 total 10.00 last 79.07) - open/close 1574 (time 1788841382.43 total 20.01 last 78.28) - open/close 2288 (time 1788841392.43 total 30.01 last 71.36) - open/close 3074 (time 1788841402.45 total 40.03 last 78.43) - open/close 3793 (time 1788841412.46 total 50.04 last 71.84) - open/close 4554 (time 1788841422.46 total 60.04 last 76.09) - open/close 5458 (time 1788841432.48 total 70.06 last 90.20) - open/close 6277 (time 1788841442.49 total 80.07 last 81.84) - open/close 6957 (time 1788841452.52 total 90.10 last 67.83) - open/close 7730 (time 1788841462.52 total 100.10 last 77.29) - open/close 8489 (time 1788841472.52 total 110.10 last 75.85) - open/close 9209 (time 1788841482.53 total 120.11 last 71.94) total: 10000 open/close in 129.67 seconds: 77.12 ops/second - unlinked 0 (time 1788841496 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.121@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.121@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.121@tcp sys.timeout=20 start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg221-server unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local oleg221-server: kmemleak disabled modules unloaded. pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2