== conf-sanity test 69: replace an OST with the same index ========================================================== 15:24:09 (1789673049) start mds service on oleg124-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 17 12:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: kmemleak disabled oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: detected 4 online CPUs by sysfs oleg124-server: MODOPTS_LIBCFS= oleg124-server: Force libcfs to create 2 CPU partitions oleg124-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg124-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg124-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg124-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg124-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg124-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 605 (time 1789673163.34 total 10.01 last 60.43) - open/close 1369 (time 1789673173.35 total 20.02 last 76.33) - open/close 2209 (time 1789673183.36 total 30.03 last 83.95) - open/close 3145 (time 1789673193.36 total 40.03 last 93.55) - open/close 3968 (time 1789673203.37 total 50.04 last 82.26) - open/close 4678 (time 1789673213.37 total 60.04 last 70.97) - open/close 5503 (time 1789673223.38 total 70.05 last 82.42) - open/close 6316 (time 1789673233.40 total 80.07 last 81.13) - open/close 7112 (time 1789673243.40 total 90.08 last 79.57) - open/close 7803 (time 1789673253.42 total 100.10 last 68.97) - open/close 8730 (time 1789673263.43 total 110.11 last 92.61) - open/close 9600 (time 1789673273.44 total 120.12 last 86.91) total: 10000 open/close in 125.38 seconds: 79.76 ops/second - unlinked 0 (time 1789673283 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 705 (time 1789673366.77 total 10.00 last 70.49) - open/close 1325 (time 1789673376.78 total 20.01 last 61.95) - open/close 1944 (time 1789673386.79 total 30.03 last 61.80) - open/close 2758 (time 1789673396.80 total 40.03 last 81.34) - open/close 3516 (time 1789673406.80 total 50.03 last 75.79) - open/close 4363 (time 1789673416.80 total 60.04 last 84.67) - open/close 5088 (time 1789673426.81 total 70.04 last 72.46) - open/close 5748 (time 1789673436.82 total 80.05 last 65.94) - open/close 6500 (time 1789673446.83 total 90.07 last 75.09) - open/close 7089 (time 1789673456.83 total 100.07 last 58.89) - open/close 7885 (time 1789673466.84 total 110.07 last 79.59) - open/close 8818 (time 1789673476.84 total 120.07 last 93.28) - open/close 9717 (time 1789673486.85 total 130.08 last 89.79) total: 10000 open/close in 133.92 seconds: 74.67 ops/second - unlinked 0 (time 1789673495 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 600 (time 1789673583.53 total 10.00 last 59.99) - open/close 1384 (time 1789673593.56 total 20.02 last 78.23) - open/close 2191 (time 1789673603.56 total 30.03 last 80.66) - open/close 3049 (time 1789673613.57 total 40.03 last 85.74) - open/close 3894 (time 1789673623.59 total 50.06 last 84.30) - open/close 4750 (time 1789673633.59 total 60.06 last 85.58) - open/close 5606 (time 1789673643.59 total 70.06 last 85.59) - open/close 6370 (time 1789673653.60 total 80.07 last 76.34) - open/close 7036 (time 1789673663.60 total 90.07 last 66.60) - open/close 7647 (time 1789673673.64 total 100.10 last 60.89) - open/close 8256 (time 1789673683.64 total 110.11 last 60.89) - open/close 8947 (time 1789673693.65 total 120.11 last 69.05) - open/close 9613 (time 1789673703.65 total 130.12 last 66.56) total: 10000 open/close in 135.33 seconds: 73.89 ops/second - unlinked 0 (time 1789673713 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 815 (time 1789673797.61 total 10.01 last 81.43) - open/close 1434 (time 1789673807.62 total 20.02 last 61.84) - open/close 2134 (time 1789673817.64 total 30.04 last 69.88) - open/close 2802 (time 1789673827.65 total 40.04 last 66.77) - open/close 3579 (time 1789673837.65 total 50.04 last 77.68) - open/close 4422 (time 1789673847.66 total 60.06 last 84.17) - open/close 5221 (time 1789673857.67 total 70.07 last 79.84) - open/close 5952 (time 1789673867.68 total 80.07 last 73.06) - open/close 6633 (time 1789673877.68 total 90.08 last 68.05) - open/close 7348 (time 1789673887.69 total 100.08 last 71.48) - open/close 8147 (time 1789673897.69 total 110.09 last 79.82) - open/close 8916 (time 1789673907.70 total 120.09 last 76.89) - open/close 9740 (time 1789673917.70 total 130.09 last 82.36) total: 10000 open/close in 133.00 seconds: 75.19 ops/second - unlinked 0 (time 1789673925 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 860 (time 1789674010.26 total 10.01 last 85.92) - open/close 1718 (time 1789674020.26 total 20.01 last 85.77) - open/close 2503 (time 1789674030.26 total 30.01 last 78.49) - open/close 3369 (time 1789674040.26 total 40.02 last 86.58) - open/close 4204 (time 1789674050.27 total 50.02 last 83.48) - open/close 5014 (time 1789674060.27 total 60.02 last 80.97) - open/close 5913 (time 1789674070.27 total 70.02 last 89.89) - open/close 6799 (time 1789674080.27 total 80.02 last 88.60) - open/close 7631 (time 1789674090.28 total 90.03 last 83.14) - open/close 8475 (time 1789674100.28 total 100.03 last 84.40) - open/close 9242 (time 1789674110.30 total 110.05 last 76.57) - open/close 9956 (time 1789674120.30 total 120.05 last 71.34) total: 10000 open/close in 120.57 seconds: 82.94 ops/second - unlinked 0 (time 1789674125 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 717 (time 1789674214.36 total 10.01 last 71.66) - open/close 1567 (time 1789674224.37 total 20.01 last 84.93) - open/close 2526 (time 1789674234.37 total 30.02 last 95.83) - open/close 3302 (time 1789674244.40 total 40.04 last 77.42) - open/close 4082 (time 1789674254.41 total 50.06 last 77.91) - open/close 4683 (time 1789674264.41 total 60.06 last 60.09) - open/close 5469 (time 1789674274.41 total 70.06 last 78.59) - open/close 6214 (time 1789674284.43 total 80.08 last 74.34) - open/close 7206 (time 1789674294.45 total 90.10 last 99.02) - open/close 7985 (time 1789674304.46 total 100.11 last 77.84) - open/close 8663 (time 1789674314.47 total 110.11 last 67.74) - open/close 9301 (time 1789674324.47 total 120.12 last 63.77) total: 10000 open/close in 130.10 seconds: 76.87 ops/second - unlinked 0 (time 1789674339 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 808 (time 1789674428.42 total 10.01 last 80.70) - open/close 1567 (time 1789674438.43 total 20.02 last 75.83) - open/close 2299 (time 1789674448.44 total 30.03 last 73.12) - open/close 3148 (time 1789674458.44 total 40.04 last 84.86) - open/close 3977 (time 1789674468.45 total 50.04 last 82.86) - open/close 4802 (time 1789674478.46 total 60.05 last 82.43) - open/close 5629 (time 1789674488.46 total 70.05 last 82.67) - open/close 6405 (time 1789674498.47 total 80.06 last 77.53) - open/close 7297 (time 1789674508.48 total 90.07 last 89.13) - open/close 8033 (time 1789674518.48 total 100.08 last 73.54) - open/close 8657 (time 1789674528.49 total 110.09 last 62.34) - open/close 9518 (time 1789674538.50 total 120.10 last 86.05) total: 10000 open/close in 124.89 seconds: 80.07 ops/second - unlinked 0 (time 1789674547 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 931 (time 1789674626.18 total 10.00 last 93.07) - open/close 1850 (time 1789674636.18 total 20.01 last 91.85) - open/close 2737 (time 1789674646.19 total 30.02 last 88.63) - open/close 3542 (time 1789674656.19 total 40.02 last 80.47) - open/close 4484 (time 1789674666.21 total 50.04 last 94.03) - open/close 5300 (time 1789674676.22 total 60.04 last 81.56) - open/close 6144 (time 1789674686.22 total 70.05 last 84.34) - open/close 6880 (time 1789674696.25 total 80.07 last 73.44) - open/close 7653 (time 1789674706.26 total 90.08 last 77.20) - open/close 8520 (time 1789674716.27 total 100.09 last 86.62) - open/close 9256 (time 1789674726.28 total 110.10 last 73.52) total: 10000 open/close in 118.15 seconds: 84.64 ops/second - unlinked 0 (time 1789674738 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 740 (time 1789674823.77 total 10.01 last 73.89) - open/close 1502 (time 1789674833.78 total 20.02 last 76.12) - open/close 2377 (time 1789674843.79 total 30.03 last 87.46) - open/close 3219 (time 1789674853.79 total 40.03 last 84.18) - open/close 3988 (time 1789674863.79 total 50.03 last 76.88) - open/close 4764 (time 1789674873.80 total 60.04 last 77.54) - open/close 5435 (time 1789674883.80 total 70.04 last 67.08) - open/close 6055 (time 1789674893.82 total 80.06 last 61.90) - open/close 6694 (time 1789674903.82 total 90.06 last 63.88) - open/close 7471 (time 1789674913.84 total 100.08 last 77.57) - open/close 8200 (time 1789674923.85 total 110.09 last 72.84) - open/close 9035 (time 1789674933.85 total 120.09 last 83.47) - open/close 9673 (time 1789674943.86 total 130.10 last 63.73) total: 10000 open/close in 135.63 seconds: 73.73 ops/second - unlinked 0 (time 1789674955 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 818 (time 1789675049.26 total 10.02 last 81.67) - open/close 1566 (time 1789675059.26 total 20.02 last 74.74) - open/close 2120 (time 1789675069.28 total 30.04 last 55.32) - open/close 2787 (time 1789675079.29 total 40.05 last 66.61) - open/close 3485 (time 1789675089.30 total 50.06 last 69.74) - open/close 4204 (time 1789675099.30 total 60.06 last 71.89) - open/close 4944 (time 1789675109.32 total 70.08 last 73.84) - open/close 5801 (time 1789675119.34 total 80.09 last 85.60) - open/close 6686 (time 1789675129.34 total 90.10 last 88.49) - open/close 7289 (time 1789675139.35 total 100.11 last 60.22) - open/close 7805 (time 1789675149.37 total 110.12 last 51.52) - open/close 8468 (time 1789675159.37 total 120.13 last 66.27) - open/close 9271 (time 1789675169.37 total 130.13 last 80.28) - open/close 9994 (time 1789675179.39 total 140.15 last 72.17) total: 10000 open/close in 140.25 seconds: 71.30 ops/second - unlinked 0 (time 1789675184 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.124@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg124-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg124-server: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg124-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg124-server: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg124-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.124@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.124@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.124@tcp sys.timeout=20 start ost1 service on oleg124-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg124-server: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg124-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg124-server: oleg124-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg124-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg124-server unloading modules via unload_modules_local on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing unload_modules_local oleg124-server: kmemleak disabled modules unloaded. pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2