== conf-sanity test 69: replace an OST with the same index ========================================================== 16:02:03 (1790971323) start mds service on oleg145-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Oct 2 13:55 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: kmemleak disabled oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: MODOPTS_LIBCFS= oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg145-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg145-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg145-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg145-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 651 (time 1790971453.28 total 10.01 last 65.05) - open/close 1289 (time 1790971463.28 total 20.01 last 63.77) - open/close 1938 (time 1790971473.29 total 30.02 last 64.86) - open/close 2716 (time 1790971483.30 total 40.03 last 77.72) - open/close 3546 (time 1790971493.30 total 50.03 last 82.99) - open/close 4395 (time 1790971503.30 total 60.03 last 84.86) - open/close 5220 (time 1790971513.32 total 70.05 last 82.40) - open/close 6117 (time 1790971523.32 total 80.05 last 89.62) - open/close 6904 (time 1790971533.33 total 90.06 last 78.70) - open/close 7521 (time 1790971543.33 total 100.06 last 61.69) - open/close 8266 (time 1790971553.35 total 110.08 last 74.34) - open/close 8827 (time 1790971563.37 total 120.10 last 55.98) - open/close 9453 (time 1790971573.38 total 130.11 last 62.52) total: 10000 open/close in 137.40 seconds: 72.78 ops/second - unlinked 0 (time 1790971585 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 808 (time 1790971674.96 total 10.01 last 80.74) - open/close 1724 (time 1790971684.96 total 20.01 last 91.56) - open/close 2630 (time 1790971694.97 total 30.03 last 90.48) - open/close 3316 (time 1790971704.98 total 40.03 last 68.58) - open/close 3974 (time 1790971714.98 total 50.03 last 65.77) - open/close 4576 (time 1790971725.00 total 60.05 last 60.09) - open/close 5198 (time 1790971735.01 total 70.06 last 62.15) - open/close 5925 (time 1790971745.02 total 80.07 last 72.64) - open/close 6482 (time 1790971755.02 total 90.07 last 55.70) - open/close 7278 (time 1790971765.04 total 100.09 last 79.40) - open/close 8100 (time 1790971775.04 total 110.10 last 82.18) - open/close 8811 (time 1790971785.06 total 120.11 last 71.02) - open/close 9512 (time 1790971795.06 total 130.11 last 70.05) total: 10000 open/close in 135.81 seconds: 73.63 ops/second - unlinked 0 (time 1790971806 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 608 (time 1790971894.66 total 10.01 last 60.74) - open/close 1268 (time 1790971904.67 total 20.02 last 65.91) - open/close 1994 (time 1790971914.67 total 30.03 last 72.57) - open/close 2762 (time 1790971924.69 total 40.04 last 76.67) - open/close 3463 (time 1790971934.69 total 50.05 last 70.08) - open/close 4291 (time 1790971944.70 total 60.05 last 82.75) - open/close 5235 (time 1790971954.70 total 70.06 last 94.38) - open/close 5942 (time 1790971964.72 total 80.07 last 70.58) - open/close 6609 (time 1790971974.72 total 90.07 last 66.70) - open/close 7409 (time 1790971984.72 total 100.07 last 80.00) - open/close 8260 (time 1790971994.72 total 110.08 last 85.08) - open/close 9007 (time 1790972004.72 total 120.08 last 74.70) - open/close 9820 (time 1790972014.72 total 130.08 last 81.30) total: 10000 open/close in 132.01 seconds: 75.75 ops/second - unlinked 0 (time 1790972020 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 607 (time 1790972105.37 total 10.01 last 60.62) - open/close 1170 (time 1790972115.41 total 20.05 last 56.08) - open/close 1792 (time 1790972125.42 total 30.06 last 62.15) - open/close 2504 (time 1790972135.43 total 40.07 last 71.15) - open/close 3180 (time 1790972145.44 total 50.08 last 67.52) - open/close 3992 (time 1790972155.44 total 60.08 last 81.19) - open/close 4803 (time 1790972165.45 total 70.09 last 81.03) - open/close 5671 (time 1790972175.45 total 80.09 last 86.79) - open/close 6596 (time 1790972185.46 total 90.10 last 92.46) - open/close 7443 (time 1790972195.46 total 100.10 last 84.64) - open/close 8412 (time 1790972205.46 total 110.10 last 96.90) - open/close 9234 (time 1790972215.49 total 120.13 last 81.94) total: 10000 open/close in 129.07 seconds: 77.48 ops/second - unlinked 0 (time 1790972228 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 871 (time 1790972307.66 total 10.00 last 87.08) - open/close 1740 (time 1790972317.68 total 20.02 last 86.77) - open/close 2539 (time 1790972327.68 total 30.02 last 79.88) - open/close 3409 (time 1790972337.69 total 40.03 last 86.91) - open/close 4235 (time 1790972347.70 total 50.04 last 82.53) - open/close 5105 (time 1790972357.70 total 60.04 last 86.97) - open/close 5928 (time 1790972367.70 total 70.04 last 82.29) - open/close 6703 (time 1790972377.71 total 80.05 last 77.45) - open/close 7576 (time 1790972387.72 total 90.06 last 87.19) - open/close 8500 (time 1790972397.73 total 100.07 last 92.32) - open/close 9396 (time 1790972407.74 total 110.08 last 89.55) total: 10000 open/close in 116.77 seconds: 85.64 ops/second - unlinked 0 (time 1790972418 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 688 (time 1790972496.01 total 10.01 last 68.73) - open/close 1522 (time 1790972506.02 total 20.02 last 83.33) - open/close 2327 (time 1790972516.02 total 30.02 last 80.49) - open/close 3258 (time 1790972526.03 total 40.03 last 93.05) - open/close 4286 (time 1790972536.03 total 50.03 last 102.79) - open/close 4904 (time 1790972546.04 total 60.04 last 61.73) - open/close 5710 (time 1790972556.05 total 70.05 last 80.53) - open/close 6538 (time 1790972566.05 total 80.05 last 82.78) - open/close 7258 (time 1790972576.07 total 90.06 last 71.89) - open/close 7979 (time 1790972586.07 total 100.07 last 72.06) - open/close 8551 (time 1790972596.08 total 110.08 last 57.14) - open/close 9205 (time 1790972606.10 total 120.10 last 65.29) - open/close 9977 (time 1790972616.10 total 130.10 last 77.18) total: 10000 open/close in 130.43 seconds: 76.67 ops/second - unlinked 0 (time 1790972621 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 954 (time 1790972703.86 total 10.01 last 95.30) - open/close 1730 (time 1790972713.87 total 20.02 last 77.52) - open/close 2546 (time 1790972723.87 total 30.02 last 81.58) - open/close 3201 (time 1790972733.88 total 40.03 last 65.45) - open/close 3925 (time 1790972743.88 total 50.03 last 72.37) - open/close 4677 (time 1790972753.89 total 60.05 last 75.10) - open/close 5415 (time 1790972763.91 total 70.07 last 73.66) - open/close 6253 (time 1790972773.92 total 80.07 last 83.77) - open/close 7089 (time 1790972783.92 total 90.07 last 83.58) - open/close 7866 (time 1790972793.92 total 100.07 last 77.70) - open/close 8631 (time 1790972803.93 total 110.08 last 76.41) - open/close 9341 (time 1790972813.93 total 120.09 last 70.98) total: 10000 open/close in 129.89 seconds: 76.99 ops/second - unlinked 0 (time 1790972828 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 740 (time 1790972917.03 total 10.02 last 73.88) - open/close 1424 (time 1790972927.04 total 20.02 last 68.39) - open/close 2118 (time 1790972937.04 total 30.03 last 69.34) - open/close 2762 (time 1790972947.05 total 40.04 last 64.34) - open/close 3193 (time 1790972957.06 total 50.04 last 43.09) - open/close 3744 (time 1790972967.08 total 60.07 last 54.96) - open/close 4396 (time 1790972977.09 total 70.07 last 65.14) - open/close 5207 (time 1790972987.10 total 80.08 last 81.03) - open/close 5947 (time 1790972997.12 total 90.10 last 73.88) - open/close 6764 (time 1790973007.12 total 100.10 last 81.65) - open/close 7500 (time 1790973017.13 total 110.11 last 73.53) - open/close 8301 (time 1790973027.14 total 120.12 last 80.07) - open/close 9058 (time 1790973037.15 total 130.13 last 75.60) - open/close 9924 (time 1790973047.16 total 140.14 last 86.54) total: 10000 open/close in 141.00 seconds: 70.92 ops/second - unlinked 0 (time 1790973052 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 773 (time 1790973136.51 total 10.00 last 77.27) - open/close 1446 (time 1790973146.51 total 20.01 last 67.24) - open/close 2224 (time 1790973156.52 total 30.02 last 77.74) - open/close 3015 (time 1790973166.53 total 40.02 last 79.07) - open/close 3941 (time 1790973176.54 total 50.03 last 92.51) - open/close 4878 (time 1790973186.54 total 60.04 last 93.63) - open/close 5719 (time 1790973196.54 total 70.04 last 84.10) - open/close 6519 (time 1790973206.55 total 80.05 last 79.92) - open/close 7217 (time 1790973216.56 total 90.06 last 69.74) - open/close 8059 (time 1790973226.57 total 100.06 last 84.16) - open/close 9074 (time 1790973236.57 total 110.07 last 101.42) - open/close 9867 (time 1790973246.59 total 120.08 last 79.20) total: 10000 open/close in 121.83 seconds: 82.08 ops/second - unlinked 0 (time 1790973252 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 771 (time 1790973350.38 total 10.01 last 77.02) - open/close 1542 (time 1790973360.39 total 20.02 last 77.00) - open/close 2345 (time 1790973370.40 total 30.03 last 80.21) - open/close 3169 (time 1790973380.41 total 40.05 last 82.29) - open/close 4010 (time 1790973390.43 total 50.06 last 84.00) - open/close 5023 (time 1790973400.43 total 60.06 last 101.26) - open/close 5852 (time 1790973410.44 total 70.07 last 82.85) - open/close 6706 (time 1790973420.44 total 80.08 last 85.34) - open/close 7588 (time 1790973430.45 total 90.08 last 88.15) - open/close 8518 (time 1790973440.45 total 100.08 last 92.97) - open/close 9328 (time 1790973450.46 total 110.09 last 80.96) total: 10000 open/close in 119.92 seconds: 83.39 ops/second - unlinked 0 (time 1790973465 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg145-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.145@tcp sys.timeout=20 start ost1 service on oleg145-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg145-server: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg145-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg145-server unloading modules via unload_modules_local on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local oleg145-server: kmemleak disabled modules unloaded. pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2