== conf-sanity test 69: replace an OST with the same index ========================================================== 11:18:07 (1788535087) start mds service on oleg612-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1923184 Sep 3 20:45 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg612-server' oleg612-server: oleg612-server.virtnet: executing load_modules_local oleg612-server: kmemleak disabled oleg612-server: Loading modules from /home/green/git/lustre-release/lustre oleg612-server: detected 4 online CPUs by sysfs oleg612-server: MODOPTS_LIBCFS= oleg612-server: Force libcfs to create 2 CPU partitions oleg612-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg612-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg612-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg612-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg612-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg612-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Started lustre-OST0000 oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg612-client.virtnet: -o user_xattr,flock 192.168.206.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 639 (time 1788535203.53 total 10.02 last 63.77) - open/close 1310 (time 1788535213.54 total 20.03 last 67.02) - open/close 1917 (time 1788535223.56 total 30.05 last 60.60) - open/close 2646 (time 1788535233.56 total 40.06 last 72.85) - open/close 3325 (time 1788535243.57 total 50.07 last 67.83) - open/close 4126 (time 1788535253.58 total 60.07 last 80.06) - open/close 4853 (time 1788535263.58 total 70.07 last 72.70) - open/close 5601 (time 1788535273.59 total 80.08 last 74.72) - open/close 6322 (time 1788535283.59 total 90.08 last 72.09) - open/close 7041 (time 1788535293.61 total 100.10 last 71.79) - open/close 7780 (time 1788535303.61 total 110.10 last 73.88) - open/close 8464 (time 1788535313.61 total 120.11 last 68.38) - open/close 9149 (time 1788535323.61 total 130.11 last 68.50) - open/close 9894 (time 1788535333.62 total 140.11 last 74.46) total: 10000 open/close in 141.97 seconds: 70.44 ops/second - unlinked 0 (time 1788535341 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 820 (time 1788535434.25 total 10.00 last 81.98) - open/close 1590 (time 1788535444.27 total 20.02 last 76.84) - open/close 2452 (time 1788535454.27 total 30.02 last 86.19) - open/close 3291 (time 1788535464.27 total 40.02 last 83.89) - open/close 4016 (time 1788535474.28 total 50.03 last 72.47) - open/close 4836 (time 1788535484.28 total 60.03 last 81.99) - open/close 5723 (time 1788535494.29 total 70.04 last 88.64) - open/close 6542 (time 1788535504.29 total 80.04 last 81.88) - open/close 7356 (time 1788535514.29 total 90.04 last 81.38) - open/close 8184 (time 1788535524.30 total 100.05 last 82.75) - open/close 9017 (time 1788535534.30 total 110.06 last 83.23) - open/close 9842 (time 1788535544.30 total 120.06 last 82.50) total: 10000 open/close in 121.71 seconds: 82.17 ops/second - unlinked 0 (time 1788535550 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 774 (time 1788535633.46 total 10.01 last 77.31) - open/close 1595 (time 1788535643.47 total 20.02 last 82.07) - open/close 2371 (time 1788535653.48 total 30.03 last 77.47) - open/close 3115 (time 1788535663.49 total 40.04 last 74.33) - open/close 3917 (time 1788535673.51 total 50.05 last 80.10) - open/close 4668 (time 1788535683.52 total 60.07 last 74.98) - open/close 5370 (time 1788535693.53 total 70.08 last 70.15) - open/close 6001 (time 1788535703.53 total 80.08 last 63.09) - open/close 6624 (time 1788535713.55 total 90.10 last 62.19) - open/close 7288 (time 1788535723.56 total 100.11 last 66.33) - open/close 7966 (time 1788535733.58 total 110.13 last 67.64) - open/close 8723 (time 1788535743.59 total 120.13 last 75.66) - open/close 9445 (time 1788535753.59 total 130.14 last 72.14) total: 10000 open/close in 136.97 seconds: 73.01 ops/second - unlinked 0 (time 1788535765 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 779 (time 1788535854.47 total 10.00 last 77.86) - open/close 1612 (time 1788535864.47 total 20.01 last 83.24) - open/close 2363 (time 1788535874.48 total 30.02 last 75.05) - open/close 3070 (time 1788535884.49 total 40.03 last 70.64) - open/close 3773 (time 1788535894.51 total 50.05 last 70.18) - open/close 4478 (time 1788535904.52 total 60.06 last 70.43) - open/close 5223 (time 1788535914.52 total 70.06 last 74.46) - open/close 6020 (time 1788535924.53 total 80.07 last 79.64) - open/close 6739 (time 1788535934.53 total 90.07 last 71.88) - open/close 7463 (time 1788535944.54 total 100.08 last 72.33) - open/close 8270 (time 1788535954.55 total 110.08 last 80.66) - open/close 9095 (time 1788535964.55 total 120.09 last 82.49) - open/close 9909 (time 1788535974.55 total 130.09 last 81.35) total: 10000 open/close in 131.24 seconds: 76.20 ops/second - unlinked 0 (time 1788535980 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 701 (time 1788536064.13 total 10.01 last 70.06) - open/close 1485 (time 1788536074.14 total 20.01 last 78.37) - open/close 2246 (time 1788536084.14 total 30.02 last 76.05) - open/close 3027 (time 1788536094.15 total 40.02 last 78.07) - open/close 3861 (time 1788536104.16 total 50.03 last 83.31) - open/close 4676 (time 1788536114.17 total 60.04 last 81.40) - open/close 5458 (time 1788536124.18 total 70.05 last 78.15) - open/close 6220 (time 1788536134.18 total 80.05 last 76.17) - open/close 6956 (time 1788536144.19 total 90.06 last 73.57) - open/close 7795 (time 1788536154.19 total 100.06 last 83.87) - open/close 8724 (time 1788536164.20 total 110.07 last 92.84) - open/close 9483 (time 1788536174.20 total 120.07 last 75.86) total: 10000 open/close in 126.07 seconds: 79.32 ops/second - unlinked 0 (time 1788536184 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 862 (time 1788536272.39 total 10.01 last 86.14) - open/close 1711 (time 1788536282.40 total 20.01 last 84.84) - open/close 2523 (time 1788536292.40 total 30.02 last 81.17) - open/close 3355 (time 1788536302.41 total 40.03 last 83.12) - open/close 4251 (time 1788536312.42 total 50.04 last 89.49) - open/close 5058 (time 1788536322.43 total 60.04 last 80.67) - open/close 5956 (time 1788536332.44 total 70.05 last 89.71) - open/close 6807 (time 1788536342.44 total 80.06 last 85.04) - open/close 7659 (time 1788536352.45 total 90.07 last 85.14) - open/close 8350 (time 1788536362.45 total 100.07 last 69.10) - open/close 9040 (time 1788536372.47 total 110.09 last 68.87) - open/close 9645 (time 1788536382.47 total 120.09 last 60.49) total: 10000 open/close in 124.90 seconds: 80.07 ops/second - unlinked 0 (time 1788536392 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 802 (time 1788536483.60 total 10.01 last 80.16) - open/close 1617 (time 1788536493.62 total 20.02 last 81.40) - open/close 2428 (time 1788536503.62 total 30.02 last 81.08) - open/close 3237 (time 1788536513.62 total 40.02 last 80.90) - open/close 4120 (time 1788536523.63 total 50.03 last 88.21) - open/close 5057 (time 1788536533.64 total 60.04 last 93.63) - open/close 5873 (time 1788536543.64 total 70.04 last 81.59) - open/close 6558 (time 1788536553.65 total 80.05 last 68.41) - open/close 7470 (time 1788536563.66 total 90.06 last 91.10) - open/close 8314 (time 1788536573.67 total 100.07 last 84.38) - open/close 9108 (time 1788536583.68 total 110.09 last 79.24) - open/close 9960 (time 1788536593.69 total 120.09 last 85.20) total: 10000 open/close in 120.49 seconds: 83.00 ops/second - unlinked 0 (time 1788536597 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 692 (time 1788536681.75 total 10.02 last 69.09) - open/close 1465 (time 1788536691.76 total 20.02 last 77.23) - open/close 2330 (time 1788536701.77 total 30.04 last 86.39) - open/close 3034 (time 1788536711.78 total 40.04 last 70.35) - open/close 3841 (time 1788536721.80 total 50.06 last 80.54) - open/close 4676 (time 1788536731.81 total 60.07 last 83.41) - open/close 5360 (time 1788536741.81 total 70.08 last 68.38) - open/close 6032 (time 1788536751.82 total 80.08 last 67.17) - open/close 6821 (time 1788536761.82 total 90.09 last 78.85) - open/close 7657 (time 1788536771.83 total 100.09 last 83.57) - open/close 8417 (time 1788536781.84 total 110.11 last 75.91) - open/close 9072 (time 1788536791.84 total 120.11 last 65.50) - open/close 9724 (time 1788536801.85 total 130.12 last 65.13) total: 10000 open/close in 134.04 seconds: 74.61 ops/second - unlinked 0 (time 1788536810 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 895 (time 1788536899.08 total 10.01 last 89.41) - open/close 1812 (time 1788536909.08 total 20.02 last 91.65) - open/close 2577 (time 1788536919.08 total 30.02 last 76.50) - open/close 3332 (time 1788536929.09 total 40.02 last 75.49) - open/close 4217 (time 1788536939.10 total 50.03 last 88.35) - open/close 5077 (time 1788536949.11 total 60.04 last 85.94) - open/close 5900 (time 1788536959.12 total 70.05 last 82.19) - open/close 6692 (time 1788536969.12 total 80.05 last 79.19) - open/close 7439 (time 1788536979.13 total 90.06 last 74.68) - open/close 8185 (time 1788536989.13 total 100.06 last 74.57) - open/close 9031 (time 1788536999.14 total 110.07 last 84.52) - open/close 9867 (time 1788537009.14 total 120.07 last 83.57) total: 10000 open/close in 121.60 seconds: 82.24 ops/second - unlinked 0 (time 1788537015 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 806 (time 1788537099.30 total 10.01 last 80.56) - open/close 1330 (time 1788537109.30 total 20.01 last 52.38) - open/close 2046 (time 1788537119.31 total 30.02 last 71.50) - open/close 2853 (time 1788537129.32 total 40.03 last 80.63) - open/close 3721 (time 1788537139.33 total 50.04 last 86.71) - open/close 4527 (time 1788537149.34 total 60.05 last 80.53) - open/close 5318 (time 1788537159.34 total 70.05 last 79.09) - open/close 6105 (time 1788537169.35 total 80.06 last 78.66) - open/close 6879 (time 1788537179.36 total 90.07 last 77.32) - open/close 7564 (time 1788537189.36 total 100.07 last 68.46) - open/close 8374 (time 1788537199.36 total 110.07 last 81.00) - open/close 9247 (time 1788537209.36 total 120.07 last 87.28) total: 10000 open/close in 129.61 seconds: 77.15 ops/second - unlinked 0 (time 1788537224 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg612-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg612-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg612-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.112@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg612-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Started lustre-OST0000 oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg612-server: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg612-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg612-server: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg612-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg612-client.virtnet: -o user_xattr,flock 192.168.206.112@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg612-client: oleg612-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg612-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg612-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg612-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.112@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.112@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.112@tcp sys.timeout=20 start ost1 service on oleg612-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Started lustre-OST0000 oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg612-server: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg612-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg612-server: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg612-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg612-client.virtnet: -o user_xattr,flock 192.168.206.112@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg612-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg612-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg612-server stop mds service on oleg612-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg612-server stop mds service on oleg612-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg612-server unloading modules via unload_modules_local on: 'oleg612-server' oleg612-server: oleg612-server.virtnet: executing unload_modules_local oleg612-server: kmemleak disabled modules unloaded. pdsh@oleg612-client: oleg612-client: ssh exited with exit code 2 pdsh@oleg612-client: oleg612-server: ssh exited with exit code 2 pdsh@oleg612-client: oleg612-client: ssh exited with exit code 2 pdsh@oleg612-client: oleg612-server: ssh exited with exit code 2