== conf-sanity test 69: replace an OST with the same index ========================================================== 01:26:09 (1789709169) start mds service on oleg226-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 18 00:56 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing load_modules_local oleg226-server: kmemleak disabled oleg226-server: Loading modules from /home/green/git/lustre-release/lustre oleg226-server: detected 4 online CPUs by sysfs oleg226-server: MODOPTS_LIBCFS= oleg226-server: Force libcfs to create 2 CPU partitions oleg226-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg226-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg226-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg226-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg226-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 1329 (time 1789709264.69 total 10.00 last 132.85) - open/close 2557 (time 1789709274.70 total 20.01 last 122.68) - open/close 3720 (time 1789709284.70 total 30.02 last 116.24) - open/close 4934 (time 1789709294.71 total 40.02 last 121.36) - open/close 6036 (time 1789709304.72 total 50.03 last 110.11) - open/close 7254 (time 1789709314.72 total 60.03 last 121.78) - open/close 8387 (time 1789709324.72 total 70.04 last 113.25) - open/close 9439 (time 1789709334.72 total 80.04 last 105.18) total: 10000 open/close in 85.44 seconds: 117.04 ops/second - unlinked 0 (time 1789709343 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1191 (time 1789709401.32 total 10.00 last 119.04) - open/close 2429 (time 1789709411.33 total 20.01 last 123.75) - open/close 3711 (time 1789709421.33 total 30.01 last 128.16) - open/close 4974 (time 1789709431.33 total 40.02 last 126.24) - open/close 6201 (time 1789709441.34 total 50.02 last 122.62) - open/close 7515 (time 1789709451.34 total 60.03 last 131.36) - open/close 8945 (time 1789709461.35 total 70.03 last 142.95) total: 10000 open/close in 77.23 seconds: 129.48 ops/second - unlinked 0 (time 1789709470 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1588 (time 1789709515.19 total 10.00 last 158.73) - open/close 3479 (time 1789709525.19 total 20.00 last 189.09) - open/close 4982 (time 1789709535.20 total 30.01 last 150.19) - open/close 6337 (time 1789709545.20 total 40.01 last 135.47) - open/close 7734 (time 1789709555.21 total 50.02 last 139.62) - open/close 9185 (time 1789709565.21 total 60.03 last 145.01) total: 10000 open/close in 65.62 seconds: 152.40 ops/second - unlinked 0 (time 1789709572 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1387 (time 1789709623.01 total 10.01 last 138.58) - open/close 2524 (time 1789709633.01 total 20.01 last 113.67) - open/close 3864 (time 1789709643.01 total 30.01 last 133.97) - open/close 4986 (time 1789709653.02 total 40.02 last 112.16) - open/close 6400 (time 1789709663.02 total 50.02 last 141.33) - open/close 7866 (time 1789709673.02 total 60.02 last 146.57) - open/close 9476 (time 1789709683.02 total 70.02 last 161.00) total: 10000 open/close in 73.56 seconds: 135.94 ops/second - unlinked 0 (time 1789709688 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1381 (time 1789709733.07 total 10.01 last 137.99) - open/close 3075 (time 1789709743.07 total 20.01 last 169.36) - open/close 4696 (time 1789709753.08 total 30.02 last 161.98) - open/close 6296 (time 1789709763.09 total 40.02 last 159.89) - open/close 7860 (time 1789709773.09 total 50.03 last 156.39) - open/close 9378 (time 1789709783.09 total 60.03 last 151.75) total: 10000 open/close in 63.94 seconds: 156.41 ops/second - unlinked 0 (time 1789709788 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1953 (time 1789709832.37 total 10.00 last 195.24) - open/close 3630 (time 1789709842.37 total 20.01 last 167.55) - open/close 5345 (time 1789709852.38 total 30.01 last 171.46) - open/close 6840 (time 1789709862.38 total 40.01 last 149.50) - open/close 9028 (time 1789709872.38 total 50.02 last 218.72) total: 10000 open/close in 55.37 seconds: 180.61 ops/second - unlinked 0 (time 1789709879 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1671 (time 1789709922.66 total 10.00 last 167.08) - open/close 3186 (time 1789709932.67 total 20.00 last 151.46) - open/close 4931 (time 1789709942.67 total 30.00 last 174.50) - open/close 6534 (time 1789709952.67 total 40.01 last 160.19) - open/close 8686 (time 1789709962.68 total 50.01 last 215.16) total: 10000 open/close in 56.02 seconds: 178.52 ops/second - unlinked 0 (time 1789709970 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1554 (time 1789710014.64 total 10.00 last 155.39) - open/close 2976 (time 1789710024.65 total 20.01 last 142.05) - open/close 4382 (time 1789710034.65 total 30.02 last 140.51) - open/close 5805 (time 1789710044.65 total 40.02 last 142.29) - open/close 7296 (time 1789710054.66 total 50.03 last 148.99) - open/close 8739 (time 1789710064.66 total 60.03 last 144.29) total: 10000 open/close in 68.94 seconds: 145.06 ops/second - unlinked 0 (time 1789710075 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1692 (time 1789710118.03 total 10.00 last 169.19) - open/close 3335 (time 1789710128.04 total 20.01 last 164.18) - open/close 5037 (time 1789710138.04 total 30.01 last 170.17) - open/close 6867 (time 1789710148.04 total 40.01 last 182.99) - open/close 8867 (time 1789710158.04 total 50.01 last 199.96) total: 10000 open/close in 54.53 seconds: 183.37 ops/second - unlinked 0 (time 1789710164 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 1946 (time 1789710198.41 total 10.00 last 194.52) - open/close 3737 (time 1789710208.41 total 20.01 last 179.07) - open/close 5448 (time 1789710218.41 total 30.01 last 171.08) - open/close 7175 (time 1789710228.41 total 40.01 last 172.67) - open/close 8965 (time 1789710238.41 total 50.01 last 178.91) total: 10000 open/close in 56.31 seconds: 177.60 ops/second - unlinked 0 (time 1789710246 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre On OST0, 18562 used inodes rc=0 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg226-server unloading modules via unload_modules_local on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local oleg226-server: kmemleak disabled modules unloaded. pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2