[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 515482955 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003208] x2apic enabled [ 0.004000] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007608] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012044] Yama: becoming mindful. [ 0.013025] SELinux: Initializing. [ 0.014050] *** VALIDATE selinux *** [ 0.023029] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028478] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031099] *** VALIDATE tmpfs *** [ 0.032475] *** VALIDATE proc *** [ 0.033211] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.036184] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039023] Spectre V2 : User space: Vulnerable [ 0.040006] Speculative Store Bypass: Vulnerable [ 0.043316] debug: unmapping init [mem 0xffffffffa5c59000-0xffffffffa5c60fff] [ 0.045212] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046607] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047013] ... version: 2 [ 0.047967] ... bit width: 48 [ 0.048007] ... generic registers: 4 [ 0.048995] ... value mask: 0000ffffffffffff [ 0.049010] ... max period: 00007fffffffffff [ 0.050007] ... fixed-purpose events: 3 [ 0.050961] ... event mask: 000000070000000f [ 0.051260] rcu: Hierarchical SRCU implementation. [ 0.053367] smp: Bringing up secondary CPUs ... [ 0.054522] x86: Booting SMP configuration: [ 0.055012] .... node #0, CPUs: #1 #2 #3 [ 0.064141] smp: Brought up 1 node, 4 CPUs [ 0.066007] smpboot: Max logical packages: 1 [ 0.067010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.096038] node 0 deferred pages initialised in 28ms [ 0.099183] devtmpfs: initialized [ 0.100314] x86/mm: Memory block size: 128MB [ 0.103382] gcov: version magic: 0x41383552 [ 0.105091] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.106062] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.107222] pinctrl core: initialized pinctrl subsystem [ 0.108169] [ 0.108739] ************************************************************* [ 0.109010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.110009] ** ** [ 0.111008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.112011] ** ** [ 0.113013] ** This means that this kernel is built to expose internal ** [ 0.114027] ** IOMMU data structures, which may compromise security on ** [ 0.115015] ** your system. ** [ 0.116011] ** ** [ 0.117009] ** If you see this message and you are not debugging the ** [ 0.118007] ** kernel, report this immediately to your vendor! ** [ 0.119009] ** ** [ 0.120007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.121008] ************************************************************* [ 0.122696] NET: Registered protocol family 16 [ 0.123421] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.124038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.125047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.128061] cpuidle: using governor menu [ 0.129899] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.131435] PCI: Using configuration type 1 for base access [ 0.133180] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.138161] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.141077] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.144289] cryptd: max_cpu_qlen set to 1000 [ 0.147651] ACPI: Added _OSI(Module Device) [ 0.149009] ACPI: Added _OSI(Processor Device) [ 0.150009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.151058] ACPI: Added _OSI(Processor Aggregator Device) [ 0.156679] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.164128] ACPI: Interpreter enabled [ 0.165057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.166008] ACPI: Using IOAPIC for interrupt routing [ 0.167177] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.168379] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.179470] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.180028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.181014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.182067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.184437] acpiphp: Slot [2] registered [ 0.185077] acpiphp: Slot [5] registered [ 0.186073] acpiphp: Slot [6] registered [ 0.187076] acpiphp: Slot [7] registered [ 0.188131] acpiphp: Slot [8] registered [ 0.189154] acpiphp: Slot [9] registered [ 0.190079] acpiphp: Slot [10] registered [ 0.191070] acpiphp: Slot [3] registered [ 0.192088] acpiphp: Slot [4] registered [ 0.193070] acpiphp: Slot [11] registered [ 0.194087] acpiphp: Slot [12] registered [ 0.195078] acpiphp: Slot [13] registered [ 0.196113] acpiphp: Slot [14] registered [ 0.197062] acpiphp: Slot [15] registered [ 0.198064] acpiphp: Slot [16] registered [ 0.199063] acpiphp: Slot [17] registered [ 0.200089] acpiphp: Slot [18] registered [ 0.201069] acpiphp: Slot [19] registered [ 0.202064] acpiphp: Slot [20] registered [ 0.203077] acpiphp: Slot [21] registered [ 0.204066] acpiphp: Slot [22] registered [ 0.205086] acpiphp: Slot [23] registered [ 0.206065] acpiphp: Slot [24] registered [ 0.207086] acpiphp: Slot [25] registered [ 0.208061] acpiphp: Slot [26] registered [ 0.209148] acpiphp: Slot [27] registered [ 0.210066] acpiphp: Slot [28] registered [ 0.211063] acpiphp: Slot [29] registered [ 0.212072] acpiphp: Slot [30] registered [ 0.213105] acpiphp: Slot [31] registered [ 0.214045] PCI host bridge to bus 0000:00 [ 0.215012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.216011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.217011] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.218014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.219013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.220019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.221151] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.222575] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.225470] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.236941] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.242555] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.246019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.248011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.251014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.253528] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.256117] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.259039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.262858] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.268000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.280012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.284012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.293307] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.302016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.312017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.333018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.345000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.353014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.360019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.377012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.388235] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.394013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.402016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.432067] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.444279] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.450018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.458017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.473082] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.485821] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.494017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.501018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.518015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.529402] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.539018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.549030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.573101] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.588811] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.591416] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.594448] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.595336] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.596157] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.609201] iommu: Default domain type: Passthrough [ 0.613000] SCSI subsystem initialized [ 0.616403] ACPI: bus type USB registered [ 0.619639] usbcore: registered new interface driver usbfs [ 0.621166] usbcore: registered new interface driver hub [ 0.624463] usbcore: registered new device driver usb [ 0.627377] pps_core: LinuxPPS API ver. 1 registered [ 0.628000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.632211] PTP clock support registered [ 0.635572] EDAC MC: Ver: 3.0.0 [ 0.640234] PCI: Using ACPI for IRQ routing [ 0.644260] NetLabel: Initializing [ 0.647034] NetLabel: domain hash size = 128 [ 0.652012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.658159] NetLabel: unlabeled traffic allowed by default [ 0.663882] vgaarb: loaded [ 0.668841] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.673287] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.685430] clocksource: Switched to clocksource kvm-clock [ 0.928248] VFS: Disk quotas dquot_6.6.0 [ 0.930765] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.938975] *** VALIDATE ramfs *** [ 0.943513] *** VALIDATE hugetlbfs *** [ 0.945226] pnp: PnP ACPI init [ 0.947790] pnp: PnP ACPI: found 6 devices [ 0.970393] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.972709] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.975589] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.982534] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.984332] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.986033] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.988490] NET: Registered protocol family 2 [ 0.991393] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.998971] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.003429] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.011331] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.018316] TCP: Hash tables configured (established 65536 bind 65536) [ 1.023243] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.027522] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.033409] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.038925] NET: Registered protocol family 1 [ 1.042408] RPC: Registered named UNIX socket transport module. [ 1.045450] RPC: Registered udp transport module. [ 1.047619] RPC: Registered tcp transport module. [ 1.049992] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.053384] NET: Registered protocol family 44 [ 1.055759] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.058069] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.060409] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.062260] PCI: CLS 0 bytes, default 64 [ 1.063463] Unpacking initramfs... [ 2.841920] debug: unmapping init [mem 0xffff9b35fcc54000-0xffff9b35fffbffff] [ 2.845835] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.847640] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.850221] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.412684] Initialise system trusted keyrings [ 3.414406] Key type blacklist registered [ 3.416332] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.425589] zbud: loaded [ 3.428534] *** VALIDATE nfs *** [ 3.429667] *** VALIDATE nfs4 *** [ 3.431233] pstore: using deflate compression [ 3.434889] Platform Keyring initialized [ 3.546414] NET: Registered protocol family 38 [ 3.548449] Key type asymmetric registered [ 3.550244] Asymmetric key parser 'x509' registered [ 3.552283] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.555566] io scheduler mq-deadline registered [ 3.557394] io scheduler kyber registered [ 3.559210] io scheduler bfq registered [ 3.560986] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.564289] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.567492] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.572493] ACPI: Power Button [PWRF] [ 3.578546] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.585618] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.597913] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.605563] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.630722] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.677038] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.746163] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.776548] Non-volatile memory driver v1.3 [ 3.780597] Linux agpgart interface v0.103 [ 3.918341] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.924162] vda: detected capacity change from 0 to 75100160 [ 3.972554] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.978380] vdb: detected capacity change from 0 to 1073741824 [ 4.010687] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.016318] vdc: detected capacity change from 0 to 2621440000 [ 4.067864] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.071280] vdd: detected capacity change from 0 to 2621440000 [ 4.098290] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.108741] vde: detected capacity change from 0 to 4294967296 [ 4.152984] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.161191] vdf: detected capacity change from 0 to 4294967296 [ 4.176619] libphy: Fixed MDIO Bus: probed [ 4.205351] usbcore: registered new interface driver usbserial_generic [ 4.212325] usbserial: USB Serial support registered for generic [ 4.219724] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.234933] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.241408] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.249634] mousedev: PS/2 mouse device common for all mice [ 4.252218] rtc_cmos 00:05: RTC can wake from S4 [ 4.254933] rtc_cmos 00:05: registered as rtc0 [ 4.256290] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.264851] intel_pstate: CPU model not supported [ 4.270713] hid: raw HID events driver (C) Jiri Kosina [ 4.275696] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.277274] usbcore: registered new interface driver usbhid [ 4.293207] usbhid: USB HID core driver [ 4.296342] drop_monitor: Initializing network drop monitor service [ 4.301335] Initializing XFRM netlink socket [ 4.308450] NET: Registered protocol family 10 [ 4.320537] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.322134] Segment Routing with IPv6 [ 4.332676] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.334875] NET: Registered protocol family 17 [ 4.335409] mpls_gso: MPLS GSO support [ 4.360881] RAS: Correctable Errors collector initialized. [ 4.363940] AVX version of gcm_enc/dec engaged. [ 4.369137] AES CTR mode by8 optimization enabled [ 4.592470] sched_clock: Marking stable (4592147525, 0)->(5662398278, -1070250753) [ 4.601373] registered taskstats version 1 [ 4.607433] Loading compiled-in X.509 certificates [ 4.611781] zswap: loaded using pool lzo/zbud [ 4.716695] Key type big_key registered [ 4.762569] Key type encrypted registered [ 4.766472] ima: No TPM chip found, activating TPM-bypass! [ 4.770750] ima: Allocated hash algorithm: sha1 [ 4.779204] ima: No architecture policies found [ 4.783383] evm: Initialising EVM extended attributes: [ 4.789208] evm: security.selinux [ 4.792167] evm: security.ima [ 4.794397] evm: security.capability [ 4.797859] evm: HMAC attrs: 0x1 [ 4.808453] rtc_cmos 00:05: setting system clock to 2026-08-27 14:21:44 UTC (1787840504) [ 4.820089] debug: unmapping init [mem 0xffffffffa6c03000-0xffffffffa6dfffff] [ 4.834341] debug: unmapping init [mem 0xffffffffa5982000-0xffffffffa5c58fff] [ 4.851445] Write protecting the kernel read-only data: 28672k [ 4.864287] debug: unmapping init [mem 0xffffffffa4003000-0xffffffffa41fffff] [ 4.870325] debug: unmapping init [mem 0xffffffffa4914000-0xffffffffa49fffff] [ 5.008230] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.028427] systemd[1]: Detected virtualization kvm. [ 5.033291] systemd[1]: Detected architecture x86-64. [ 5.039225] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.109622] systemd[1]: No hostname configured. [ 5.111385] systemd[1]: Set hostname to . [ 5.113379] random: systemd: uninitialized urandom read (16 bytes read) [ 5.118469] systemd[1]: Initializing machine ID from random generator. [ 5.335280] random: ln: uninitialized urandom read (6 bytes read) [ 5.568748] random: systemd: uninitialized urandom read (16 bytes read) [ 5.571412] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.583321] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.597396] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.407330] device-mapper: uevent: version 1.0.3 [ 7.414295] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.711991] virtio_net virtio0 ens2: renamed from eth0 [ 9.949125] random: fast init done [ 10.507962] scsi host0: ata_piix [ 10.841521] scsi host1: ata_piix [ 10.845989] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.855924] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.319613] random: crng init done [ 15.322336] random: 7 urandom warning(s) missed due to ratelimiting [ 17.688605] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.976357] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.324881] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.229424] SELinux: Disabled at runtime. [ 22.412322] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.434392] systemd[1]: Detected virtualization kvm. [ 22.452204] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.735347] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.755595] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.785754] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.805856] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.830384] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.868655] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.897116] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 25.447740] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 27.279759] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.304419] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.385414] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.166210] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.445465] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit)[ 38.678985] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (15s / no limit)[ 40.332358] hrtimer: interrupt took 5106504 ns [ ***] A start job is running for Configur…only root support (16s / no limit)[ 41.062757] NFS: Registering the id_resolver key type [ 41.072851] Key type id_resolver registered [ 41.083273] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg252-server login: [ 142.829703] libcfs: loading out-of-tree module taints kernel. [ 142.899474] Key type ._llcrypt registered [ 142.901811] Key type .llcrypt registered [ 142.996740] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 167.759602] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 171.229426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 171.272591] alg: No test for adler32 (adler32-zlib) [ 172.900736] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 173.667143] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 175.376112] Key type lgssc registered [ 177.124772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 197.143076] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 239.628548] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 252.977711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 253.014644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 254.286759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 254.315590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 254.411403] Lustre: lustre-MDT0000: new disk, initializing [ 254.540608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 254.574891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 259.300970] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.193259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 273.294809] Lustre: 6524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 273.318387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 273.321681] Lustre: Skipped 1 previous similar message [ 273.402689] Lustre: lustre-MDT0001: new disk, initializing [ 273.485781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 273.537855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 273.554614] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 277.755907] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.772258] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 292.324367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.657626] Lustre: lustre-OST0000: new disk, initializing [ 292.665880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 292.672961] Lustre: 8463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 292.748867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.054163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 295.066509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 295.108121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 300.465171] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 316.834050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 317.001552] Lustre: lustre-OST0001: new disk, initializing [ 317.011969] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 317.034044] Lustre: 9535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 317.146890] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 323.695793] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 325.158711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 325.163995] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 325.232930] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 336.364576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 348.656886] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 356.588259] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing check_logdir /tmp/testlogs/ [ 362.799747] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing yml_node [ 367.364592] Lustre: DEBUG MARKER: Client: 2.17.57.69 [ 370.177129] Lustre: DEBUG MARKER: MDS: 2.17.57.69 [ 372.558385] Lustre: DEBUG MARKER: OSS: 2.17.57.69 [ 374.118590] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:27:52 EDT 2026 [ 393.374989] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 395.215145] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 412.137067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 412.154819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 412.157037] Lustre: Skipped 3 previous similar messages [ 412.175700] Lustre: Skipped 3 previous similar messages [ 417.248435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 417.250436] Lustre: Skipped 3 previous similar messages [ 418.335885] Lustre: server umount lustre-MDT0000 complete [ 421.830151] LustreError: 7482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840921 with bad export cookie 16806112537540004441 [ 421.832800] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 421.836760] LustreError: 7482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 422.368939] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.369547] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 422.372406] Lustre: Skipped 1 previous similar message [ 426.409264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 426.416657] Lustre: Skipped 2 previous similar messages [ 431.536451] LustreError: 9033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.539805] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 431.562542] LustreError: 9033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 431.570483] Lustre: Skipped 1 previous similar message [ 436.191183] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 436.602295] Lustre: server umount lustre-MDT0001 complete [ 440.984698] Lustre: server umount lustre-OST0000 complete [ 444.384139] Lustre: server umount lustre-OST0001 complete [ 450.523615] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 463.874555] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 517.126607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.377809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 517.411785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 517.465458] Lustre: lustre-MDT0000: new disk, initializing [ 517.546439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.563968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 523.668114] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 540.727071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.957283] Lustre: 15103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 541.088081] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 541.094242] Lustre: Skipped 1 previous similar message [ 541.216425] Lustre: lustre-MDT0001: new disk, initializing [ 541.389996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 541.437804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 541.452912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 545.384663] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 555.398838] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 564.280789] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 576.069760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 576.373941] Lustre: lustre-OST0000: new disk, initializing [ 576.376981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 576.381720] Lustre: 17074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 576.484293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.316462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 578.338072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 578.604382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 588.662599] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 603.642907] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 609.756383] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 609.960796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 614.995888] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 615.100963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 619.505070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 619.509714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.513849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 619.531045] Lustre: Skipped 1 previous similar message [ 631.263185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 631.753540] Lustre: server umount lustre-OST0000 complete [ 644.064492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 644.087507] LustreError: Skipped 1 previous similar message [ 644.106542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.149075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 644.162362] Lustre: Skipped 5 previous similar messages [ 655.329892] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 656.326767] Lustre: server umount lustre-MDT0000 complete [ 659.436242] LustreError: 15115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.452136] LustreError: 15115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 661.987830] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841161 with bad export cookie 16806112537540005967 [ 661.992600] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.005973] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 662.469279] Lustre: server umount lustre-MDT0001 complete [ 668.790303] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:32:46 (1787841166) [ 671.777133] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 673.772180] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:32:51 (1787841171) [ 681.814351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 682.372973] LustreError: 19491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 682.455222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 687.592976] LustreError: 19492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.628509] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 692.707910] LustreError: 19491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.504234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.825259] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 697.840928] Lustre: Skipped 6 previous similar messages [ 703.365343] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 712.054766] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 718.417382] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 726.022165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 734.102872] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 743.834307] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 753.121396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 753.132915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.143020] Lustre: Skipped 1 previous similar message [ 758.367630] Lustre: server umount lustre-OST0000 complete [ 764.384911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 764.403128] LustreError: Skipped 1 previous similar message [ 764.408787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.420915] Lustre: Skipped 1 previous similar message [ 764.430751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 764.436261] Lustre: Skipped 4 previous similar messages [ 769.005623] Lustre: server umount lustre-MDT0000 complete [ 771.554773] LustreError: 21158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.569664] LustreError: 21158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 773.554599] LustreError: 21338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841273 with bad export cookie 16806112537540006856 [ 773.582020] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 773.905640] Lustre: server umount lustre-MDT0001 complete [ 787.658788] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 791.863648] Key type lgssc unregistered [ 792.296619] LNet: 23030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.308510] LNetError: 23030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.321680] LNet: Removed LNI 192.168.202.152@tcp [ 794.560166] Key type .llcrypt unregistered [ 794.561956] Key type ._llcrypt unregistered [ 820.316383] Key type ._llcrypt registered [ 820.320050] Key type .llcrypt registered [ 820.441527] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 821.955169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.014377] alg: No test for adler32 (adler32-zlib) [ 823.615885] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 823.991524] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 825.695300] Key type lgssc registered [ 827.643860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 839.123678] loop: module loaded [ 839.222348] loop0: detected capacity change from 0 to 204800000 [ 839.387691] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 839.395617] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 839.593879] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 845.722225] loop0: detected capacity change from 0 to 204800000 [ 846.064904] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 846.090570] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 846.116283] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 852.801540] loop0: detected capacity change from 0 to 204800000 [ 859.992743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 860.013265] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.217971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 861.278898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 861.417839] Lustre: lustre-MDT0000: new disk, initializing [ 861.528612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.558653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 866.294977] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 877.159859] loop1: detected capacity change from 0 to 204800000 [ 884.888593] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 885.364294] Lustre: lustre-OST07c6: new disk, initializing [ 885.373587] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 885.378442] Lustre: Skipped 1 previous similar message [ 885.386715] Lustre: 26218:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 885.471479] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 890.973049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 890.984034] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 891.020237] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 894.111788] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 900.081927] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 900.104032] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 900.119584] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 900.132609] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 900.141951] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 900.156926] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 915.038943] Lustre: 25202:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 915.043084] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 915.045310] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 915.048078] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 915.051512] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 915.054679] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 920.173101] Lustre: 25202:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 920.184664] Lustre: 25202:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 920.188996] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 920.194295] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 920.198544] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 920.202405] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 920.207222] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 920.211779] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 920.216165] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 920.220063] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 920.224709] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 920.229164] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 923.385751] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 923.394416] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 923.398505] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 923.403280] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 923.410737] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 923.420948] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 923.427732] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 923.433185] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 923.435839] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 923.442024] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 923.444982] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 923.450654] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 928.290936] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 928.313052] Lustre: 25203:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 928.324438] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 928.337961] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 928.354808] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 928.366343] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 928.380158] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 928.398611] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 928.414358] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 928.430728] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 928.448804] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 928.464340] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 940.658298] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 940.669036] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 940.676045] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 940.682423] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 940.692961] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 940.700203] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 940.709816] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 940.721802] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 940.732218] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 940.745320] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 940.758427] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 940.769904] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 993.249718] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 993.260585] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.283839] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 995.085322] Lustre: server umount lustre-OST07c6 complete [ 1005.388109] Lustre: server umount lustre-MDT0000 complete [ 1025.057602] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1028.855994] Key type lgssc unregistered [ 1029.144620] LNet: 28779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.153188] LNetError: 28779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.166949] LNet: Removed LNI 192.168.202.152@tcp [ 1030.358821] Key type .llcrypt unregistered [ 1030.365886] Key type ._llcrypt unregistered [ 1050.882927] Key type ._llcrypt registered [ 1050.884512] Key type .llcrypt registered [ 1051.083588] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1052.495640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1052.623977] alg: No test for adler32 (adler32-zlib) [ 1053.744371] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1053.940710] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1055.607237] Key type lgssc registered [ 1057.254990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1066.556360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1066.570409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.218100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.224118] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1077.731620] LustreError: 30051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.751025] LustreError: 30051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1082.861923] LustreError: 30052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.591949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.170840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1093.134452] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1102.036661] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1107.656875] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1113.831667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.178380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.251091] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1130.990817] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1137.797856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1156.065961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1156.073327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.088739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.624855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1158.632369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.634125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.666084] Lustre: Skipped 1 previous similar message [ 1161.211787] Lustre: server umount lustre-OST0000 complete [ 1167.846037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1167.855527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.874413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1172.036279] Lustre: server umount lustre-MDT0000 complete [ 1173.997645] LustreError: 32779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.013332] LustreError: 32779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1175.799436] LustreError: 30032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841675 with bad export cookie 11682176283844575653 [ 1175.820774] LustreError: 30032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1175.821056] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1176.132586] Lustre: server umount lustre-MDT0001 complete [ 1188.538323] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1192.104238] Key type lgssc unregistered [ 1192.600923] LNet: 33813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.604886] LNetError: 33813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1193.638121] LNet: Removed LNI 192.168.202.152@tcp [ 1194.646936] Key type .llcrypt unregistered [ 1194.649289] Key type ._llcrypt unregistered [ 1215.138339] Key type ._llcrypt registered [ 1215.140340] Key type .llcrypt registered [ 1215.282216] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1216.998234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1217.030949] alg: No test for adler32 (adler32-zlib) [ 1218.100093] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1218.494257] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1220.199188] Key type lgssc registered [ 1221.322879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1229.470886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1229.486505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.019386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.129570] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1240.560581] LustreError: 35086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.577211] LustreError: 35086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1244.645708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.939862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1249.627443] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1257.758318] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1263.834638] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1272.483086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1272.994321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1276.091753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1281.264387] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1290.674691] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1305.083563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1305.098377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.104304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.156479] Lustre: Skipped 1 previous similar message [ 1308.506816] Lustre: server umount lustre-OST0000 complete [ 1315.298271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.326258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1315.348079] Lustre: Skipped 2 previous similar messages [ 1318.953973] Lustre: server umount lustre-MDT0000 complete [ 1320.421478] LustreError: 35814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1320.438908] LustreError: 35814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1323.656685] LustreError: 35065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841823 with bad export cookie 13511635032687765154 [ 1323.668411] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1323.678142] LustreError: 35065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1324.001259] Lustre: server umount lustre-MDT0001 complete [ 1338.387985] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1341.773400] Key type lgssc unregistered [ 1342.162134] LNet: 38624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.177499] LNetError: 38624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.206714] LNet: Removed LNI 192.168.202.152@tcp [ 1344.322943] Key type .llcrypt unregistered [ 1344.324898] Key type ._llcrypt unregistered [ 1370.200275] Key type ._llcrypt registered [ 1370.202699] Key type .llcrypt registered [ 1370.404773] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1371.956159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1372.121978] alg: No test for adler32 (adler32-zlib) [ 1373.322316] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1373.733554] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1375.423469] Key type lgssc registered [ 1376.992362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1386.520252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1386.552493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.945877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.646194] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1398.249807] LustreError: 40043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.296159] LustreError: 40043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1403.377828] LustreError: 40044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1404.991626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.572827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1413.312757] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1425.368929] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1431.806184] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1439.622077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1439.963774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.591290] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1457.277467] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1457.405420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1464.276961] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 1466.848396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1466.858629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.873130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.885476] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.886676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.906859] Lustre: Skipped 1 previous similar message [ 1467.940884] Lustre: Skipped 2 previous similar messages [ 1471.133951] Lustre: server umount lustre-MDT0000 complete [ 1472.999347] LustreError: 40038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.025768] LustreError: 40038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1475.602045] LustreError: 40024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841975 with bad export cookie 6442761976318341778 [ 1475.611817] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1475.617476] LustreError: 40024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1478.112813] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.113909] LustreError: 40040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.123893] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1478.123902] Lustre: Skipped 1 previous similar message [ 1478.138038] LustreError: 40040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1482.231911] Lustre: server umount lustre-MDT0001 complete [ 1499.164442] Lustre: server umount lustre-OST0000 complete [ 1517.984960] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1520.969248] Key type lgssc unregistered [ 1521.299714] LNet: 43910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.305592] LNetError: 43910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.342943] LNet: Removed LNI 192.168.202.152@tcp [ 1523.364481] Key type .llcrypt unregistered [ 1523.369087] Key type ._llcrypt unregistered [ 1549.046444] Key type ._llcrypt registered [ 1549.048848] Key type .llcrypt registered [ 1549.266816] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1550.904396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1550.931904] alg: No test for adler32 (adler32-zlib) [ 1552.070389] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1552.309612] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1554.032182] Key type lgssc registered [ 1555.514196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1565.340852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1565.354310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.964859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.296025] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1576.436673] LustreError: 45330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1576.455362] LustreError: 45330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1581.548978] LustreError: 45331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1584.241874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.609968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1589.872958] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1598.829521] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1605.302894] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1613.041185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1613.459803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.623090] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1629.536185] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1637.044320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1646.575127] Lustre: lustre-OST0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 1647.600934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.605630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1647.626224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1651.636510] Lustre: lustre-OST0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 1651.646633] Lustre: Skipped 1 previous similar message [ 1652.553982] Lustre: server umount lustre-OST0000 complete [ 1671.648691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1671.664585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.684175] Lustre: Skipped 1 previous similar message [ 1671.695227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.194508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.273877] Lustre: server umount lustre-MDT0000 complete [ 1678.304094] LustreError: 46060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.322432] LustreError: 46060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1680.116624] LustreError: 45309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842179 with bad export cookie 14361734491958477082 [ 1680.124375] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1680.130769] LustreError: 45309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1686.614658] Lustre: server umount lustre-MDT0001 complete [ 1698.750339] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1701.732878] Key type lgssc unregistered [ 1702.031545] LNet: 49063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.036636] LNetError: 49063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.057296] LNet: Removed LNI 192.168.202.152@tcp [ 1703.098088] Key type .llcrypt unregistered [ 1703.099886] Key type ._llcrypt unregistered [ 1730.470394] Key type ._llcrypt registered [ 1730.473224] Key type .llcrypt registered [ 1730.557375] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1731.885206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1731.983122] alg: No test for adler32 (adler32-zlib) [ 1733.256518] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1733.573174] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1735.312117] Key type lgssc registered [ 1736.525546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1746.104704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1746.133100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1747.785874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.164731] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1758.191799] LustreError: 50487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1758.236317] LustreError: 50487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1763.301727] LustreError: 50486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1763.803509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1764.212340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1769.151978] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1777.506376] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1784.479803] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1790.840634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1791.123464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.198933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1797.546986] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1806.129927] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1812.525988] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1816.568021] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1829.119262] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2710h52m06s [ 1831.127946] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1834.459102] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 1834.479138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.507341] Lustre: Skipped 1 previous similar message [ 1835.532305] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 1835.534759] Lustre: Skipped 3 previous similar messages [ 1839.549875] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 1839.560737] Lustre: Skipped 3 previous similar messages [ 1839.951000] Lustre: server umount lustre-MDT0000 complete [ 1841.637221] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1841.673904] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1844.051886] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1845.687200] LustreError: 50482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.542558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.743757] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1851.837528] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (not set up) [ 1852.023776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.083410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1853.046817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1858.216201] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1862.521755] Lustre: DEBUG MARKER: Wait for df (36789) ... [ 1864.317814] Lustre: DEBUG MARKER: done [ 1870.313380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1870.323179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.344392] Lustre: Skipped 1 previous similar message [ 1870.357961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1872.356765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1872.364386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.599296] Lustre: server umount lustre-OST0000 complete [ 1884.130165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1884.132070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.154132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.165234] Lustre: Skipped 1 previous similar message [ 1884.185621] Lustre: Skipped 4 previous similar messages [ 1894.367458] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1894.555214] Lustre: server umount lustre-MDT0000 complete [ 1897.464493] LustreError: 51214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.493324] LustreError: 51214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1898.394695] LustreError: 50468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842398 with bad export cookie 5227089944457470846 [ 1898.398182] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1898.402270] LustreError: 50468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1898.646532] Lustre: server umount lustre-MDT0001 complete [ 1911.020567] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1913.929105] Key type lgssc unregistered [ 1914.216546] LNet: 55417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.226829] LNetError: 55417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.242742] LNet: Removed LNI 192.168.202.152@tcp [ 1915.084078] Key type .llcrypt unregistered [ 1915.089412] Key type ._llcrypt unregistered [ 1956.848684] Key type ._llcrypt registered [ 1956.851410] Key type .llcrypt registered [ 1957.025291] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1958.655407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.728605] alg: No test for adler32 (adler32-zlib) [ 1959.804522] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1960.028869] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1961.671660] Key type lgssc registered [ 1963.208518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1972.431842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1972.466068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.846772] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1973.888416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1974.106916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.312071] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1989.366753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.571364] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1989.608345] Lustre: 57622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1989.826785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1994.694420] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2003.319494] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2009.268723] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2017.412907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2017.564988] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2017.779522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.177755] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2028.271145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2035.172452] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2041.984702] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2710h55m39s [ 2043.845411] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2053.193481] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2056.890859] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2056.952484] LustreError: 60155:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2077.148737] Lustre: lustre-MDT0000: Client 52b23e07-1a10-4a12-a502-06f056ab8333 (at 192.168.202.52@tcp) reconnecting [ 2097.015141] LustreError: 60155:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2099.005312] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2101.878290] Lustre: DEBUG MARKER: done [ 2109.086838] Lustre: server umount lustre-OST0000 complete [ 2115.554550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.557951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2115.563857] Lustre: Skipped 1 previous similar message [ 2116.648676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.680039] Lustre: Skipped 2 previous similar messages [ 2122.729251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2126.815725] Lustre: 56948:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842610/real 1787842610] req@00000000f78c4760 x1874688694345344/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787842626 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2126.845590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.907911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2127.913157] Lustre: Skipped 2 previous similar messages [ 2128.351375] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2128.625453] Lustre: server umount lustre-MDT0000 complete [ 2131.999101] Lustre: 56948:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842615/real 1787842615] req@00000000f1db92ea x1874688694347776/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787842631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2133.264517] LustreError: 57614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842632 with bad export cookie 8272560455797702504 [ 2133.273511] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2133.285199] LustreError: 57614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2133.854703] Lustre: server umount lustre-MDT0001 complete [ 2145.410773] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2148.360277] Key type lgssc unregistered [ 2148.718503] LNet: 61959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.729138] LNetError: 61959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.745566] LNet: Removed LNI 192.168.202.152@tcp [ 2149.706481] Key type .llcrypt unregistered [ 2149.709155] Key type ._llcrypt unregistered [ 2193.211433] Key type ._llcrypt registered [ 2193.212787] Key type .llcrypt registered [ 2193.318776] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2194.437811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2194.454239] alg: No test for adler32 (adler32-zlib) [ 2195.614961] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2195.898355] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2197.599226] Key type lgssc registered [ 2198.821994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2201.893337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2201.909202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.101447] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2203.128613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2203.282446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.066939] Lustre: Failing over lustre-MDT0000 [ 2204.311692] Lustre: server umount lustre-MDT0000 complete [ 2210.453460] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:58:28 (1787842708) [ 2217.858732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.337722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.043418] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2233.592053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.746209] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2233.809662] Lustre: 64760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2234.165253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2239.471155] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2248.079321] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2254.427463] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2261.964030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2262.189073] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2262.413338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.441365] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2272.434425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2277.511479] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2284.215153] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2293.216727] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.216727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.227943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2293.239322] Lustre: Skipped 1 previous similar message [ 2293.258844] Lustre: Skipped 1 previous similar message [ 2297.096761] Lustre: server umount lustre-MDT0000 complete [ 2300.906363] LustreError: 64751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842800 with bad export cookie 5356509945530579634 [ 2300.916227] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.921542] LustreError: 64751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2301.286354] Lustre: server umount lustre-MDT0001 complete [ 2305.583692] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2310.813050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2319.848130] Lustre: 63685:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842803/real 1787842803] req@00000000fa574ff9 x1874688941794048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2319.890350] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.464521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.887671] Lustre: 63682:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842805/real 1787842805] req@00000000cbb49253 x1874688941794176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2324.962296] Lustre: 63683:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842808/real 1787842808] req@00000000aca4d7b1 x1874688941794304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2326.462058] LustreError: 66610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.474379] LustreError: 66610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2326.573464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2326.693613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2327.588942] Lustre: 63685:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842810/real 1787842810] req@00000000556fcbec x1874688941794560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2328.618371] LustreError: 66611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2329.637146] LustreError: 68533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2329.665590] LustreError: 68533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2331.659195] Lustre: 63685:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842815/real 1787842815] req@00000000abfdde0b x1874688941794688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842831 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2332.367555] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2334.696390] LustreError: 68531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.716362] LustreError: 68531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2339.816255] LustreError: 66610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.830953] LustreError: 66610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2342.441043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.843590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2348.041735] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2348.479935] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2349.063700] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2357.985357] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2365.701564] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2378.722294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2378.729435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.742832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.761391] Lustre: Skipped 1 previous similar message [ 2379.748938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2379.759867] Lustre: Skipped 2 previous similar messages [ 2381.732708] Lustre: server umount lustre-MDT0000 complete [ 2384.869413] LustreError: 68531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.900905] LustreError: 68531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2386.924327] LustreError: 64751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842886 with bad export cookie 5356509945530581909 [ 2386.934677] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2386.940084] LustreError: 64751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2387.398302] Lustre: server umount lustre-MDT0001 complete [ 2395.190041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2405.218378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.303199] Lustre: 63684:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842889/real 1787842889] req@00000000deea09d5 x1874688941825920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2406.325520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.338834] Lustre: Skipped 2 previous similar messages [ 2411.491545] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4a56256a5b3288f8 [ 2411.508471] Lustre: MGC192.168.202.152@tcp: Connection restored to 0@lo (at 0@lo) [ 2411.779246] LustreError: 69269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.907549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.976972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2417.022618] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2426.554927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.962963] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2432.495402] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2440.676584] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2447.565101] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2457.571694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2457.575631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.588035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2461.928490] Lustre: server umount lustre-MDT0000 complete [ 2465.278504] LustreError: 71233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.310288] LustreError: 71233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2466.714518] LustreError: 64750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842966 with bad export cookie 5356509945530583288 [ 2466.728797] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2467.209837] Lustre: server umount lustre-MDT0001 complete [ 2473.949723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2477.309905] Lustre: DEBUG MARKER: files compared the same [ 2490.368947] Lustre: server umount lustre-OST0000 complete [ 2509.026692] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2512.611878] Key type lgssc unregistered [ 2513.007868] LNet: 74514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2513.016878] LNetError: 74514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2513.035400] LNet: Removed LNI 192.168.202.152@tcp [ 2514.471405] Key type .llcrypt unregistered [ 2514.473690] Key type ._llcrypt unregistered [ 2542.229697] Key type ._llcrypt registered [ 2542.231317] Key type .llcrypt registered [ 2542.340338] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2543.540576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.625974] alg: No test for adler32 (adler32-zlib) [ 2544.850798] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2545.163825] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2546.807400] Key type lgssc registered [ 2548.046296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2555.670697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2555.699791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.183979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.195080] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug +malloc all [ 2566.627173] LustreError: 75936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.647322] LustreError: 75936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2571.744874] LustreError: 75935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.477626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.808157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2579.719358] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug +malloc all [ 2588.009820] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2595.763856] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2602.863594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2603.099883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.726912] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug +malloc all [ 2618.431715] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2626.631894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2639.844989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2639.851059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.864900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.888225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.888907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.239225] Lustre: server umount lustre-OST0000 complete [ 2650.592671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2650.602365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.615120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2650.624684] Lustre: Skipped 1 previous similar message [ 2657.249554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.253137] Lustre: Skipped 3 previous similar messages [ 2662.367449] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2662.368328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.398234] Lustre: Skipped 1 previous similar message [ 2662.583876] Lustre: server umount lustre-MDT0000 complete [ 2666.467097] LustreError: 76663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.485061] LustreError: 76663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2666.977469] LustreError: 75916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843166 with bad export cookie 17147480686516302628 [ 2666.991126] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.401492] Lustre: server umount lustre-MDT0001 complete [ 2678.877842] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2681.843389] Key type lgssc unregistered [ 2682.240687] LNet: 79493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2682.247286] LNetError: 79493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2682.267816] LNet: Removed LNI 192.168.202.152@tcp [ 2683.515333] Key type .llcrypt unregistered [ 2683.517625] Key type ._llcrypt unregistered [ 2709.556065] Key type ._llcrypt registered [ 2709.558623] Key type .llcrypt registered [ 2709.678262] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2710.796967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2710.948147] alg: No test for adler32 (adler32-zlib) [ 2711.960270] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2712.211825] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2713.864121] Key type lgssc registered [ 2715.184701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2724.575576] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2724.601597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2740.191617] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2745.311324] LustreError: 80887:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 4: rc = -110 [ 2772.001130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.417623] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2786.995947] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2795.622562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.747898] Lustre: *** cfs_fail_loc=706, val=0*** [ 2795.988770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2796.025785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2801.134198] LustreError: 80912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.161549] LustreError: 80912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2801.876069] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2806.244626] LustreError: 80913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.313285] LustreError: 81971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.392305] LustreError: 80912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.404737] LustreError: 80912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2813.182276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.683054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2818.855308] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2826.825737] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2833.004437] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2836.960768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2836.975178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.994628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.528666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.534444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.670499] Lustre: server umount lustre-OST0000 complete [ 2849.762318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2849.764973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.773890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2849.793109] Lustre: Skipped 1 previous similar message [ 2849.798198] Lustre: Skipped 1 previous similar message [ 2852.651541] Lustre: server umount lustre-MDT0000 complete [ 2854.889100] LustreError: 81976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.923370] LustreError: 80893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843356 with bad export cookie 15278380264065876159 [ 2856.941941] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2856.946932] LustreError: 80893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2857.318216] Lustre: server umount lustre-MDT0001 complete [ 2868.361890] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2870.966551] Key type lgssc unregistered [ 2871.178266] LNet: 84507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2871.183664] LNetError: 84507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2871.193967] LNet: Removed LNI 192.168.202.152@tcp [ 2872.048628] Key type .llcrypt unregistered [ 2872.053955] Key type ._llcrypt unregistered [ 2901.940881] Key type ._llcrypt registered [ 2901.942895] Key type .llcrypt registered [ 2902.077052] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2903.448599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2903.455843] alg: No test for adler32 (adler32-zlib) [ 2904.415354] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2904.593900] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2906.399191] Key type lgssc registered [ 2907.708996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.628514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2915.648027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.893271] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2929.175862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.831339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2934.592429] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2935.266448] LustreError: 86613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.295765] LustreError: 86613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2939.372772] LustreError: 86617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.398315] LustreError: 86617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2943.804269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2944.130527] LustreError: 86616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2944.149755] LustreError: 86616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2944.173870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2949.605789] LustreError: 86612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.626437] LustreError: 86612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2951.625932] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2954.729549] LustreError: 86617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.760546] LustreError: 86617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2960.401583] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2960.413666] Lustre: MGS: Client 4bf8d3e8-4b3f-4c99-b46b-beac233f7744 (at 0@lo) reconnecting [ 2960.429031] LustreError: 86627:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2960.568316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2960.630662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2965.806992] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2977.772233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2977.782422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.789125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2981.345675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2981.354607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.361560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2983.920983] Lustre: server umount lustre-OST0000 complete [ 2991.586295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2991.592283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.597821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.597830] Lustre: Skipped 1 previous similar message [ 2991.653778] Lustre: Skipped 1 previous similar message [ 2994.022280] Lustre: server umount lustre-MDT0000 complete [ 2996.704872] LustreError: 87608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.718683] LustreError: 87608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2999.131675] Lustre: server umount lustre-MDT0001 complete [ 3003.024772] Lustre: server umount MGS complete [ 3015.685490] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:11:53 (1787843513) [ 3047.212216] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3056.103917] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3090.892237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.337572] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3105.467613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.628783] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3105.659938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3105.715439] Lustre: lustre-MDT0001: new disk, initializing [ 3105.776654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3105.801554] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3105.806294] Lustre: 93783:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3105.816773] LustreError: 93783:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3110.811223] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3122.261412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.537802] Lustre: lustre-OST0000: new disk, initializing [ 3122.546524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3122.556860] Lustre: 94551:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3122.684602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.036201] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3130.978271] Lustre: 94885:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ec58abcf x1874689685233792/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787843646 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3139.393761] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3146.737801] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3146.752674] Lustre: MGS: Client 2bfc2a41-713e-46e0-820d-29e70c8e90fa (at 0@lo) reconnecting [ 3146.803703] Lustre: 95596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3146.821640] LustreError: 93769:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3146.848459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3146.978300] Lustre: lustre-MDT0000: new disk, initializing [ 3147.166171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.214927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3147.250219] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3148.336454] Lustre: 95621:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000622ca3f5 x1874689685240704/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787843664 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3152.842834] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3153.457781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3153.467968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3153.559804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3183.594605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3183.601179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3183.607608] Lustre: Skipped 1 previous similar message [ 3183.623086] Lustre: Skipped 2 previous similar messages [ 3188.713708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3188.719837] Lustre: Skipped 1 previous similar message [ 3194.335184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3194.505810] Lustre: server umount lustre-OST0000 complete [ 3198.944762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3198.946588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.954089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3198.969277] Lustre: Skipped 1 previous similar message [ 3198.988344] Lustre: Skipped 3 previous similar messages [ 3204.513524] Lustre: server umount lustre-MDT0000 complete [ 3215.113541] Lustre: server umount lustre-MDT0001 complete [ 3219.035797] Lustre: server umount MGS complete [ 3232.929474] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:15:30 (1787843730) [ 3254.472358] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3257.773813] Key type lgssc unregistered [ 3258.064978] LNet: 98125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3258.076077] LNetError: 98125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3258.096596] LNet: Removed LNI 192.168.202.152@tcp [ 3259.028935] Key type .llcrypt unregistered [ 3259.031672] Key type ._llcrypt unregistered [ 3278.951275] Key type ._llcrypt registered [ 3278.954368] Key type .llcrypt registered [ 3279.041249] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3280.061322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3280.238521] alg: No test for adler32 (adler32-zlib) [ 3281.265885] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3281.494540] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3283.159193] Key type lgssc registered [ 3284.438728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3294.497881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3294.512273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.093983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.786657] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3305.452090] LustreError: 99348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.477396] LustreError: 99348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3310.081926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.377707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3315.856578] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3325.005986] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3332.246274] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3340.276152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3340.712780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.743929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3347.785654] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3356.303731] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3365.348742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3365.356599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3365.397674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3365.407882] Lustre: Skipped 1 previous similar message [ 3365.434309] Lustre: Skipped 1 previous similar message [ 3370.465564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3370.976384] Lustre: server umount lustre-OST0000 complete [ 3377.121203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3377.131287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.158183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.160867] Lustre: Skipped 1 previous similar message [ 3380.704059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3380.710394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.882870] Lustre: server umount lustre-MDT0000 complete [ 3385.824082] LustreError: 101792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.841134] LustreError: 101792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3389.334025] LustreError: 102131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843889 with bad export cookie 14372487275452065585 [ 3389.344374] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.352260] LustreError: 102131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3389.734369] Lustre: server umount lustre-MDT0001 complete [ 3402.083748] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3404.785529] Key type lgssc unregistered [ 3404.997698] LNet: 102885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3405.002919] LNetError: 102885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3406.054637] LNet: Removed LNI 192.168.202.152@tcp [ 3406.837710] Key type .llcrypt unregistered [ 3406.839412] Key type ._llcrypt unregistered [ 3422.209760] Key type ._llcrypt registered [ 3422.217324] Key type .llcrypt registered [ 3422.377949] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3423.969786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3424.013684] alg: No test for adler32 (adler32-zlib) [ 3425.030628] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3425.276813] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3426.975568] Key type lgssc registered [ 3428.378214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3436.129716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3436.136832] LustreError: 103959:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3436.210671] LustreError: 103969:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3436.216758] LustreError: 103959:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4925 [ 3436.225974] LustreError: 103959:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3436.241802] LustreError: 103959:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3436.248369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.882292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.729707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3445.988642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3450.644277] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3456.644354] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing lsmod [ 3460.170346] LustreError: 105173:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3460.186295] LustreError: 105176:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3460.192701] LustreError: 105173:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 3460.205777] LustreError: 105173:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3460.212452] LustreError: 105173:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3460.223919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.847448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.928091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3466.720978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3466.727554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.743205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.258574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.261408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.278045] Lustre: Skipped 1 previous similar message [ 3468.291857] Lustre: Skipped 2 previous similar messages [ 3470.370620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3470.382324] Lustre: Skipped 1 previous similar message [ 3471.214856] Lustre: server umount lustre-MDT0000 complete [ 3475.425975] LustreError: 104008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.449039] LustreError: 104008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3475.708555] LustreError: 103986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843975 with bad export cookie 17535803280397958100 [ 3475.722062] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.732516] LustreError: 103986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3476.639286] LustreError: 104438:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000565eb94 x1874690230475136/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3476.953756] Lustre: server umount lustre-MDT0001 complete [ 3492.048466] Lustre: server umount lustre-OST0000 complete [ 3501.983320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.476089] LustreError: 106240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.610873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.687046] LustreError: 106241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3508.076797] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3512.809305] LustreError: 106240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.921192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.339833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3521.883782] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3531.999723] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3538.475856] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3546.147669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3546.560511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.585060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3553.110901] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3561.255795] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3567.075798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3567.082666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.099910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3567.114026] Lustre: Skipped 1 previous similar message [ 3569.004816] Lustre: server umount lustre-OST0000 complete [ 3573.728194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3573.739266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.750123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3573.755359] Lustre: Skipped 1 previous similar message [ 3579.402230] Lustre: server umount lustre-MDT0000 complete [ 3582.432389] LustreError: 106241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3582.460995] LustreError: 106241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3583.409758] LustreError: 106221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844083 with bad export cookie 17535803280397959038 [ 3583.413440] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3583.431682] LustreError: 106221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3583.812823] Lustre: server umount lustre-MDT0001 complete [ 3596.588961] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3599.927598] Key type lgssc unregistered [ 3600.225946] LNet: 109778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3600.232995] LNetError: 109778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3600.261391] LNet: Removed LNI 192.168.202.152@tcp [ 3601.452664] Key type .llcrypt unregistered [ 3601.457403] Key type ._llcrypt unregistered [ 3625.719697] Key type ._llcrypt registered [ 3625.724675] Key type .llcrypt registered [ 3625.892539] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3627.199148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3627.373708] alg: No test for adler32 (adler32-zlib) [ 3628.489863] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3628.737990] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3630.415113] Key type lgssc registered [ 3631.597669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3639.171649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3639.188833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.796976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.566502] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3651.052816] LustreError: 111201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.083598] LustreError: 111201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3655.657641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.147191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3660.339163] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3670.505188] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3676.763345] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3683.423989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3683.825366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3685.908242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3690.981337] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3699.494274] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3705.859142] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3710.139825] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3716.579668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.598885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3716.611503] Lustre: Skipped 1 previous similar message [ 3716.636615] Lustre: Skipped 1 previous similar message [ 3721.703198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3721.715502] Lustre: Skipped 1 previous similar message [ 3726.826128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3726.843456] Lustre: Skipped 1 previous similar message [ 3728.351190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3728.646661] Lustre: server umount lustre-OST0000 complete [ 3732.968465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3732.979718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3732.989666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.072499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.078054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.090995] Lustre: Skipped 1 previous similar message [ 3738.923314] Lustre: server umount lustre-MDT0000 complete [ 3742.176475] LustreError: 111931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.205284] LustreError: 111931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3743.133109] LustreError: 111181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844242 with bad export cookie 8117746551530269754 [ 3743.145110] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3743.413840] Lustre: server umount lustre-MDT0001 complete [ 3755.365452] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3758.745372] Key type lgssc unregistered [ 3759.057369] LNet: 114843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3759.075177] LNetError: 114843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3759.105780] LNet: Removed LNI 192.168.202.152@tcp [ 3760.236633] Key type .llcrypt unregistered [ 3760.238415] Key type ._llcrypt unregistered [ 3781.029738] Key type ._llcrypt registered [ 3781.032295] Key type .llcrypt registered [ 3781.206309] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3782.409488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3782.491364] alg: No test for adler32 (adler32-zlib) [ 3783.664146] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3783.976070] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3785.697130] Key type lgssc registered [ 3787.109925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3796.062734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3796.088845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.858214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3803.685348] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3808.238612] LustreError: 116066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3808.273796] LustreError: 116066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3813.350183] LustreError: 116067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.621933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.884027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3821.224619] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3830.727107] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3840.416993] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3851.437773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3852.163409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3863.446443] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3867.574722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3876.131917] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3886.048871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3886.054787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.063448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3888.096729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3888.102624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3888.108164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3888.138859] Lustre: Skipped 1 previous similar message [ 3893.221439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3893.233699] Lustre: Skipped 1 previous similar message [ 3898.340465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3898.342914] Lustre: Skipped 1 previous similar message [ 3899.365808] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3899.612554] Lustre: server umount lustre-OST0000 complete [ 3908.069768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3908.075664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3908.089815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3911.205843] Lustre: server umount lustre-MDT0000 complete [ 3913.697549] LustreError: 117030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3913.721671] LustreError: 117030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3916.065497] LustreError: 116048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844415 with bad export cookie 14610727422240281466 [ 3916.078588] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3916.082596] LustreError: 116048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3916.426645] Lustre: server umount lustre-MDT0001 complete [ 3931.197573] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3935.623211] Key type lgssc unregistered [ 3936.085048] LNet: 119608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3936.097972] LNetError: 119608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3936.125826] LNet: Removed LNI 192.168.202.152@tcp [ 3937.465375] Key type .llcrypt unregistered [ 3937.467126] Key type ._llcrypt unregistered [ 3966.206792] Key type ._llcrypt registered [ 3966.213596] Key type .llcrypt registered [ 3966.310835] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3967.878905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3967.934656] alg: No test for adler32 (adler32-zlib) [ 3969.102697] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3969.414286] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3971.263128] Key type lgssc registered [ 3972.742955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3982.341563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3982.369245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.959407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3989.260356] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3994.083504] LustreError: 121029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.097438] LustreError: 121029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3998.501106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3998.831029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4004.505815] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4012.226972] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4019.098254] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4027.124925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4027.630988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4028.677826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 4035.037595] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4043.459164] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4049.977436] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4054.217883] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4054.227138] Lustre: Skipped 2 previous similar messages [ 4059.114742] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4059.123907] Lustre: Skipped 2 previous similar messages [ 4059.940913] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4060.007181] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4064.538546] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.52@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4064.546717] Lustre: Skipped 2 previous similar messages [ 4069.469643] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.52@tcp 192.168.0.[2,10]@tcp [ 4075.716704] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.52/24@tcp in log lustre-MDT0000 [ 4075.727510] Lustre: Skipped 2 previous similar messages [ 4081.202266] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.52/24@tcp [ 4081.209780] Lustre: Skipped 1 previous similar message [ 4088.822635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4088.825532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.830785] LustreError: Skipped 1 previous similar message [ 4088.844452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4090.336216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4090.338491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.357248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4090.380941] LustreError: Skipped 4 previous similar messages [ 4090.812463] Lustre: server umount lustre-OST0000 complete [ 4095.457602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.474769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4095.477281] Lustre: Skipped 1 previous similar message [ 4100.577789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4100.582323] Lustre: Skipped 1 previous similar message [ 4101.479816] Lustre: server umount lustre-MDT0000 complete [ 4105.697651] LustreError: 121029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4105.721100] LustreError: 121029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4105.786633] LustreError: 121009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844605 with bad export cookie 8817171205075689659 [ 4105.796711] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4105.799498] LustreError: 121009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4106.202178] Lustre: server umount lustre-MDT0001 complete [ 4120.199924] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4123.646429] Key type lgssc unregistered [ 4123.973980] LNet: 125871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4123.980057] LNetError: 125871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4124.005485] LNet: Removed LNI 192.168.202.152@tcp [ 4125.103062] Key type .llcrypt unregistered [ 4125.112378] Key type ._llcrypt unregistered [ 4155.235793] Key type ._llcrypt registered [ 4155.238281] Key type .llcrypt registered [ 4155.333567] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4156.359287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4156.450144] alg: No test for adler32 (adler32-zlib) [ 4157.710494] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 4157.982068] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4159.655136] Key type lgssc registered [ 4160.689303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4169.469386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4169.497628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.035381] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4171.060899] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.52/24@tcp [ 4171.084977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.358415] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4180.467445] LustreError: 127431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.495850] LustreError: 127431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4185.504861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4185.576350] LustreError: 127432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4186.010364] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4186.076649] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.52/24@tcp [ 4186.135657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4191.109437] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4199.503957] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4206.256340] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4213.662619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4214.241038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4217.350728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4222.742189] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4231.886991] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4242.920792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4242.929290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.949844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4247.011307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4247.015001] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4247.039049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4247.072957] Lustre: Skipped 1 previous similar message [ 4248.494517] Lustre: server umount lustre-OST0000 complete [ 4252.129353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.145394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4252.149086] Lustre: Skipped 1 previous similar message [ 4257.250311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4257.256540] Lustre: Skipped 1 previous similar message [ 4258.641747] Lustre: server umount lustre-MDT0000 complete [ 4262.374708] LustreError: 127431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.392395] LustreError: 127431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4262.561466] LustreError: 127412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844762 with bad export cookie 13433633499249316304 [ 4262.563257] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4262.570391] LustreError: 127412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4262.823327] Lustre: server umount lustre-MDT0001 complete [ 4275.474561] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4278.454268] Key type lgssc unregistered [ 4278.808270] LNet: 131036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4278.826202] LNetError: 131036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4279.846651] LNet: Removed LNI 192.168.202.152@tcp [ 4280.746895] Key type .llcrypt unregistered [ 4280.756988] Key type ._llcrypt unregistered [ 4310.255539] Key type ._llcrypt registered [ 4310.259348] Key type .llcrypt registered [ 4310.431201] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4327.549911] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4328.680406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4328.887092] alg: No test for adler32 (adler32-zlib) [ 4330.052596] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 4330.271514] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4331.967615] Key type lgssc registered [ 4333.301261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4373.475350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4373.505369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.829736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4374.871760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4374.965851] Lustre: lustre-MDT0000: new disk, initializing [ 4375.101375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4375.135772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4380.430880] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4391.174929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.318087] Lustre: 134440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4391.372725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4391.376364] Lustre: Skipped 1 previous similar message [ 4391.455234] Lustre: lustre-MDT0001: new disk, initializing [ 4391.664584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4391.725468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4391.731452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4397.438698] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4408.174941] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4414.163201] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4421.528445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4421.855986] Lustre: lustre-OST0000: new disk, initializing [ 4421.860903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4421.870418] Lustre: 136406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4421.974214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.235689] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4429.388950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4429.399569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4429.609406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4439.414935] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4446.001579] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4446.301111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4452.743996] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4453.109169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4460.516057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.548564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.554422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4461.787945] Lustre: server umount lustre-OST0000 complete [ 4468.704518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4468.713248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.725924] Lustre: Skipped 1 previous similar message [ 4468.729362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4468.731500] Lustre: Skipped 1 previous similar message [ 4470.763903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4470.764516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.721415] Lustre: server umount lustre-MDT0000 complete [ 4477.222202] LustreError: 134430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844976 with bad export cookie 12005347227195822847 [ 4477.231083] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.241737] LustreError: 134430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4477.689924] Lustre: server umount lustre-MDT0001 complete [ 4487.155121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.535545] LustreError: 138623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.553206] LustreError: 138623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4487.656835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4492.710194] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4492.771211] LustreError: 138624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.890883] LustreError: 138623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.890294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.009252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4503.011317] Lustre: Skipped 1 previous similar message [ 4503.166429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4509.416768] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4518.395929] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4525.264701] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4533.491705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4533.941701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4541.479492] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4550.489225] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4556.023588] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4558.522687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4564.145853] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4564.413237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4575.471071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4575.665649] Lustre: lustre-OST0001: new disk, initializing [ 4575.671528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4575.680830] Lustre: 142011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4575.804903] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4583.639777] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4584.592214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4584.608334] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4584.751413] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4592.853166] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4593.047718] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4597.580855] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4597.903428] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4603.852399] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f74454f4800.ost_server_uuid 50 [ 4606.052306] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f74454f4800.ost_server_uuid in IDLE state after 0 sec [ 4620.791604] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4620.797405] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.808218] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4620.815468] LustreError: Skipped 1 previous similar message [ 4620.864578] Lustre: Skipped 1 previous similar message [ 4625.008950] Lustre: server umount lustre-OST0001 complete [ 4625.895909] LustreError: 140446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.912943] LustreError: 140446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4631.015761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4631.019119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4631.022179] LustreError: 140447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.054093] Lustre: Skipped 1 previous similar message [ 4631.099656] Lustre: Skipped 1 previous similar message [ 4637.331628] Lustre: server umount lustre-OST0000 complete [ 4646.369528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4646.402606] Lustre: Skipped 1 previous similar message [ 4649.577169] Lustre: server umount lustre-MDT0000 complete [ 4651.488969] LustreError: 143313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.545375] LustreError: 143313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4653.518814] LustreError: 138603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845153 with bad export cookie 12005347227195823736 [ 4653.520600] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4653.525438] LustreError: 138603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4654.102388] Lustre: server umount lustre-MDT0001 complete [ 4678.147923] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4681.135455] Key type lgssc unregistered [ 4681.484462] LNet: 145033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4681.493126] LNetError: 145033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4681.511306] LNet: Removed LNI 192.168.202.152@tcp [ 4682.538761] Key type .llcrypt unregistered [ 4682.542429] Key type ._llcrypt unregistered [ 4731.710456] Key type ._llcrypt registered [ 4731.717755] Key type .llcrypt registered [ 4731.899980] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4749.638827] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4751.011345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4751.065995] alg: No test for adler32 (adler32-zlib) [ 4752.299308] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 4752.607834] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4754.351256] Key type lgssc registered [ 4755.439217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4795.014952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4795.045509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.282202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4796.309550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4796.398457] Lustre: lustre-MDT0000: new disk, initializing [ 4796.518430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4796.537945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4802.138922] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4814.459316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4814.619247] Lustre: 149189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4814.680412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4814.684632] Lustre: Skipped 1 previous similar message [ 4814.770758] Lustre: lustre-MDT0001: new disk, initializing [ 4814.879955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4814.920962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4814.937645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4819.761895] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4830.258822] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4836.720215] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4843.848104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4844.139195] Lustre: lustre-OST0000: new disk, initializing [ 4844.142909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4844.150314] Lustre: 151161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4844.298527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4850.286158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4850.303858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4850.444927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4850.753206] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4860.796807] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4865.975705] Lustre: Failing over lustre-OST0000 [ 4866.024752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4866.035624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.038764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4866.058124] Lustre: Skipped 1 previous similar message [ 4866.082373] Lustre: Skipped 1 previous similar message [ 4866.374263] Lustre: server umount lustre-OST0000 complete [ 4884.952874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4885.220606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4885.240196] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4886.260766] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4892.261544] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4895.265341] Lustre: lustre-OST0000: Recovery over after 0:09, of 3 clients 3 recovered and 0 were evicted. [ 4895.268676] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4897.094354] Lustre: Failing over lustre-MDT0000 [ 4897.434657] Lustre: server umount lustre-MDT0000 complete [ 4901.856952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4901.862831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.875520] LustreError: Skipped 1 previous similar message [ 4901.905764] Lustre: Skipped 2 previous similar messages [ 4905.396862] LustreError: 149197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4905.417726] LustreError: 149197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4906.992264] LustreError: 149199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.001785] LustreError: 149199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4910.515440] LustreError: 149197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4915.469185] LustreError: 149198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4915.487146] LustreError: 149198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4917.629907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4917.767642] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4918.010192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.066769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4920.769055] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4923.098338] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4923.387473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4923.403591] Lustre: Skipped 1 previous similar message [ 4923.430697] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4923.478882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4933.604448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.610237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4933.630027] Lustre: Skipped 1 previous similar message [ 4938.102695] Lustre: server umount lustre-OST0000 complete [ 4943.843443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4943.843671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.853679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4943.853688] Lustre: Skipped 1 previous similar message [ 4943.889413] Lustre: Skipped 1 previous similar message [ 4948.552828] Lustre: server umount lustre-MDT0000 complete [ 4948.962354] LustreError: 149203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4948.981649] LustreError: 149203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4951.901623] LustreError: 151160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845451 with bad export cookie 5065446038632511583 [ 4951.908430] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4951.915668] LustreError: 151160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4952.338281] Lustre: server umount lustre-MDT0001 complete [ 4965.799850] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4969.543732] Key type lgssc unregistered [ 4969.877910] LNet: 154817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4969.884834] LNetError: 154817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4969.909671] LNet: Removed LNI 192.168.202.152@tcp [ 4971.420538] Key type .llcrypt unregistered [ 4971.425515] Key type ._llcrypt unregistered [ 5013.170958] Key type ._llcrypt registered [ 5013.185941] Key type .llcrypt registered [ 5013.395127] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5014.678317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5014.765629] alg: No test for adler32 (adler32-zlib) [ 5015.922207] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 5016.142825] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5017.791240] Key type lgssc registered [ 5018.885048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5027.853482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5027.880438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5029.364518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5034.471831] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5038.563414] LustreError: 156340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5038.577224] LustreError: 156340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5043.688295] LustreError: 156339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5044.713968] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5048.805876] LustreError: 156340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.458525] LustreError: 156368:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5049.468341] LustreError: 156368:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5049.478292] LustreError: 156368:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 5050.136762] Lustre: server umount lustre-MDT0000 complete [ 5067.764764] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5079.607913] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5122.688733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.972634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5123.000829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5123.095258] Lustre: lustre-MDT0000: new disk, initializing [ 5123.186695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.202832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5128.902408] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5140.684340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.881056] Lustre: 159673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5140.994356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5141.009026] Lustre: Skipped 1 previous similar message [ 5141.147953] Lustre: lustre-MDT0001: new disk, initializing [ 5141.305342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5141.365988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5141.395849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5148.046783] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5159.084935] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5166.115602] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5174.018593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5174.499761] Lustre: lustre-OST0000: new disk, initializing [ 5174.506829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5174.515594] Lustre: 161641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5174.617811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5182.358406] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5183.044507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5183.056321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5183.223565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5193.726591] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5198.492565] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5200.781670] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:48:18 (1787845698) [ 5490.656196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5490.663186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5490.688902] Lustre: Skipped 1 previous similar message [ 5490.710760] Lustre: Skipped 2 previous similar messages [ 5495.785033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5495.792135] Lustre: Skipped 2 previous similar messages [ 5496.193311] Lustre: server umount lustre-MDT0000 complete [ 5500.239424] LustreError: 159665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845999 with bad export cookie 14544933311163923366 [ 5500.240487] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5500.254844] LustreError: 159665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5500.895829] LustreError: 159679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5500.897196] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.932676] Lustre: Skipped 1 previous similar message [ 5500.937148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5506.017688] LustreError: 159681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.017886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5507.024548] Lustre: server umount lustre-MDT0001 complete [ 5525.983278] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5526.124636] Lustre: server umount lustre-OST0000 complete [ 5536.575824] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5549.704880] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5591.621219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.963052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5591.992893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5592.064732] Lustre: lustre-MDT0000: new disk, initializing [ 5592.143625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5592.160858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5597.064617] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5608.755041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5608.951662] Lustre: 166315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5609.030128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5609.036471] Lustre: Skipped 1 previous similar message [ 5609.154325] Lustre: lustre-MDT0001: new disk, initializing [ 5609.274154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5609.347334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5609.367055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5614.039792] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5625.584019] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5632.667923] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5640.001209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5640.305834] Lustre: lustre-OST0000: new disk, initializing [ 5640.313395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5640.323283] Lustre: 168286:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5640.475457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5647.475234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5647.493978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5647.645840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5648.376274] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5659.529774] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5664.753781] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5664.987661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5669.955684] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5670.382360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5673.440776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5673.458097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.465299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5673.469139] Lustre: Skipped 1 previous similar message [ 5678.572119] Lustre: server umount lustre-OST0000 complete [ 5683.097580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5683.100439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5683.105929] Lustre: Skipped 2 previous similar messages [ 5683.114289] Lustre: Skipped 1 previous similar message [ 5689.511227] Lustre: server umount lustre-MDT0000 complete [ 5693.408585] LustreError: 166328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.439348] LustreError: 166328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5693.574779] LustreError: 166308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846193 with bad export cookie 14544933311164082196 [ 5693.577639] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5693.598573] LustreError: 166308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5694.079435] Lustre: server umount lustre-MDT0001 complete [ 5705.006743] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:56:42 (1787846202) [ 5723.564667] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5734.931924] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5778.369189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.755028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5778.807766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5778.934067] Lustre: lustre-MDT0000: new disk, initializing [ 5779.060499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.077913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5784.672655] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5797.327519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.498843] Lustre: 173043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5797.802806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5803.978244] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5815.552966] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5823.691846] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5831.503700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5831.803772] Lustre: lustre-OST0000: new disk, initializing [ 5831.806708] Lustre: Skipped 1 previous similar message [ 5831.810396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5831.816607] Lustre: Skipped 2 previous similar messages [ 5831.830335] Lustre: 175023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5838.063970] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5840.512369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5840.525196] Lustre: Skipped 1 previous similar message [ 5840.544014] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5840.749240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5851.442475] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5861.359875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5861.365182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5861.383469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5861.383479] Lustre: Skipped 2 previous similar messages [ 5861.408167] LustreError: Skipped 2 previous similar messages [ 5866.808257] Lustre: server umount lustre-OST0000 complete [ 5871.586362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5871.593157] Lustre: Skipped 1 previous similar message [ 5885.407136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5885.576740] Lustre: server umount lustre-MDT0000 complete [ 5886.948746] LustreError: 173052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5886.969028] LustreError: 173052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5890.293142] LustreError: 173038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846389 with bad export cookie 14544933311164083078 [ 5890.303714] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5890.856744] Lustre: server umount lustre-MDT0001 complete [ 5902.244994] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:59:59 (1787846399) [ 5920.818124] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5935.650266] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5979.091263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.322419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5979.367400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5979.463176] Lustre: lustre-MDT0000: new disk, initializing [ 5979.601091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5979.605752] Lustre: Skipped 2 previous similar messages [ 5979.624908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5984.873976] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5996.541913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5996.676556] Lustre: 179379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5996.899880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6001.644694] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6011.997524] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6019.066553] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6026.500244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6026.763469] Lustre: 181357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6034.251852] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6035.529119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6035.729490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6044.821640] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6056.419354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6056.422593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.433960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6056.447353] Lustre: Skipped 2 previous similar messages [ 6056.452518] Lustre: Skipped 9 previous similar messages [ 6067.167216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6067.516353] Lustre: server umount lustre-OST0000 complete [ 6073.832635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6073.845558] LustreError: Skipped 1 previous similar message [ 6078.514865] Lustre: server umount lustre-MDT0000 complete [ 6082.019960] LustreError: 179385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6082.032915] LustreError: 179385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6082.643419] LustreError: 180189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846582 with bad export cookie 14544933311164084163 [ 6082.646231] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6082.651930] LustreError: 180189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6083.014600] Lustre: server umount lustre-MDT0001 complete [ 6097.839920] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6101.185273] Key type lgssc unregistered [ 6101.584741] LNet: 183216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6101.589781] LNetError: 183216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6101.603886] LNet: Removed LNI 192.168.202.152@tcp [ 6102.696320] Key type .llcrypt unregistered [ 6102.697987] Key type ._llcrypt unregistered [ 6130.058916] Key type ._llcrypt registered [ 6130.061499] Key type .llcrypt registered [ 6130.164495] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6131.860628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6131.890697] alg: No test for adler32 (adler32-zlib) [ 6133.118608] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 6133.355934] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6135.039716] Key type lgssc registered [ 6136.403911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6144.581318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6144.608471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6146.237533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.433515] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6155.760123] LustreError: 184635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.798136] LustreError: 184635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6160.006025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6160.426842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6165.109338] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6174.144131] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6180.908330] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6188.194911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6188.675964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6190.742562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6196.549875] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6206.132665] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6226.931102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.936670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.946081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6226.953781] Lustre: Skipped 1 previous similar message [ 6227.003243] Lustre: Skipped 1 previous similar message [ 6232.061693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6232.065885] Lustre: Skipped 1 previous similar message [ 6232.877384] Lustre: server umount lustre-OST0000 complete [ 6242.284713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.287498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6242.290552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6242.297620] Lustre: Skipped 1 previous similar message [ 6242.301554] LustreError: Skipped 1 previous similar message [ 6243.746091] Lustre: server umount lustre-MDT0000 complete [ 6247.399791] LustreError: 186560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.429611] LustreError: 186560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6248.024635] LustreError: 184616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846747 with bad export cookie 1187664911188026411 [ 6248.034195] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6248.038212] LustreError: 184616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6248.241911] Lustre: server umount lustre-MDT0001 complete [ 6260.165585] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 6263.147683] Key type lgssc unregistered [ 6263.425772] LNet: 188202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6263.429294] LNetError: 188202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6263.457285] LNet: Removed LNI 192.168.202.152@tcp [ 6264.146500] Key type .llcrypt unregistered [ 6264.148569] Key type ._llcrypt unregistered [ 6295.993907] Key type ._llcrypt registered [ 6295.995876] Key type .llcrypt registered [ 6296.074292] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6296.767047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6296.890066] alg: No test for adler32 (adler32-zlib) [ 6297.904307] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 6298.071390] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 6299.728109] Key type lgssc registered [ 6301.496260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6311.794944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6311.840849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.368553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6319.178693] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6323.681513] LustreError: 189620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6323.717577] LustreError: 189620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6328.804928] LustreError: 189619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.578336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6332.116939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6338.161812] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6348.264340] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6355.890241] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6363.955891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6364.534776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6372.721500] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6381.992528] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6384.010770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6393.818493] Lustre: lustre-OST0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 6394.340547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6394.344829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6394.349093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6394.350518] Lustre: Skipped 1 previous similar message [ 6394.373659] Lustre: Skipped 1 previous similar message [ 6398.902833] Lustre: lustre-OST0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 6399.400860] Lustre: server umount lustre-OST0000 complete [ 6409.515722] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f7446783800.ost_server_uuid 50 [ 6411.763833] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6429.664945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6429.667846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6429.692242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6432.019817] Lustre: server umount lustre-MDT0000 complete [ 6435.298168] LustreError: 189616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6435.320831] LustreError: 189616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6437.127946] LustreError: 189601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846936 with bad export cookie 17779852196504242572 [ 6437.135859] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6437.142906] LustreError: 189601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6437.491559] Lustre: server umount lustre-MDT0001 complete [ 6450.748478] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:09:08 (1787846948) [ 6458.447308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6459.382967] LustreError: 193537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.522358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6464.185655] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6465.004815] LustreError: 193538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.137937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6473.490570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6478.879870] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6486.207479] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6494.253977] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6504.460085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.053198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6512.432897] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6519.293685] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6526.745216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.980145] Lustre: lustre-OST0001: new disk, initializing [ 6526.984177] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6526.989791] Lustre: 196380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6527.085229] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6530.158863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6533.858228] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6535.809940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6535.819855] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6535.989079] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6545.259316] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6549.473652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6549.480444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.487925] Lustre: Skipped 1 previous similar message [ 6549.493861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6549.499224] Lustre: Skipped 2 previous similar messages [ 6554.757914] Lustre: server umount lustre-OST0000 complete [ 6556.594293] LustreError: 195360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6556.626802] LustreError: 195360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6562.498692] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6562.710500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6567.836976] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6568.041212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6577.114615] LustreError: 195361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6577.160173] LustreError: 195361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6587.360079] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6587.368436] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.374154] LustreError: Skipped 2 previous similar messages [ 6587.377420] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6587.377429] Lustre: Skipped 3 previous similar messages [ 6587.445788] Lustre: Skipped 2 previous similar messages [ 6590.867351] Lustre: server umount lustre-OST0001 complete [ 6596.576181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6596.584850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.842291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6607.854409] Lustre: Skipped 7 previous similar messages [ 6610.399249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6610.669834] Lustre: server umount lustre-MDT0000 complete [ 6612.961078] LustreError: 193533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6612.987857] LustreError: 193533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6614.487206] LustreError: 193518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847114 with bad export cookie 17779852196504243643 [ 6614.491334] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6614.512510] LustreError: 193518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6614.991110] Lustre: server umount lustre-MDT0001 complete [ 6643.329938] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:12:20 (1787847140) [ 6650.607298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6650.911185] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6650.957094] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6651.202615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6656.271709] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6665.961898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.026635] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6666.048579] Lustre: 199831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6671.540938] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6681.564615] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6688.393850] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6695.853513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6695.954618] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6702.117885] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6702.232235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6711.142573] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6718.611069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.694180] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6718.906935] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6718.913557] Lustre: Skipped 2 previous similar messages [ 6725.664160] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6725.925448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6733.081198] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6736.359266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.361993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.380599] Lustre: Skipped 2 previous similar messages [ 6736.400287] Lustre: Skipped 1 previous similar message [ 6742.303835] Lustre: server umount lustre-OST0000 complete [ 6743.994461] LustreError: 202064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6761.955057] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.961486] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6761.981594] Lustre: Skipped 1 previous similar message [ 6765.313235] Lustre: server umount lustre-OST0001 complete [ 6775.813579] Lustre: server umount lustre-MDT0000 complete [ 6780.297270] LustreError: 201700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847279 with bad export cookie 17779852196504245050 [ 6780.298358] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6780.313918] LustreError: 201700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6780.645135] Lustre: server umount lustre-MDT0001 complete [ 6809.668746] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:15:07 (1787847307) [ 6829.175290] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 6839.832251] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 6883.092232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.323640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6883.355988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6883.482610] Lustre: lustre-MDT0000: new disk, initializing [ 6883.598620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6883.622759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6889.181938] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6902.487101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6902.612416] Lustre: 207925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6902.683865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6902.687182] Lustre: Skipped 1 previous similar message [ 6902.772320] Lustre: lustre-MDT0001: new disk, initializing [ 6902.947237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6902.965868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6908.391804] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6919.390837] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6925.386961] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6933.034484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.436416] Lustre: lustre-OST0000: new disk, initializing [ 6933.440893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6933.451475] Lustre: 209898:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6939.279078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6939.289428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6939.410048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6940.497954] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 6950.079635] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6954.928984] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6955.174597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6959.454952] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6959.750001] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6965.216598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6965.225722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.234227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6965.236206] Lustre: Skipped 3 previous similar messages [ 6965.239143] Lustre: Skipped 7 previous similar messages [ 6967.918848] Lustre: server umount lustre-OST0000 complete [ 6978.586684] Lustre: server umount lustre-MDT0000 complete [ 6980.582540] LustreError: 207938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6980.616228] LustreError: 207938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6982.862794] LustreError: 209897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847482 with bad export cookie 17779852196504246828 [ 6982.866167] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6982.868790] LustreError: 209897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6983.401838] Lustre: server umount lustre-MDT0001 complete [ 6991.890888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.447611] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7006.315233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7011.698437] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7020.943970] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7027.312783] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7034.236575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7041.610866] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7049.333634] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7054.079452] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7054.261763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7058.896850] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7059.139991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7062.496627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7062.501986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7062.511687] LustreError: Skipped 2 previous similar messages [ 7062.530435] Lustre: Skipped 3 previous similar messages [ 7067.534199] Lustre: server umount lustre-OST0000 complete [ 7074.945980] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7080.199655] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7110.927986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.180292] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7127.842080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7127.868123] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7127.890090] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7129.998452] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7131.641949] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7138.278529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7138.279729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7138.291083] LustreError: Skipped 1 previous similar message [ 7138.304331] Lustre: Skipped 7 previous similar messages [ 7142.609449] Lustre: server umount lustre-OST0000 complete [ 7161.311761] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7161.514878] Lustre: server umount lustre-MDT0000 complete [ 7165.607138] LustreError: 212849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847665 with bad export cookie 17779852196504247717 [ 7165.613773] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7165.619155] LustreError: 212849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7165.977854] Lustre: server umount lustre-MDT0001 complete [ 7175.104970] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:21:13 (1787847673) [ 7182.313920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7182.852257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7182.860183] Lustre: Skipped 6 previous similar messages [ 7188.646106] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7200.349933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.573482] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7214.744559] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7221.232213] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7229.205737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7236.675941] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7245.069535] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7250.290362] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7253.748644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7259.090584] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7259.470905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7267.091495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7267.284161] Lustre: lustre-OST0001: new disk, initializing [ 7267.287632] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7267.292440] Lustre: 221478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7274.164656] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7275.625883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7275.632241] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7275.797683] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7283.970284] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7289.568907] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7289.858407] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7295.449285] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7295.736306] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7301.616341] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7301.626488] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.636662] LustreError: Skipped 1 previous similar message [ 7301.661544] Lustre: Skipped 5 previous similar messages [ 7304.594797] Lustre: server umount lustre-OST0001 complete [ 7306.746950] LustreError: 219907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.782183] LustreError: 219907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7312.704644] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7313.060271] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7319.023345] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7319.227787] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7350.054629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7355.399899] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7355.409828] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7355.410443] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7355.453968] Lustre: Skipped 2 previous similar messages [ 7358.862477] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7368.699626] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7370.601855] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7379.025714] Lustre: server umount lustre-OST0001 complete [ 7399.904804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7399.907060] Lustre: Skipped 10 previous similar messages [ 7403.368615] Lustre: server umount lustre-OST0000 complete [ 7414.508822] Lustre: server umount lustre-MDT0000 complete [ 7419.102471] LustreError: 218064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847918 with bad export cookie 17779852196504248683 [ 7419.121193] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7426.077804] Lustre: server umount lustre-MDT0001 complete [ 7455.320763] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:25:53 (1787847953) [ 7462.707989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.001746] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7463.050825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7468.904745] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7479.888042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.014688] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7480.051250] Lustre: 226722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7485.552449] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7495.307192] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7503.533130] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7512.563067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.783040] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7520.889292] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7530.420028] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7541.016058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.065690] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7548.820595] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7558.933264] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7561.187893] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7567.310807] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7567.618540] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7572.538743] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7572.775705] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7580.830145] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f7447d5c800.ost_server_uuid 50 [ 7582.972967] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f7447d5c800.ost_server_uuid in IDLE state after 0 sec [ 7584.164906] Lustre: Permanently deactivating lustre-OST0001 [ 7584.204642] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7589.153905] Lustre: Permanently reactivating lustre-OST0001 [ 7589.167818] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7589.175356] Lustre: Skipped 2 previous similar messages [ 7594.992973] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7595.004959] LustreError: Skipped 2 previous similar messages [ 7595.006322] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7595.024639] Lustre: Skipped 6 previous similar messages [ 7609.318268] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7609.634693] Lustre: server umount lustre-OST0001 complete [ 7620.966758] Lustre: server umount lustre-OST0000 complete [ 7633.374231] Lustre: server umount lustre-MDT0000 complete [ 7638.400878] LustreError: 229603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848138 with bad export cookie 17779852196504249971 [ 7638.403203] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7638.426031] LustreError: 229603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7639.165145] Lustre: server umount lustre-MDT0001 complete [ 7670.421725] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:29:28 (1787848168) [ 7680.796140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7681.063862] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7681.107429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7681.122979] Lustre: Skipped 2 previous similar messages [ 7686.128091] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7696.535318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7696.655515] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7696.702455] Lustre: 233430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7697.054803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7697.070805] Lustre: Skipped 9 previous similar messages [ 7701.967470] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7710.459447] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7717.040924] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7724.436628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7724.719960] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7724.746737] Lustre: Permanently deactivating lustre-OST0000 [ 7724.782218] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7731.250977] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7732.405462] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7740.579350] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7748.162044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7748.271431] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7755.525015] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7756.106393] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7765.766833] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7768.678319] Lustre: Permanently reactivating lustre-OST0000 [ 7768.700161] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7768.704844] Lustre: Skipped 2 previous similar messages [ 7774.870110] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7774.884835] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7774.894422] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7774.948196] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7774.991099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7791.730366] Lustre: server umount lustre-OST0001 complete [ 7802.055892] Lustre: server umount lustre-OST0000 complete [ 7812.752875] Lustre: server umount lustre-MDT0000 complete [ 7817.565786] LustreError: 233422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848317 with bad export cookie 17779852196504251637 [ 7817.568035] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7817.574306] LustreError: 233422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7817.898416] Lustre: server umount lustre-MDT0001 complete [ 7832.248602] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7835.488026] Key type lgssc unregistered [ 7835.795689] LNet: 238313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7835.800301] LNetError: 238313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7835.815878] LNet: Removed LNI 192.168.202.152@tcp [ 7836.799465] Key type .llcrypt unregistered [ 7836.806782] Key type ._llcrypt unregistered [ 7867.036796] Key type ._llcrypt registered [ 7867.038126] Key type .llcrypt registered [ 7867.192413] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7868.194295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7868.420901] alg: No test for adler32 (adler32-zlib) [ 7869.551858] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 7869.759990] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7871.479199] Key type lgssc registered [ 7872.432200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7884.105199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7884.118894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7885.673716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7892.153043] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7896.051390] LustreError: 239779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7896.086902] LustreError: 239779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7901.169844] LustreError: 239780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.211626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7903.381575] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7903.480661] Lustre: 239766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7903.499728] Lustre: Permanently deactivating lustre-MDT0001 [ 7903.530658] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7904.019322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7909.404596] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7910.874589] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7921.590974] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7928.245307] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7937.469180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7938.019618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7946.312640] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7949.298367] LustreError: 241624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7949.316839] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7954.400167] LustreError: 241624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7955.475770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7956.009631] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7960.555090] LustreError: 241623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.577126] LustreError: 241623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7965.410846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7965.643645] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7970.845034] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7970.855745] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7975.156722] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7986.411275] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7989.617700] Lustre: Permanently reactivating lustre-MDT0001 [ 7989.641711] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7989.648252] Lustre: Skipped 2 previous similar messages [ 7996.527291] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7996.541528] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7996.551870] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7996.566770] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7996.909983] Lustre: lustre-MDT0001: Client ea38235c-ea14-4047-80d9-68ce7acdc599 (at 192.168.202.52@tcp) reconnecting [ 8004.889948] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8007.157219] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8014.354691] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8014.688640] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8022.692232] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8022.914545] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8025.810797] Lustre: Permanently deactivating lustre-MDT0001 [ 8025.825524] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8025.833372] Lustre: Skipped 2 previous similar messages [ 8034.932581] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8047.589923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8047.600722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8047.618551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8047.621824] Lustre: Skipped 2 previous similar messages [ 8047.647198] Lustre: Skipped 3 previous similar messages [ 8051.120488] Lustre: server umount lustre-MDT0000 complete [ 8052.724318] LustreError: 239780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8052.749412] LustreError: 239780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8055.960911] LustreError: 239760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848555 with bad export cookie 8536210552604328152 [ 8055.971918] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8055.980601] LustreError: 239760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8056.439673] Lustre: server umount lustre-MDT0001 complete [ 8063.593405] Lustre: server umount lustre-OST0000 complete [ 8077.837321] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:36:15 (1787848575) [ 8093.756588] Lustre: server umount lustre-OST0001 complete [ 8103.359186] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8114.837605] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8159.320510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8159.544348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8159.549761] Lustre: Skipped 2 previous similar messages [ 8159.567085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8159.628904] Lustre: lustre-MDT0000: new disk, initializing [ 8159.692816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8159.715853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8165.518243] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8178.538991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8178.783964] Lustre: 248098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8178.920947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8178.931447] Lustre: Skipped 1 previous similar message [ 8179.029976] Lustre: lustre-MDT0001: new disk, initializing [ 8179.233131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8179.317647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8179.332968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8186.240885] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8198.820468] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8206.582525] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8215.170156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8215.442936] Lustre: lustre-OST0000: new disk, initializing [ 8215.448864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8215.455860] Lustre: 250068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8215.560126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8223.091803] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8225.349767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8225.358254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8225.480343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8235.676883] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8238.207432] LustreError: 248105:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8248.270671] LustreError: 248105:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8248.317426] LustreError: 248104:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8249.209225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8249.418841] Lustre: lustre-OST0001: new disk, initializing [ 8249.422572] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8249.428530] Lustre: 251183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8249.531093] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8256.910416] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8257.700532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8257.717409] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8257.846291] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8258.327123] LustreError: 248104:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8268.614257] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8270.980737] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8278.499897] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.528348] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8278.533139] Lustre: Skipped 2 previous similar messages [ 8278.551412] Lustre: Skipped 1 previous similar message [ 8279.779163] Lustre: server umount lustre-OST0001 complete [ 8283.620185] LustreError: 250532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8302.567403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8302.584797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8302.611549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8302.625702] Lustre: Skipped 1 previous similar message [ 8306.486728] Lustre: server umount lustre-OST0000 complete [ 8312.319939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8312.333663] LustreError: Skipped 1 previous similar message [ 8312.347828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8312.371222] Lustre: Skipped 1 previous similar message [ 8312.385123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8312.394729] Lustre: Skipped 2 previous similar messages [ 8318.700245] Lustre: server umount lustre-MDT0000 complete [ 8319.456900] LustreError: 248109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8319.483339] LustreError: 248109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8323.392351] LustreError: 251182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848823 with bad export cookie 8536210552604330966 [ 8323.408345] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8323.409384] LustreError: 251182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8330.039683] Lustre: server umount lustre-MDT0001 complete [ 8343.208799] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8346.334289] Key type lgssc unregistered [ 8346.770505] LNet: 253220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8346.780582] LNetError: 253220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8346.806226] LNet: Removed LNI 192.168.202.152@tcp [ 8347.910401] Key type .llcrypt unregistered [ 8347.912891] Key type ._llcrypt unregistered [ 8394.133972] Key type ._llcrypt registered [ 8394.138932] Key type .llcrypt registered [ 8394.309605] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8396.208198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8396.350877] alg: No test for adler32 (adler32-zlib) [ 8397.509936] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 8397.901170] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 8399.631194] Key type lgssc registered [ 8401.038583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8411.599826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8411.645205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8413.109038] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8413.142337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8413.425541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.524884] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8430.334920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8430.483326] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8430.527913] Lustre: 255433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8430.919248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8436.500179] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8445.867363] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8452.958652] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8462.251541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8462.397862] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8462.611385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8470.223591] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8471.729343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8479.541374] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8538.600603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8538.618092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8538.634528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8538.647653] Lustre: Skipped 1 previous similar message [ 8538.683283] Lustre: Skipped 1 previous similar message [ 8543.471705] Lustre: server umount lustre-OST0000 complete [ 8549.020735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8565.606705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8566.455464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8575.375494] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8578.737316] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8578.755596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8578.763380] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8585.436481] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8628.194514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.203475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8628.224372] Lustre: Skipped 1 previous similar message [ 8628.247188] Lustre: Skipped 1 previous similar message [ 8633.231094] Lustre: server umount lustre-OST0000 complete [ 8638.434727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8638.435394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8638.444773] Lustre: Skipped 1 previous similar message [ 8643.553167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8643.568721] Lustre: Skipped 2 previous similar messages [ 8644.060467] Lustre: server umount lustre-MDT0000 complete [ 8648.735150] LustreError: 255425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849148 with bad export cookie 9219955445876850499 [ 8648.736363] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8648.744595] LustreError: 255425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8649.502700] Lustre: server umount lustre-MDT0001 complete [ 8664.311128] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8668.456747] Key type lgssc unregistered [ 8668.749974] LNet: 260859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8668.756927] LNetError: 260859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8668.797632] LNet: Removed LNI 192.168.202.152@tcp [ 8669.732762] Key type .llcrypt unregistered [ 8669.734571] Key type ._llcrypt unregistered [ 8698.965932] Key type ._llcrypt registered [ 8698.972483] Key type .llcrypt registered [ 8699.200390] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8701.053169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8701.132753] alg: No test for adler32 (adler32-zlib) [ 8702.159341] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 8702.564115] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 8704.427711] Key type lgssc registered [ 8705.861708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8715.106936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8715.145968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.909311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8723.814738] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8726.508932] LustreError: 262279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8726.545897] LustreError: 262279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8731.617788] LustreError: 262280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8735.162306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8735.741270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8740.687392] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8751.727563] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8758.295685] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8767.957338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8768.434620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8777.405406] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8786.899423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8788.021651] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8817.635863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8817.643239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8817.646287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8817.662295] Lustre: Skipped 1 previous similar message [ 8822.764508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8822.780538] Lustre: Skipped 1 previous similar message [ 8823.530771] Lustre: server umount lustre-OST0000 complete [ 8832.992418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8832.992443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8832.995178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8832.995186] Lustre: Skipped 1 previous similar message [ 8833.006634] Lustre: Skipped 1 previous similar message [ 8833.009151] LustreError: Skipped 1 previous similar message [ 8838.115049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8838.122829] Lustre: Skipped 1 previous similar message [ 8843.234252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8843.240261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8843.275062] Lustre: Skipped 1 previous similar message [ 8843.615748] Lustre: server umount lustre-MDT0000 complete [ 8847.332769] LustreError: 262279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8847.364994] LustreError: 262279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8849.022263] LustreError: 262261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849348 with bad export cookie 12445075464728080535 [ 8849.032745] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8849.046605] LustreError: 262261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8849.952639] Lustre: server umount lustre-MDT0001 complete [ 8863.807976] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8867.800777] Key type lgssc unregistered [ 8868.181100] LNet: 266592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8868.189214] LNetError: 266592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8868.215084] LNet: Removed LNI 192.168.202.152@tcp [ 8869.408499] Key type .llcrypt unregistered [ 8869.417935] Key type ._llcrypt unregistered [ 8889.390970] Key type ._llcrypt registered [ 8889.399600] Key type .llcrypt registered [ 8889.609584] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8891.622217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8891.645945] alg: No test for adler32 (adler32-zlib) [ 8892.975464] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 8893.287536] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 8895.103156] Key type lgssc registered [ 8896.916480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8906.585306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8906.616406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8908.355341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8913.762886] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8917.542239] LustreError: 267815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8917.581253] LustreError: 267815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8922.594428] LustreError: 267816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8923.634653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8924.081340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8931.490541] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8942.462662] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8951.856354] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8962.142559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8962.705534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8971.837715] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8978.045127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8983.252409] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8998.893678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8998.899514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8998.931547] Lustre: Skipped 1 previous similar message [ 8998.935649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8998.949826] Lustre: Skipped 1 previous similar message [ 9002.955280] Lustre: server umount lustre-OST0000 complete [ 9009.133773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9009.163296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9009.177492] Lustre: Skipped 1 previous similar message [ 9014.243729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9014.260090] Lustre: Skipped 1 previous similar message [ 9015.726448] Lustre: server umount lustre-MDT0000 complete [ 9018.338354] LustreError: 268544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9018.363681] LustreError: 268544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9020.834063] LustreError: 267797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849520 with bad export cookie 7073480699866608964 [ 9020.843208] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9020.847399] LustreError: 267797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9021.375124] Lustre: server umount lustre-MDT0001 complete [ 9040.599831] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 9044.769655] Key type lgssc unregistered [ 9045.269084] LNet: 271522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9045.282385] LNetError: 271522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9045.319694] LNet: Removed LNI 192.168.202.152@tcp [ 9046.550456] Key type .llcrypt unregistered [ 9046.559395] Key type ._llcrypt unregistered [ 9075.944441] Key type ._llcrypt registered [ 9075.946589] Key type .llcrypt registered [ 9076.064904] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9077.580687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9077.708600] alg: No test for adler32 (adler32-zlib) [ 9079.179393] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 9079.540599] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 9081.335463] Key type lgssc registered [ 9082.793252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9091.801895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9091.818390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.494075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.967392] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9103.845240] LustreError: 272943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9103.853044] LustreError: 272943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9108.966965] LustreError: 272942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9109.384178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9109.732656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9115.082941] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9123.603864] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9131.390059] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9139.329656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9139.828355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9148.838982] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9158.265655] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9163.384538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9194.474506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9194.485849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9194.501609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9194.521198] Lustre: Skipped 1 previous similar message [ 9194.548699] Lustre: Skipped 1 previous similar message [ 9199.594642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9199.611165] Lustre: Skipped 1 previous similar message [ 9200.192366] Lustre: server umount lustre-OST0000 complete [ 9204.705370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9204.721484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9204.736583] Lustre: Skipped 1 previous similar message [ 9209.825638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9209.862225] Lustre: Skipped 1 previous similar message [ 9211.118529] Lustre: server umount lustre-MDT0000 complete [ 9214.944287] LustreError: 272943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9214.959824] LustreError: 272943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9216.267170] LustreError: 272922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849715 with bad export cookie 15126567205508280974 [ 9216.277844] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9216.282226] LustreError: 272922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9216.920482] Lustre: server umount lustre-MDT0001 complete [ 9229.963440] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 9233.147920] Key type lgssc unregistered [ 9233.463128] LNet: 277272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9233.469959] LNetError: 277272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9233.489180] LNet: Removed LNI 192.168.202.152@tcp [ 9234.526677] Key type .llcrypt unregistered [ 9234.529764] Key type ._llcrypt unregistered [ 9254.180804] Key type ._llcrypt registered [ 9254.182950] Key type .llcrypt registered [ 9254.322618] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9255.522747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9256.133718] alg: No test for adler32 (adler32-zlib) [ 9257.179604] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 9257.484956] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 9259.127240] Key type lgssc registered [ 9260.711307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9270.424959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9270.455418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9272.085035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9278.140623] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9281.511539] LustreError: 278511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9281.541907] LustreError: 278511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9286.627401] LustreError: 278513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9288.919469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.273479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9294.841364] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9304.019936] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9310.800277] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9318.954201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9319.418377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9327.579607] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9337.210934] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9341.999883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9354.209146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9354.214649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.221988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9356.329169] Lustre: server umount lustre-OST0000 complete [ 9360.884107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9360.895110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9360.915988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9362.404142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9362.410183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9362.447487] Lustre: Skipped 1 previous similar message [ 9367.521855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9367.531973] Lustre: Skipped 1 previous similar message [ 9371.616897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9371.631722] Lustre: Skipped 1 previous similar message [ 9375.199221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9375.590556] Lustre: server umount lustre-MDT0000 complete [ 9376.736900] LustreError: 278497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9376.764284] LustreError: 278497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9379.820404] LustreError: 278478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849879 with bad export cookie 6081143120559826037 [ 9379.830766] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9379.835787] LustreError: 278478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9380.150793] Lustre: server umount lustre-MDT0001 complete [ 9394.603513] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 9397.963725] Key type lgssc unregistered [ 9398.401290] LNet: 282219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9398.422416] LNetError: 282219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9398.446244] LNet: Removed LNI 192.168.202.152@tcp [ 9399.526624] Key type .llcrypt unregistered [ 9399.530242] Key type ._llcrypt unregistered [ 9414.264086] Key type ._llcrypt registered [ 9414.266499] Key type .llcrypt registered [ 9414.396440] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9434.131397] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 9453.448355] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9455.224934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9455.317121] alg: No test for adler32 (adler32-zlib) [ 9456.496853] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 9456.825908] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 9458.531457] Key type lgssc registered [ 9459.986680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9506.166565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9506.215977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9507.636931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9507.664226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9507.764284] Lustre: lustre-MDT0000: new disk, initializing [ 9507.838378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.863664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9511.631439] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9525.801594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9526.039987] Lustre: 285776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9526.095242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9526.099381] Lustre: Skipped 1 previous similar message [ 9526.191717] Lustre: lustre-MDT0001: new disk, initializing [ 9526.326747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9526.367901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9526.381263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9531.922687] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9543.896179] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9551.753179] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9560.886844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9561.144689] Lustre: lustre-OST0000: new disk, initializing [ 9561.147643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9561.151871] Lustre: 287744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9561.237433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9568.207513] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9568.857503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9568.872638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9569.039622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9579.145316] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9585.296209] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9585.645600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9591.186868] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9591.440838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9594.867668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9594.873680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9594.880263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9594.885877] LustreError: Skipped 1 previous similar message [ 9594.928938] Lustre: Skipped 1 previous similar message [ 9599.979516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9599.998919] Lustre: Skipped 2 previous similar messages [ 9605.096805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9605.111609] Lustre: Skipped 1 previous similar message [ 9608.159690] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9608.408701] Lustre: server umount lustre-OST0000 complete [ 9615.331528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9615.347825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9615.353182] Lustre: Skipped 1 previous similar message [ 9615.365676] Lustre: Skipped 1 previous similar message [ 9620.124598] Lustre: server umount lustre-MDT0000 complete [ 9625.558278] LustreError: 287743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850125 with bad export cookie 3145552173397564547 [ 9625.566521] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9625.589737] LustreError: 287743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9626.079163] Lustre: server umount lustre-MDT0001 complete [ 9637.864804] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:02:15 (1787850135) [ 9646.285101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9646.781625] LustreError: 290171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9646.802651] LustreError: 290171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9646.851397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9651.733581] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9652.195854] LustreError: 290172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.314285] LustreError: 290171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9661.752860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9662.350589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9668.258196] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9676.787301] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9683.500889] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9691.263488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9691.688992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9698.892682] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9707.464482] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9722.340451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9722.352601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9722.367047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9722.370755] Lustre: Skipped 1 previous similar message [ 9724.624553] Lustre: server umount lustre-OST0000 complete [ 9728.995378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9728.999190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9729.005208] Lustre: Skipped 1 previous similar message [ 9732.578143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9732.594588] Lustre: Skipped 1 previous similar message [ 9743.327667] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9743.850874] Lustre: server umount lustre-MDT0000 complete [ 9747.946425] LustreError: 291110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.968859] LustreError: 291110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9749.574826] LustreError: 290906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850249 with bad export cookie 3145552173397565429 [ 9749.584684] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9749.587859] LustreError: 290906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9749.970718] Lustre: server umount lustre-MDT0001 complete [ 9768.872977] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 9773.724063] Key type lgssc unregistered [ 9774.043762] LNet: 293739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9774.052903] LNetError: 293739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9774.072307] LNet: Removed LNI 192.168.202.152@tcp [ 9774.944762] Key type .llcrypt unregistered [ 9774.948524] Key type ._llcrypt unregistered [ 9807.469606] Key type ._llcrypt registered [ 9807.471568] Key type .llcrypt registered [ 9807.535371] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 9826.459706] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9827.771718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9827.838339] alg: No test for adler32 (adler32-zlib) [ 9828.907716] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 9829.171892] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 9831.008536] Key type lgssc registered [ 9832.910038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9886.039149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9886.087228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9887.378829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9887.409599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9887.475792] Lustre: lustre-MDT0000: new disk, initializing [ 9887.552854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9887.566301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9891.268458] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9904.782545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9904.959869] Lustre: 297575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9905.035022] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9905.039282] Lustre: Skipped 1 previous similar message [ 9905.200166] Lustre: lustre-MDT0001: new disk, initializing [ 9905.458531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9905.532708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9905.549513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9911.020698] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9922.730340] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9929.180704] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9937.114237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9937.439927] Lustre: lustre-OST2710: new disk, initializing [ 9937.444088] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9937.453343] Lustre: 299544:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9937.592579] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9945.731487] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9946.704874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9946.720281] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9946.938520] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9957.282925] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9965.354370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9965.522351] Lustre: lustre-OST03e8: new disk, initializing [ 9965.524740] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9965.529459] Lustre: 300606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9965.636425] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9972.359914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9972.372373] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9972.481527] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9973.819490] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9982.666632] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9990.941703] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9991.288761] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9996.802203] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9997.025853] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10003.478953] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [10003.769197] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10009.651682] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [10010.110168] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10015.711381] Lustre: server umount lustre-MDT0000 complete [10018.272509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10018.280550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10018.792265] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10020.271831] LustreError: 299542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850519 with bad export cookie 8407848457026650253 [10020.302696] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10020.638953] Lustre: server umount lustre-MDT0001 complete [10036.364136] Lustre: server umount lustre-OST2710 complete [10050.338911] Lustre: server umount lustre-OST03e8 complete [10073.269262] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [10085.795685] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10129.024557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10129.311438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10129.357966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10129.537396] Lustre: lustre-MDT0000: new disk, initializing [10129.659517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10129.695411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10135.209815] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10148.496049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10148.738994] Lustre: 305862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10148.824253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10148.835469] Lustre: Skipped 1 previous similar message [10148.982503] Lustre: lustre-MDT0001: new disk, initializing [10149.138265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10149.211529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10149.234581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10155.489744] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10166.413796] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10172.976921] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10181.561918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10181.877742] Lustre: lustre-OST0000: new disk, initializing [10181.883977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10181.890079] Lustre: 307833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10182.046095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10189.919728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10189.935349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10189.960547] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10190.144832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10201.788948] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10207.608768] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10207.925459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10214.600753] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10214.762118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10221.030654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10221.040105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.044035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10221.047700] LustreError: Skipped 1 previous similar message [10221.071879] Lustre: Skipped 3 previous similar messages [10226.150076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10226.162221] Lustre: Skipped 2 previous similar messages [10231.263172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10231.274032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10231.290682] Lustre: Skipped 1 previous similar message [10231.525714] Lustre: server umount lustre-OST0000 complete [10241.504709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10241.513530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10241.529124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10241.560853] Lustre: Skipped 1 previous similar message [10243.442299] Lustre: server umount lustre-MDT0000 complete [10246.625113] LustreError: 307014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10246.648832] LustreError: 307014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10247.821015] LustreError: 305855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850747 with bad export cookie 8407848457026651933 [10247.825602] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10247.833020] LustreError: 305855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10248.227119] Lustre: server umount lustre-MDT0001 complete [10257.807421] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:12:35 (1787850755) [10259.594632] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10261.186870] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:12:39 (1787850759) [10265.565272] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10294.966149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10295.226672] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10295.248764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10295.578583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10300.731654] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10310.642964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.797635] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10310.858293] Lustre: 311693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10316.329492] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10341.111708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10341.538383] Lustre: Denying initial registration attempt from nid 192.168.202.152@tcp, specified as failover [10341.546060] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10341.568920] LustreError: 313147:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10341.575056] LustreError: 313147:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10341.585192] LustreError: 313147:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10341.721367] Lustre: server umount lustre-OST0000 complete [10341.723387] LustreError: 313147:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10352.098726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10352.110600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10352.133185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.143642] Lustre: Skipped 1 previous similar message [10363.876223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10363.884954] Lustre: Skipped 5 previous similar messages [10364.383541] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10364.564582] Lustre: server umount lustre-MDT0000 complete [10368.995384] LustreError: 311704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10369.026926] LustreError: 311704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10369.636957] LustreError: 311684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850869 with bad export cookie 8407848457026652815 [10369.651285] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10369.653042] LustreError: 311684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10370.084908] Lustre: server umount lustre-MDT0001 complete [10383.271338] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [10386.739216] Key type lgssc unregistered [10387.257320] LNet: 314274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10387.265623] LNetError: 314274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10387.297088] LNet: Removed LNI 192.168.202.152@tcp [10388.201218] Key type .llcrypt unregistered [10388.202829] Key type ._llcrypt unregistered [10402.759826] Key type ._llcrypt registered [10402.761856] Key type .llcrypt registered [10402.939206] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10404.430111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10404.554895] alg: No test for adler32 (adler32-zlib) [10405.742565] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [10406.061284] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [10407.751194] Key type lgssc registered [10409.244930] Lustre: Echo OBD driver; http://www.lustre.org/ [10461.849749] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10473.632695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10473.663611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10475.329844] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10475.374034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10475.683048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10481.427684] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10493.841759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10493.927716] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10493.962699] Lustre: 317043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10494.252217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10500.708853] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10509.779717] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10515.179871] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10534.704183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10534.987069] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10535.184502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10543.020560] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10550.816564] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10556.402785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10556.414268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.428072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10556.914120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10559.547341] Lustre: server umount lustre-OST0000 complete [10566.116701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10566.124404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10566.130139] Lustre: Skipped 1 previous similar message [10566.134123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10566.136752] Lustre: Skipped 1 previous similar message [10567.139956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10567.146600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10570.146095] Lustre: server umount lustre-MDT0000 complete [10574.414428] LustreError: 317036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851074 with bad export cookie 5531902732015158465 [10574.424223] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10574.431098] LustreError: 317036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10575.132486] Lustre: server umount lustre-MDT0001 complete [10588.107196] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [10591.542387] Key type lgssc unregistered [10591.941789] LNet: 320623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10591.953851] LNetError: 320623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10591.988249] LNet: Removed LNI 192.168.202.152@tcp [10592.946818] Key type .llcrypt unregistered [10592.950358] Key type ._llcrypt unregistered [10622.869824] Key type ._llcrypt registered [10622.871790] Key type .llcrypt registered [10622.982692] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10624.750739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10624.795812] alg: No test for adler32 (adler32-zlib) [10625.965226] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [10626.272773] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [10628.063979] Key type lgssc registered [10629.328395] Lustre: Echo OBD driver; http://www.lustre.org/ [10638.483986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10638.507111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10640.015788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10645.084773] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10649.577946] LustreError: 322040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10649.608397] LustreError: 322040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10654.694276] LustreError: 322041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10655.464416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10656.142226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10661.968901] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10669.967213] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10676.812387] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10687.095851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10687.497993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10695.259301] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10703.763422] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10721.252397] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [10722.785224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10722.800817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10722.838643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10723.212529] Lustre: server umount lustre-MDT0000 complete [10725.386750] LustreError: 322041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.398958] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10725.432587] LustreError: 322041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10725.474237] Lustre: Skipped 1 previous similar message [10727.980103] LustreError: 322022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851227 with bad export cookie 7169376519528517336 [10727.991596] LustreError: 322022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10727.998418] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10730.464198] LustreError: 322036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10730.474925] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10730.478390] LustreError: 322036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10730.501140] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10732.512763] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10732.520758] Lustre: Skipped 1 previous similar message [10734.555696] Lustre: server umount lustre-MDT0001 complete [10742.053524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10752.156921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10753.505431] LustreError: 323874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10753.522505] LustreError: 323874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10753.646674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10753.771061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10759.472702] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10762.164936] LustreError: 325656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10762.197282] LustreError: 325656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10762.871616] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10769.259698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10769.774868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10774.813300] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10775.034050] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10784.095068] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10785.913582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10791.890850] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10793.590934] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10799.588571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10799.593543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10799.605394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10800.610390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10805.623554] Lustre: server umount lustre-OST0000 complete [10810.849509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10810.857157] Lustre: Skipped 1 previous similar message [10810.860735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10810.861267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10810.861272] Lustre: Skipped 2 previous similar messages [10823.135146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10823.301905] Lustre: server umount lustre-MDT0000 complete [10826.209259] LustreError: 325661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10826.238694] LustreError: 325661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10827.847917] LustreError: 324646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851327 with bad export cookie 7169376519528524476 [10827.854050] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10827.873925] LustreError: 324646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10828.323291] Lustre: server umount lustre-MDT0001 complete [10840.722078] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [10843.678515] Key type lgssc unregistered [10843.925487] LNet: 328335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10843.935757] LNetError: 328335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10843.950113] LNet: Removed LNI 192.168.202.152@tcp [10845.014922] Key type .llcrypt unregistered [10845.016727] Key type ._llcrypt unregistered [10872.930593] Key type ._llcrypt registered [10872.936644] Key type .llcrypt registered [10873.078093] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10874.842289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10874.998676] alg: No test for adler32 (adler32-zlib) [10876.155588] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [10876.434428] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [10878.143847] Key type lgssc registered [10879.582699] Lustre: Echo OBD driver; http://www.lustre.org/ [10888.089706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10888.110112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10889.897670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10896.320624] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10899.427332] LustreError: 329753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.466602] LustreError: 329753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10904.546079] LustreError: 329752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10909.681877] LustreError: 329753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10911.039916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10911.538545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10917.807547] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10927.863694] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10935.277791] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10942.433258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10942.434325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10942.447112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10942.459301] Lustre: Skipped 1 previous similar message [10944.414756] Lustre: server umount lustre-MDT0000 complete [10947.556087] LustreError: 329747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.567992] LustreError: 329747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10948.656293] LustreError: 329732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851448 with bad export cookie 12535152877964683299 [10948.680737] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10949.120593] Lustre: server umount lustre-MDT0001 complete [10966.185099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10966.584016] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10966.650837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10966.855681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10971.867515] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10981.489911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10981.654217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10981.724899] Lustre: 332343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10982.075664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10987.405604] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10996.214686] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11003.264527] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11014.190137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11014.485502] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11014.498980] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11014.510259] LustreError: 334225:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [11014.517970] LustreError: 334225:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [11014.525870] LustreError: 334225:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [11014.657556] Lustre: server umount lustre-OST0000 complete [11014.668080] LustreError: 334225:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [11026.423063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11026.607294] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11026.866797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11032.722447] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11035.403178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [11042.935772] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11053.838377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11053.972220] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11054.207869] Lustre: lustre-OST0001: new disk, initializing [11054.222344] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11054.243158] Lustre: 335771:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11054.458281] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11062.411442] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11063.958319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11063.976832] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11064.128549] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11070.853764] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11074.530135] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11074.544522] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11074.573749] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11074.591613] Lustre: Skipped 2 previous similar messages [11079.654305] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11079.665524] Lustre: Skipped 1 previous similar message [11084.783119] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11084.798280] Lustre: Skipped 1 previous similar message [11088.351242] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11088.589043] Lustre: server umount lustre-OST0001 complete [11089.926497] LustreError: 334695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11089.945518] LustreError: 334695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11095.011562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11095.019773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11095.039611] Lustre: Skipped 2 previous similar messages [11095.052490] Lustre: Skipped 1 previous similar message [11100.151133] LustreError: 334692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11100.184664] LustreError: 334692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11105.256214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11105.275332] Lustre: Skipped 3 previous similar messages [11107.295975] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11107.742631] Lustre: server umount lustre-OST0000 complete [11115.490910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.526146] Lustre: Skipped 1 previous similar message [11125.728598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11125.736406] Lustre: Skipped 5 previous similar messages [11127.263096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11127.572690] Lustre: server umount lustre-MDT0000 complete [11130.851848] LustreError: 333548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.883285] LustreError: 333548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11131.946164] LustreError: 332333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851631 with bad export cookie 12535152877964683852 [11131.950434] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11131.956467] LustreError: 332333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11132.386825] Lustre: server umount lustre-MDT0001 complete [11146.553934] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [11150.017917] Key type lgssc unregistered [11150.431572] LNet: 337736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11150.442734] LNetError: 337736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11150.460746] LNet: Removed LNI 192.168.202.152@tcp [11151.560107] Key type .llcrypt unregistered [11151.562349] Key type ._llcrypt unregistered [11242.534705] Key type ._llcrypt registered [11242.541680] Key type .llcrypt registered [11242.655353] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [11263.114833] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [11265.507985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11265.569601] alg: No test for adler32 (adler32-zlib) [11267.016759] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [11267.523287] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [11269.359671] Key type lgssc registered [11271.495331] Lustre: Echo OBD driver; http://www.lustre.org/ [11311.398413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11311.445095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11312.929453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11312.972129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11313.096813] Lustre: lustre-MDT0000: new disk, initializing [11313.229917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11313.279333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11319.959751] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11331.596292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11331.748935] Lustre: 342678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11331.830315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11331.833886] Lustre: Skipped 1 previous similar message [11331.943840] Lustre: lustre-MDT0001: new disk, initializing [11332.095680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11332.142624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11332.150333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11337.579328] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11347.138580] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11353.674097] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11362.584102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11363.205976] Lustre: lustre-OST0000: new disk, initializing [11363.221640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11363.250409] Lustre: 344648:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11363.457971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11370.588723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11370.615188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11370.744026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11371.660241] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11382.478509] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11387.594784] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11387.959380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11395.361382] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11395.779506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11401.702322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11401.713585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11401.752145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11401.761595] Lustre: Skipped 1 previous similar message [11406.196125] Lustre: server umount lustre-OST0000 complete [11413.987395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11413.998077] LustreError: Skipped 1 previous similar message [11414.005494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11414.036943] Lustre: Skipped 1 previous similar message [11414.057100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11417.057448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11417.063323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11419.778071] Lustre: server umount lustre-MDT0000 complete [11423.848083] LustreError: 342669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851923 with bad export cookie 5621408746114772316 [11423.851043] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11423.861089] LustreError: 342669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11424.556478] Lustre: server umount lustre-MDT0001 complete [11434.180936] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:32:11 (1787851931) [11444.121913] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:32:21 (1787851941) [11454.713158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11455.281301] LustreError: 347404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11455.295895] LustreError: 347404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11455.365196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11460.583207] LustreError: 347405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11461.092761] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11464.678033] LustreError: 347404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11469.799747] LustreError: 347405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.516521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11472.005375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11478.180170] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11487.315661] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11493.512510] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11501.268850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11501.746205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11508.980159] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11517.876872] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11537.548774] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11539.650993] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11541.270862] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11543.238114] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11545.442307] Lustre: Failing over lustre-MDT0000 [11545.568734] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [11545.575978] Lustre: Skipped 1 previous similar message [11545.820278] Lustre: server umount lustre-MDT0000 complete [11546.083406] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11546.087392] LustreError: 349709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11546.096768] Lustre: Skipped 1 previous similar message [11546.125714] LustreError: 349709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11555.763808] LustreError: 347401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11555.789333] LustreError: 347401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11562.466768] Lustre: 340900:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852045/real 1787852045] req@000000009a946f89 x1874698454513152/t0(0) o400->MGC192.168.202.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787852061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11562.491704] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11566.593932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11572.151868] LustreError: 347399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11572.177403] LustreError: 347399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [11572.724457] LustreError: 340899:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e6108f9a x1874698454519296/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11573.211148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11573.309639] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11577.272777] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11577.620969] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11578.345952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11578.361560] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11578.399658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11588.659721] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11591.231705] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11597.097665] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11600.396778] Lustre: Failing over lustre-MDT0000 [11600.921263] Lustre: server umount lustre-MDT0000 complete [11603.938071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11603.952796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11603.975567] Lustre: Skipped 2 previous similar messages [11607.987426] LustreError: 347399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11608.021953] LustreError: 347399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11620.194420] Lustre: 340901:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852103/real 1787852103] req@000000002d5c3746 x1874698454536832/t0(0) o400->MGC192.168.202.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787852119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11620.225605] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11621.775821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11629.536065] LustreError: 340899:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f1e1ee50 x1874698454542464/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11629.838492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11629.895823] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11633.597799] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11635.190961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11635.203887] Lustre: Skipped 2 previous similar messages [11635.278287] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11635.375410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11635.774539] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11648.039821] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11650.366202] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11660.774638] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11660.779451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11660.789529] Lustre: Skipped 2 previous similar messages [11660.796513] Lustre: Skipped 2 previous similar messages [11664.333490] Lustre: server umount lustre-MDT0000 complete [11668.664623] LustreError: 347386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852168 with bad export cookie 5621408746114775032 [11668.673420] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11668.685987] LustreError: 347386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11669.174794] Lustre: server umount lustre-MDT0001 complete [11684.404553] Lustre: server umount lustre-OST0000 complete [11702.093726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11703.322531] LustreError: 353773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11703.356473] LustreError: 353773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [11703.469252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11709.441889] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11722.393661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11726.749943] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11735.650610] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11743.071720] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11750.809967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11759.689833] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11770.575869] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11773.786351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11784.162949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11784.171372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11784.171496] Lustre: Skipped 2 previous similar messages [11789.854314] Lustre: server umount lustre-OST0000 complete [11799.521421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11800.792967] LustreError: 356577:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (000000004326d946) refcount nonzero (1) after lock cleanup; forcing cleanup. [11801.013790] Lustre: server umount lustre-MDT0000 complete [11805.679987] LustreError: 353754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852305 with bad export cookie 5621408746114775774 [11805.687127] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11805.705125] LustreError: 353754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11806.475426] Lustre: server umount lustre-MDT0001 complete [11821.370066] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [11827.312110] Key type lgssc unregistered [11827.811627] LNet: 357335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11827.817100] LNetError: 357335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11827.847399] LNet: Removed LNI 192.168.202.152@tcp [11828.800534] Key type .llcrypt unregistered [11828.801595] Key type ._llcrypt unregistered [11866.429594] Key type ._llcrypt registered [11866.432166] Key type .llcrypt registered [11866.552527] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [11868.008095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11868.089456] alg: No test for adler32 (adler32-zlib) [11869.136863] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [11869.352952] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [11871.087171] Key type lgssc registered [11872.563226] Lustre: Echo OBD driver; http://www.lustre.org/ [11882.458733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11882.480542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11884.190831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11890.311289] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11893.730557] LustreError: 358859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11893.741387] LustreError: 358859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11898.867137] LustreError: 358858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.032779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11900.498882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11906.255721] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11914.108199] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11921.025048] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11928.465398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11928.914117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11929.973781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11936.215123] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11944.904918] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11956.401233] Lustre: server umount lustre-MDT0000 complete [11956.711335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11956.721176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.742236] LustreError: 359587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11960.801158] LustreError: 358859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11960.803766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11960.922734] LustreError: 358841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852460 with bad export cookie 7394270744654360828 [11960.934349] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11961.464847] Lustre: server umount lustre-MDT0001 complete [11977.368796] Lustre: server umount lustre-OST0000 complete [11992.221555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11992.908882] LustreError: 362568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11992.928582] LustreError: 362568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11992.984364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11997.599434] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12003.305607] LustreError: 362568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12003.328692] LustreError: 362568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12007.462849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12008.175534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12012.683648] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12024.528572] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12032.469212] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12040.975553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12041.420931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12050.864686] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12062.159859] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12062.972903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [12078.569445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12078.573923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12078.590906] Lustre: Skipped 3 previous similar messages [12078.609136] Lustre: Skipped 2 previous similar messages [12083.691029] LustreError: 362563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.707034] LustreError: 362563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12085.698277] Lustre: server umount lustre-MDT0000 complete [12090.264113] LustreError: 362550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852589 with bad export cookie 7394270744654361948 [12090.280842] LustreError: 362550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12090.282714] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12090.890640] Lustre: server umount lustre-MDT0001 complete [12106.916461] Lustre: server umount lustre-OST0000 complete [12126.583572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12127.047513] LustreError: 366484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12127.066534] LustreError: 366484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12127.127433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12132.421408] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12144.901631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12151.241322] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12162.773216] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12171.388317] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12179.070727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12179.459794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12179.472476] Lustre: Skipped 1 previous similar message [12186.761623] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12196.857629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [12197.663079] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12207.072209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12207.082742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12207.106432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12207.110379] Lustre: Skipped 1 previous similar message [12207.143926] Lustre: Skipped 1 previous similar message [12210.001461] Lustre: server umount lustre-OST0000 complete [12217.313860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12217.318031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12217.329306] Lustre: Skipped 1 previous similar message [12217.334786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12217.350462] LustreError: Skipped 1 previous similar message [12217.373735] Lustre: Skipped 1 previous similar message [12220.970056] Lustre: server umount lustre-MDT0000 complete [12222.445041] LustreError: 366484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12222.476607] LustreError: 366484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12226.380647] LustreError: 366466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852726 with bad export cookie 7394270744654362963 [12226.397602] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12226.415296] LustreError: 366466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12227.149357] Lustre: server umount lustre-MDT0001 complete [12248.478527] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12251.225980] Key type lgssc unregistered [12251.710282] LNet: 370048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12251.720473] LNetError: 370048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12251.738472] LNet: Removed LNI 192.168.202.152@tcp [12252.952120] Key type .llcrypt unregistered [12252.954807] Key type ._llcrypt unregistered [12281.394553] Key type ._llcrypt registered [12281.398134] Key type .llcrypt registered [12281.498506] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [12283.188858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12283.244224] alg: No test for adler32 (adler32-zlib) [12284.523282] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [12284.795575] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [12286.519170] Key type lgssc registered [12288.083182] Lustre: Echo OBD driver; http://www.lustre.org/ [12297.527748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12297.565337] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12297.580838] LustreError: 371492:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12297.621207] LustreError: 371492:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12297.643610] LustreError: 371492:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12297.654880] LustreError: 371492:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12297.673289] LustreError: 371492:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12309.827845] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12309.839504] LustreError: 371889:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12309.868407] LustreError: 371889:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12309.882806] LustreError: 371889:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12309.893738] LustreError: 371889:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12309.927568] LustreError: 371889:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12337.103847] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12340.048729] Key type lgssc unregistered [12340.397030] LNet: 372896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12340.415114] LNetError: 372896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12340.435808] LNet: Removed LNI 192.168.202.152@tcp [12341.383728] Key type .llcrypt unregistered [12341.386890] Key type ._llcrypt unregistered [12366.355686] Key type ._llcrypt registered [12366.362418] Key type .llcrypt registered [12366.596990] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [12386.910367] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [12388.283178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12388.380560] alg: No test for adler32 (adler32-zlib) [12389.361720] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [12389.554054] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [12391.168577] Key type lgssc registered [12392.016640] Lustre: Echo OBD driver; http://www.lustre.org/ [12429.834343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12429.908170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12431.315947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12431.356340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12431.424471] Lustre: lustre-MDT0000: new disk, initializing [12431.506939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12431.533305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12436.836284] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12448.984624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12449.199967] Lustre: 376051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12449.282202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12449.286755] Lustre: Skipped 1 previous similar message [12449.389585] Lustre: lustre-MDT0001: new disk, initializing [12449.492042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12449.542281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12449.554489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12452.266839] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12463.482072] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12470.174514] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12477.567645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12477.798317] Lustre: lustre-OST0000: new disk, initializing [12477.801637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12477.806856] Lustre: 378023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12477.870394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12483.718251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12483.739955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12483.855590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12485.310244] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12496.205787] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12501.596153] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12501.854416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12506.647528] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12506.952912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12509.681331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12509.696473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12509.706239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12509.714646] Lustre: Skipped 1 previous similar message [12514.788495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12514.793637] Lustre: Skipped 2 previous similar messages [12515.331864] Lustre: server umount lustre-OST0000 complete [12519.905971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12519.906744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12519.913912] Lustre: Skipped 1 previous similar message [12519.918703] Lustre: Skipped 1 previous similar message [12525.024523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12525.649096] Lustre: server umount lustre-MDT0000 complete [12530.629152] LustreError: 376045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853030 with bad export cookie 6081135048145271643 [12530.636709] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12530.640306] LustreError: 376045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12531.256936] Lustre: server umount lustre-MDT0001 complete [12543.177308] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12545.264643] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:50:42 (1787853042) [12553.811507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12554.584652] LustreError: 380485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12554.611624] LustreError: 380485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12554.718884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12559.869159] LustreError: 380486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12560.213123] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12564.965070] LustreError: 380485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12571.109499] LustreError: 380486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12571.134646] LustreError: 380486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12571.519531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12572.085304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12577.434922] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12587.091429] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12593.991794] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12602.552255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12603.015226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12610.234078] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12619.544039] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12628.009214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12628.342414] Lustre: lustre-OST0001: new disk, initializing [12628.350601] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12628.372649] Lustre: 383323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12628.567628] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12637.255850] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12637.833269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12637.853641] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12637.987395] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12648.081711] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12653.538856] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12653.568217] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12653.572269] LustreError: Skipped 2 previous similar messages [12653.589875] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12653.589884] Lustre: Skipped 1 previous similar message [12653.644358] Lustre: Skipped 1 previous similar message [12658.666093] LustreError: 382309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12658.750809] Lustre: server umount lustre-OST0001 complete [12668.907317] LustreError: 383710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12668.950116] LustreError: 383710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12679.137388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12679.144338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12679.149753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12679.149766] Lustre: Skipped 2 previous similar messages [12679.182150] Lustre: Skipped 1 previous similar message [12684.090563] Lustre: server umount lustre-OST0000 complete [12689.377801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12695.094314] Lustre: server umount lustre-MDT0000 complete [12698.592805] LustreError: 380480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.627695] LustreError: 380480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12699.547887] LustreError: 380467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853199 with bad export cookie 6081135048145272525 [12699.556467] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12699.573104] LustreError: 380467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12705.985274] Lustre: server umount lustre-MDT0001 complete [12717.756511] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12721.715041] Key type lgssc unregistered [12722.220958] LNet: 385313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12722.231749] LNetError: 385313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12722.261108] LNet: Removed LNI 192.168.202.152@tcp [12723.608272] Key type .llcrypt unregistered [12723.610130] Key type ._llcrypt unregistered [12784.516482] Key type ._llcrypt registered [12784.518877] Key type .llcrypt registered [12784.662614] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [12785.922991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12786.178783] alg: No test for adler32 (adler32-zlib) [12787.225801] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [12787.552607] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [12789.279918] Key type lgssc registered [12790.802391] Lustre: Echo OBD driver; http://www.lustre.org/ [12800.254809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12800.285740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12801.665687] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12801.714043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12802.043534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12808.171481] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12818.889090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12819.114967] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12819.164452] Lustre: 388118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12819.453517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12825.825064] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12835.088972] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12843.145752] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12846.566110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12846.589305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12846.596136] Lustre: Skipped 1 previous similar message [12851.684797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12851.691744] Lustre: Skipped 1 previous similar message [12856.802389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12856.820019] Lustre: Skipped 1 previous similar message [12860.386958] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12860.715984] Lustre: server umount lustre-MDT0000 complete [12866.114868] LustreError: 388111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853365 with bad export cookie 15295835645044391410 [12866.122614] LustreError: 388111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12866.123729] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12866.640126] Lustre: server umount lustre-MDT0001 complete [12878.950729] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:56:16 (1787853376) [12887.129654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12887.828154] LustreError: 390635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12887.841788] LustreError: 390635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12887.905368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12893.161143] LustreError: 390636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12893.712823] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12898.273455] LustreError: 390635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12903.399032] LustreError: 390636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12905.564704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12906.273327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12913.493750] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12922.236445] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12928.957986] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12935.607662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12935.838939] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12936.177131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12944.299150] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12954.531923] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12963.196513] Lustre: Permanently deactivating lustre-OST0000 [12963.252336] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12969.100974] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12970.444959] LustreError: 393526:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12970.456448] LustreError: 393526:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12970.479936] LustreError: 393526:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12974.444389] LustreError: 393575:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12974.451598] LustreError: 393575:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12979.642408] Lustre: server umount lustre-OST0000 complete [12983.338909] LustreError: 393825:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12983.349596] LustreError: 393825:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12985.830216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12985.851016] Lustre: Skipped 1 previous similar message [12985.867043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12985.882195] Lustre: Skipped 1 previous similar message [12990.944500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12996.138536] Lustre: server umount lustre-MDT0000 complete [12999.824806] LustreError: 390616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853499 with bad export cookie 15295835645044391991 [12999.826346] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12999.843887] LustreError: 390616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13009.638266] Lustre: server umount lustre-MDT0001 complete [13019.860885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13024.696803] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13027.746525] LustreError: 394983:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [13034.934722] Lustre: 395322:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.152@tcp [13037.338579] Lustre: server umount MGS complete [13053.420748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13054.119162] LustreError: 396046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13054.156517] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13054.163485] Lustre: Skipped 1 previous similar message [13054.229367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13060.615322] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13064.673174] LustreError: 396047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13064.697641] LustreError: 396047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13071.093697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13071.488279] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13071.556333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13076.599101] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13085.963719] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13092.918891] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13095.028699] Lustre: Permanently reactivating lustre-OST0000 [13095.046803] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [13095.053225] Lustre: Skipped 2 previous similar messages [13124.575593] LustreError: 397860:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [13134.021989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13134.620828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13142.727832] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13151.460722] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13158.907877] Lustre: server umount lustre-OST0000 complete [13164.001106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13164.008808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13164.035108] Lustre: Skipped 1 previous similar message [13164.039464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13164.044270] Lustre: Skipped 3 previous similar messages [13175.712025] Lustre: 387438:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853659/real 1787853659] req@000000002438c9df x1874700048371968/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787853675 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13175.757621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13175.775818] Lustre: Skipped 1 previous similar message [13176.799178] Lustre: 387441:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853660/real 1787853660] req@00000000f733b64e x1874700048372736/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787853676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13178.335153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13178.555714] Lustre: server umount lustre-MDT0000 complete [13180.899895] LustreError: 396047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13180.929544] LustreError: 396047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13181.919090] Lustre: 387438:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853665/real 1787853665] req@0000000015e6b176 x1874700048374528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787853681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13182.868738] LustreError: 396029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853682 with bad export cookie 15295835645044393041 [13182.895598] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13182.895725] LustreError: 396029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13183.306626] Lustre: server umount lustre-MDT0001 complete [13195.832762] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13198.715221] Key type lgssc unregistered [13199.068932] LNet: 400428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13199.074832] LNetError: 400428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13199.085329] LNet: Removed LNI 192.168.202.152@tcp [13199.975216] Key type .llcrypt unregistered [13199.977791] Key type ._llcrypt unregistered [13221.105138] Key type ._llcrypt registered [13221.108812] Key type .llcrypt registered [13221.256879] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [13237.862967] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [13239.497814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13239.536402] alg: No test for adler32 (adler32-zlib) [13240.770386] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [13241.100378] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [13242.887266] Key type lgssc registered [13243.940295] Lustre: Echo OBD driver; http://www.lustre.org/ [13283.590947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13283.637098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13284.928378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13284.981625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13285.125484] Lustre: lustre-MDT0000: new disk, initializing [13285.239479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13285.262811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13291.240360] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13303.019569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13303.174249] Lustre: 403587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13303.236525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13303.240236] Lustre: Skipped 1 previous similar message [13303.326731] Lustre: lustre-MDT0001: new disk, initializing [13303.415248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13303.448684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13303.456828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13308.754302] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13318.605413] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13325.117479] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13332.334065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13332.517748] Lustre: lustre-OST0000: new disk, initializing [13332.521166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13332.526212] Lustre: 405553:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13332.629199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13339.009647] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13339.770637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13339.789082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13339.921821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13349.754845] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13355.356535] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13355.578236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13361.004222] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13361.303075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13365.736746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13365.748902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13365.781134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13365.790628] Lustre: Skipped 1 previous similar message [13369.960534] Lustre: server umount lustre-OST0000 complete [13374.965421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13374.980283] LustreError: Skipped 1 previous similar message [13374.982381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13374.991472] Lustre: Skipped 1 previous similar message [13374.997497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13375.987607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13376.022532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13376.029830] Lustre: Skipped 1 previous similar message [13380.924451] Lustre: server umount lustre-MDT0000 complete [13385.360347] LustreError: 405564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853885 with bad export cookie 14211543607889543232 [13385.369270] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13385.376668] LustreError: 405564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13385.863561] Lustre: server umount lustre-MDT0001 complete [13395.701203] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13397.546215] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:04:55 (1787853895) [13405.672142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13406.286403] LustreError: 408014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13406.311748] LustreError: 408014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13406.394308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13411.813149] LustreError: 408015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13411.907337] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13416.938340] LustreError: 408014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13422.055114] LustreError: 408015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13422.717667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13423.123112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13429.204366] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13438.236056] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13444.697560] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13452.336135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13452.735405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13459.715564] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13467.737518] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13469.701567] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13477.621626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13484.515640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13484.529304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13484.549904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13487.619043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13489.097978] Lustre: server umount lustre-OST0000 complete [13494.775132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13494.792387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13494.809022] Lustre: Skipped 1 previous similar message [13494.815823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13494.829338] Lustre: Skipped 2 previous similar messages [13499.712986] Lustre: server umount lustre-MDT0000 complete [13502.945596] LustreError: 409801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13502.961878] LustreError: 409801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13503.525091] LustreError: 407996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854003 with bad export cookie 14211543607889544114 [13503.534542] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13503.536084] LustreError: 407996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13503.975256] Lustre: server umount lustre-MDT0001 complete [13515.547037] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13518.719463] Key type lgssc unregistered [13518.983943] LNet: 411745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13518.989508] LNetError: 411745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13519.008891] LNet: Removed LNI 192.168.202.152@tcp [13519.924333] Key type .llcrypt unregistered [13519.932807] Key type ._llcrypt unregistered [13548.426870] Key type ._llcrypt registered [13548.432036] Key type .llcrypt registered [13548.528397] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13548.832686] Key type .llcrypt unregistered [13548.836471] Key type ._llcrypt unregistered [13568.220905] Key type ._llcrypt registered [13568.223454] Key type .llcrypt registered [13568.383560] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [13569.644795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13569.869314] alg: No test for adler32 (adler32-zlib) [13570.993818] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [13571.233228] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [13572.967179] Key type lgssc registered [13574.193767] Lustre: Echo OBD driver; http://www.lustre.org/ [13582.651525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13582.692982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13584.498728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13589.585621] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13595.125190] LustreError: 413965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13595.159628] LustreError: 413965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13599.177362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13599.592633] LustreError: 413964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13599.648401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13604.833261] LustreError: 413965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13604.851890] LustreError: 413965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13607.261680] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13609.957491] LustreError: 413959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13609.967296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13609.974433] LustreError: 413959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13615.076355] LustreError: 413964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13615.094924] LustreError: 413964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13615.960365] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13625.040772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13625.501212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13630.989519] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13640.679033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13640.683470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13640.688394] LustreError: Skipped 1 previous similar message [13640.722561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13646.317877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13646.333341] Lustre: Skipped 3 previous similar messages [13646.830404] Lustre: server umount lustre-OST0000 complete [13650.921679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13650.928908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13650.941896] Lustre: Skipped 1 previous similar message [13650.954693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13650.957541] Lustre: Skipped 1 previous similar message [13656.546689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13656.552138] Lustre: Skipped 3 previous similar messages [13657.333040] Lustre: server umount lustre-MDT0000 complete [13661.166377] LustreError: 413946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854160 with bad export cookie 1876124483434198647 [13661.175983] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13661.578262] Lustre: server umount lustre-MDT0001 complete [13672.327686] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13675.230487] Key type lgssc unregistered [13675.522556] LNet: 417113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13675.530901] LNetError: 417113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13676.587359] LNet: Removed LNI 192.168.202.152@tcp [13677.632219] Key type .llcrypt unregistered [13677.635512] Key type ._llcrypt unregistered [13703.093670] Key type ._llcrypt registered [13703.097417] Key type .llcrypt registered [13703.197945] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [13704.176439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13704.253642] alg: No test for adler32 (adler32-zlib) [13705.464491] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [13705.782992] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [13707.464067] Key type lgssc registered [13708.748939] Lustre: Echo OBD driver; http://www.lustre.org/ [13717.206878] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13717.241814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13732.832803] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13737.951457] LustreError: 418507:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 4: rc = -110 [13764.641731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13771.357300] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13778.952902] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13785.878899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13786.141075] LustreError: 417842:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007568a9b1 x1874701010864384/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13786.670373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13786.758584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13791.730176] LustreError: 418532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13791.745505] LustreError: 418532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13791.865938] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13796.835222] LustreError: 418533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13799.933071] LustreError: 418532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13799.956445] LustreError: 418532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13801.220120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13801.485500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13806.237944] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13814.997862] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13822.186332] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13831.977841] Lustre: server umount lustre-OST0000 complete [13836.771387] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13836.777828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13836.792989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13837.281750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13837.292721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13837.301366] Lustre: Skipped 1 previous similar message [13842.408914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13842.419086] Lustre: Skipped 1 previous similar message [13845.590686] Lustre: server umount lustre-MDT0000 complete [13847.522526] LustreError: 420755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13847.524296] Lustre: 417846:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854331/real 1787854331] req@00000000841cbe89 x1874701010893440/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787854347 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13847.552944] LustreError: 420755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13847.575819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13849.721552] LustreError: 418515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854349 with bad export cookie 12038232530323003542 [13849.735210] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13849.736566] LustreError: 418515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13850.091883] Lustre: server umount lustre-MDT0001 complete [13860.646366] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13863.248689] Key type lgssc unregistered [13863.552347] LNet: 422085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13863.562339] LNetError: 422085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13864.615565] LNet: Removed LNI 192.168.202.152@tcp [13865.490847] Key type .llcrypt unregistered [13865.495550] Key type ._llcrypt unregistered [13891.940881] Key type ._llcrypt registered [13891.944799] Key type .llcrypt registered [13892.070829] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [13893.368943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13893.440096] alg: No test for adler32 (adler32-zlib) [13894.530668] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [13894.827571] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [13896.535228] Key type lgssc registered [13897.697889] Lustre: Echo OBD driver; http://www.lustre.org/ [13906.272423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13906.310930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13907.888318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13912.534071] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13918.177652] LustreError: 423507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13918.194743] LustreError: 423507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13921.835806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13922.286030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13927.142794] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13935.199392] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13940.427910] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13947.389888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13947.680855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13952.809986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13954.348750] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13961.654699] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13967.398084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13967.401490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13967.409421] Lustre: Skipped 1 previous similar message [13967.434865] Lustre: Skipped 2 previous similar messages [13969.883489] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [13972.450914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13972.461118] Lustre: Skipped 1 previous similar message [13972.635706] Lustre: server umount lustre-MDT0000 complete [13975.006620] LustreError: 423490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.52@tcp arrived at 1787854474 with bad export cookie 12725383501707634233 [13976.722174] LustreError: 423490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854476 with bad export cookie 12725383501707633344 [13976.737905] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13976.745467] LustreError: 423490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13977.569198] LustreError: 424246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13977.573172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13977.591792] LustreError: 424246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13977.610446] Lustre: Skipped 1 previous similar message [13977.616575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13981.603604] LustreError: 423508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13981.617609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13981.635138] LustreError: 423508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13982.994603] Lustre: server umount lustre-OST0000 complete [13986.722107] LustreError: 425996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13986.738543] LustreError: 425996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13989.983425] LustreError: 424236:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009ec5de94 x1874701209074560/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13990.150490] Lustre: server umount lustre-MDT0001 complete [14000.684649] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [14003.288395] Key type lgssc unregistered [14003.685989] LNet: 427194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14003.697973] LNetError: 427194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14003.714685] LNet: Removed LNI 192.168.202.152@tcp [14004.578820] Key type .llcrypt unregistered [14004.581310] Key type ._llcrypt unregistered [14028.676241] Key type ._llcrypt registered [14028.678604] Key type .llcrypt registered [14028.816481] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [14029.962602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14030.052493] alg: No test for adler32 (adler32-zlib) [14031.054716] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [14031.253035] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [14032.895275] Key type lgssc registered [14033.970286] Lustre: Echo OBD driver; http://www.lustre.org/ [14040.775361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14040.792298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14042.236807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14046.697457] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14051.810595] LustreError: 428612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14051.840097] LustreError: 428612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14055.908568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14056.315699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14060.662803] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14067.923792] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14074.240414] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14081.150389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14081.593577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14086.726062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14088.655090] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14096.955123] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14102.496652] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14102.504291] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14102.514469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14104.502963] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [14106.088309] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14106.103095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14108.085712] Lustre: server umount lustre-MDT0001 complete [14109.618106] LustreError: 428607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14111.203658] LustreError: 428612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14111.218561] LustreError: 428612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14116.321754] LustreError: 429658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14116.325595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14116.352859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14116.359583] Lustre: Skipped 1 previous similar message [14120.964638] Lustre: server umount lustre-OST0000 complete [14121.444304] LustreError: 428612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14126.194200] Lustre: server umount lustre-MDT0000 complete [14141.202969] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [14144.471982] Key type lgssc unregistered [14144.826794] LNet: 432298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14144.840720] LNetError: 432298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14144.879368] LNet: Removed LNI 192.168.202.152@tcp [14145.909672] Key type .llcrypt unregistered [14145.918424] Key type ._llcrypt unregistered [14172.844168] Key type ._llcrypt registered [14172.846495] Key type .llcrypt registered [14172.946707] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [14188.746627] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [14189.648425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14189.756917] alg: No test for adler32 (adler32-zlib) [14190.780621] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [14191.012859] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [14192.695175] Key type lgssc registered [14193.652147] Lustre: Echo OBD driver; http://www.lustre.org/ [14231.786860] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [14248.310148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14248.342385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14249.747531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14249.812254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14249.935104] Lustre: lustre-MDT0000: new disk, initializing [14250.043440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14250.066222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14255.268964] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14270.906701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14271.149609] Lustre: lustre-OST0000: new disk, initializing [14271.154573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14271.161275] Lustre: Skipped 1 previous similar message [14271.165667] Lustre: 437335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14271.242234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14277.115867] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14280.797650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14280.806661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14280.827477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14287.733730] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14304.479585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14304.648569] Lustre: 436316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14304.726597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14304.798828] Lustre: lustre-MDT0001: new disk, initializing [14304.944260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14305.035035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14305.047121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14310.067362] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14319.549529] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14319.761510] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14323.699359] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14323.964078] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14330.846433] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [14333.410176] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14333.415242] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14333.420301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14333.442738] Lustre: Skipped 1 previous similar message [14334.811984] Lustre: server umount lustre-MDT0001 complete [14343.648897] LustreError: 436323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14343.661796] LustreError: 436323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [14348.778590] LustreError: 436324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14348.797924] LustreError: 436324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14352.354545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14352.370353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14352.388496] Lustre: Skipped 1 previous similar message [14352.395718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14353.891697] LustreError: 436328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14359.013973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14359.014494] LustreError: 436327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14359.018212] Lustre: Skipped 1 previous similar message [14364.130184] LustreError: 436328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14364.131075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14366.175263] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14366.379925] Lustre: server umount lustre-OST0000 complete [14378.411716] Lustre: server umount lustre-MDT0000 complete [14393.200736] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [14396.139256] Key type lgssc unregistered [14396.491803] LNet: 441088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14396.498876] LNetError: 441088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14396.520918] LNet: Removed LNI 192.168.202.152@tcp [14397.506293] Key type .llcrypt unregistered [14397.507498] Key type ._llcrypt unregistered [14445.014757] Key type ._llcrypt registered [14445.016734] Key type .llcrypt registered [14445.152905] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [14445.541542] Key type .llcrypt unregistered [14445.545822] Key type ._llcrypt unregistered [14464.367916] Key type ._llcrypt registered [14464.370454] Key type .llcrypt registered [14464.597893] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [14465.980766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14466.011260] alg: No test for adler32 (adler32-zlib) [14467.668602] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [14468.184769] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [14469.991600] Key type lgssc registered [14471.818803] Lustre: Echo OBD driver; http://www.lustre.org/ [14500.204779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14500.228444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14501.432960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14501.463127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14501.533172] Lustre: lustre-MDT0000: new disk, initializing [14501.592844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14501.611388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14506.919832] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14518.227446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14518.365446] Lustre: 444471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14518.429871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14518.434890] Lustre: Skipped 1 previous similar message [14518.531395] Lustre: lustre-MDT0001: new disk, initializing [14518.643834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14518.685449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14518.697497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14522.810086] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14533.223984] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14542.572746] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14552.181936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14552.664409] Lustre: lustre-OST0000: new disk, initializing [14552.677758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14552.696772] Lustre: 446439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14552.926593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14558.309621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14558.322680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14558.552373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14561.641325] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14572.439964] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14579.178719] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14579.190872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14579.192139] Lustre: Skipped 1 previous similar message [14579.202605] Lustre: Skipped 2 previous similar messages [14584.293626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14584.306028] Lustre: Skipped 2 previous similar messages [14585.356825] Lustre: server umount lustre-MDT0000 complete [14589.066266] LustreError: 446449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855088 with bad export cookie 1133746305554112538 [14589.083461] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14589.410482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14589.420498] Lustre: Skipped 1 previous similar message [14589.697988] Lustre: server umount lustre-MDT0001 complete [14595.638677] Lustre: server umount lustre-OST0000 complete [14606.510577] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:25:04 (1787855104) [14614.779358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14615.296707] LustreError: 448467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14615.323625] LustreError: 448467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14615.397515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14619.990958] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14620.671329] LustreError: 448468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14625.778621] LustreError: 448467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14629.393066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14629.849748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14634.684784] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14643.358897] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14649.579800] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14670.776702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14671.123864] Lustre: Found index 0 for lustre-OST0000, updating log [14671.340993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14678.757151] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [14687.752196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14687.954873] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14694.887502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14694.898547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14694.936231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14700.636620] Lustre: server umount lustre-OST0000 complete [14716.898446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14716.907360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14716.917814] Lustre: Skipped 1 previous similar message [14716.923365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14716.928397] Lustre: Skipped 3 previous similar messages [14721.509833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14721.525881] Lustre: Skipped 3 previous similar messages [14731.232087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14731.366816] Lustre: server umount lustre-MDT0000 complete [14731.487179] LNet: 1 local NIs in recovery (showing 1): 192.168.202.152@tcp [14732.771678] LustreError: 448462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14732.788443] LustreError: 448462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14735.862881] LustreError: 448448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855235 with bad export cookie 1133746305554113931 [14735.868701] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14735.884774] LustreError: 448448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [14736.320493] Lustre: server umount lustre-MDT0001 complete [14746.276587] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:27:24 (1787855244) [15369.030639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15369.618383] LustreError: 454880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15369.693456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15374.750646] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [15374.825894] LustreError: 454881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15379.936789] LustreError: 454880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15381.985492] LustreError: 454881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15382.012249] LustreError: 454881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15384.017560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15384.323946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15389.216985] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [15396.965638] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15404.133340] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15422.879286] LNet: 1 local NIs in recovery (showing 1): 192.168.202.152@tcp [15425.992092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15426.367602] Lustre: Found index 0 for lustre-OST0000, updating log [15426.747607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15434.999258] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [15444.949381] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15448.743686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15448.777366] Lustre: Skipped 1 previous similar message [15448.789873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15448.802157] Lustre: Skipped 4 previous similar messages [15461.855172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15462.024477] Lustre: server umount lustre-OST0000 complete [15466.164186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15466.176300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15466.186604] Lustre: Skipped 1 previous similar message [15468.259702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15468.267076] Lustre: Skipped 4 previous similar messages [15471.463399] Lustre: server umount lustre-MDT0000 complete [15472.352734] LustreError: 454881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15472.373639] LustreError: 454881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15475.025432] LustreError: 454862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855974 with bad export cookie 1133746305554115016 [15475.026277] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15475.039267] LustreError: 454862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15475.868803] Lustre: server umount lustre-MDT0001 complete [15483.295278] LNet: 1 local NIs in recovery (showing 1): 192.168.202.152@tcp [15511.405733] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [15522.394389] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [15564.404527] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:41:02 (1787856062) [15586.195505] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [15589.538412] Key type lgssc unregistered [15589.876927] LNet: 461501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15589.885959] LNetError: 461501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15689.252902] LNet: Removed LNI 192.168.202.152@tcp [15690.183524] Key type .llcrypt unregistered [15690.188127] Key type ._llcrypt unregistered [15712.818234] Key type ._llcrypt registered [15712.826254] Key type .llcrypt registered [15712.994489] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [15715.086903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15715.098453] alg: No test for adler32 (adler32-zlib) [15716.215585] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [15716.518994] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [15718.223215] Key type lgssc registered [15719.582436] Lustre: Echo OBD driver; http://www.lustre.org/ [15723.703618] LNet: Added LNI 192.168.202.152@tcp100 [8/256/0/180] [15724.829948] LNet: Added LNI 192.168.202.152@tcp101 [8/256/0/180] [15726.030491] LNet: Added LNI 192.168.202.152@tcp102 [8/256/0/180] [15727.027459] LNet: Added LNI 192.168.202.152@tcp103 [8/256/0/180] [15728.198664] LNet: Added LNI 192.168.202.152@tcp104 [8/256/0/180] [15729.247960] LNet: Added LNI 192.168.202.152@tcp105 [8/256/0/180] [15730.223568] LNet: Added LNI 192.168.202.152@tcp106 [8/256/0/180] [15731.247267] LNet: Added LNI 192.168.202.152@tcp107 [8/256/0/180] [15732.258982] LNet: Added LNI 192.168.202.152@tcp108 [8/256/0/180] [15733.132273] LNet: Added LNI 192.168.202.152@tcp109 [8/256/0/180] [15734.300050] LNet: Added LNI 192.168.202.152@tcp110 [8/256/0/180] [15735.320306] LNet: Added LNI 192.168.202.152@tcp111 [8/256/0/180] [15736.542367] LNet: Added LNI 192.168.202.152@tcp112 [8/256/0/180] [15737.854756] LNet: Added LNI 192.168.202.152@tcp113 [8/256/0/180] [15738.988323] LNet: Added LNI 192.168.202.152@tcp114 [8/256/0/180] [15739.950970] LNet: Added LNI 192.168.202.152@tcp115 [8/256/0/180] [15740.885758] LNet: Added LNI 192.168.202.152@tcp116 [8/256/0/180] [15741.838344] LNet: Added LNI 192.168.202.152@tcp117 [8/256/0/180] [15742.835967] LNet: Added LNI 192.168.202.152@tcp118 [8/256/0/180] [15743.817243] LNet: Added LNI 192.168.202.152@tcp119 [8/256/0/180] [15744.616140] LNet: Added LNI 192.168.202.152@tcp120 [8/256/0/180] [15745.512899] LNet: Added LNI 192.168.202.152@tcp121 [8/256/0/180] [15746.295293] LNet: Added LNI 192.168.202.152@tcp122 [8/256/0/180] [15747.202462] LNet: Added LNI 192.168.202.152@tcp123 [8/256/0/180] [15748.068168] LNet: Added LNI 192.168.202.152@tcp124 [8/256/0/180] [15748.938906] LNet: Added LNI 192.168.202.152@tcp125 [8/256/0/180] [15749.882537] LNet: Added LNI 192.168.202.152@tcp126 [8/256/0/180] [15750.786839] LNet: Added LNI 192.168.202.152@tcp127 [8/256/0/180] [15751.792418] LNet: Added LNI 192.168.202.152@tcp128 [8/256/0/180] [15752.923430] LNet: Added LNI 192.168.202.152@tcp129 [8/256/0/180] [15753.930564] LNet: Added LNI 192.168.202.152@tcp130 [8/256/0/180] [15754.798900] LNet: Added LNI 192.168.202.152@tcp131 [8/256/0/180] [15755.847864] LNet: Added LNI 192.168.202.152@tcp132 [8/256/0/180] [15756.981971] LNet: Added LNI 192.168.202.152@tcp133 [8/256/0/180] [15758.137552] LNet: Added LNI 192.168.202.152@tcp134 [8/256/0/180] [15759.121361] LNet: Added LNI 192.168.202.152@tcp135 [8/256/0/180] [15767.784313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15767.804864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15768.946229] Lustre: 464533:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.152@tcp: can't add NID 192.168.202.152@tcp130: rc = -75 [15769.043063] LNetError: 464559:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.152 [15769.056122] LNetError: Protocol error connecting to 192.168.202.152@tcp115 at host 192.168.202.152:988: is it running a compatible version of Lustre? [15769.064705] LNetError: Skipped 1 previous similar message [15769.155832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15769.176695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15769.270663] Lustre: lustre-MDT0000: new disk, initializing [15769.390223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15769.412378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15769.763244] LNet: 10 local NIs in recovery (showing 10): 192.168.202.152@tcp123, 192.168.202.152@tcp124, 192.168.202.152@tcp126, 192.168.202.152@tcp127, 192.168.202.152@tcp125, 192.168.202.152@tcp130, 192.168.202.152@tcp129, 192.168.202.152@tcp128, 192.168.202.152@tcp131, 192.168.202.152@tcp132 [15770.144446] LNetError: 464554:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.152 [15770.160204] LNetError: 464554:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [15770.176641] LNetError: Protocol error connecting to 192.168.202.152@tcp121 at host 192.168.202.152:988: is it running a compatible version of Lustre? [15770.195166] LNetError: Skipped 15 previous similar messages [15772.901589] LNetError: 461993:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.152 [15772.924387] LNetError: 461993:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [15772.947301] LNetError: Protocol error connecting to 192.168.202.152@tcp127 at host 192.168.202.152:988: is it running a compatible version of Lustre? [15772.970695] LNetError: Skipped 13 previous similar messages [15774.836438] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [15777.001673] LNetError: 461992:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.152 [15777.018102] LNetError: 461992:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15777.031650] LNetError: Protocol error connecting to 192.168.202.152@tcp127 at host 192.168.202.152:988: is it running a compatible version of Lustre? [15787.515859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15787.692288] Lustre: 464551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15787.770477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15787.777082] Lustre: Skipped 1 previous similar message [15787.858425] Lustre: lustre-MDT0001: new disk, initializing [15787.889763] LNetError: 464560:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.152 [15787.898893] Lustre: 465351:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.152@tcp: can't add NID 192.168.202.152@tcp130: rc = -75 [15787.899154] LNetError: Protocol error connecting to 192.168.202.152@tcp133 at host 192.168.202.152:988: is it running a compatible version of Lustre? [15787.902286] Lustre: 465351:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15787.998261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15788.030801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15788.042851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15793.380792] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [15795.812937] Lustre: 465715:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.152@tcp: can't add NID 192.168.202.152@tcp130: rc = -75 [15795.833418] Lustre: 465715:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15799.478106] LNetError: 461994:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.152 [15799.485355] LNetError: 461994:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [15799.491882] LNetError: Protocol error connecting to 192.168.202.152@tcp117 at host 192.168.202.152:988: is it running a compatible version of Lustre? [15799.505042] LNetError: Skipped 11 previous similar messages [15805.433712] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15812.169739] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15815.909674] LNetError: 464566:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.152 [15815.928642] LNetError: 464566:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [15815.938783] LNetError: Protocol error connecting to 192.168.202.152@tcp123 at host 192.168.202.152:988: is it running a compatible version of Lustre? [15815.957019] LNetError: Skipped 18 previous similar messages [15820.511799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15820.876362] Lustre: lustre-OST0000: new disk, initializing [15820.883846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15820.897479] Lustre: 466526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15821.065763] Lustre: 466531:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.152@tcp: can't add NID 192.168.202.152@tcp130: rc = -75 [15821.071803] Lustre: 466531:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15821.129995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15828.078358] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [15830.070875] Lustre: 466972:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.152@tcp: can't add NID 192.168.202.152@tcp130: rc = -75 [15830.083232] Lustre: 466972:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15830.177770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15830.191202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15830.239200] LNet: 3 local NIs in recovery (showing 3): 192.168.202.152@tcp133, 192.168.202.152@tcp134, 192.168.202.152@tcp127 [15830.333745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15839.303256] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15848.673553] LNetError: 464560:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.152 [15848.692559] LNetError: 464560:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 28 previous similar messages [15848.703253] LNetError: Protocol error connecting to 192.168.202.152@tcp132 at host 192.168.202.152:988: is it running a compatible version of Lustre? [15848.716724] LNetError: Skipped 28 previous similar messages [15850.977502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15850.998229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15851.019339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15853.473343] Lustre: server umount lustre-OST0000 complete [15859.695897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15859.721450] LustreError: Skipped 1 previous similar message [15859.730083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15859.748307] Lustre: Skipped 1 previous similar message [15859.753507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15859.763152] Lustre: Skipped 1 previous similar message [15861.223158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15861.225953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15864.473082] Lustre: server umount lustre-MDT0000 complete [15868.849371] LustreError: 466525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856368 with bad export cookie 10058922516254665734 [15868.858962] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15868.862861] LustreError: 466525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15869.336750] Lustre: server umount lustre-MDT0001 complete [15881.590843] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [15884.775778] Key type lgssc unregistered [15885.408919] LNet: 468441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15885.431539] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15885.445795] LNet: Removed LNI 192.168.202.152@tcp [15886.495412] LNet: Removed LNI 192.168.202.152@tcp100 [15886.511050] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15886.521844] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15886.535717] LNet: Removed LNI 192.168.202.152@tcp101 [15887.583171] LNet: Removed LNI 192.168.202.152@tcp102 [15887.585906] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15887.603890] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15887.615423] LNet: Removed LNI 192.168.202.152@tcp103 [15888.673622] LNet: Removed LNI 192.168.202.152@tcp104 [15889.696148] LNet: Removed LNI 192.168.202.152@tcp105 [15889.723390] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15889.743158] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15889.750696] LNet: Removed LNI 192.168.202.152@tcp106 [15890.783158] LNet: Removed LNI 192.168.202.152@tcp107 [15891.871438] LNet: Removed LNI 192.168.202.152@tcp108 [15892.895475] LNet: Removed LNI 192.168.202.152@tcp109 [15893.983298] LNet: Removed LNI 192.168.202.152@tcp110 [15893.986168] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15893.999409] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15894.016848] LNet: Removed LNI 192.168.202.152@tcp111 [15895.071168] LNet: Removed LNI 192.168.202.152@tcp112 [15896.096231] LNet: Removed LNI 192.168.202.152@tcp113 [15897.119288] LNet: Removed LNI 192.168.202.152@tcp114 [15898.143389] LNet: Removed LNI 192.168.202.152@tcp115 [15899.167457] LNet: Removed LNI 192.168.202.152@tcp116 [15900.194426] LNet: Removed LNI 192.168.202.152@tcp117 [15901.215288] LNet: Removed LNI 192.168.202.152@tcp118 [15902.303174] LNet: Removed LNI 192.168.202.152@tcp119 [15902.309786] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15902.327102] LNetError: 468441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15902.341643] LNet: Removed LNI 192.168.202.152@tcp120 [15903.391162] LNet: Removed LNI 192.168.202.152@tcp121 [15904.417224] LNet: Removed LNI 192.168.202.152@tcp122 [15905.440498] LNet: Removed LNI 192.168.202.152@tcp123 [15906.463559] LNet: Removed LNI 192.168.202.152@tcp124 [15907.551389] LNet: Removed LNI 192.168.202.152@tcp125 [15908.575179] LNet: Removed LNI 192.168.202.152@tcp126 [15908.579790] LNet: Removed LNI 192.168.202.152@tcp127 [15909.599181] LNet: Removed LNI 192.168.202.152@tcp128 [15910.623717] LNet: Removed LNI 192.168.202.152@tcp129 [15911.647229] LNet: Removed LNI 192.168.202.152@tcp130 [15912.671250] LNet: Removed LNI 192.168.202.152@tcp131 [15913.700443] LNet: Removed LNI 192.168.202.152@tcp132 [15913.706683] LNet: Removed LNI 192.168.202.152@tcp133 [15913.709615] LNet: Removed LNI 192.168.202.152@tcp134 [15914.726377] LNet: Removed LNI 192.168.202.152@tcp135 [15915.680969] Key type .llcrypt unregistered [15915.686753] Key type ._llcrypt unregistered [15941.852937] Key type ._llcrypt registered [15941.854809] Key type .llcrypt registered [15941.958555] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [15959.044127] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [15959.754975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15959.782113] alg: No test for adler32 (adler32-zlib) [15960.682942] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [15960.864076] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [15962.471222] Key type lgssc registered [15963.899646] Lustre: Echo OBD driver; http://www.lustre.org/ [16006.336811] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:48:24 (1787856504) [16027.481599] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [16030.852151] Key type lgssc unregistered [16031.175760] LNet: 472536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16031.182749] LNetError: 472536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16031.222933] LNet: Removed LNI 192.168.202.152@tcp [16032.242494] Key type .llcrypt unregistered [16032.248080] Key type ._llcrypt unregistered [16048.467876] Key type ._llcrypt registered [16048.472882] Key type .llcrypt registered [16048.570841] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [16049.970415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16050.200447] alg: No test for adler32 (adler32-zlib) [16051.298537] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [16051.544503] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [16053.271319] Key type lgssc registered [16054.679849] Lustre: Echo OBD driver; http://www.lustre.org/ [16066.200280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16066.237090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16067.712633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16067.753720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16067.835416] Lustre: lustre-MDT0000: new disk, initializing [16067.954329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16068.000221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16072.516063] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16085.364672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16085.518837] Lustre: 473751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16085.582657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16085.593412] Lustre: Skipped 1 previous similar message [16085.700113] Lustre: lustre-MDT0001: new disk, initializing [16085.831594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16085.895072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16085.912450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16091.065522] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16101.536536] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16108.098697] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16116.006869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16116.327538] Lustre: lustre-OST0000: new disk, initializing [16116.333470] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16116.341650] Lustre: 475720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16116.444693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16122.600737] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16125.537465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16125.556960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16125.686764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16132.140704] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16136.461453] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing lnet_if_list [16142.392382] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [16142.418780] LNet: Added LNI 192.168.202.152@tcp10 [8/256/0/180] [16148.267900] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16148.318537] LNet: Added LNI 192.168.202.152@tcp11 [8/256/0/180] [16154.085732] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16154.142119] LNet: Added LNI 192.168.202.152@tcp12 [8/256/0/180] [16159.657417] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16159.706863] LNet: Added LNI 192.168.202.152@tcp13 [8/256/0/180] [16164.264933] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16164.293181] LNet: Added LNI 192.168.202.152@tcp14 [8/256/0/180] [16169.172803] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16169.218959] LNet: Added LNI 192.168.202.152@tcp15 [8/256/0/180] [16175.046976] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16175.093611] LNet: Added LNI 192.168.202.152@tcp16 [8/256/0/180] [16181.251246] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16181.281812] LNet: Added LNI 192.168.202.152@tcp17 [8/256/0/180] [16186.924442] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16186.951573] LNet: Added LNI 192.168.202.152@tcp18 [8/256/0/180] [16193.101945] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16193.133420] LNet: Added LNI 192.168.202.152@tcp19 [8/256/0/180] [16200.739448] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16200.784355] LNet: Added LNI 192.168.202.152@tcp20 [8/256/0/180] [16207.568371] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16207.601585] LNet: Added LNI 192.168.202.152@tcp21 [8/256/0/180] [16214.738817] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16214.816983] LNet: Added LNI 192.168.202.152@tcp22 [8/256/0/180] [16222.164088] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16222.203172] LNet: Added LNI 192.168.202.152@tcp23 [8/256/0/180] [16228.660248] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16228.683152] LNet: Added LNI 192.168.202.152@tcp24 [8/256/0/180] [16235.102457] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16235.132589] LNet: Added LNI 192.168.202.152@tcp25 [8/256/0/180] [16241.898285] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16241.919717] LNet: Added LNI 192.168.202.152@tcp26 [8/256/0/180] [16249.396787] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16249.427610] LNet: Added LNI 192.168.202.152@tcp27 [8/256/0/180] [16255.902924] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16255.936024] LNet: Added LNI 192.168.202.152@tcp28 [8/256/0/180] [16262.928467] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16263.003113] LNet: Added LNI 192.168.202.152@tcp29 [8/256/0/180] [16269.866493] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16269.895459] LNet: Added LNI 192.168.202.152@tcp30 [8/256/0/180] [16278.534691] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16278.567106] LNet: Added LNI 192.168.202.152@tcp31 [8/256/0/180] [16286.239704] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16286.290124] LNet: Added LNI 192.168.202.152@tcp32 [8/256/0/180] [16293.749918] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16293.770413] LNet: Added LNI 192.168.202.152@tcp33 [8/256/0/180] [16301.248439] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16301.296316] LNet: Added LNI 192.168.202.152@tcp34 [8/256/0/180] [16309.324276] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16309.354123] LNet: Added LNI 192.168.202.152@tcp35 [8/256/0/180] [16317.521584] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16317.545288] LNet: Added LNI 192.168.202.152@tcp36 [8/256/0/180] [16326.022755] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16326.045722] LNet: Added LNI 192.168.202.152@tcp37 [8/256/0/180] [16333.992720] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16334.037761] LNet: Added LNI 192.168.202.152@tcp38 [8/256/0/180] [16340.811259] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16340.837831] LNet: Added LNI 192.168.202.152@tcp39 [8/256/0/180] [16349.052248] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16349.066996] LNet: Added LNI 192.168.202.152@tcp40 [8/256/0/180] [16357.582108] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16357.607922] LNet: Added LNI 192.168.202.152@tcp41 [8/256/0/180] [16366.638873] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16366.664241] LNet: Added LNI 192.168.202.152@tcp42 [8/256/0/180] [16377.234386] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16377.267963] LNet: Added LNI 192.168.202.152@tcp43 [8/256/0/180] [16389.692057] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16389.733768] LNet: Added LNI 192.168.202.152@tcp44 [8/256/0/180] [16397.578721] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16397.664896] LNet: Added LNI 192.168.202.152@tcp45 [8/256/0/180] [16406.552759] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16406.588525] LNet: Added LNI 192.168.202.152@tcp46 [8/256/0/180] [16415.602067] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16415.625670] LNet: Added LNI 192.168.202.152@tcp47 [8/256/0/180] [16425.423259] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16425.446804] LNet: Added LNI 192.168.202.152@tcp48 [8/256/0/180] [16433.726392] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16433.741462] LNet: Added LNI 192.168.202.152@tcp49 [8/256/0/180] [16442.508558] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16442.528109] LNet: Added LNI 192.168.202.152@tcp50 [8/256/0/180] [16463.841557] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16463.846801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16463.855178] Lustre: Skipped 1 previous similar message [16463.874811] Lustre: Skipped 2 previous similar messages [16468.979176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16468.990216] Lustre: Skipped 2 previous similar messages [16470.311704] Lustre: server umount lustre-MDT0000 complete [16474.416160] LustreError: 473741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856974 with bad export cookie 11018828060253625841 [16474.417546] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16474.427130] LustreError: 473741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16474.839979] Lustre: server umount lustre-MDT0001 complete [16494.400616] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [16494.560873] Lustre: 473024:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856978/real 1787856978] req@000000006e89de99 x1874703471007360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787856994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16494.598729] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16494.615796] Lustre: Skipped 1 previous similar message [16500.703147] Lustre: 473023:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856984/real 1787856984] req@0000000048a03b6d x1874703471007488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16503.666570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16505.823234] Lustre: 473022:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856989/real 1787856989] req@000000009e651f18 x1874703471007744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16510.943300] Lustre: 473021:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856994/real 1787856994] req@00000000b363e00e x1874703471007872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857010 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16520.229928] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x98eabdbd9486286a [16520.259070] Lustre: MGC192.168.202.152@tcp: Connection restored to 0@lo (at 0@lo) [16520.604649] LustreError: 475713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16520.631780] LustreError: 475713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16520.698429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16525.794621] LustreError: 475714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16525.854656] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16528.885728] LustreError: 475713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16528.901503] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16533.999241] LustreError: 475714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16536.326573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16536.671107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16539.782200] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16542.798076] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16551.311685] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16559.368654] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16581.093736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16581.129238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16585.813163] Lustre: server umount lustre-OST0000 complete [16591.333123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16591.338705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16591.353234] Lustre: Skipped 2 previous similar messages [16591.371957] Lustre: Skipped 2 previous similar messages [16596.448929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16596.460256] Lustre: Skipped 1 previous similar message [16596.707212] Lustre: server umount lustre-MDT0000 complete [16600.549514] LustreError: 499976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16600.571340] LustreError: 499976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16601.252460] LustreError: 473741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857100 with bad export cookie 11018828060253628522 [16601.259757] LustreError: 473741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16601.260021] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16601.779608] Lustre: server umount lustre-MDT0001 complete [16612.479912] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [16615.450823] Key type lgssc unregistered [16615.809256] LNet: 502756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16615.815588] LNetError: 502756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16616.863861] LNet: Removed LNI 192.168.202.152@tcp [16616.865804] LNetError: 502756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16616.870350] LNet: Removed LNI 192.168.202.152@tcp10 [16616.871981] LNet: Removed LNI 192.168.202.152@tcp11 [16616.876557] LNet: Removed LNI 192.168.202.152@tcp12 [16616.879493] LNet: Removed LNI 192.168.202.152@tcp13 [16616.882685] LNet: Removed LNI 192.168.202.152@tcp14 [16616.890041] LNet: Removed LNI 192.168.202.152@tcp15 [16616.892674] LNet: Removed LNI 192.168.202.152@tcp16 [16616.896243] LNet: Removed LNI 192.168.202.152@tcp17 [16616.899756] LNet: Removed LNI 192.168.202.152@tcp18 [16616.903217] LNet: Removed LNI 192.168.202.152@tcp19 [16616.909092] LNet: Removed LNI 192.168.202.152@tcp20 [16616.915931] LNet: Removed LNI 192.168.202.152@tcp21 [16616.918688] LNet: Removed LNI 192.168.202.152@tcp22 [16616.921895] LNet: Removed LNI 192.168.202.152@tcp23 [16616.924618] LNet: Removed LNI 192.168.202.152@tcp24 [16616.926992] LNet: Removed LNI 192.168.202.152@tcp25 [16616.931466] LNet: Removed LNI 192.168.202.152@tcp26 [16616.933400] LNet: Removed LNI 192.168.202.152@tcp27 [16616.935875] LNet: Removed LNI 192.168.202.152@tcp28 [16616.938075] LNet: Removed LNI 192.168.202.152@tcp29 [16616.946611] LNet: Removed LNI 192.168.202.152@tcp30 [16616.950386] LNet: Removed LNI 192.168.202.152@tcp31 [16616.956379] LNet: Removed LNI 192.168.202.152@tcp32 [16616.957677] LNet: Removed LNI 192.168.202.152@tcp33 [16616.959915] LNet: Removed LNI 192.168.202.152@tcp34 [16616.962243] LNet: Removed LNI 192.168.202.152@tcp35 [16616.964563] LNet: Removed LNI 192.168.202.152@tcp36 [16616.967352] LNet: Removed LNI 192.168.202.152@tcp37 [16616.971874] LNet: Removed LNI 192.168.202.152@tcp38 [16616.973485] LNet: Removed LNI 192.168.202.152@tcp39 [16616.980415] LNet: Removed LNI 192.168.202.152@tcp40 [16616.983114] LNet: Removed LNI 192.168.202.152@tcp41 [16616.985410] LNet: Removed LNI 192.168.202.152@tcp42 [16616.987608] LNet: Removed LNI 192.168.202.152@tcp43 [16616.989807] LNet: Removed LNI 192.168.202.152@tcp44 [16616.992129] LNet: Removed LNI 192.168.202.152@tcp45 [16616.994378] LNet: Removed LNI 192.168.202.152@tcp46 [16617.000223] LNet: Removed LNI 192.168.202.152@tcp47 [16617.006888] LNet: Removed LNI 192.168.202.152@tcp48 [16617.011975] LNet: Removed LNI 192.168.202.152@tcp49 [16617.029788] LNet: Removed LNI 192.168.202.152@tcp50 [16617.877906] Key type .llcrypt unregistered [16617.879917] Key type ._llcrypt unregistered [16638.347159] Key type ._llcrypt registered [16638.349376] Key type .llcrypt registered [16638.472725] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [16656.277825] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [16657.942052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16657.954594] alg: No test for adler32 (adler32-zlib) [16659.245528] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [16659.673611] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [16661.359276] Key type lgssc registered [16663.263480] Lustre: Echo OBD driver; http://www.lustre.org/ [16705.348721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16705.405917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16706.695769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16706.724325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16706.855312] Lustre: lustre-MDT0000: new disk, initializing [16706.942196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16706.963407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16712.248330] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16723.819870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16723.956154] Lustre: 505953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16724.010599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16724.015256] Lustre: Skipped 1 previous similar message [16724.095087] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [16724.132672] Lustre: lustre-MDT0001: new disk, initializing [16724.225886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16724.271794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16724.285877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16729.777753] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16739.935894] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16746.330643] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16755.372383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16755.671223] Lustre: *** cfs_fail_loc=90f, val=0*** [16755.754427] Lustre: lustre-OST0000: new disk, initializing [16755.756915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16755.763660] Lustre: 507974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16755.857207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16762.224538] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16764.038235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16764.041327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16764.166017] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16773.103497] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16784.869584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16784.900082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16784.919519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16784.928545] Lustre: Skipped 1 previous similar message [16787.856809] Lustre: server umount lustre-OST0000 complete [16795.104191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16795.106359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16795.113311] Lustre: Skipped 2 previous similar messages [16795.140735] Lustre: Skipped 1 previous similar message [16797.607226] Lustre: server umount lustre-MDT0000 complete [16801.925700] LustreError: 505946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857301 with bad export cookie 6536749469466280271 [16801.932337] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16801.940344] LustreError: 505946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16802.377294] Lustre: server umount lustre-MDT0001 complete [16814.394364] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [16817.818891] Key type lgssc unregistered [16818.141223] LNet: 509894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16818.155292] LNetError: 509894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16818.168779] LNet: Removed LNI 192.168.202.152@tcp [16819.317624] Key type .llcrypt unregistered [16819.322742] Key type ._llcrypt unregistered [16841.397867] Key type ._llcrypt registered [16841.399891] Key type .llcrypt registered [16841.500507] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [16842.991697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16843.298617] alg: No test for adler32 (adler32-zlib) [16844.577793] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [16844.917520] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [16846.615346] Key type lgssc registered [16848.244373] Lustre: Echo OBD driver; http://www.lustre.org/ [16857.665154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16857.713123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16859.233979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16864.781456] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16868.325075] LustreError: 511181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16868.353648] LustreError: 511181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16873.440474] LustreError: 511182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16873.809818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16874.103779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16879.344810] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16887.423733] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16894.126874] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16901.814703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16902.150724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16908.830891] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16917.906417] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16929.731828] loop0: detected capacity change from 0 to 204800000 [16929.991594] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16930.013295] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16930.036639] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16938.148472] loop0: detected capacity change from 0 to 204800000 [16938.424714] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16938.437396] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16938.449195] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16944.546392] loop0: detected capacity change from 0 to 204800000 [16950.841869] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16950.992730] Lustre: fs15246-OST0000: new disk, initializing [16950.997513] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16951.003738] Lustre: 514775:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16951.077218] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16956.892172] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16967.622884] loop1: detected capacity change from 0 to 204800000 [16973.416986] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16973.555870] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16973.619861] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16973.697236] Lustre: fs15246-MDT0000: new disk, initializing [16973.836574] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16973.877638] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16973.947713] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16975.015186] Lustre: 515895:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000002384d94d x1874704302391680/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787857490 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16979.505769] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [16984.108796] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16984.118610] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16984.147047] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17001.953669] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17001.973204] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [17006.267335] Lustre: server umount fs15246-MDT0000 complete [17007.091702] LustreError: 513125:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17012.195610] LustreError: 511176:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17016.331767] Lustre: server umount fs15246-OST0000 complete [17031.138279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17031.149322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17031.171691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17032.168462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17032.176230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17032.189140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17033.916692] Lustre: server umount lustre-OST0000 complete [17037.794550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17037.798241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17037.800863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17037.821175] Lustre: Skipped 1 previous similar message [17037.836620] Lustre: Skipped 2 previous similar messages [17042.923932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17042.934953] Lustre: Skipped 1 previous similar message [17043.670109] Lustre: server umount lustre-MDT0000 complete [17047.686753] LustreError: 511163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857547 with bad export cookie 14305591260860145695 [17047.694341] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17047.707200] LustreError: 511163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17048.169997] Lustre: server umount lustre-MDT0001 complete [17060.854688] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [17063.176669] Key type lgssc unregistered [17063.412483] LNet: 518995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17063.421643] LNetError: 518995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17064.497746] LNet: Removed LNI 192.168.202.152@tcp [17065.257932] Key type .llcrypt unregistered [17065.259424] Key type ._llcrypt unregistered [17087.570850] Key type ._llcrypt registered [17087.577265] Key type .llcrypt registered [17087.752930] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [17090.064938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17090.199977] alg: No test for adler32 (adler32-zlib) [17091.616047] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [17092.194125] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [17094.098612] Key type lgssc registered [17095.919614] Lustre: Echo OBD driver; http://www.lustre.org/ [17137.615656] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [17148.560859] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [17186.615285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17186.626409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17188.001573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17188.067606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17188.187372] Lustre: lustre-MDT0000: new disk, initializing [17188.292850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17188.329735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17192.020962] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [17204.755909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17205.107871] Lustre: 523472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17205.208588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17205.212576] Lustre: Skipped 1 previous similar message [17205.278128] Lustre: lustre-MDT0001: new disk, initializing [17205.510643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17205.596707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17205.599731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17212.847871] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [17225.456660] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17232.592750] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17239.788094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17240.299690] Lustre: lustre-OST0000: new disk, initializing [17240.305788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17240.319077] Lustre: 525444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17240.446417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17246.807638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17246.839157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17247.002965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17247.394298] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [17258.302363] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17263.885687] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17264.136056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17269.353767] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17269.591857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17272.806715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17272.808747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17272.822572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17272.856957] Lustre: Skipped 1 previous similar message [17277.925913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17277.939208] Lustre: Skipped 2 previous similar messages [17283.045491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17283.055922] Lustre: Skipped 1 previous similar message [17286.111165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17286.379315] Lustre: server umount lustre-OST0000 complete [17292.769319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17292.781516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17292.807152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17296.828969] Lustre: server umount lustre-MDT0000 complete [17301.237867] LustreError: 523464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857800 with bad export cookie 12139011554032848946 [17301.239507] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17301.244081] LustreError: 523464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17301.651619] Lustre: server umount lustre-MDT0001 complete [17330.587538] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [17344.726367] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [17382.136906] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 17007 sec ========= 15:11:20 (1787857880) [17384.374810] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:11:22 (1787857882) === [17388.739112] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:11:26 (1787857886) === [17434.933787] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [17438.097278] Key type lgssc unregistered [17438.518480] LNet: 532109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17438.525584] LNetError: 532109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17438.548451] LNet: Removed LNI 192.168.202.152@tcp [17439.499467] Key type .llcrypt unregistered [17439.502198] Key type ._llcrypt unregistered