[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 674501545 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002231] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006651] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.010011] LSM: Security Framework initializing [ 0.011036] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013048] *** VALIDATE selinux *** [ 0.021093] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026447] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029123] *** VALIDATE tmpfs *** [ 0.031267] *** VALIDATE proc *** [ 0.032233] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039014] Speculative Store Bypass: Vulnerable [ 0.041834] debug: unmapping init [mem 0xffffffff93459000-0xffffffff93460fff] [ 0.043166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044699] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.053242] rcu: Hierarchical SRCU implementation. [ 0.055407] smp: Bringing up secondary CPUs ... [ 0.056605] x86: Booting SMP configuration: [ 0.057030] .... node #0, CPUs: #1 #2 #3 [ 0.060787] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.254166] node 0 deferred pages initialised in 188ms [ 0.258013] devtmpfs: initialized [ 0.259220] x86/mm: Memory block size: 128MB [ 0.261487] gcov: version magic: 0x41383552 [ 0.262688] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.266250] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.269591] pinctrl core: initialized pinctrl subsystem [ 0.272249] [ 0.273009] ************************************************************* [ 0.276020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278011] ** ** [ 0.280013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.282015] ** ** [ 0.284012] ** This means that this kernel is built to expose internal ** [ 0.286013] ** IOMMU data structures, which may compromise security on ** [ 0.287022] ** your system. ** [ 0.289020] ** ** [ 0.292023] ** If you see this message and you are not debugging the ** [ 0.295023] ** kernel, report this immediately to your vendor! ** [ 0.298018] ** ** [ 0.301020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.304019] ************************************************************* [ 0.308902] NET: Registered protocol family 16 [ 0.310625] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.314094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.316093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.320121] cpuidle: using governor menu [ 0.321737] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.322835] PCI: Using configuration type 1 for base access [ 0.324112] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.331255] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.333043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.335159] cryptd: max_cpu_qlen set to 1000 [ 0.337307] ACPI: Added _OSI(Module Device) [ 0.339019] ACPI: Added _OSI(Processor Device) [ 0.340017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.342017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.346755] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.351316] ACPI: Interpreter enabled [ 0.353062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.354013] ACPI: Using IOAPIC for interrupt routing [ 0.356155] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.359332] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.368011] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.372051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.376020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.381103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.384489] acpiphp: Slot [2] registered [ 0.385157] acpiphp: Slot [5] registered [ 0.387113] acpiphp: Slot [6] registered [ 0.388102] acpiphp: Slot [7] registered [ 0.389135] acpiphp: Slot [8] registered [ 0.390094] acpiphp: Slot [9] registered [ 0.392134] acpiphp: Slot [10] registered [ 0.393103] acpiphp: Slot [3] registered [ 0.394124] acpiphp: Slot [4] registered [ 0.396083] acpiphp: Slot [11] registered [ 0.397077] acpiphp: Slot [12] registered [ 0.398095] acpiphp: Slot [13] registered [ 0.399166] acpiphp: Slot [14] registered [ 0.401082] acpiphp: Slot [15] registered [ 0.402075] acpiphp: Slot [16] registered [ 0.403100] acpiphp: Slot [17] registered [ 0.404116] acpiphp: Slot [18] registered [ 0.406181] acpiphp: Slot [19] registered [ 0.408118] acpiphp: Slot [20] registered [ 0.409111] acpiphp: Slot [21] registered [ 0.410106] acpiphp: Slot [22] registered [ 0.412139] acpiphp: Slot [23] registered [ 0.413116] acpiphp: Slot [24] registered [ 0.415100] acpiphp: Slot [25] registered [ 0.416110] acpiphp: Slot [26] registered [ 0.417105] acpiphp: Slot [27] registered [ 0.419111] acpiphp: Slot [28] registered [ 0.420194] acpiphp: Slot [29] registered [ 0.422093] acpiphp: Slot [30] registered [ 0.423112] acpiphp: Slot [31] registered [ 0.425069] PCI host bridge to bus 0000:00 [ 0.426019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.429023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.431022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.434026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.436036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.438025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.439186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.442190] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.445565] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.460018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.466065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.469029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.471037] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.473030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.476749] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.479817] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.483057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.484987] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.490021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.500000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.506020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.513000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.522020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.532023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.548044] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.558228] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.578025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.597026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.655036] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.680542] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.690020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.698023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.721019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.733935] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.749025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.766038] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.809026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.828526] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.836024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.842022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.860025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.871827] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.881020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.887020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.899018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.907589] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.909305] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.911247] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.912277] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.914168] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.918222] iommu: Default domain type: Passthrough [ 0.919600] SCSI subsystem initialized [ 0.921128] ACPI: bus type USB registered [ 0.922000] usbcore: registered new interface driver usbfs [ 0.923055] usbcore: registered new interface driver hub [ 0.924071] usbcore: registered new device driver usb [ 0.925169] pps_core: LinuxPPS API ver. 1 registered [ 0.926008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.928060] PTP clock support registered [ 0.929152] EDAC MC: Ver: 3.0.0 [ 0.931108] PCI: Using ACPI for IRQ routing [ 0.932646] NetLabel: Initializing [ 0.933010] NetLabel: domain hash size = 128 [ 0.933913] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.935083] NetLabel: unlabeled traffic allowed by default [ 0.936141] vgaarb: loaded [ 0.937242] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.938009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.946462] clocksource: Switched to clocksource kvm-clock [ 1.045486] VFS: Disk quotas dquot_6.6.0 [ 1.047087] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.049669] *** VALIDATE ramfs *** [ 1.050927] *** VALIDATE hugetlbfs *** [ 1.052182] pnp: PnP ACPI init [ 1.054703] pnp: PnP ACPI: found 6 devices [ 1.074516] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.076974] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.078584] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.079946] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.081914] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.084296] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.087243] NET: Registered protocol family 2 [ 1.089585] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.094612] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.098533] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.102961] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.105663] TCP: Hash tables configured (established 65536 bind 65536) [ 1.107908] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.110162] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.113353] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.116164] NET: Registered protocol family 1 [ 1.118995] RPC: Registered named UNIX socket transport module. [ 1.120489] RPC: Registered udp transport module. [ 1.121666] RPC: Registered tcp transport module. [ 1.122836] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.124717] NET: Registered protocol family 44 [ 1.125928] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.127824] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.129179] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.130667] PCI: CLS 0 bytes, default 64 [ 1.131747] Unpacking initramfs... [ 2.546047] debug: unmapping init [mem 0xffff98883cc54000-0xffff98883ffbffff] [ 2.552170] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.554380] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.558589] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.038773] Initialise system trusted keyrings [ 3.040122] Key type blacklist registered [ 3.041937] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.050599] zbud: loaded [ 3.053699] *** VALIDATE nfs *** [ 3.055637] *** VALIDATE nfs4 *** [ 3.057655] pstore: using deflate compression [ 3.062295] Platform Keyring initialized [ 3.153766] NET: Registered protocol family 38 [ 3.155111] Key type asymmetric registered [ 3.156080] Asymmetric key parser 'x509' registered [ 3.157380] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.159721] io scheduler mq-deadline registered [ 3.161139] io scheduler kyber registered [ 3.162244] io scheduler bfq registered [ 3.163379] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.165643] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.167594] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.169246] ACPI: Power Button [PWRF] [ 3.172859] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.178217] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.268817] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.306692] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.328481] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.356703] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.383716] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.388579] Non-volatile memory driver v1.3 [ 3.390631] Linux agpgart interface v0.103 [ 3.428725] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.431611] vda: detected capacity change from 0 to 74645504 [ 3.445428] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.447455] vdb: detected capacity change from 0 to 1073741824 [ 3.463527] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.466565] vdc: detected capacity change from 0 to 2621440000 [ 3.488139] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.491703] vdd: detected capacity change from 0 to 2621440000 [ 3.518101] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.521886] vde: detected capacity change from 0 to 4294967296 [ 3.544235] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.548180] vdf: detected capacity change from 0 to 4294967296 [ 3.558481] libphy: Fixed MDIO Bus: probed [ 3.570640] usbcore: registered new interface driver usbserial_generic [ 3.573986] usbserial: USB Serial support registered for generic [ 3.577339] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.582216] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.584294] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.586816] mousedev: PS/2 mouse device common for all mice [ 3.589748] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.593506] rtc_cmos 00:05: RTC can wake from S4 [ 3.597586] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.600147] rtc_cmos 00:05: registered as rtc0 [ 3.606149] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.609754] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.610265] intel_pstate: CPU model not supported [ 3.617406] hid: raw HID events driver (C) Jiri Kosina [ 3.619542] usbcore: registered new interface driver usbhid [ 3.621581] usbhid: USB HID core driver [ 3.623890] drop_monitor: Initializing network drop monitor service [ 3.626516] Initializing XFRM netlink socket [ 3.629716] NET: Registered protocol family 10 [ 3.632503] Segment Routing with IPv6 [ 3.634312] NET: Registered protocol family 17 [ 3.638041] mpls_gso: MPLS GSO support [ 3.643145] RAS: Correctable Errors collector initialized. [ 3.644650] AVX version of gcm_enc/dec engaged. [ 3.645963] AES CTR mode by8 optimization enabled [ 3.719073] sched_clock: Marking stable (3719035099, 0)->(4630144131, -911109032) [ 3.722466] registered taskstats version 1 [ 3.725305] Loading compiled-in X.509 certificates [ 3.727796] zswap: loaded using pool lzo/zbud [ 3.753850] Key type big_key registered [ 3.766848] Key type encrypted registered [ 3.768442] ima: No TPM chip found, activating TPM-bypass! [ 3.770723] ima: Allocated hash algorithm: sha1 [ 3.772474] ima: No architecture policies found [ 3.774427] evm: Initialising EVM extended attributes: [ 3.776739] evm: security.selinux [ 3.778351] evm: security.ima [ 3.779731] evm: security.capability [ 3.781095] evm: HMAC attrs: 0x1 [ 3.783517] rtc_cmos 00:05: setting system clock to 2026-07-13 18:46:17 UTC (1783968377) [ 3.790539] debug: unmapping init [mem 0xffffffff94403000-0xffffffff945fffff] [ 3.793736] debug: unmapping init [mem 0xffffffff93182000-0xffffffff93458fff] [ 3.803089] Write protecting the kernel read-only data: 28672k [ 3.806649] debug: unmapping init [mem 0xffffffff91803000-0xffffffff919fffff] [ 3.809488] debug: unmapping init [mem 0xffffffff92114000-0xffffffff921fffff] [ 3.847258] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.856311] systemd[1]: Detected virtualization kvm. [ 3.857712] systemd[1]: Detected architecture x86-64. [ 3.860264] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.884910] systemd[1]: No hostname configured. [ 3.886050] systemd[1]: Set hostname to . [ 3.887645] random: systemd: uninitialized urandom read (16 bytes read) [ 3.889530] systemd[1]: Initializing machine ID from random generator. [ 3.933289] random: ln: uninitialized urandom read (6 bytes read) [ 4.021439] random: systemd: uninitialized urandom read (16 bytes read) [ 4.023650] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.028348] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.035201] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK [ 4.238027] random: fast init done ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.651646] device-mapper: uevent: version 1.0.3 [ 4.653938] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.358885] scsi host0: ata_piix [ 5.409843] virtio_net virtio0 ens2: renamed from eth0 [ 5.413238] scsi host1: ata_piix [ 5.416029] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.421338] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.067803] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.288768] random: crng init done [ 10.290164] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.844546] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.102781] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.378042] SELinux: Disabled at runtime. [ 12.442676] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.450415] systemd[1]: Detected virtualization kvm. [ 12.452273] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.977578] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.981520] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.987268] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.992352] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.995208] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.002097] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.007674] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ 13.248085] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.565338] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.014807] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.039373] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.205367] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.227171] EDAC sbridge: Ver: 1.1.2 [ 16.088908] Key type dns_resolver registered [ 16.401225] NFS: Registering the id_resolver key type [ 16.403392] Key type id_resolver registered [ 16.405098] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg203-server login: [ 52.494486] hrtimer: interrupt took 2569604 ns [ 70.276855] libcfs: loading out-of-tree module taints kernel. [ 70.354820] Key type ._llcrypt registered [ 70.359242] Key type .llcrypt registered [ 70.495473] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 91.396303] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 93.667693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.684242] alg: No test for adler32 (adler32-zlib) [ 95.605596] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 96.907523] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 98.785970] Key type lgssc registered [ 101.073350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 118.643018] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.737211] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 180.068477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 180.109826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.412995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 181.441610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 181.537240] Lustre: lustre-MDT0000: new disk, initializing [ 181.633222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.649389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.488285] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.173100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.291512] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.353193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.357303] Lustre: Skipped 1 previous similar message [ 203.493912] Lustre: lustre-MDT0001: new disk, initializing [ 203.594662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.688623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.709934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.841711] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.501402] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 226.216440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.543542] Lustre: lustre-OST0000: new disk, initializing [ 226.554647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.568267] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.690409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.509067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.516039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 232.629983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 234.331641] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.486934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.708989] Lustre: lustre-OST0001: new disk, initializing [ 251.715100] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.727326] Lustre: 9528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.817674] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.825225] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.759410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.771185] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.874778] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 274.133388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 287.213171] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 293.911555] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing check_logdir /tmp/testlogs/ [ 301.626370] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing yml_node [ 308.984618] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 312.694724] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 315.432445] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 317.372308] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 13 14:51:29 EDT 2026 [ 337.532020] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 339.531528] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 357.345445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 357.356231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.371517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.393266] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.398208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.412961] Lustre: Skipped 2 previous similar messages [ 359.418780] Lustre: Skipped 2 previous similar messages [ 362.834217] Lustre: server umount lustre-MDT0000 complete [ 366.697756] LustreError: 7460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968740 with bad export cookie 7890451888488656814 [ 366.719101] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.722821] LustreError: 7460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 369.634561] LustreError: 7829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.634902] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.638081] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.638093] Lustre: Skipped 1 previous similar message [ 369.651088] LustreError: 7829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 369.687371] Lustre: Skipped 1 previous similar message [ 371.685710] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.686682] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.715539] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 371.727222] Lustre: Skipped 2 previous similar messages [ 373.086734] Lustre: server umount lustre-MDT0001 complete [ 377.587318] Lustre: server umount lustre-OST0000 complete [ 382.076369] Lustre: server umount lustre-OST0001 complete [ 390.041896] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 402.323323] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 446.552458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.098855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 447.178260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 447.314079] Lustre: lustre-MDT0000: new disk, initializing [ 447.524781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.549942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 453.311294] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 465.727611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.955328] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 466.117993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 466.126374] Lustre: Skipped 1 previous similar message [ 466.245300] Lustre: lustre-MDT0001: new disk, initializing [ 466.376271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.409261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.422520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 472.114651] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 484.116167] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 491.488784] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 499.223733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.689212] Lustre: lustre-OST0000: new disk, initializing [ 499.696397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 499.705768] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 499.853711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.455353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 501.466296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 501.667655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 507.254705] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 517.936180] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 524.170826] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 524.488257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 528.774479] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 529.020652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 532.461468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.470992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.488685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 532.505581] Lustre: Skipped 1 previous similar message [ 532.541380] Lustre: Skipped 1 previous similar message [ 542.702724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.711532] Lustre: Skipped 3 previous similar messages [ 545.248745] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 545.634777] Lustre: server umount lustre-OST0000 complete [ 552.936064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.958394] Lustre: Skipped 1 previous similar message [ 555.975677] Lustre: server umount lustre-MDT0000 complete [ 558.063347] LustreError: 15093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.094363] LustreError: 15093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 560.471914] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968934 with bad export cookie 7890451888488658347 [ 560.487464] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.013317] Lustre: server umount lustre-MDT0001 complete [ 565.379605] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:55:37 (1783968937) [ 567.874613] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 569.830444] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:55:42 (1783968942) [ 576.856690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.332453] LustreError: 19462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.418605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 582.627813] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.783035] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 592.460066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 598.458237] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 606.005520] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 612.114977] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 620.416878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 629.565701] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 638.964278] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 647.654124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 647.665149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.666613] LustreError: Skipped 1 previous similar message [ 647.670908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 647.696873] Lustre: Skipped 1 previous similar message [ 647.736860] Lustre: Skipped 3 previous similar messages [ 653.470686] Lustre: server umount lustre-OST0000 complete [ 659.427106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 664.361348] Lustre: server umount lustre-MDT0000 complete [ 668.138374] LustreError: 19462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.163527] LustreError: 19462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 668.919965] LustreError: 21308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969042 with bad export cookie 7890451888488659222 [ 668.934769] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 668.940686] LustreError: 21308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 669.445300] Lustre: server umount lustre-MDT0001 complete [ 682.816454] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 686.341406] Key type lgssc unregistered [ 686.668631] LNet: 22999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.687635] LNetError: 22999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.703640] LNet: Removed LNI 192.168.202.103@tcp [ 687.829303] Key type .llcrypt unregistered [ 687.831107] Key type ._llcrypt unregistered [ 713.703195] Key type ._llcrypt registered [ 713.714617] Key type .llcrypt registered [ 713.871819] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 715.549394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 715.706960] alg: No test for adler32 (adler32-zlib) [ 716.909167] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 717.189054] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 718.953276] Key type lgssc registered [ 720.358552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 730.130053] loop: module loaded [ 730.235087] loop0: detected capacity change from 0 to 204800000 [ 730.484957] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 730.506972] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 731.079499] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.844378] loop0: detected capacity change from 0 to 204800000 [ 738.061404] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.076437] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.089234] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 743.694811] loop0: detected capacity change from 0 to 204800000 [ 750.790572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 750.825678] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.250759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 752.360208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 752.514917] Lustre: lustre-MDT0000: new disk, initializing [ 752.594986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.624400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 757.980548] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 768.555317] loop1: detected capacity change from 0 to 204800000 [ 774.619565] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 774.964413] Lustre: lustre-OST07c6: new disk, initializing [ 774.969606] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 774.979285] Lustre: Skipped 1 previous similar message [ 774.985126] Lustre: 26188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 775.060932] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 782.517031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 782.529958] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 782.614049] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 783.976775] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 789.668279] Lustre: 26398:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 789.699734] Lustre: 26398:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 789.713131] Lustre: 26398:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 789.732380] Lustre: 26398:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 789.747149] Lustre: 26398:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 789.762948] Lustre: 26398:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 809.433201] Lustre: 26398:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 809.445827] Lustre: 26398:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 809.458750] Lustre: 26398:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 809.475187] Lustre: 26398:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 809.484920] Lustre: 26398:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 809.493502] Lustre: 26398:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 814.406134] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 814.412700] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 814.421091] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 814.432095] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.439418] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 814.448172] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.455278] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 814.466075] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.474028] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 814.482953] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.496048] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 814.504426] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.799522] Lustre: 25174:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 817.816709] Lustre: 25174:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 817.829747] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 817.859847] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.868525] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 817.877101] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.885336] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 817.898113] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.911268] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 817.919633] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.931482] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 817.944867] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.721574] Lustre: 26398:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 822.752125] Lustre: 26398:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 822.768064] Lustre: 26398:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 822.787256] Lustre: 26398:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.812023] Lustre: 26398:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 822.822228] Lustre: 26398:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.836301] Lustre: 26398:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 822.843891] Lustre: 26398:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.850665] Lustre: 26398:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 822.860325] Lustre: 26398:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.873618] Lustre: 26398:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 822.886586] Lustre: 26398:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.476309] Lustre: 25174:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 834.503961] Lustre: 25174:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 834.515072] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 834.529904] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 834.535415] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 834.549853] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 834.563155] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 834.570548] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 834.579565] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 834.587311] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 834.592346] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 834.601385] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 889.834615] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.839326] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.863229] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 891.938130] Lustre: server umount lustre-OST07c6 complete [ 905.002464] Lustre: server umount lustre-MDT0000 complete [ 927.476054] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 931.234371] Key type lgssc unregistered [ 931.534560] LNet: 28772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.549303] LNetError: 28772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.560802] LNet: Removed LNI 192.168.202.103@tcp [ 932.756173] Key type .llcrypt unregistered [ 932.763577] Key type ._llcrypt unregistered [ 956.404350] Key type ._llcrypt registered [ 956.406453] Key type .llcrypt registered [ 956.500434] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 958.345489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 958.415105] alg: No test for adler32 (adler32-zlib) [ 959.674331] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 960.006879] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 961.752703] Key type lgssc registered [ 963.443820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.302451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.323666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.816659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.137412] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 985.057441] LustreError: 30045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.081769] LustreError: 30045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 989.132654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.555043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.637220] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1004.032348] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1011.460954] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1018.694450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1019.033675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.102728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1025.517284] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1034.860769] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1050.090951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.094246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1050.123410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1050.148177] Lustre: Skipped 1 previous similar message [ 1055.205693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1055.227712] Lustre: Skipped 1 previous similar message [ 1056.490588] Lustre: server umount lustre-OST0000 complete [ 1065.454675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.464950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1065.473655] Lustre: Skipped 2 previous similar messages [ 1065.500176] Lustre: Skipped 1 previous similar message [ 1068.628991] Lustre: server umount lustre-MDT0000 complete [ 1070.570379] LustreError: 30041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1070.601466] LustreError: 30041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1074.732423] LustreError: 30779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969448 with bad export cookie 18438833847787314543 [ 1074.741315] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1074.762066] LustreError: 30779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1075.266925] Lustre: server umount lustre-MDT0001 complete [ 1088.992809] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1094.215771] Key type lgssc unregistered [ 1094.636778] LNet: 33785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.655851] LNetError: 33785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.675759] LNet: Removed LNI 192.168.202.103@tcp [ 1096.168847] Key type .llcrypt unregistered [ 1096.176385] Key type ._llcrypt unregistered [ 1121.484720] Key type ._llcrypt registered [ 1121.490519] Key type .llcrypt registered [ 1121.608799] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1122.990313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1123.006956] alg: No test for adler32 (adler32-zlib) [ 1124.473823] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1124.802819] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1126.568193] Key type lgssc registered [ 1127.985227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1137.354893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1137.383197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1139.031136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1144.361648] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1148.400871] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.424282] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1153.105784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1153.652534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1159.205553] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1167.682788] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1175.362651] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1184.175421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1184.767297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.343832] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1205.444285] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1208.329338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1223.651820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.694939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1223.697395] Lustre: Skipped 1 previous similar message [ 1228.268667] Lustre: server umount lustre-OST0000 complete [ 1233.893942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.909654] Lustre: Skipped 1 previous similar message [ 1233.915177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1238.496717] Lustre: server umount lustre-MDT0000 complete [ 1239.010578] LustreError: 35052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.022327] LustreError: 35052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1242.035369] LustreError: 35035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969615 with bad export cookie 14538746328719685354 [ 1242.044288] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.047995] LustreError: 35035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1242.483514] Lustre: server umount lustre-MDT0001 complete [ 1253.096537] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1256.363767] Key type lgssc unregistered [ 1256.815400] LNet: 38611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.834928] LNetError: 38611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.858858] LNet: Removed LNI 192.168.202.103@tcp [ 1257.940279] Key type .llcrypt unregistered [ 1257.942376] Key type ._llcrypt unregistered [ 1283.486495] Key type ._llcrypt registered [ 1283.496153] Key type .llcrypt registered [ 1283.651363] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1285.045981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1285.359655] alg: No test for adler32 (adler32-zlib) [ 1286.511834] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1286.792202] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1288.496252] Key type lgssc registered [ 1290.115444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1299.619596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1299.649847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.552766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.518454] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1311.718663] LustreError: 40030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.742415] LustreError: 40030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1316.845803] LustreError: 40029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1317.089453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.707687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1323.325347] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1331.293385] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1336.810462] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1344.248063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1344.880077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.946973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1352.059317] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1360.237675] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1373.324673] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1373.676604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.704031] Lustre: Skipped 2 previous similar messages [ 1377.529577] Lustre: server umount lustre-MDT0000 complete [ 1378.415028] LustreError: 42522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.490857] LustreError: 40011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969755 with bad export cookie 8718194253216693214 [ 1381.492900] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1381.498045] LustreError: 40011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1383.525228] LustreError: 42522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.553204] LustreError: 42522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1383.561872] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.3@tcp (stopping) [ 1383.573783] Lustre: Skipped 3 previous similar messages [ 1383.912870] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.914911] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1386.917160] Lustre: Skipped 1 previous similar message [ 1388.094516] Lustre: server umount lustre-MDT0001 complete [ 1403.369951] Lustre: server umount lustre-OST0000 complete [ 1420.560630] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1423.936290] Key type lgssc unregistered [ 1424.250606] LNet: 43891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1424.260099] LNetError: 43891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.288317] LNet: Removed LNI 192.168.202.103@tcp [ 1425.448368] Key type .llcrypt unregistered [ 1425.449740] Key type ._llcrypt unregistered [ 1451.949957] Key type ._llcrypt registered [ 1451.958424] Key type .llcrypt registered [ 1452.073527] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1453.625676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1453.729087] alg: No test for adler32 (adler32-zlib) [ 1454.901917] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1455.243587] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1457.024954] Key type lgssc registered [ 1458.568533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1467.759427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1467.782678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.405058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.544088] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1479.662848] LustreError: 45311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.691246] LustreError: 45311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1483.700792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.156963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1488.693577] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1497.227236] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1503.627886] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1511.521354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1511.941771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.017978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1518.529611] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1527.097325] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1533.409231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1533.414771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.432097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1533.441191] Lustre: Skipped 1 previous similar message [ 1533.471030] Lustre: Skipped 1 previous similar message [ 1534.085777] Lustre: lustre-OST0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1538.533869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1538.544031] Lustre: Skipped 1 previous similar message [ 1539.362565] Lustre: server umount lustre-OST0000 complete [ 1559.009800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.034544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1559.037315] Lustre: Skipped 1 previous similar message [ 1564.132062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1564.140929] Lustre: Skipped 1 previous similar message [ 1564.795819] Lustre: server umount lustre-MDT0000 complete [ 1565.171757] LustreError: 46323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.199042] LustreError: 46323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1569.248059] LustreError: 45292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969942 with bad export cookie 971456779918785235 [ 1569.288637] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1575.791332] Lustre: server umount lustre-MDT0001 complete [ 1588.240844] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1591.716682] Key type lgssc unregistered [ 1591.978636] LNet: 49018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.984678] LNetError: 49018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.002448] LNet: Removed LNI 192.168.202.103@tcp [ 1592.825237] Key type .llcrypt unregistered [ 1592.827045] Key type ._llcrypt unregistered [ 1619.655506] Key type ._llcrypt registered [ 1619.662333] Key type .llcrypt registered [ 1619.831844] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1621.400419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1621.472868] alg: No test for adler32 (adler32-zlib) [ 1623.084865] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1623.551561] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1625.404754] Key type lgssc registered [ 1627.396298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1637.828201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1637.861212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.673855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.052539] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1650.161180] LustreError: 50435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.187182] LustreError: 50435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1654.720360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.149541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1661.120181] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1669.624308] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1676.395656] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1683.403821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1683.788556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.797554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1690.673966] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1698.960937] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1704.353217] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1708.416267] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1721.303723] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-1315h14m52s [ 1722.821522] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1725.924650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.932434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.946832] Lustre: Skipped 2 previous similar messages [ 1725.966990] Lustre: Skipped 2 previous similar messages [ 1726.595555] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1727.645334] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1727.657127] Lustre: Skipped 3 previous similar messages [ 1731.370468] Lustre: server umount lustre-MDT0000 complete [ 1731.697952] LustreError: 50432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.104969] LustreError: 50432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.117175] LustreError: 50432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1735.292780] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1737.842505] LustreError: 50429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.657617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.901169] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1743.334351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.439385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1744.380724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1749.119554] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1753.712321] Lustre: DEBUG MARKER: Wait for df (36794) ... [ 1756.113327] Lustre: DEBUG MARKER: done [ 1763.809910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1763.829662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.861984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1764.842223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.418582] Lustre: server umount lustre-OST0000 complete [ 1775.076454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1775.080870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.090198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.101330] Lustre: Skipped 1 previous similar message [ 1775.118419] Lustre: Skipped 3 previous similar messages [ 1785.314215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.320984] Lustre: Skipped 4 previous similar messages [ 1788.390595] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1788.653022] Lustre: server umount lustre-MDT0000 complete [ 1790.375956] LustreError: 50429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.399847] LustreError: 50429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1792.908580] LustreError: 50417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970166 with bad export cookie 10072487719225818376 [ 1792.913668] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1792.922095] LustreError: 50417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1793.305941] Lustre: server umount lustre-MDT0001 complete [ 1807.797589] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1811.339715] Key type lgssc unregistered [ 1811.619688] LNet: 55364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.627579] LNetError: 55364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.639192] LNet: Removed LNI 192.168.202.103@tcp [ 1813.139860] Key type .llcrypt unregistered [ 1813.144306] Key type ._llcrypt unregistered [ 1862.948218] Key type ._llcrypt registered [ 1862.962215] Key type .llcrypt registered [ 1863.321410] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1865.682281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.714249] alg: No test for adler32 (adler32-zlib) [ 1866.719991] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1867.321398] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1869.122186] Key type lgssc registered [ 1871.940298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1882.887087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1882.898322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.082055] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1884.092646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1884.212518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.487567] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1899.545809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.689966] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1899.779152] Lustre: 57574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1900.198467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.331082] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1914.156888] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1919.831630] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1927.080839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1927.264851] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1927.480856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.485117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1934.203303] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1942.591719] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1948.989443] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-1315h18m41s [ 1950.834822] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1957.885558] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1960.793493] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1960.865182] LustreError: 57581:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1981.053542] Lustre: lustre-MDT0000: Client 4b28bcc4-9b63-4b78-a9cd-2c3f1c6aad70 (at 192.168.202.3@tcp) reconnecting [ 2000.888098] LustreError: 57581:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2003.280338] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2005.813874] Lustre: DEBUG MARKER: done [ 2014.162901] Lustre: server umount lustre-OST0000 complete [ 2020.833489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.850856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.857713] Lustre: Skipped 1 previous similar message [ 2021.923965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.930525] Lustre: Skipped 1 previous similar message [ 2028.008937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.205960] Lustre: server umount lustre-MDT0000 complete [ 2031.072195] Lustre: 56898:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783970388/real 1783970388] req@00000000c0414513 x1870628379482624/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783970404 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2031.107360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.117616] Lustre: Skipped 1 previous similar message [ 2032.660269] LustreError: 59435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970406 with bad export cookie 2668433565957090035 [ 2032.663790] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2032.687064] LustreError: 59435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2033.558769] Lustre: server umount lustre-MDT0001 complete [ 2049.544899] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2054.102171] Key type lgssc unregistered [ 2054.747536] LNet: 61908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.750887] LNetError: 61908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.778009] LNet: Removed LNI 192.168.202.103@tcp [ 2055.887086] Key type .llcrypt unregistered [ 2055.888627] Key type ._llcrypt unregistered [ 2108.219624] Key type ._llcrypt registered [ 2108.222508] Key type .llcrypt registered [ 2108.356701] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2109.365960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2109.519629] alg: No test for adler32 (adler32-zlib) [ 2110.582273] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2110.911215] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2112.592516] Key type lgssc registered [ 2113.679337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2117.554086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2117.570388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.854334] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2118.870536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2119.059425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.163846] Lustre: Failing over lustre-MDT0000 [ 2120.413725] Lustre: server umount lustre-MDT0000 complete [ 2127.240314] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:21:39 (1783970499) [ 2134.435926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.979171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.725639] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2148.970306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.081706] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2149.127584] Lustre: 64709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2149.408872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2154.393725] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2161.754539] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2167.437451] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2176.343526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2176.676554] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2177.080102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.978359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2185.864314] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2193.906873] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2200.572737] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2208.743463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.760324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.768260] Lustre: Skipped 1 previous similar message [ 2213.775210] Lustre: server umount lustre-MDT0000 complete [ 2217.891654] LustreError: 64703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970591 with bad export cookie 16113461702227098549 [ 2217.902936] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2217.914263] LustreError: 64703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2218.327120] Lustre: server umount lustre-MDT0001 complete [ 2221.707575] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2227.014763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2234.336324] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783970592/real 1783970592] req@00000000aa7bd195 x1870628634273664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783970608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2234.382095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.409306] Lustre: Skipped 1 previous similar message [ 2236.638329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.432690] Lustre: 63632:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783970596/real 1783970596] req@000000007ba188ea x1870628634273792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783970612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2242.657654] LustreError: 63631:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dcd70c45 x1870628634274560/t0(0) o250->MGC192.168.202.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2243.081926] LustreError: 66561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.100248] LustreError: 66561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2243.203647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.333908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2244.256199] Lustre: 63633:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783970601/real 1783970601] req@000000005fa031d2 x1870628634274048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783970617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2245.287488] LustreError: 68485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.338022] LustreError: 68485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2249.187516] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2249.376106] Lustre: 63634:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783970607/real 1783970607] req@0000000020a52dbf x1870628634274176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783970623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2250.721427] LustreError: 66561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.731603] LustreError: 66561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2255.858194] LustreError: 68484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.875730] LustreError: 68484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2259.134356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.770302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2263.865285] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2265.084369] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2274.116988] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2281.482043] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2289.635497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.640354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.660707] Lustre: Skipped 2 previous similar messages [ 2289.668542] Lustre: Skipped 3 previous similar messages [ 2294.756790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.773696] Lustre: Skipped 3 previous similar messages [ 2295.253309] Lustre: server umount lustre-MDT0000 complete [ 2299.880510] LustreError: 68484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.916677] LustreError: 68484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2300.096869] LustreError: 64702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970673 with bad export cookie 16113461702227100831 [ 2300.108309] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.119445] LustreError: 64702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2300.434836] Lustre: server umount lustre-MDT0001 complete [ 2307.298572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2317.623543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.356338] Lustre: 63633:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783970678/real 1783970678] req@0000000077c359ba x1870628634302848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783970694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2320.401832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.517468] LustreError: 63631:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000075cb678f x1870628634303616/t0(0) o250->MGC192.168.202.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2324.899806] LustreError: 69217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.011995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.103920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2330.299321] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2331.107095] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783970688/real 1783970688] req@000000006b9a50d0 x1870628634303232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783970704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2331.154604] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2331.162448] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2331.191174] Lustre: Skipped 1 previous similar message [ 2340.063864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.963281] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2346.276200] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2354.751823] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2361.547146] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2371.041508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2371.058227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.108992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2375.373007] Lustre: server umount lustre-MDT0000 complete [ 2376.680640] LustreError: 71168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.704083] LustreError: 71168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2379.402139] LustreError: 64704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970753 with bad export cookie 16113461702227102210 [ 2379.416462] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.431168] LustreError: 64704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2379.793874] Lustre: server umount lustre-MDT0001 complete [ 2386.943675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2391.097379] Lustre: DEBUG MARKER: files compared the same [ 2403.958542] Lustre: server umount lustre-OST0000 complete [ 2419.351066] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2423.272142] Key type lgssc unregistered [ 2423.650910] LNet: 74445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2423.668288] LNetError: 74445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2423.695300] LNet: Removed LNI 192.168.202.103@tcp [ 2424.799660] Key type .llcrypt unregistered [ 2424.805071] Key type ._llcrypt unregistered [ 2454.158731] Key type ._llcrypt registered [ 2454.163536] Key type .llcrypt registered [ 2454.308179] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2455.636176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2455.838789] alg: No test for adler32 (adler32-zlib) [ 2456.989576] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2457.289325] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2459.048183] Key type lgssc registered [ 2460.528252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2469.084557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2469.121996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.837426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.076513] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug +malloc all [ 2481.123354] LustreError: 75860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.135802] LustreError: 75860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2485.469392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.857796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2491.357755] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug +malloc all [ 2500.099506] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2506.075491] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2514.438747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2514.983812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.033781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2522.982930] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug +malloc all [ 2532.056366] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2540.513348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.516306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2540.528778] Lustre: Skipped 1 previous similar message [ 2540.548365] Lustre: Skipped 1 previous similar message [ 2545.648048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2545.654601] Lustre: Skipped 1 previous similar message [ 2546.579375] Lustre: server umount lustre-OST0000 complete [ 2552.289573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2552.305216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.351233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2555.873449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2555.883808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.472331] Lustre: server umount lustre-MDT0000 complete [ 2560.995719] LustreError: 75861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.018751] LustreError: 75861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2561.856158] LustreError: 75841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970935 with bad export cookie 12793638741724603098 [ 2561.869901] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2562.217906] Lustre: server umount lustre-MDT0001 complete [ 2573.211866] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2575.827593] Key type lgssc unregistered [ 2576.128652] LNet: 79393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.137347] LNetError: 79393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2576.156933] LNet: Removed LNI 192.168.202.103@tcp [ 2577.169653] Key type .llcrypt unregistered [ 2577.175463] Key type ._llcrypt unregistered [ 2601.061465] Key type ._llcrypt registered [ 2601.063734] Key type .llcrypt registered [ 2601.125484] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2602.506829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2602.557045] alg: No test for adler32 (adler32-zlib) [ 2603.715394] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2603.918545] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2605.570367] Key type lgssc registered [ 2607.508917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2618.049264] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2618.091918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2633.825389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2638.945299] LustreError: 80787:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 2665.696411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.294472] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2680.285966] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2688.916333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.112801] Lustre: *** cfs_fail_loc=706, val=0*** [ 2689.476036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.620484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2693.736866] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2694.632394] LustreError: 80813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.663145] LustreError: 80813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2699.787851] LustreError: 80812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.845087] LustreError: 81873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.880549] LustreError: 81873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2703.118621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.365253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2708.041511] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2716.425903] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2723.417952] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2728.936715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.947804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2728.948021] Lustre: Skipped 1 previous similar message [ 2728.956098] Lustre: Skipped 1 previous similar message [ 2732.504065] Lustre: server umount lustre-OST0000 complete [ 2739.170733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2739.174786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.190392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.200551] Lustre: Skipped 1 previous similar message [ 2739.224138] Lustre: Skipped 1 previous similar message [ 2743.286713] Lustre: server umount lustre-MDT0000 complete [ 2744.295089] LustreError: 81877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.321346] LustreError: 81877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2748.205859] LustreError: 80794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971121 with bad export cookie 1598121038281104757 [ 2748.213697] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2748.222779] LustreError: 80794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2748.714333] Lustre: server umount lustre-MDT0001 complete [ 2761.949571] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2765.050981] Key type lgssc unregistered [ 2765.376658] LNet: 84403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2765.386612] LNetError: 84403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2766.448886] LNet: Removed LNI 192.168.202.103@tcp [ 2767.512060] Key type .llcrypt unregistered [ 2767.515078] Key type ._llcrypt unregistered [ 2796.807942] Key type ._llcrypt registered [ 2796.812021] Key type .llcrypt registered [ 2796.956328] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2798.294377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2798.483709] alg: No test for adler32 (adler32-zlib) [ 2799.639391] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2799.958800] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2801.721792] Key type lgssc registered [ 2803.165595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2811.938270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2811.964892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.797681] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2827.013120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2827.971907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2833.389993] LustreError: 86509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.428036] LustreError: 86509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2834.008093] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2837.484770] LustreError: 86513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.505711] LustreError: 86513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2842.597723] LustreError: 86512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.610497] LustreError: 86512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2847.713331] LustreError: 86513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.740570] LustreError: 86513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2847.988668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2848.216686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.324210] LustreError: 86509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.366044] LustreError: 86509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2855.501163] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2862.565230] LustreError: 86512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.592896] LustreError: 86512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2865.258145] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2865.272495] Lustre: MGS: Client 14c6675a-1a67-4294-bdba-15cf959142ca (at 0@lo) reconnecting [ 2865.294180] LustreError: 86523:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2865.478237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.557458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2870.656195] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2886.124600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2886.128284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.130721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2886.135868] Lustre: Skipped 1 previous similar message [ 2886.147906] Lustre: Skipped 1 previous similar message [ 2891.237929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2891.246554] Lustre: Skipped 1 previous similar message [ 2892.305597] Lustre: server umount lustre-OST0000 complete [ 2896.357064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2896.363538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.374589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.423344] Lustre: Skipped 1 previous similar message [ 2901.473626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.479590] Lustre: Skipped 2 previous similar messages [ 2902.416282] Lustre: server umount lustre-MDT0000 complete [ 2906.595130] LustreError: 86508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.618795] LustreError: 86508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2908.026172] Lustre: server umount lustre-MDT0001 complete [ 2912.661459] Lustre: server umount MGS complete [ 2927.140340] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:34:59 (1783971299) [ 2959.626403] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 2971.869121] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3010.648794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.592852] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3026.621690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.900888] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3026.968235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3027.058661] Lustre: lustre-MDT0001: new disk, initializing [ 3027.163197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3027.232428] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3027.256759] Lustre: 93698:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3027.272621] LustreError: 93698:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3032.556147] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3043.773696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3044.184688] Lustre: lustre-OST0000: new disk, initializing [ 3044.214930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3044.222241] Lustre: 94463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3044.363037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.308500] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3054.243747] Lustre: 94857:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000739cebfa x1870629356742144/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783971443 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3062.047430] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3069.029874] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3069.039685] Lustre: MGS: Client 0582448f-b163-465b-8599-5efb87249a95 (at 0@lo) reconnecting [ 3069.097305] Lustre: 95509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3069.108562] LustreError: 93684:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3069.139801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3069.308478] Lustre: lustre-MDT0000: new disk, initializing [ 3069.507525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.555373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3069.585740] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3070.627690] Lustre: 95533:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000038f78e64 x1870629356748928/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1783971460 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3074.351349] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3075.815440] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3079.809819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3079.823723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3079.890856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3105.762776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3105.776691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.796562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3111.587335] Lustre: server umount lustre-OST0000 complete [ 3114.991461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3114.995654] LustreError: Skipped 1 previous similar message [ 3114.998467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3114.999303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3115.010484] Lustre: Skipped 2 previous similar messages [ 3115.025444] Lustre: Skipped 4 previous similar messages [ 3129.312725] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3129.786596] Lustre: server umount lustre-MDT0000 complete [ 3130.349651] LustreError: 93669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.373529] LustreError: 93669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3141.276963] Lustre: server umount lustre-MDT0001 complete [ 3145.821755] Lustre: server umount MGS complete [ 3159.072923] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:38:51 (1783971531) [ 3179.501812] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3184.903421] Key type lgssc unregistered [ 3185.671131] LNet: 98034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3185.703601] LNetError: 98034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3185.737234] LNet: Removed LNI 192.168.202.103@tcp [ 3187.347840] Key type .llcrypt unregistered [ 3187.350257] Key type ._llcrypt unregistered [ 3206.775652] Key type ._llcrypt registered [ 3206.777637] Key type .llcrypt registered [ 3206.899964] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3208.509560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3208.885306] alg: No test for adler32 (adler32-zlib) [ 3210.159742] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 3210.573058] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3212.336210] Key type lgssc registered [ 3214.219857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3223.831183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3223.865037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3225.657186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.810231] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3235.814072] LustreError: 99258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.840566] LustreError: 99258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3240.407053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.809150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3245.425754] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3252.962205] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3259.133743] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3266.710996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3267.168368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3271.300157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3274.526984] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3282.507097] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3296.261053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3296.261683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3296.280864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3296.295144] Lustre: Skipped 1 previous similar message [ 3301.349211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3301.352046] Lustre: Skipped 1 previous similar message [ 3301.687480] Lustre: server umount lustre-OST0000 complete [ 3306.466140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3306.474849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3306.491497] Lustre: Skipped 1 previous similar message [ 3306.495123] Lustre: Skipped 1 previous similar message [ 3311.585954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.598728] Lustre: Skipped 1 previous similar message [ 3313.160399] Lustre: server umount lustre-MDT0000 complete [ 3316.707737] LustreError: 99254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.773820] LustreError: 99254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3318.351970] LustreError: 99240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971692 with bad export cookie 10089995296479296502 [ 3318.394832] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3319.083837] Lustre: server umount lustre-MDT0001 complete [ 3333.601353] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3337.344621] Key type lgssc unregistered [ 3337.907930] LNet: 102804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3337.912547] LNetError: 102804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3337.944699] LNet: Removed LNI 192.168.202.103@tcp [ 3339.557395] Key type .llcrypt unregistered [ 3339.564961] Key type ._llcrypt unregistered [ 3359.504275] Key type ._llcrypt registered [ 3359.508853] Key type .llcrypt registered [ 3359.719888] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3361.394250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3361.504940] alg: No test for adler32 (adler32-zlib) [ 3362.638529] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 3362.943082] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3364.624194] Key type lgssc registered [ 3365.966858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3373.385196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3373.385253] LustreError: 103897:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3373.398058] Lustre: Skipped 1 previous similar message [ 3373.414107] LustreError: 103896:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3373.421046] LustreError: 103897:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 3373.429350] LustreError: 103897:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3373.438465] LustreError: 103897:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3373.466496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.105828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.288838] LustreError: 103934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.308684] LustreError: 103934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3384.455086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.891792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3390.054293] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3396.656988] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing lsmod [ 3399.815083] LustreError: 105104:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3399.821180] LustreError: 105103:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3399.827594] LustreError: 105104:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4999 [ 3399.840273] LustreError: 105104:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3399.858866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3399.864270] LustreError: 105104:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3400.282839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.281456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.285375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.303773] Lustre: Skipped 1 previous similar message [ 3405.320870] Lustre: Skipped 2 previous similar messages [ 3409.443452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.470478] Lustre: Skipped 2 previous similar messages [ 3410.381525] Lustre: server umount lustre-MDT0000 complete [ 3414.461096] LustreError: 103912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971788 with bad export cookie 1232959933254481835 [ 3414.467620] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3414.471706] LustreError: 103912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3414.496947] LustreError: 105289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.498245] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3414.531918] Lustre: Skipped 1 previous similar message [ 3414.537210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3415.713435] LustreError: 104365:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000038f78e64 x1870629947067776/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3415.990457] Lustre: server umount lustre-MDT0001 complete [ 3420.463757] Lustre: server umount lustre-OST0000 complete [ 3430.867482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.491479] LustreError: 106165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.605666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.033580] LustreError: 106166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3437.651448] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3442.152577] LustreError: 106165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.070383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.710497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3454.924833] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3465.943119] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3473.916254] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3481.426563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3481.782414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.668987] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3499.924606] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3502.271932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3512.807971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.811131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3512.826720] Lustre: Skipped 1 previous similar message [ 3512.841263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3517.929517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3517.933516] Lustre: Skipped 1 previous similar message [ 3518.471876] Lustre: server umount lustre-OST0000 complete [ 3525.603614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3525.611666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3528.163833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3528.176140] Lustre: Skipped 3 previous similar messages [ 3529.839052] Lustre: server umount lustre-MDT0000 complete [ 3533.289881] LustreError: 106166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.329345] LustreError: 106166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3534.424859] LustreError: 106146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971908 with bad export cookie 1232959933254482535 [ 3534.439896] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3534.444575] LustreError: 106146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3534.916828] Lustre: server umount lustre-MDT0001 complete [ 3548.340451] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3551.538453] Key type lgssc unregistered [ 3552.047061] LNet: 109700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3552.055409] LNetError: 109700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3553.135821] LNet: Removed LNI 192.168.202.103@tcp [ 3554.393397] Key type .llcrypt unregistered [ 3554.401388] Key type ._llcrypt unregistered [ 3583.430732] Key type ._llcrypt registered [ 3583.434699] Key type .llcrypt registered [ 3583.585462] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3584.742315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3584.893098] alg: No test for adler32 (adler32-zlib) [ 3586.048134] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 3586.356447] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3588.136214] Key type lgssc registered [ 3589.482877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3598.059452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3598.073332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3599.576450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.034912] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3610.082407] LustreError: 111118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.095250] LustreError: 111118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3615.208393] LustreError: 111119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3615.517129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.986912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3620.757174] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3630.113722] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3637.578926] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3648.720126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3649.119178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3656.396272] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3666.086534] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3669.640837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3677.613500] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3682.283334] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3690.474919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3690.475919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.486208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3690.511322] Lustre: Skipped 1 previous similar message [ 3692.942961] Lustre: server umount lustre-OST0000 complete [ 3698.162367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3698.184206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3698.202603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3698.208568] Lustre: Skipped 1 previous similar message [ 3700.706461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3700.710480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.329238] Lustre: server umount lustre-MDT0000 complete [ 3705.839168] LustreError: 111119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.867247] LustreError: 111119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3709.689301] LustreError: 111099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972083 with bad export cookie 17790426942428600615 [ 3709.690206] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3709.701898] LustreError: 111099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3710.182151] Lustre: server umount lustre-MDT0001 complete [ 3721.329926] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3724.484082] Key type lgssc unregistered [ 3724.776888] LNet: 114773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3724.784422] LNetError: 114773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3725.799241] LNet: Removed LNI 192.168.202.103@tcp [ 3726.727831] Key type .llcrypt unregistered [ 3726.730254] Key type ._llcrypt unregistered [ 3745.653864] Key type ._llcrypt registered [ 3745.657250] Key type .llcrypt registered [ 3745.802723] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3747.080777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3747.178294] alg: No test for adler32 (adler32-zlib) [ 3748.240290] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 3748.536936] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3750.320242] Key type lgssc registered [ 3751.754513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3760.388945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3760.426728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.867805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.610737] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3771.376620] LustreError: 115998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.402301] LustreError: 115998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3776.268372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3776.489908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3776.754762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3782.275556] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3790.390945] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3796.275804] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3803.450416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3803.945559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.018522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3810.155323] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3816.492466] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3827.689233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3827.707374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.721743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3831.787531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3831.789212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3831.814772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.444576] Lustre: server umount lustre-OST0000 complete [ 3838.433891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3838.443225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.460584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3838.463343] Lustre: Skipped 1 previous similar message [ 3842.029450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.968473] Lustre: server umount lustre-MDT0000 complete [ 3847.138862] LustreError: 115999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3847.154794] LustreError: 115999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3847.811103] LustreError: 115978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972221 with bad export cookie 9676688960184844499 [ 3847.815214] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3847.835749] LustreError: 115978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3848.237314] Lustre: server umount lustre-MDT0001 complete [ 3860.665604] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3863.950834] Key type lgssc unregistered [ 3864.390616] LNet: 119530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3864.404536] LNetError: 119530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3864.443738] LNet: Removed LNI 192.168.202.103@tcp [ 3865.743938] Key type .llcrypt unregistered [ 3865.745802] Key type ._llcrypt unregistered [ 3894.590951] Key type ._llcrypt registered [ 3894.594178] Key type .llcrypt registered [ 3894.722162] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3896.237689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3896.456361] alg: No test for adler32 (adler32-zlib) [ 3897.687812] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 3897.975730] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3899.680264] Key type lgssc registered [ 3901.562410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3911.361251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3911.405565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3913.148631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.083310] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3922.408954] LustreError: 120958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3922.418600] LustreError: 120958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3927.533330] LustreError: 120957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.443633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.821933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3933.633709] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3941.508190] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3947.987689] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3955.486773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3955.765851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3957.811920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3963.017598] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3971.398966] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3981.407551] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3986.368072] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3986.383700] Lustre: Skipped 2 previous similar messages [ 3990.716709] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3990.795240] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3991.764990] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3991.774499] Lustre: Skipped 2 previous similar messages [ 3996.773722] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3996.780792] Lustre: Skipped 1 previous similar message [ 4002.550513] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4002.556814] Lustre: Skipped 2 previous similar messages [ 4004.406893] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.3@tcp 192.168.0.[2,10]@tcp [ 4013.058277] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.3/24@tcp in log lustre-MDT0000 [ 4013.064918] Lustre: Skipped 2 previous similar messages [ 4021.461833] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.3/24@tcp [ 4021.480299] Lustre: Skipped 1 previous similar message [ 4029.409586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4029.421886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.443047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4029.796600] Lustre: server umount lustre-OST0000 complete [ 4036.577538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.601617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4036.603812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4036.632412] Lustre: Skipped 1 previous similar message [ 4041.699354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4041.710073] Lustre: Skipped 1 previous similar message [ 4046.841132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4046.844861] Lustre: Skipped 1 previous similar message [ 4047.904287] Lustre: 120269:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783972405/real 1783972405] req@00000000d5297d59 x1870630508104832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783972421 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4047.931285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.940236] Lustre: Skipped 1 previous similar message [ 4048.352792] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4048.842941] Lustre: server umount lustre-MDT0000 complete [ 4052.960368] Lustre: 120267:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783972410/real 1783972410] req@000000009391b7ce x1870630508106368/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783972426 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4052.963990] LustreError: 122741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.981353] Lustre: 120267:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4053.000250] LustreError: 122741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4053.662549] LustreError: 123783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972427 with bad export cookie 8762274452176253662 [ 4053.670382] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4053.676373] LustreError: 123783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4054.190824] Lustre: server umount lustre-MDT0001 complete [ 4068.913529] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4071.983804] Key type lgssc unregistered [ 4072.473829] LNet: 125858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4072.482332] LNetError: 125858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4072.518195] LNet: Removed LNI 192.168.202.103@tcp [ 4073.645747] Key type .llcrypt unregistered [ 4073.648100] Key type ._llcrypt unregistered [ 4108.655813] Key type ._llcrypt registered [ 4108.657446] Key type .llcrypt registered [ 4108.748766] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4110.053061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4110.129964] alg: No test for adler32 (adler32-zlib) [ 4111.546944] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 4111.923943] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4113.728205] Key type lgssc registered [ 4115.086765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4125.227790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4125.292658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.210567] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4127.274626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.279636] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.3/24@tcp [ 4134.146628] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4136.441680] LustreError: 127442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4136.470043] LustreError: 127442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4141.546303] LustreError: 127443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4144.456696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4144.701517] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4144.741342] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.3/24@tcp [ 4144.874977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4151.184096] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4160.685577] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4166.861554] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4174.983919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4175.678706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.935746] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4192.543405] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4198.221360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4208.619338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.622513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4208.629770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.638855] Lustre: Skipped 1 previous similar message [ 4213.740121] Lustre: server umount lustre-OST0000 complete [ 4218.849786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.863585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4218.868708] Lustre: Skipped 1 previous similar message [ 4218.882592] Lustre: Skipped 2 previous similar messages [ 4223.969558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4223.987148] Lustre: Skipped 1 previous similar message [ 4229.091502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4229.096576] Lustre: Skipped 1 previous similar message [ 4232.160298] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4232.634227] Lustre: server umount lustre-MDT0000 complete [ 4234.210624] LustreError: 127439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.230306] LustreError: 127439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4236.273248] LustreError: 127423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972609 with bad export cookie 14719138216387303843 [ 4236.277986] LustreError: 127423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4236.278341] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4236.576788] Lustre: server umount lustre-MDT0001 complete [ 4252.002339] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4255.562666] Key type lgssc unregistered [ 4255.887073] LNet: 131030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4255.898644] LNetError: 131030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4255.918888] LNet: Removed LNI 192.168.202.103@tcp [ 4257.028275] Key type .llcrypt unregistered [ 4257.030654] Key type ._llcrypt unregistered [ 4288.338784] Key type ._llcrypt registered [ 4288.340909] Key type .llcrypt registered [ 4288.437891] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 4306.046673] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4307.600265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4307.643588] alg: No test for adler32 (adler32-zlib) [ 4308.743855] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 4309.092522] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4310.832328] Key type lgssc registered [ 4312.587398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4351.946994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4351.983282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.287540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4353.319410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4353.396271] Lustre: lustre-MDT0000: new disk, initializing [ 4353.509985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.525573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4357.788123] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4368.644679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.735221] Lustre: 134431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4368.806987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4368.810465] Lustre: Skipped 1 previous similar message [ 4368.907595] Lustre: lustre-MDT0001: new disk, initializing [ 4369.057537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4369.107140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4369.128437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4374.095366] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4384.122521] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4390.407741] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4398.148693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4398.556731] Lustre: lustre-OST0000: new disk, initializing [ 4398.563982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4398.577304] Lustre: 136400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4398.686931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4404.854025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4404.870446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4405.098698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4406.172294] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4416.443892] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4421.606899] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4421.837412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4427.422370] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4427.653456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4430.822975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4430.827083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.833963] LustreError: Skipped 1 previous similar message [ 4430.841154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4430.869063] Lustre: Skipped 1 previous similar message [ 4435.776986] Lustre: server umount lustre-OST0000 complete [ 4441.057904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4441.058367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4441.092263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4441.114183] Lustre: Skipped 2 previous similar messages [ 4446.178227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.637579] Lustre: server umount lustre-MDT0000 complete [ 4450.705289] LustreError: 134426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972824 with bad export cookie 2025599728945131403 [ 4450.716063] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4450.719307] LustreError: 134426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4451.506175] Lustre: server umount lustre-MDT0001 complete [ 4461.397419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.989749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.735800] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4467.184700] LustreError: 138617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4467.221781] LustreError: 138617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4472.291497] LustreError: 138616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.684539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.005540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4482.420332] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4490.610212] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4497.228420] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4503.918375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.218574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4511.104637] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4519.910798] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4525.170976] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4525.466822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4530.726566] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4530.982969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4540.050927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4540.261707] Lustre: lustre-OST0001: new disk, initializing [ 4540.265259] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4540.282916] Lustre: 141999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4540.512137] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4547.428403] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4548.825252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4548.844321] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4548.954883] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4558.017880] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4558.317328] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4563.777980] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4564.292072] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4572.416572] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b06436b8800.ost_server_uuid 50 [ 4574.163617] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b06436b8800.ost_server_uuid in IDLE state after 0 sec [ 4587.795894] Lustre: server umount lustre-OST0001 complete [ 4590.049717] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4590.058762] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4590.065971] LustreError: Skipped 1 previous similar message [ 4590.067545] LustreError: 140447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.123920] Lustre: Skipped 2 previous similar messages [ 4593.640157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4593.643834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.683621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4593.692480] Lustre: Skipped 1 previous similar message [ 4595.188476] LustreError: 140449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4595.229361] LustreError: 140449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4597.785814] Lustre: server umount lustre-OST0000 complete [ 4604.920473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4604.929872] LustreError: Skipped 1 previous similar message [ 4604.935557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.958756] Lustre: Skipped 1 previous similar message [ 4604.969064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4604.975801] Lustre: Skipped 3 previous similar messages [ 4609.909577] Lustre: server umount lustre-MDT0000 complete [ 4610.534648] LustreError: 142268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.556743] LustreError: 142268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4614.922663] LustreError: 141998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972988 with bad export cookie 2025599728945132285 [ 4614.924162] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4614.932182] LustreError: 141998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4615.463110] Lustre: server umount lustre-MDT0001 complete [ 4636.001767] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4638.732382] Key type lgssc unregistered [ 4639.091642] LNet: 145002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4639.096708] LNetError: 145002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4640.167491] LNet: Removed LNI 192.168.202.103@tcp [ 4641.096404] Key type .llcrypt unregistered [ 4641.098704] Key type ._llcrypt unregistered [ 4678.696685] Key type ._llcrypt registered [ 4678.698812] Key type .llcrypt registered [ 4678.803538] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 4693.140470] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4694.555381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4694.568791] alg: No test for adler32 (adler32-zlib) [ 4695.620482] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 4695.900027] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4697.616692] Key type lgssc registered [ 4698.795828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4740.109118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4740.128335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.395717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4741.427151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4741.489595] Lustre: lustre-MDT0000: new disk, initializing [ 4741.589096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4741.612370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4746.179453] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4756.545657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4756.748194] Lustre: 149161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4756.823528] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4756.826849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4756.834959] Lustre: Skipped 1 previous similar message [ 4756.896476] Lustre: lustre-MDT0001: new disk, initializing [ 4756.970552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4757.036380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4757.056753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4761.725032] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4771.876322] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4777.874942] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4785.442966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4785.776511] Lustre: lustre-OST0000: new disk, initializing [ 4785.786009] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4785.794736] Lustre: 151129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4785.912435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.734587] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4794.986018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4795.004443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4795.131065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4802.112302] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4807.764388] Lustre: Failing over lustre-OST0000 [ 4808.211167] Lustre: server umount lustre-OST0000 complete [ 4824.608491] Lustre: 147383:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783973182/real 1783973182] req@00000000adc8ba65 x1870631344834304/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1783973198 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4824.642384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4826.592166] Lustre: 147382:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783973184/real 1783973184] req@0000000091720549 x1870631344835200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783973200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4826.625698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4827.399293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4827.645256] Lustre: 147383:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783973184/real 1783973184] req@00000000c2e2c7b8 x1870631344835328/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783973200 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4827.670294] Lustre: 147383:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4827.678638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4827.691213] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4831.712286] Lustre: 147384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783973189/real 1783973189] req@00000000eecb3080 x1870631344836480/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783973205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4831.731111] Lustre: 147384:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4834.003424] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4834.815718] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4836.834380] Lustre: 147381:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783973194/real 1783973194] req@00000000f4a3367c x1870631344837632/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783973210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4836.961044] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4836.967964] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4838.051390] Lustre: Failing over lustre-MDT0000 [ 4838.329815] Lustre: server umount lustre-MDT0000 complete [ 4838.881578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4838.889499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4845.028343] LustreError: 149168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4845.066443] LustreError: 149168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4846.685815] LustreError: 151802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4850.146042] LustreError: 150241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4850.160758] LustreError: 150241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4855.278773] LustreError: 149171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.307138] LustreError: 149171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4856.288143] Lustre: 147384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783973213/real 1783973213] req@000000004cf8b75d x1870631344850048/t0(0) o400->MGC192.168.202.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783973229 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4856.320183] Lustre: 147384:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4856.324143] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4857.698123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.411076] LustreError: 149171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.436611] LustreError: 149171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4867.036623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4867.113054] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4867.169844] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4872.209923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4872.223247] Lustre: Skipped 1 previous similar message [ 4872.263786] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4872.390562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4873.460565] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4885.796742] Lustre: server umount lustre-OST0000 complete [ 4892.643085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4892.653955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4892.654707] Lustre: Skipped 3 previous similar messages [ 4892.677946] Lustre: Skipped 1 previous similar message [ 4897.781400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4899.931395] Lustre: server umount lustre-MDT0000 complete [ 4902.886698] LustreError: 149172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4902.914213] LustreError: 149172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4903.905467] Lustre: 147384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783973261/real 1783973261] req@00000000b131f240 x1870631344871296/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783973277 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4903.926305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4904.666160] LustreError: 149152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973278 with bad export cookie 15458016435958692078 [ 4904.667636] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4904.687324] LustreError: 149152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4905.066304] Lustre: server umount lustre-MDT0001 complete [ 4918.825758] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4921.997795] Key type lgssc unregistered [ 4922.264236] LNet: 154786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4922.274665] LNetError: 154786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4922.287934] LNet: Removed LNI 192.168.202.103@tcp [ 4923.192910] Key type .llcrypt unregistered [ 4923.199384] Key type ._llcrypt unregistered [ 4962.283195] Key type ._llcrypt registered [ 4962.285748] Key type .llcrypt registered [ 4962.405680] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4964.044889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4964.104994] alg: No test for adler32 (adler32-zlib) [ 4965.476645] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 4965.737248] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4967.424274] Key type lgssc registered [ 4968.785321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4976.172587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4976.184798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4977.716924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4982.103296] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4987.889747] LustreError: 156307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4987.903978] LustreError: 156307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4991.739515] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4992.993409] LustreError: 156306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.849740] LustreError: 156335:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4995.860597] LustreError: 156335:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4995.867320] LustreError: 156335:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4996.286467] Lustre: server umount lustre-MDT0000 complete [ 5012.747719] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5021.895766] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5055.585674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5055.781462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5055.805175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5055.864671] Lustre: lustre-MDT0000: new disk, initializing [ 5055.941467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5055.971794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5060.696585] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5071.307982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.426099] Lustre: 159642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5071.491930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5071.501128] Lustre: Skipped 1 previous similar message [ 5071.616093] Lustre: lustre-MDT0001: new disk, initializing [ 5071.739307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5071.767127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5071.774562] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5076.824931] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5087.761034] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5094.128650] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5101.133320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5101.483725] Lustre: lustre-OST0000: new disk, initializing [ 5101.487747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5101.493256] Lustre: 161613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5101.585663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5107.285108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5107.298641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5107.415870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5108.447318] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5119.305841] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5124.334577] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5127.278142] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:11:38 (1783973498) [ 5358.560631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5358.570402] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.572718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5358.606156] Lustre: Skipped 1 previous similar message [ 5362.089769] Lustre: server umount lustre-MDT0000 complete [ 5363.686563] LustreError: 163395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5363.722566] LustreError: 163395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5366.225457] LustreError: 159633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973739 with bad export cookie 15063329562564398005 [ 5366.243962] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5368.800970] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.801273] LustreError: 159649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.822379] Lustre: Skipped 1 previous similar message [ 5368.825799] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5368.875817] Lustre: Skipped 2 previous similar messages [ 5371.811614] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5372.753560] Lustre: server umount lustre-MDT0001 complete [ 5383.616815] Lustre: server umount lustre-OST0000 complete [ 5394.192958] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5404.741289] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5444.591857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5444.836189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5444.864568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5444.969610] Lustre: lustre-MDT0000: new disk, initializing [ 5445.033979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5445.046085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5449.689491] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5461.731362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5461.897325] Lustre: 166271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5461.969296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5461.975638] Lustre: Skipped 1 previous similar message [ 5462.027406] Lustre: lustre-MDT0001: new disk, initializing [ 5462.105904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5462.153461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5462.167191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5467.198225] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5477.265423] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5483.944837] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5491.581032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5491.961871] Lustre: lustre-OST0000: new disk, initializing [ 5491.965727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5491.974480] Lustre: 168239:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5492.059260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5500.538950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5500.551376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5500.776656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5501.106922] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5513.389321] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5517.662809] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5517.953550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5523.301262] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5523.585780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5526.497987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5526.501938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5526.516178] LustreError: Skipped 1 previous similar message [ 5526.532329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5526.544299] Lustre: Skipped 1 previous similar message [ 5526.572471] Lustre: Skipped 1 previous similar message [ 5531.673845] Lustre: server umount lustre-OST0000 complete [ 5536.737548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5536.749216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5536.751665] Lustre: Skipped 1 previous similar message [ 5536.765641] Lustre: Skipped 1 previous similar message [ 5542.928441] Lustre: server umount lustre-MDT0000 complete [ 5546.946107] LustreError: 166262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973920 with bad export cookie 15063329562564556835 [ 5546.955607] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5546.973684] LustreError: 166262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5547.015828] LustreError: 155619:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 5547.262830] Lustre: server umount lustre-MDT0001 complete [ 5556.453271] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:18:48 (1783973928) [ 5575.179367] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5585.650537] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5623.423704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.628369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5623.650507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5623.718600] Lustre: lustre-MDT0000: new disk, initializing [ 5623.828571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5623.846353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5629.136540] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5640.058946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.375287] Lustre: 172998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5640.683484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5645.514133] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5656.435175] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5662.176529] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5672.931434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5673.246201] Lustre: lustre-OST0000: new disk, initializing [ 5673.249213] Lustre: Skipped 1 previous similar message [ 5673.254271] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5673.257304] Lustre: Skipped 2 previous similar messages [ 5673.264423] Lustre: 174975:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5680.876196] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5683.259485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5683.281790] Lustre: Skipped 1 previous similar message [ 5683.284811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5683.334549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5690.605710] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5699.046903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.048561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5699.049184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5699.049189] Lustre: Skipped 2 previous similar messages [ 5699.061889] Lustre: Skipped 1 previous similar message [ 5699.064650] LustreError: Skipped 1 previous similar message [ 5702.610447] Lustre: server umount lustre-OST0000 complete [ 5707.237228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5707.249143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.498421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.504835] Lustre: Skipped 7 previous similar messages [ 5721.056673] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5721.304505] Lustre: server umount lustre-MDT0000 complete [ 5723.625769] LustreError: 173008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.673896] LustreError: 173008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5725.586646] LustreError: 172988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974099 with bad export cookie 15063329562564557717 [ 5725.591368] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5725.595985] LustreError: 172988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5725.996395] Lustre: server umount lustre-MDT0001 complete [ 5737.600881] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:21:49 (1783974109) [ 5759.108592] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5771.781479] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5811.498231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.982836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5812.046611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5812.180828] Lustre: lustre-MDT0000: new disk, initializing [ 5812.293187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5812.305843] Lustre: Skipped 2 previous similar messages [ 5812.348287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5818.006820] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5830.156794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.370963] Lustre: 179300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5830.699061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5836.284793] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5847.519467] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5854.884722] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5864.195849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5864.388765] Lustre: 181279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5869.810670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5870.060126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5873.217043] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5884.105701] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5895.654067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5895.662961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.669503] LustreError: Skipped 1 previous similar message [ 5895.670986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.701748] Lustre: Skipped 2 previous similar messages [ 5897.652568] Lustre: server umount lustre-OST0000 complete [ 5909.401588] Lustre: server umount lustre-MDT0000 complete [ 5911.023170] LustreError: 179309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.035943] LustreError: 179309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5914.172583] LustreError: 179293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974287 with bad export cookie 15063329562564558802 [ 5914.179253] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5914.182230] LustreError: 179293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5914.754676] Lustre: server umount lustre-MDT0001 complete [ 5923.885238] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5926.585841] Key type lgssc unregistered [ 5927.108398] LNet: 183135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5927.125193] LNetError: 183135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5927.148924] LNet: Removed LNI 192.168.202.103@tcp [ 5928.825729] Key type .llcrypt unregistered [ 5928.835258] Key type ._llcrypt unregistered [ 5957.901534] Key type ._llcrypt registered [ 5957.904445] Key type .llcrypt registered [ 5958.072834] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5959.425751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5959.656991] alg: No test for adler32 (adler32-zlib) [ 5960.853541] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 5961.180835] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5962.945574] Key type lgssc registered [ 5964.600247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5973.951454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5973.978218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5975.460380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5981.199249] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5985.765811] LustreError: 184557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5985.796065] LustreError: 184557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5990.889606] LustreError: 184556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5991.024870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.557540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5997.239318] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6005.348531] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6011.470789] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6019.543176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6019.885983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.927526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6026.531402] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6035.497721] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6056.933569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.941299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6056.949772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6056.957320] Lustre: Skipped 1 previous similar message [ 6061.120264] Lustre: server umount lustre-OST0000 complete [ 6067.183165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6067.205122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6067.217066] Lustre: Skipped 2 previous similar messages [ 6070.435856] Lustre: server umount lustre-MDT0000 complete [ 6072.289624] LustreError: 185286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6072.301683] LustreError: 185286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6074.122265] LustreError: 184536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974447 with bad export cookie 13973555313179955994 [ 6074.136920] LustreError: 184536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6074.138817] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6074.632112] Lustre: server umount lustre-MDT0001 complete [ 6087.018603] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6090.350741] Key type lgssc unregistered [ 6090.673825] LNet: 188114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6090.681220] LNetError: 188114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6090.693404] LNet: Removed LNI 192.168.202.103@tcp [ 6091.544636] Key type .llcrypt unregistered [ 6091.547670] Key type ._llcrypt unregistered [ 6119.802186] Key type ._llcrypt registered [ 6119.803747] Key type .llcrypt registered [ 6120.003483] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6121.704651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6121.733754] alg: No test for adler32 (adler32-zlib) [ 6122.933087] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 6123.112400] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6124.784235] Key type lgssc registered [ 6125.986879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6134.385897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6134.401657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6135.983694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6140.922883] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6145.508696] LustreError: 189536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.527307] LustreError: 189536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6150.036106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6150.724241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6155.928806] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6164.400656] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6170.602846] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6177.231143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6177.549686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.663485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6182.683779] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6190.761983] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6201.317945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6201.319203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6201.336214] LustreError: Skipped 1 previous similar message [ 6201.339771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6201.364518] Lustre: Skipped 1 previous similar message [ 6202.479047] Lustre: lustre-OST0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 6202.483431] Lustre: Skipped 1 previous similar message [ 6205.247330] Lustre: server umount lustre-OST0000 complete [ 6216.513637] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b065167d800.ost_server_uuid 50 [ 6219.330633] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6237.154251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.159434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6237.169546] Lustre: Skipped 1 previous similar message [ 6237.183816] Lustre: Skipped 1 previous similar message [ 6238.974664] Lustre: server umount lustre-MDT0000 complete [ 6241.316906] LustreError: 191544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6241.353717] LustreError: 191544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6243.373270] LustreError: 189516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974617 with bad export cookie 14261685925112871702 [ 6243.375244] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6243.390741] LustreError: 189516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6244.049877] Lustre: server umount lustre-MDT0001 complete [ 6252.978554] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 16:30:25 (1783974625) [ 6259.744664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6260.324242] LustreError: 193448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6260.394912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.699845] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6265.838534] LustreError: 193449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.950174] LustreError: 193448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6274.221554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.633297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6279.327490] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6287.141717] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6293.456239] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6302.400993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6302.797627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6305.953366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6310.803971] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6320.585805] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6331.528589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6332.077411] Lustre: lustre-OST0001: new disk, initializing [ 6332.079829] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6332.098343] Lustre: 196295:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6332.404259] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6340.256403] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6341.285264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6341.293364] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6341.401916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6351.205801] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6356.449853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6356.464203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.486539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6362.084154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6362.104266] Lustre: Skipped 3 previous similar messages [ 6368.224114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6368.350066] Lustre: server umount lustre-OST0000 complete [ 6372.324530] LustreError: 195274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6372.349191] LustreError: 195274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6375.346719] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6375.615374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6381.097238] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6381.311546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6392.805335] LustreError: 195274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.832178] LustreError: 195274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6397.921193] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6397.930554] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6397.939294] LustreError: Skipped 1 previous similar message [ 6397.943147] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6397.943156] Lustre: Skipped 4 previous similar messages [ 6397.980874] Lustre: Skipped 2 previous similar messages [ 6401.920934] Lustre: server umount lustre-OST0001 complete [ 6407.650717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6407.661837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.136378] Lustre: server umount lustre-MDT0000 complete [ 6416.389470] LustreError: 193429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974790 with bad export cookie 14261685925112872780 [ 6416.390165] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6416.402707] LustreError: 193429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6417.021132] Lustre: server umount lustre-MDT0001 complete [ 6444.370546] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:33:36 (1783974816) [ 6453.018337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.347701] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6453.406144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6453.721968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6459.633771] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6467.859412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6467.983534] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6468.075372] Lustre: 199739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6474.078507] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6482.088424] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6488.502548] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6496.382558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6496.541396] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6505.852301] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6505.875231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6516.346957] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6523.795953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6523.942471] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6524.289228] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6524.292960] Lustre: Skipped 2 previous similar messages [ 6532.670692] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6541.720601] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6546.914635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6546.920662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6546.923873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6546.936924] Lustre: Skipped 2 previous similar messages [ 6546.955670] Lustre: Skipped 4 previous similar messages [ 6551.085129] Lustre: server umount lustre-OST0000 complete [ 6552.679916] LustreError: 201971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6552.709843] LustreError: 201971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6569.455228] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6569.462291] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6569.473821] LustreError: Skipped 2 previous similar messages [ 6569.530751] Lustre: Skipped 1 previous similar message [ 6575.353795] Lustre: server umount lustre-OST0001 complete [ 6581.218904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6581.221875] Lustre: Skipped 8 previous similar messages [ 6586.267689] Lustre: server umount lustre-MDT0000 complete [ 6590.570216] LustreError: 199730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974964 with bad export cookie 14261685925112874187 [ 6590.594976] LustreError: 199730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6590.596876] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6591.003385] Lustre: server umount lustre-MDT0001 complete [ 6621.681825] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:36:33 (1783974993) [ 6642.415729] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 6656.067077] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6697.380289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6697.747265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6697.800634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6697.911586] Lustre: lustre-MDT0000: new disk, initializing [ 6698.076585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6698.115599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6704.148460] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6717.746516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.916110] Lustre: 207838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6717.994446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6718.002343] Lustre: Skipped 1 previous similar message [ 6718.149216] Lustre: lustre-MDT0001: new disk, initializing [ 6718.329595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6718.360926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6723.480555] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6733.610741] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6740.288947] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6751.375581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6751.896481] Lustre: lustre-OST0000: new disk, initializing [ 6751.900246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6751.905579] Lustre: 209812:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6757.998098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6758.012974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6758.156533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6762.256404] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6773.068059] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6778.763937] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6779.207622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6786.240039] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6786.587995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6794.224908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6794.231183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6794.237602] Lustre: Skipped 3 previous similar messages [ 6794.238437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6794.238441] Lustre: Skipped 2 previous similar messages [ 6794.259467] LustreError: Skipped 2 previous similar messages [ 6795.541625] Lustre: server umount lustre-OST0000 complete [ 6806.891911] Lustre: server umount lustre-MDT0000 complete [ 6809.570663] LustreError: 207850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6809.585547] LustreError: 207850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 6811.319358] LustreError: 208645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975185 with bad export cookie 14261685925112875748 [ 6811.326858] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6811.337440] LustreError: 208645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6812.034391] Lustre: server umount lustre-MDT0001 complete [ 6822.153934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.069542] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6838.774041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.648423] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6853.599324] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6860.888298] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6868.712878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6877.446875] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6885.630629] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6891.469470] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6893.906841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6899.359742] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6899.679552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6903.273230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.284438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6903.289470] Lustre: Skipped 2 previous similar messages [ 6908.279809] Lustre: server umount lustre-OST0000 complete [ 6915.563283] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6920.740055] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6951.655268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6959.511916] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6968.633390] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6968.682761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6968.716640] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6968.721953] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6968.748473] Lustre: Skipped 1 previous similar message [ 6970.608713] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6979.043960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6979.050101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6979.054018] LustreError: Skipped 1 previous similar message [ 6979.076180] Lustre: Skipped 6 previous similar messages [ 6981.093672] Lustre: server umount lustre-OST0000 complete [ 6991.689780] Lustre: server umount lustre-MDT0000 complete [ 6994.428845] LustreError: 213001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.449911] LustreError: 213001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6996.209697] LustreError: 212013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975369 with bad export cookie 14261685925112876630 [ 6996.230203] LustreError: 212013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6996.235774] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6996.561761] Lustre: server umount lustre-MDT0001 complete [ 7006.055122] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:42:58 (1783975378) [ 7013.438336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7014.048903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7014.053587] Lustre: Skipped 6 previous similar messages [ 7019.143873] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7028.792511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.493561] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7042.249504] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7050.022958] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7059.802654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7066.760869] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7074.272472] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7079.465665] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7084.157323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 7089.935077] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7090.222806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7097.770705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7097.932498] Lustre: lustre-OST0001: new disk, initializing [ 7097.934566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7097.940491] Lustre: 221413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7104.676101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7104.707099] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7104.830363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7106.106747] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7116.552630] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7122.570215] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7122.953725] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7127.722449] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7128.124827] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7130.600274] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7130.605941] LustreError: Skipped 1 previous similar message [ 7130.607279] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7130.626719] Lustre: Skipped 6 previous similar messages [ 7137.042082] Lustre: server umount lustre-OST0001 complete [ 7145.534713] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7145.934159] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7152.150162] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7152.394370] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7182.509102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7187.979681] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7187.985912] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7187.991414] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7189.242623] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7197.802718] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7199.354064] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7207.754851] Lustre: server umount lustre-OST0001 complete [ 7239.164571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7239.181298] Lustre: Skipped 16 previous similar messages [ 7240.163633] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7240.392820] Lustre: server umount lustre-OST0000 complete [ 7252.067581] Lustre: server umount lustre-MDT0000 complete [ 7254.519994] LustreError: 218014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7254.567908] LustreError: 218014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 43 previous similar messages [ 7256.068316] LustreError: 217995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975629 with bad export cookie 14261685925112877596 [ 7256.089628] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7256.092183] LustreError: 217995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7262.620243] Lustre: server umount lustre-MDT0001 complete [ 7290.570619] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:47:42 (1783975662) [ 7297.650863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7298.013823] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7298.065800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7303.708748] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7312.534930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.704274] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7312.767192] Lustre: 226664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7318.847879] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7330.807160] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7337.633415] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7346.281536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7346.467248] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7354.518659] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7363.819314] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7374.699438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.883568] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7382.688619] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7390.732933] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7392.559130] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7397.953746] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7398.203839] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7403.961540] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7404.276181] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7410.913355] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b0650e76800.ost_server_uuid 50 [ 7412.701667] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b0650e76800.ost_server_uuid in IDLE state after 0 sec [ 7413.923059] Lustre: Permanently deactivating lustre-OST0001 [ 7413.942226] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7418.580256] Lustre: Permanently reactivating lustre-OST0001 [ 7418.606536] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7418.612472] Lustre: Skipped 2 previous similar messages [ 7423.977457] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7423.993899] LustreError: Skipped 6 previous similar messages [ 7423.996880] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7424.019726] Lustre: Skipped 6 previous similar messages [ 7437.280658] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7437.424080] Lustre: server umount lustre-OST0001 complete [ 7447.545354] Lustre: server umount lustre-OST0000 complete [ 7457.715443] Lustre: server umount lustre-MDT0000 complete [ 7461.706765] LustreError: 231077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975835 with bad export cookie 14261685925112878884 [ 7461.708497] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7461.736737] LustreError: 231077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7462.515208] Lustre: server umount lustre-MDT0001 complete [ 7491.272733] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:51:03 (1783975863) [ 7500.440238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7500.657449] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7500.678699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7500.684829] Lustre: Skipped 2 previous similar messages [ 7505.504678] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7514.518101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7514.707993] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7514.757279] Lustre: 233366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7520.546895] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7528.409631] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7534.714652] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7541.401640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.598960] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7541.627564] Lustre: Permanently deactivating lustre-OST0000 [ 7541.651307] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7541.823344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7541.829238] Lustre: Skipped 10 previous similar messages [ 7549.537409] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7549.544062] LustreError: 188844:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000003baff67 x1870632841612672/t0(0) o8->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7549.667189] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7558.276987] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7565.478991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7565.587931] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7573.227552] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7574.252806] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7582.344833] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7584.580219] Lustre: Permanently reactivating lustre-OST0000 [ 7584.598935] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7584.607488] Lustre: Skipped 2 previous similar messages [ 7591.531123] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 000000000b60acdd, cur 1783975965 deadline 1783975953 last 1783975923 [ 7591.632612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7608.025502] Lustre: server umount lustre-OST0001 complete [ 7618.483732] Lustre: server umount lustre-OST0000 complete [ 7629.834790] Lustre: server umount lustre-MDT0000 complete [ 7634.640973] LustreError: 233357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976008 with bad export cookie 14261685925112880550 [ 7634.663715] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7635.153706] Lustre: server umount lustre-MDT0001 complete [ 7648.416523] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7652.010862] Key type lgssc unregistered [ 7652.444039] LNet: 238246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7652.463348] LNetError: 238246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7653.490685] LNet: Removed LNI 192.168.202.103@tcp [ 7654.724378] Key type .llcrypt unregistered [ 7654.729888] Key type ._llcrypt unregistered [ 7682.428458] Key type ._llcrypt registered [ 7682.429910] Key type .llcrypt registered [ 7682.601833] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7684.131334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7684.219378] alg: No test for adler32 (adler32-zlib) [ 7685.597910] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 7685.889548] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7687.640213] Key type lgssc registered [ 7689.238158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7701.499454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7701.545612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7703.096839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7708.899979] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7713.251913] LustreError: 239714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7713.280949] LustreError: 239714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7718.354424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.374982] LustreError: 239715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7718.572058] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7718.590201] Lustre: 239702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7718.596258] Lustre: Permanently deactivating lustre-MDT0001 [ 7718.607737] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7718.840912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7725.711788] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7726.665529] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7735.874458] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7742.740113] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7750.530277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7751.224056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7757.451180] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7763.497737] LustreError: 241559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7763.528034] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7766.870307] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7768.552630] LustreError: 242169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.672452] LustreError: 242169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.683954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7778.756874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7779.333053] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7784.449324] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7784.476594] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7789.222660] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7798.234995] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7802.756664] Lustre: Permanently reactivating lustre-MDT0001 [ 7802.780495] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7802.785611] Lustre: Skipped 2 previous similar messages [ 7806.504258] Lustre: lustre-MDT0001: Client 205e69c5-243d-4e11-b3b5-793f7623d865 (at 192.168.202.3@tcp) reconnecting [ 7812.273143] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.307279] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7812.322779] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7812.381127] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7815.592599] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7817.764276] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7824.212905] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7824.436340] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7830.400821] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7830.847721] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7834.350847] Lustre: Permanently deactivating lustre-MDT0001 [ 7834.384566] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7834.390361] Lustre: Skipped 2 previous similar messages [ 7842.948739] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7850.040914] Lustre: server umount lustre-MDT0000 complete [ 7850.979765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7851.001277] LustreError: 240802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7851.001289] LustreError: 240802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7851.034793] Lustre: Skipped 3 previous similar messages [ 7854.364566] LustreError: 239694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976228 with bad export cookie 3000420119107151310 [ 7854.372269] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7854.378640] LustreError: 239694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7854.722952] Lustre: server umount lustre-MDT0001 complete [ 7858.951187] Lustre: server umount lustre-OST0000 complete [ 7872.284650] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:57:24 (1783976244) [ 7884.571294] Lustre: server umount lustre-OST0001 complete [ 7892.393927] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 7903.828501] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7944.297691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7944.555240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7944.565291] Lustre: Skipped 2 previous similar messages [ 7944.588131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7944.671408] Lustre: lustre-MDT0000: new disk, initializing [ 7944.738258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7944.757455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7950.903216] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7963.751926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7963.897465] Lustre: 247982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7963.985138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7963.989836] Lustre: Skipped 1 previous similar message [ 7964.060698] Lustre: lustre-MDT0001: new disk, initializing [ 7964.214988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7964.253636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7964.264529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7971.082944] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7982.380219] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7989.332472] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7997.311931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7997.703711] Lustre: lustre-OST0000: new disk, initializing [ 7997.710328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7997.726721] Lustre: 249952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7997.862973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8005.733773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8005.749656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8005.940893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8006.055901] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8017.670334] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8020.531826] LustreError: 247988:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8030.592388] LustreError: 247988:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8030.651644] LustreError: 247989:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8031.101960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8031.351520] Lustre: lustre-OST0001: new disk, initializing [ 8031.356633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8031.363669] Lustre: 251066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8031.547298] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8037.019945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8037.037033] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8037.227229] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8039.578620] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8040.664451] LustreError: 247989:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8050.844486] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8052.691362] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8057.827856] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8057.831799] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8057.888422] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8061.117952] Lustre: server umount lustre-OST0001 complete [ 8062.953459] LustreError: 250295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8062.987741] LustreError: 250295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8078.309971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8078.312230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8078.321237] Lustre: Skipped 2 previous similar messages [ 8078.325129] Lustre: Skipped 2 previous similar messages [ 8083.432684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8083.450047] Lustre: Skipped 1 previous similar message [ 8084.339659] Lustre: server umount lustre-OST0000 complete [ 8092.130544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8092.158916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8092.176791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8098.789649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8098.803333] Lustre: Skipped 3 previous similar messages [ 8103.392239] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8103.780580] Lustre: server umount lustre-MDT0000 complete [ 8103.907685] LustreError: 247994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.926337] LustreError: 247994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8107.740820] LustreError: 247973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976481 with bad export cookie 3000420119107154124 [ 8107.744876] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8107.763952] LustreError: 247973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8122.336313] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8122.668372] Lustre: server umount lustre-MDT0001 complete [ 8134.149871] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8137.741975] Key type lgssc unregistered [ 8138.118801] LNet: 253114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8138.129751] LNetError: 253114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8138.158843] LNet: Removed LNI 192.168.202.103@tcp [ 8139.276851] Key type .llcrypt unregistered [ 8139.278798] Key type ._llcrypt unregistered [ 8185.482637] Key type ._llcrypt registered [ 8185.484906] Key type .llcrypt registered [ 8185.588817] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8187.388602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8187.478154] alg: No test for adler32 (adler32-zlib) [ 8188.668715] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 8188.994033] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8190.712197] Key type lgssc registered [ 8192.424919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8201.530987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8201.600810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8203.321848] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8203.353597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8203.564417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8208.254824] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8217.521318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.625204] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8217.662988] Lustre: 255324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8217.944122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8224.391357] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8232.517709] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8238.910162] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8246.731278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8246.951307] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8247.188633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.726430] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8256.179771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8263.560096] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8317.410290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8317.430556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8317.488683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8317.939145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8317.948560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8323.045029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8323.062884] Lustre: Skipped 2 previous similar messages [ 8323.529692] Lustre: server umount lustre-OST0000 complete [ 8328.427807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8344.909716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8345.381434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8347.461995] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8347.492966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8347.498586] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8357.088264] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8366.646120] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8409.059988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8409.066331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8409.073043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8409.079906] LustreError: Skipped 1 previous similar message [ 8409.107367] Lustre: Skipped 1 previous similar message [ 8413.364168] Lustre: server umount lustre-OST0000 complete [ 8419.298528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8419.327227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8419.338022] Lustre: Skipped 2 previous similar messages [ 8424.417747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8424.439751] Lustre: Skipped 1 previous similar message [ 8425.508696] Lustre: server umount lustre-MDT0000 complete [ 8430.064570] LustreError: 255317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976803 with bad export cookie 16689553449580998370 [ 8430.070027] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8430.088909] LustreError: 255317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8430.841301] Lustre: server umount lustre-MDT0001 complete [ 8440.357994] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8444.597896] Key type lgssc unregistered [ 8445.003705] LNet: 260748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8445.007694] LNetError: 260748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8445.023453] LNet: Removed LNI 192.168.202.103@tcp [ 8445.856284] Key type .llcrypt unregistered [ 8445.858131] Key type ._llcrypt unregistered [ 8471.869885] Key type ._llcrypt registered [ 8471.872154] Key type .llcrypt registered [ 8471.979347] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8472.876750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8473.010891] alg: No test for adler32 (adler32-zlib) [ 8474.061061] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 8474.385993] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8476.176158] Key type lgssc registered [ 8477.783365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8487.018180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8487.038547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8488.527361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8493.512550] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8498.660280] LustreError: 262168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8498.672556] LustreError: 262168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8502.252060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8502.787270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8507.949936] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8515.692268] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8522.156841] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8529.669500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8530.163740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8534.281505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8538.467819] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8545.837572] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8575.457561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8575.473175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8575.500914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8579.559188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8579.560150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8580.805771] Lustre: server umount lustre-OST0000 complete [ 8584.677853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8584.680311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8584.716692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8584.726293] Lustre: Skipped 2 previous similar messages [ 8589.793658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8589.813642] Lustre: Skipped 1 previous similar message [ 8590.737816] Lustre: server umount lustre-MDT0000 complete [ 8594.913468] LustreError: 262169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8594.953599] LustreError: 262149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976968 with bad export cookie 16205516223780060243 [ 8594.970346] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8595.255507] Lustre: server umount lustre-MDT0001 complete [ 8607.611184] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8610.768097] Key type lgssc unregistered [ 8611.139817] LNet: 266475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8611.161957] LNetError: 266475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8611.188743] LNet: Removed LNI 192.168.202.103@tcp [ 8612.133708] Key type .llcrypt unregistered [ 8612.140570] Key type ._llcrypt unregistered [ 8633.375813] Key type ._llcrypt registered [ 8633.379365] Key type .llcrypt registered [ 8633.474675] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8635.484988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8635.535519] alg: No test for adler32 (adler32-zlib) [ 8636.788300] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 8637.123438] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8638.866535] Key type lgssc registered [ 8640.633781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8650.287243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8650.300534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8651.920745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8657.554890] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8661.478467] LustreError: 267700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8661.506906] LustreError: 267700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8666.469150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.594352] LustreError: 267699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.092577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8672.421185] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8679.888210] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8687.315641] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8697.209684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8698.127083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8709.110738] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8719.979633] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8721.717121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8737.253376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8737.258423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8737.269786] LustreError: Skipped 1 previous similar message [ 8737.270482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8737.329815] Lustre: Skipped 1 previous similar message [ 8738.928772] Lustre: server umount lustre-OST0000 complete [ 8743.906045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8743.911464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8743.920522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8743.923032] Lustre: Skipped 1 previous similar message [ 8747.491841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8747.492676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8747.524188] Lustre: Skipped 1 previous similar message [ 8750.129832] Lustre: server umount lustre-MDT0000 complete [ 8752.620119] LustreError: 267696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8752.648200] LustreError: 267696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8754.596244] LustreError: 267681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977128 with bad export cookie 1521060346747198965 [ 8754.600716] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8754.611118] LustreError: 267681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8755.015649] Lustre: server umount lustre-MDT0001 complete [ 8769.327540] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8773.433849] Key type lgssc unregistered [ 8773.910653] LNet: 271382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8773.936107] LNetError: 271382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8774.950041] LNet: Removed LNI 192.168.202.103@tcp [ 8775.953455] Key type .llcrypt unregistered [ 8775.956412] Key type ._llcrypt unregistered [ 8806.935279] Key type ._llcrypt registered [ 8806.937834] Key type .llcrypt registered [ 8807.125824] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8808.378696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8808.590513] alg: No test for adler32 (adler32-zlib) [ 8809.886342] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 8810.366219] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8812.064173] Key type lgssc registered [ 8813.504752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8822.838861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8822.884966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8824.524823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8829.685568] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8835.051500] LustreError: 272800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8835.062890] LustreError: 272800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8839.004868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8839.773180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8846.341137] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8855.633407] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8862.519890] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8870.718796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8871.309481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8878.845628] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8888.216560] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8894.904361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8920.554518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.562821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8920.569107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8920.585742] LustreError: Skipped 1 previous similar message [ 8924.014938] Lustre: server umount lustre-OST0000 complete [ 8930.792702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8930.814028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8930.815239] Lustre: Skipped 2 previous similar messages [ 8930.844720] Lustre: Skipped 2 previous similar messages [ 8935.127880] Lustre: server umount lustre-MDT0000 complete [ 8935.907867] LustreError: 274371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.925258] LustreError: 274371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8939.533372] LustreError: 272782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977313 with bad export cookie 1766882433060965338 [ 8939.550144] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8939.564548] LustreError: 272782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8939.881895] Lustre: server umount lustre-MDT0001 complete [ 8953.279405] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8956.650862] Key type lgssc unregistered [ 8957.011947] LNet: 277111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8957.015794] LNetError: 277111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8957.027177] LNet: Removed LNI 192.168.202.103@tcp [ 8958.073979] Key type .llcrypt unregistered [ 8958.075529] Key type ._llcrypt unregistered [ 8979.506261] Key type ._llcrypt registered [ 8979.507803] Key type .llcrypt registered [ 8979.633567] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8981.114696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8981.209096] alg: No test for adler32 (adler32-zlib) [ 8982.565284] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 8982.944405] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8984.713864] Key type lgssc registered [ 8986.443247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8995.993913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8996.045557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8997.655360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9002.828891] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9008.104414] LustreError: 278347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9008.118989] LustreError: 278347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9011.973372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.285615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9017.732188] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9026.361306] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9032.885905] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9040.984160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9041.509582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9042.568149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9049.174330] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9058.489818] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9072.613646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9072.629076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9072.641747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9072.653766] Lustre: Skipped 1 previous similar message [ 9077.185920] Lustre: server umount lustre-OST0000 complete [ 9082.851919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9082.855546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9082.863455] Lustre: Skipped 2 previous similar messages [ 9087.605974] LustreError: 281297:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000003ec945d4) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 9087.687909] Lustre: server umount lustre-MDT0000 complete [ 9087.989125] LustreError: 278358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9088.029080] LustreError: 278358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9091.965141] LustreError: 278314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977465 with bad export cookie 14679305709843234303 [ 9091.970369] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9091.993721] LustreError: 278314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9092.396901] Lustre: server umount lustre-MDT0001 complete [ 9104.770369] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 9107.962946] Key type lgssc unregistered [ 9108.305315] LNet: 282050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9108.322044] LNetError: 282050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9108.344842] LNet: Removed LNI 192.168.202.103@tcp [ 9109.345764] Key type .llcrypt unregistered [ 9109.347346] Key type ._llcrypt unregistered [ 9122.212400] Key type ._llcrypt registered [ 9122.213887] Key type .llcrypt registered [ 9122.340898] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9139.659320] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 9156.470452] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 9157.917236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9157.970072] alg: No test for adler32 (adler32-zlib) [ 9159.093576] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 9159.355104] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 9161.152198] Key type lgssc registered [ 9162.451471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9201.455155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9201.500521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.839945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9202.868782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9202.939732] Lustre: lustre-MDT0000: new disk, initializing [ 9203.022663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9203.055526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9209.503435] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9221.593899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9221.753553] Lustre: 285603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9221.854057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9221.861532] Lustre: Skipped 1 previous similar message [ 9221.927969] Lustre: lustre-MDT0001: new disk, initializing [ 9222.019854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9222.068943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9222.084805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9227.629539] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9238.022948] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9244.556642] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9251.577319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9251.979563] Lustre: lustre-OST0000: new disk, initializing [ 9251.983543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9251.989173] Lustre: 287576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9252.070562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9257.633368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9257.654133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9257.898579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9260.328204] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9271.498212] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9276.512972] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9276.806974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9281.542825] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9281.851337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9288.681725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9288.704471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9288.705134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9288.749226] Lustre: Skipped 1 previous similar message [ 9293.803915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9293.811250] Lustre: Skipped 1 previous similar message [ 9298.400630] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9298.647459] Lustre: server umount lustre-OST0000 complete [ 9304.035525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9304.050267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9304.053392] Lustre: Skipped 2 previous similar messages [ 9304.069714] Lustre: Skipped 1 previous similar message [ 9309.556212] Lustre: server umount lustre-MDT0000 complete [ 9314.278132] LustreError: 285611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9314.331171] LustreError: 285611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9316.076612] LustreError: 285597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977689 with bad export cookie 11110544194837969560 [ 9316.092994] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9316.099804] LustreError: 285597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9317.081184] Lustre: server umount lustre-MDT0001 complete [ 9331.574547] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:21:43 (1783977703) [ 9339.191275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9339.684833] LustreError: 289991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9339.809620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9344.409222] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9345.010445] LustreError: 289992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9350.138958] LustreError: 289991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9354.135876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9354.831534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9359.819965] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9369.344849] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9376.925590] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9385.336741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9385.942420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9392.004954] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9400.397815] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9419.770553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9419.771120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9419.776125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9419.782388] LustreError: Skipped 1 previous similar message [ 9419.788579] Lustre: Skipped 1 previous similar message [ 9424.281611] Lustre: server umount lustre-OST0000 complete [ 9429.985117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9429.991467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9429.994734] Lustre: Skipped 2 previous similar messages [ 9430.002550] Lustre: Skipped 1 previous similar message [ 9435.113421] LustreError: 289991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9435.289045] Lustre: server umount lustre-MDT0000 complete [ 9441.100772] LustreError: 289971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977814 with bad export cookie 11110544194837970442 [ 9441.121476] LustreError: 289971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9441.130542] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9441.557907] Lustre: server umount lustre-MDT0001 complete [ 9454.418930] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 9457.983351] Key type lgssc unregistered [ 9458.329445] LNet: 293549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9458.340320] LNetError: 293549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9458.377875] LNet: Removed LNI 192.168.202.103@tcp [ 9459.326369] Key type .llcrypt unregistered [ 9459.333922] Key type ._llcrypt unregistered [ 9501.669184] Key type ._llcrypt registered [ 9501.676714] Key type .llcrypt registered [ 9501.893811] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 9519.322459] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 9520.869704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9520.926192] alg: No test for adler32 (adler32-zlib) [ 9522.023391] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 9522.353097] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 9524.168203] Key type lgssc registered [ 9525.582493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9573.536765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9573.565494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9575.038972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9575.108127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9575.171543] Lustre: lustre-MDT0000: new disk, initializing [ 9575.241937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9575.262938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9579.690237] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9589.893057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9589.982452] Lustre: 297380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9590.030771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9590.032881] Lustre: Skipped 1 previous similar message [ 9590.090614] Lustre: lustre-MDT0001: new disk, initializing [ 9590.198926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9590.248656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9590.267424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9594.796939] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9605.571882] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9613.331387] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9619.906061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9620.338679] Lustre: lustre-OST2710: new disk, initializing [ 9620.346994] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9620.359512] Lustre: 299349:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9620.547224] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9626.744058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9626.751991] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9626.909525] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9627.505750] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9637.527149] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9644.043710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9644.317481] Lustre: lustre-OST03e8: new disk, initializing [ 9644.322577] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9644.328756] Lustre: 300413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9644.434613] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9650.295404] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9650.798549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9650.807130] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9650.912630] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9657.394640] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9665.416499] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9665.610412] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9670.683138] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9670.869851] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9676.245788] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9676.581702] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9682.006992] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9682.350885] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9687.016504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.028580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9687.042198] Lustre: Skipped 1 previous similar message [ 9687.057051] Lustre: Skipped 3 previous similar messages [ 9689.248710] Lustre: server umount lustre-MDT0000 complete [ 9693.154246] LustreError: 297373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978066 with bad export cookie 10270065965700286472 [ 9693.159338] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9693.160344] LustreError: 297373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9693.551507] Lustre: server umount lustre-MDT0001 complete [ 9707.841253] Lustre: server umount lustre-OST2710 complete [ 9722.154830] Lustre: server umount lustre-OST03e8 complete [ 9744.922323] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 9755.850143] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 9792.679065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9792.958600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9792.981976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9793.081424] Lustre: lustre-MDT0000: new disk, initializing [ 9793.164609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.186846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9798.859270] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9810.906671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9811.100047] Lustre: 305664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9811.175409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9811.178737] Lustre: Skipped 1 previous similar message [ 9811.286772] Lustre: lustre-MDT0001: new disk, initializing [ 9811.448861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9811.518435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9811.540944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9817.347847] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9827.283707] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9834.069410] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9843.907662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9844.336814] Lustre: lustre-OST0000: new disk, initializing [ 9844.341212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9844.351451] Lustre: 307634:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9844.438167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9851.976344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9851.988034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9852.129153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9853.587595] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9867.598684] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9872.689441] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9872.962214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9878.356472] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9878.611657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9883.106705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9883.111391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9883.129044] LustreError: Skipped 1 previous similar message [ 9883.155051] Lustre: Skipped 2 previous similar messages [ 9883.164710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9886.921762] Lustre: server umount lustre-OST0000 complete [ 9893.346662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9893.350347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.373133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9893.381434] Lustre: Skipped 1 previous similar message [ 9893.391673] Lustre: Skipped 2 previous similar messages [ 9897.652584] Lustre: server umount lustre-MDT0000 complete [ 9898.467550] LustreError: 305672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9898.489888] LustreError: 305672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9902.352051] LustreError: 305659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978276 with bad export cookie 10270065965700288152 [ 9902.358271] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9902.377537] LustreError: 305659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9902.974248] Lustre: server umount lustre-MDT0001 complete [ 9914.442936] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:31:26 (1783978286) [ 9916.735206] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9919.872917] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:31:31 (1783978291) [ 9925.174708] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 9957.448575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9957.958881] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9958.025713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9958.287457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9964.824441] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9975.008908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.214269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9975.277756] Lustre: 311495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9980.398891] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10002.877109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10003.240422] Lustre: Denying initial registration attempt from nid 192.168.202.103@tcp, specified as failover [10003.258961] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10003.274332] LustreError: 312948:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10003.280193] LustreError: 312948:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10003.284890] LustreError: 312948:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10003.356549] Lustre: server umount lustre-OST0000 complete [10003.359262] LustreError: 312948:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [10011.618311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10011.633773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10011.653118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10016.712728] Lustre: server umount lustre-MDT0000 complete [10017.761922] LustreError: 311506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10017.787639] LustreError: 311506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10021.055939] LustreError: 311487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978394 with bad export cookie 10270065965700289034 [10021.063127] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10021.065630] LustreError: 311487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10021.418957] Lustre: server umount lustre-MDT0001 complete [10033.927264] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10036.019609] Key type lgssc unregistered [10036.303447] LNet: 314070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10036.316637] LNetError: 314070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10036.345736] LNet: Removed LNI 192.168.202.103@tcp [10037.209110] Key type .llcrypt unregistered [10037.210240] Key type ._llcrypt unregistered [10049.583502] Key type ._llcrypt registered [10049.595299] Key type .llcrypt registered [10049.785165] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10051.454610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10051.503872] alg: No test for adler32 (adler32-zlib) [10052.582544] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [10052.813171] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [10054.672197] Key type lgssc registered [10056.418904] Lustre: Echo OBD driver; http://www.lustre.org/ [10108.181635] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10119.038660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10119.065274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10120.534260] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10120.567764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10120.757464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10125.292833] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10134.636698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10134.877078] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10134.932139] Lustre: 316837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10135.012364] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10135.261790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10139.953689] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10148.582781] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10154.298969] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10174.645185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10174.895530] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10175.131894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10182.466950] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10190.990732] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10198.500816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10198.504299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10198.511442] Lustre: Skipped 1 previous similar message [10199.693461] Lustre: server umount lustre-OST0000 complete [10203.622047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10203.630866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10203.631547] Lustre: Skipped 1 previous similar message [10203.636186] Lustre: Skipped 1 previous similar message [10208.737321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10208.745032] Lustre: Skipped 2 previous similar messages [10209.567561] Lustre: server umount lustre-MDT0000 complete [10213.575699] LustreError: 316829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978587 with bad export cookie 13986162121422883949 [10213.582950] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10213.586731] LustreError: 316829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10214.076941] Lustre: server umount lustre-MDT0001 complete [10226.254282] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10229.554841] Key type lgssc unregistered [10229.836257] LNet: 320412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10229.846565] LNetError: 320412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10229.870685] LNet: Removed LNI 192.168.202.103@tcp [10230.818195] Key type .llcrypt unregistered [10230.824888] Key type ._llcrypt unregistered [10256.189874] Key type ._llcrypt registered [10256.192183] Key type .llcrypt registered [10256.270799] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10257.736055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10257.754339] alg: No test for adler32 (adler32-zlib) [10258.897218] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [10259.125925] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [10260.880426] Key type lgssc registered [10261.991706] Lustre: Echo OBD driver; http://www.lustre.org/ [10270.871720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10270.885165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10272.544685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10277.732884] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10282.979538] LustreError: 321837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10282.997880] LustreError: 321837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10286.294680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10286.796186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10291.972936] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10300.632404] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10306.715883] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10316.535168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10317.179251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10325.740132] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10337.403996] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10346.979918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10346.986693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10347.013856] Lustre: Skipped 2 previous similar messages [10347.039030] Lustre: Skipped 1 previous similar message [10349.742062] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [10352.104032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.107317] Lustre: Skipped 2 previous similar messages [10352.921118] Lustre: server umount lustre-MDT0000 complete [10354.817310] LustreError: 322828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.222963] LustreError: 322566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.253947] LustreError: 322566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10357.272545] LustreError: 323681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978730 with bad export cookie 9924836484595302933 [10357.277380] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10357.284240] LustreError: 323681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10359.901850] LustreError: 321833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10359.938615] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.3@tcp (stopping) [10362.352096] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10363.956976] Lustre: server umount lustre-MDT0001 complete [10371.492590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10380.475497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.109120] LustreError: 323669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10384.135446] LustreError: 323669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10384.187089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10384.275577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10388.855412] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10392.355931] LustreError: 323670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.357501] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10392.391905] LustreError: 323670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10397.684361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10398.166442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10402.834996] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10403.318368] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10411.817683] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10413.489269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10420.087205] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10421.878403] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10428.900576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10428.921325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10428.929491] Lustre: Skipped 1 previous similar message [10428.949475] Lustre: Skipped 2 previous similar messages [10433.886131] Lustre: server umount lustre-OST0000 complete [10439.142664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10439.165843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10439.174995] Lustre: Skipped 1 previous similar message [10444.720936] Lustre: server umount lustre-MDT0000 complete [10448.370273] LustreError: 325425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10448.394252] LustreError: 325425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10449.263817] LustreError: 321817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978822 with bad export cookie 9924836484595310073 [10449.268770] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10449.272689] LustreError: 321817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10449.891168] Lustre: server umount lustre-MDT0001 complete [10461.994610] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10465.217572] Key type lgssc unregistered [10465.699756] LNet: 328102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10465.710309] LNetError: 328102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10465.730765] LNet: Removed LNI 192.168.202.103@tcp [10466.683692] Key type .llcrypt unregistered [10466.685714] Key type ._llcrypt unregistered [10492.699358] Key type ._llcrypt registered [10492.703449] Key type .llcrypt registered [10492.800540] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10494.559639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10494.577914] alg: No test for adler32 (adler32-zlib) [10495.730996] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [10496.049666] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [10497.728169] Key type lgssc registered [10499.134107] Lustre: Echo OBD driver; http://www.lustre.org/ [10507.511330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10507.546349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10509.056390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10514.191315] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10518.499269] LustreError: 329520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10518.535505] LustreError: 329520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10523.353281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10523.622210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10523.798867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10528.928684] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10536.895647] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10543.639791] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10548.706101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10548.720339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10549.793722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10549.799031] Lustre: Skipped 2 previous similar messages [10553.138210] Lustre: server umount lustre-MDT0000 complete [10554.857736] LustreError: 329515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10554.874346] LustreError: 329515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10556.778735] LustreError: 329501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978930 with bad export cookie 18221201705963828002 [10556.790237] LustreError: 329501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10556.790451] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10557.335767] Lustre: server umount lustre-MDT0001 complete [10571.326346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10571.674638] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10571.717596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10572.023465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10576.224280] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10584.070318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10584.172321] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10584.195547] Lustre: 332109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10584.413944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10589.085630] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10597.577200] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10603.369191] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10612.585870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10612.786141] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10612.797635] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10612.811429] LustreError: 333992:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10612.816447] LustreError: 333992:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10612.821471] LustreError: 333992:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10612.874882] Lustre: server umount lustre-OST0000 complete [10612.878566] LustreError: 333992:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10622.076194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10622.450326] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10622.658882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10628.760500] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10632.423191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10636.798945] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10644.563989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10644.737525] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10644.807340] Lustre: lustre-OST0001: new disk, initializing [10644.810971] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10644.816932] Lustre: 335536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10644.920985] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10652.003740] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10653.248605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10653.263831] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10653.384803] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10660.713897] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10669.026519] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10669.029032] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10669.036313] LustreError: Skipped 1 previous similar message [10669.037109] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10669.076033] Lustre: Skipped 2 previous similar messages [10670.253243] Lustre: server umount lustre-OST0001 complete [10674.150499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10674.153120] LustreError: 334461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10674.156650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10674.156660] Lustre: Skipped 1 previous similar message [10674.173177] Lustre: Skipped 1 previous similar message [10674.207825] LustreError: 334461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10679.266412] LustreError: 335963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10679.287689] LustreError: 335963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10680.330483] Lustre: server umount lustre-OST0000 complete [10686.945303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10686.955346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.987394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10686.997402] Lustre: Skipped 3 previous similar messages [10690.745725] Lustre: server umount lustre-MDT0000 complete [10694.638593] LustreError: 332120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10694.667370] LustreError: 332120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10695.139138] LustreError: 334478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979068 with bad export cookie 18221201705963828555 [10695.146621] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10695.147849] LustreError: 334478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10695.741544] Lustre: server umount lustre-MDT0001 complete [10709.516531] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10712.861395] Key type lgssc unregistered [10713.180934] LNet: 337493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10713.199319] LNetError: 337493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10714.277852] LNet: Removed LNI 192.168.202.103@tcp [10715.051825] Key type .llcrypt unregistered [10715.054082] Key type ._llcrypt unregistered [10786.357538] Key type ._llcrypt registered [10786.362876] Key type .llcrypt registered [10786.577905] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [10801.943825] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10803.351906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10803.410979] alg: No test for adler32 (adler32-zlib) [10804.419726] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [10804.571686] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [10806.272172] Key type lgssc registered [10807.388920] Lustre: Echo OBD driver; http://www.lustre.org/ [10842.707580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10842.733800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.122589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10844.160332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10844.268781] Lustre: lustre-MDT0000: new disk, initializing [10844.354897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10844.371240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10849.102556] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10859.955914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10860.126116] Lustre: 342431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10860.229596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10860.232798] Lustre: Skipped 1 previous similar message [10860.303607] Lustre: lustre-MDT0001: new disk, initializing [10860.423319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10860.473180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10860.492101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10865.917726] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10876.195043] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10881.897157] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10889.469380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10889.840363] Lustre: lustre-OST0000: new disk, initializing [10889.846137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10889.852731] Lustre: 344401:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10889.953456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10896.714686] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10896.984057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10897.001175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10897.089559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10906.134402] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10911.433311] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10911.684107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10916.886486] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10917.132268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10922.979391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10922.995454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10923.001577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10923.022089] LustreError: Skipped 1 previous similar message [10925.576349] Lustre: server umount lustre-OST0000 complete [10932.202705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10932.209710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10932.218255] Lustre: Skipped 1 previous similar message [10932.228272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10932.230760] Lustre: Skipped 1 previous similar message [10933.220651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10935.489714] Lustre: server umount lustre-MDT0000 complete [10939.603800] LustreError: 343225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979313 with bad export cookie 5197708290997125172 [10939.618676] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10940.110767] Lustre: server umount lustre-MDT0001 complete [10949.314674] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:48:41 (1783979321) [10956.507340] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:48:49 (1783979329) [10964.506126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10964.988275] LustreError: 347155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10965.013453] LustreError: 347155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10965.092499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10969.953765] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10970.086950] LustreError: 347156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10970.124744] LustreError: 347156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10975.210921] LustreError: 347155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10978.134210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10978.420287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10982.679770] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10990.940590] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10997.444349] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11004.711666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11005.335699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11012.557481] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11020.785817] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11032.327283] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11034.204712] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11035.968728] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11037.971657] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11039.622740] Lustre: Failing over lustre-MDT0000 [11039.717692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11039.727622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11039.751487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11039.762907] Lustre: Skipped 2 previous similar messages [11039.905749] Lustre: server umount lustre-MDT0000 complete [11042.954975] LustreError: 347151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11044.323276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11048.029672] LustreError: 347151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.050859] LustreError: 347151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11058.275658] LustreError: 348204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.322506] LustreError: 348204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11060.055631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11060.214848] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11060.390558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11060.435479] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11063.417776] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11065.851532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11065.901470] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11065.984065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11066.602194] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11075.964203] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11077.429819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11081.595812] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11083.384427] Lustre: Failing over lustre-MDT0000 [11083.671361] Lustre: server umount lustre-MDT0000 complete [11084.432384] LustreError: 347150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11084.447506] LustreError: 347150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11086.305384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11086.315130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11086.349577] Lustre: Skipped 3 previous similar messages [11102.694739] Lustre: 340652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783979460/real 1783979460] req@00000000a3b60665 x1870637750644352/t0(0) o400->MGC192.168.202.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783979476 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11102.722125] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11103.832919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11113.966138] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4821f826eafc8ad0 [11113.992063] Lustre: MGC192.168.202.103@tcp: Connection restored to 0@lo (at 0@lo) [11113.999036] Lustre: Skipped 2 previous similar messages [11114.398500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11114.513214] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11115.614057] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11119.629294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11119.673082] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11119.718918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11120.028812] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11129.800473] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11131.610474] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11140.086224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11140.094383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11140.094471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11140.111144] Lustre: Skipped 2 previous similar messages [11140.141395] Lustre: Skipped 2 previous similar messages [11144.264269] Lustre: server umount lustre-MDT0000 complete [11145.195329] LustreError: 347884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.233042] LustreError: 347884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 32 previous similar messages [11148.210561] LustreError: 347136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979521 with bad export cookie 5197708290997127888 [11148.220497] LustreError: 347136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11148.229517] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11148.605355] Lustre: server umount lustre-MDT0001 complete [11163.413647] Lustre: server umount lustre-OST0000 complete [11180.323434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.962628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11186.025885] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11194.354282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.812543] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11208.008701] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11214.076690] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11221.702074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11226.295054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11229.081979] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11236.763327] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11250.148319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11250.182197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11253.870713] Lustre: server umount lustre-OST0000 complete [11260.428711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11260.439944] Lustre: Skipped 2 previous similar messages [11264.230919] Lustre: server umount lustre-MDT0000 complete [11265.505715] LustreError: 355551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.531449] LustreError: 355551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11268.799447] LustreError: 353495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979642 with bad export cookie 5197708290997128630 [11268.806093] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11268.815736] LustreError: 353495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11269.630393] Lustre: server umount lustre-MDT0001 complete [11282.558820] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [11285.942863] Key type lgssc unregistered [11286.356973] LNet: 357047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11286.371553] LNetError: 357047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11286.398695] LNet: Removed LNI 192.168.202.103@tcp [11287.797884] Key type .llcrypt unregistered [11287.800436] Key type ._llcrypt unregistered [11318.610460] Key type ._llcrypt registered [11318.613829] Key type .llcrypt registered [11318.754736] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11319.847680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11320.031977] alg: No test for adler32 (adler32-zlib) [11321.225854] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [11321.448447] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [11323.088294] Key type lgssc registered [11324.258873] Lustre: Echo OBD driver; http://www.lustre.org/ [11332.154661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11332.174697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11333.612244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11338.869745] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11343.859667] LustreError: 358564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11343.883956] LustreError: 358564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11348.141558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11348.592632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11353.001053] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11359.614589] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11366.433950] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11373.856508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11374.382885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11378.522761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11383.181098] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11391.920596] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11409.382307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11409.386823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11409.395616] Lustre: Skipped 2 previous similar messages [11409.414545] Lustre: Skipped 2 previous similar messages [11413.640527] Lustre: server umount lustre-MDT0000 complete [11414.510294] LustreError: 358564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11414.533381] LustreError: 358564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11417.846934] LustreError: 358544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979791 with bad export cookie 14070443770459462344 [11417.861977] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11417.862326] LustreError: 358544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11418.195797] Lustre: server umount lustre-MDT0001 complete [11433.091972] Lustre: server umount lustre-OST0000 complete [11447.254816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.923069] LustreError: 362296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.990272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11452.709352] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11453.428790] LustreError: 362297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11457.570497] LustreError: 362296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11461.990905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11462.402746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11467.762603] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11476.112638] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11482.294249] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11489.454964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11489.848184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11492.928521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11496.910485] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11504.672532] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11518.452452] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11518.463470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11518.471950] Lustre: Skipped 2 previous similar messages [11518.504204] Lustre: Skipped 2 previous similar messages [11522.631415] Lustre: server umount lustre-MDT0000 complete [11523.558927] LustreError: 362293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.583209] LustreError: 362293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11526.780102] LustreError: 363034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979900 with bad export cookie 14070443770459463464 [11526.782835] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11526.786533] LustreError: 363034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11527.309956] Lustre: server umount lustre-MDT0001 complete [11540.634611] Lustre: server umount lustre-OST0000 complete [11556.876642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11557.309386] LustreError: 366212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.424451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11562.597619] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11572.073404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11577.437940] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11584.786518] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11590.153750] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11596.518058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11596.816802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11596.824735] Lustre: Skipped 1 previous similar message [11601.356386] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11602.996122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11606.722553] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11618.276292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11618.276644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11618.279018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11618.280970] Lustre: Skipped 1 previous similar message [11620.643300] Lustre: server umount lustre-OST0000 complete [11623.905995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11623.913198] LustreError: Skipped 1 previous similar message [11623.915621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11623.929897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11623.935664] Lustre: Skipped 1 previous similar message [11628.516992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11628.520608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11638.240135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11638.329772] Lustre: server umount lustre-MDT0000 complete [11638.758414] LustreError: 368719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11638.766038] LustreError: 368719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11640.305456] LustreError: 366194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783980014 with bad export cookie 14070443770459464486 [11640.312022] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11640.312467] LustreError: 366194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11640.505199] Lustre: server umount lustre-MDT0001 complete [11646.438470] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [11648.595733] Key type lgssc unregistered [11648.831638] LNet: 369741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11648.835195] LNetError: 369741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11649.893519] LNet: Removed LNI 192.168.202.103@tcp [11650.395544] Key type .llcrypt unregistered [11650.396817] Key type ._llcrypt unregistered [11671.642890] Key type ._llcrypt registered [11671.644256] Key type .llcrypt registered [11671.696647] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11672.128124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11672.178621] alg: No test for adler32 (adler32-zlib) [11673.114419] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [11673.223368] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [11674.824125] Key type lgssc registered [11675.556883] Lustre: Echo OBD driver; http://www.lustre.org/ [11682.121510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11682.144320] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11682.147781] LustreError: 371174:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11682.157453] LustreError: 371174:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11682.159222] LustreError: 371174:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11682.161590] LustreError: 371174:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11682.165287] LustreError: 371174:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11686.800387] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11686.803602] LustreError: 371571:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11686.817461] LustreError: 371571:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11686.827497] LustreError: 371571:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11686.830107] LustreError: 371571:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11686.832184] LustreError: 371571:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11695.813846] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [11697.136096] Key type lgssc unregistered [11697.345166] LNet: 372584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11697.348544] LNetError: 372584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11697.362660] LNet: Removed LNI 192.168.202.103@tcp [11697.908249] Key type .llcrypt unregistered [11697.910144] Key type ._llcrypt unregistered [11708.799720] Key type ._llcrypt registered [11708.801909] Key type .llcrypt registered [11708.868723] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [11717.685345] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11718.615297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11718.704824] alg: No test for adler32 (adler32-zlib) [11719.951314] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [11720.161429] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [11721.817136] Key type lgssc registered [11722.763607] Lustre: Echo OBD driver; http://www.lustre.org/ [11745.782166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11745.802208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11746.994350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11747.012222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11747.058970] Lustre: lustre-MDT0000: new disk, initializing [11747.092663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11747.099530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11750.009790] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11756.664056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11756.724036] Lustre: 375717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11756.750926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11756.752781] Lustre: Skipped 1 previous similar message [11756.797318] Lustre: lustre-MDT0001: new disk, initializing [11756.844637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11756.866641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11756.869282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11758.722276] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11763.679900] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11766.089677] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11768.726378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11768.831707] Lustre: lustre-OST0000: new disk, initializing [11768.833513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11768.837539] Lustre: 377683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11768.874980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11771.240529] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11772.036373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11772.040121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11772.084169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11778.667993] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11783.026387] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11783.292711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11788.016983] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11788.176776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11792.868120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11792.876776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11792.894344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11792.905837] Lustre: Skipped 1 previous similar message [11797.991997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11797.998679] Lustre: Skipped 1 previous similar message [11803.121460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11803.129854] Lustre: Skipped 1 previous similar message [11804.129254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11804.322871] Lustre: server umount lustre-OST0000 complete [11808.225488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11808.227973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11808.248515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11808.265522] Lustre: Skipped 2 previous similar messages [11808.281409] Lustre: Skipped 1 previous similar message [11813.345606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11813.350246] Lustre: Skipped 1 previous similar message [11814.505394] Lustre: server umount lustre-MDT0000 complete [11818.447301] LustreError: 375709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783980192 with bad export cookie 15513560892159759884 [11818.455252] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11818.463159] LustreError: 375709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11818.884686] Lustre: server umount lustre-MDT0001 complete [11835.037979] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11878.663584] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:04:09 (1783980249) [11886.459341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11887.010536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11892.194363] LustreError: 380165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11892.241464] LustreError: 380165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11892.246617] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11896.295800] LustreError: 380166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11901.416934] LustreError: 380165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.184384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11902.623475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11907.267909] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11915.510910] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11921.184422] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11927.269937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11927.511222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11933.727706] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11941.025676] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11947.504986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11947.716209] Lustre: lustre-OST0001: new disk, initializing [11947.720655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11947.726593] Lustre: 383002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11947.824666] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11954.837114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11954.855201] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11954.975079] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11955.134349] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11964.951526] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11970.533613] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11970.540968] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11970.544265] LustreError: Skipped 1 previous similar message [11970.563093] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11970.565422] Lustre: Skipped 1 previous similar message [11970.578168] Lustre: Skipped 1 previous similar message [11975.485029] Lustre: server umount lustre-OST0001 complete [11975.654664] LustreError: 381986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11975.665171] LustreError: 381986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11980.775472] LustreError: 381987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.792968] LustreError: 381987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11991.014326] LustreError: 381987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11991.051537] LustreError: 381987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11996.148805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11996.153992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11996.167030] Lustre: Skipped 1 previous similar message [11996.186428] Lustre: Skipped 2 previous similar messages [11999.667130] Lustre: server umount lustre-OST0000 complete [12004.837207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12004.847387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12018.144375] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12018.535550] Lustre: server umount lustre-MDT0000 complete [12020.706463] LustreError: 380894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.741738] LustreError: 380894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12022.518101] LustreError: 380147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783980396 with bad export cookie 15513560892159760766 [12022.525414] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12022.534662] LustreError: 380147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12028.907511] Lustre: server umount lustre-MDT0001 complete [12038.997467] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [12041.930534] Key type lgssc unregistered [12042.228820] LNet: 384992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12042.235464] LNetError: 384992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12042.260612] LNet: Removed LNI 192.168.202.103@tcp [12043.025586] Key type .llcrypt unregistered [12043.028281] Key type ._llcrypt unregistered [12096.019972] Key type ._llcrypt registered [12096.025853] Key type .llcrypt registered [12096.147917] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [12097.310464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12097.371926] alg: No test for adler32 (adler32-zlib) [12098.524282] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [12098.812680] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [12100.480268] Key type lgssc registered [12101.817867] Lustre: Echo OBD driver; http://www.lustre.org/ [12109.312294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12109.347421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12110.768212] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12110.808209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12111.048829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12115.924717] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12124.155059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12124.251694] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12124.301500] Lustre: 387797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12124.573712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12128.715260] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12135.943597] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12141.583307] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12145.123603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12145.144842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12145.175939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12146.662539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12146.663572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12146.694651] Lustre: Skipped 1 previous similar message [12149.829990] Lustre: server umount lustre-MDT0000 complete [12153.804564] LustreError: 387789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783980527 with bad export cookie 16803696992049542048 [12153.824325] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12154.125351] Lustre: server umount lustre-MDT0001 complete [12162.743892] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:08:55 (1783980535) [12169.247271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12169.800779] LustreError: 390307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.828800] LustreError: 390307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12169.865513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12174.023468] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12175.329668] LustreError: 390308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12180.451620] LustreError: 390307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12181.973598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.247816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12186.220187] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12192.986878] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12198.508120] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12204.345324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12204.577610] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12204.707871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12210.893153] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12218.316451] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12227.090569] Lustre: Permanently deactivating lustre-OST0000 [12227.107808] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12236.826955] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12237.577832] LustreError: 393338:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12237.585601] LustreError: 393338:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12237.592634] LustreError: 393338:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12240.138981] LustreError: 393387:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12240.147549] LustreError: 393387:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12248.241528] Lustre: server umount lustre-OST0000 complete [12251.205168] LustreError: 393636:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12251.214978] LustreError: 393636:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12254.177842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12254.192423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12254.223246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12256.740755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12256.747752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12261.859360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12261.872034] Lustre: Skipped 2 previous similar messages [12263.978461] Lustre: server umount lustre-MDT0000 complete [12266.872096] LustreError: 390288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783980640 with bad export cookie 16803696992049542629 [12266.876691] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12266.879392] LustreError: 390288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12277.325400] Lustre: server umount lustre-MDT0001 complete [12283.348925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12286.679436] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12289.420763] LustreError: 394796:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12295.904881] Lustre: 395135:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.103@tcp [12298.424243] Lustre: server umount MGS complete [12310.810763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12311.379747] LustreError: 395860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12311.403293] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12311.411509] Lustre: Skipped 1 previous similar message [12311.531943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12316.538982] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12316.644081] LustreError: 395861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12324.865224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12325.207068] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12329.430137] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12335.858536] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12341.563336] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12343.334680] Lustre: Permanently reactivating lustre-OST0000 [12343.362075] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12343.371332] Lustre: Skipped 2 previous similar messages [12371.936211] LustreError: 397639:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12380.802058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12381.092094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12381.095114] Lustre: Skipped 1 previous similar message [12387.611456] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12396.089259] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12402.663925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12402.677029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12402.698867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12408.384032] Lustre: server umount lustre-OST0000 complete [12412.385884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12412.389279] LustreError: Skipped 1 previous similar message [12412.394718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12412.409211] Lustre: Skipped 1 previous similar message [12418.025996] LustreError: 395860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12418.040387] LustreError: 395860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12418.081871] Lustre: server umount lustre-MDT0000 complete [12422.174665] LustreError: 395839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783980795 with bad export cookie 16803696992049543672 [12422.183111] LustreError: 395839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12422.184474] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12422.705962] Lustre: server umount lustre-MDT0001 complete [12435.935552] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [12438.929868] Key type lgssc unregistered [12439.258145] LNet: 400275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12439.268330] LNetError: 400275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12439.290985] LNet: Removed LNI 192.168.202.103@tcp [12440.223488] Key type .llcrypt unregistered [12440.228528] Key type ._llcrypt unregistered [12458.922543] Key type ._llcrypt registered [12458.927348] Key type .llcrypt registered [12459.070987] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [12473.916456] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [12475.599894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12475.642685] alg: No test for adler32 (adler32-zlib) [12476.750198] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [12476.981106] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [12478.632418] Key type lgssc registered [12479.707622] Lustre: Echo OBD driver; http://www.lustre.org/ [12513.965191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12514.009205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12515.367593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12515.431653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12515.557846] Lustre: lustre-MDT0000: new disk, initializing [12515.664080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12515.686372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12520.115420] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12530.518352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12530.668803] Lustre: 403431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12530.742176] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12530.747174] Lustre: Skipped 1 previous similar message [12530.838112] Lustre: lustre-MDT0001: new disk, initializing [12530.936275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12530.961780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12530.974500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12535.666345] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12545.203909] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12550.875422] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12557.193327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12557.427365] Lustre: lustre-OST0000: new disk, initializing [12557.430491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12557.435451] Lustre: 405399:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12557.528526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12563.518504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12563.550205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12563.647983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12564.090549] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12573.108313] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12578.120572] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12578.310265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12584.691264] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12584.966610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12589.549360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12589.564826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12589.588258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12589.619274] Lustre: Skipped 1 previous similar message [12594.058950] Lustre: server umount lustre-OST0000 complete [12599.782495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12599.806751] Lustre: Skipped 1 previous similar message [12599.815290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12604.274262] Lustre: server umount lustre-MDT0000 complete [12608.290801] LustreError: 403424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783980982 with bad export cookie 18221300081682258290 [12608.292722] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12608.308329] LustreError: 403424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12608.736930] Lustre: server umount lustre-MDT0001 complete [12618.346191] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12621.307590] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:16:32 (1783980992) [12629.549481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12630.202783] LustreError: 407860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12630.214150] LustreError: 407860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12630.263760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12634.942336] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12635.623291] LustreError: 407861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12639.737783] LustreError: 407860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.012706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12644.621254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12650.391247] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12658.303885] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12663.748157] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12670.599266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12670.878081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12677.370539] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12686.741855] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12688.591993] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12691.813330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12699.634680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12699.650937] Lustre: Skipped 1 previous similar message [12699.660810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12699.660956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12699.682971] Lustre: Skipped 1 previous similar message [12703.804961] Lustre: server umount lustre-OST0000 complete [12709.859530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12709.864935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12709.900598] Lustre: Skipped 2 previous similar messages [12709.941874] Lustre: Skipped 1 previous similar message [12713.637466] Lustre: server umount lustre-MDT0000 complete [12714.978086] LustreError: 407856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12714.991952] LustreError: 407856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12717.487471] LustreError: 407840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783981091 with bad export cookie 18221300081682259172 [12717.510110] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12717.897926] Lustre: server umount lustre-MDT0001 complete [12729.583951] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [12732.870589] Key type lgssc unregistered [12733.144420] LNet: 411586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12733.154400] LNetError: 411586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12734.188336] LNet: Removed LNI 192.168.202.103@tcp [12735.413308] Key type .llcrypt unregistered [12735.417536] Key type ._llcrypt unregistered [12760.374433] Key type ._llcrypt registered [12760.379495] Key type .llcrypt registered [12760.529639] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [12760.949557] Key type .llcrypt unregistered [12760.954523] Key type ._llcrypt unregistered [12780.471601] Key type ._llcrypt registered [12780.474609] Key type .llcrypt registered [12780.620626] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [12782.010560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12782.107371] alg: No test for adler32 (adler32-zlib) [12783.143638] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [12783.466464] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [12785.152818] Key type lgssc registered [12786.475045] Lustre: Echo OBD driver; http://www.lustre.org/ [12795.534367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12795.548246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12797.035300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12802.118725] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12806.641139] LustreError: 413803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12806.667540] LustreError: 413803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12811.279826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12811.727578] LustreError: 413804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12811.748714] LustreError: 413804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12811.775515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12816.868119] LustreError: 413803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12816.874328] LustreError: 413803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12818.396828] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12821.991915] LustreError: 413799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12822.022277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12822.022589] LustreError: 413799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12826.114174] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12827.113566] LustreError: 413804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.130529] LustreError: 413804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12832.821399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12833.116982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12837.407729] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12847.597967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12847.613065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12847.643481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12848.616779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12848.620623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12848.623356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12851.086884] Lustre: server umount lustre-OST0000 complete [12858.849823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12858.853369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12858.865686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12858.865739] Lustre: Skipped 1 previous similar message [12858.913673] Lustre: Skipped 1 previous similar message [12861.508489] Lustre: server umount lustre-MDT0000 complete [12863.977653] LustreError: 413799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12864.008043] LustreError: 413799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12865.610601] LustreError: 415546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783981239 with bad export cookie 555256467488059372 [12865.618133] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12865.621732] LustreError: 415546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12866.065814] Lustre: server umount lustre-MDT0001 complete [12877.605538] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [12880.221981] Key type lgssc unregistered [12880.654579] LNet: 416952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12880.662240] LNetError: 416952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12880.674369] LNet: Removed LNI 192.168.202.103@tcp [12881.657856] Key type .llcrypt unregistered [12881.660516] Key type ._llcrypt unregistered [12907.888688] Key type ._llcrypt registered [12907.892333] Key type .llcrypt registered [12908.022102] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [12909.946457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12910.008973] alg: No test for adler32 (adler32-zlib) [12911.187819] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [12911.517854] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [12913.192201] Key type lgssc registered [12914.221782] Lustre: Echo OBD driver; http://www.lustre.org/ [12922.281180] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12922.292827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12937.824927] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12942.946096] LustreError: 418341:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [12969.632412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12975.361652] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12982.609163] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12990.132452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12990.243338] LustreError: 417677:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000edce6673 x1870639959902464/t0(0) o250->MGC192.168.202.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12990.653208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12990.727237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12995.358568] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [12996.072480] LustreError: 418366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12996.092993] LustreError: 418366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13001.193124] LustreError: 418367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13004.106735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13004.501812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13009.252692] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13017.014941] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13022.433500] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13030.370236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13030.382580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13030.389428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13031.126746] Lustre: server umount lustre-OST0000 complete [13035.490291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13035.502158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13035.519223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13036.528523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13036.543962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13036.548119] Lustre: Skipped 1 previous similar message [13041.633680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13041.640784] Lustre: Skipped 1 previous similar message [13044.797825] Lustre: server umount lustre-MDT0000 complete [13046.754091] LustreError: 419376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13046.775642] LustreError: 419376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13048.460117] LustreError: 418349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783981422 with bad export cookie 12473811729301300694 [13048.475147] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13048.985651] Lustre: server umount lustre-MDT0001 complete [13058.967280] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [13061.691479] Key type lgssc unregistered [13061.964141] LNet: 421907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13061.971903] LNetError: 421907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13062.000054] LNet: Removed LNI 192.168.202.103@tcp [13062.802788] Key type .llcrypt unregistered [13062.804256] Key type ._llcrypt unregistered [13086.805318] Key type ._llcrypt registered [13086.809961] Key type .llcrypt registered [13086.933972] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [13088.540221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13088.564106] alg: No test for adler32 (adler32-zlib) [13089.767563] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [13089.989973] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [13091.672344] Key type lgssc registered [13092.804072] Lustre: Echo OBD driver; http://www.lustre.org/ [13101.212099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13101.231898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13102.804068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13107.743702] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13112.314940] LustreError: 423326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13112.345615] LustreError: 423326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13115.897204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13116.236802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13120.481548] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13127.227484] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13132.658503] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13139.669557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13140.160589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13146.255392] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13147.348864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13154.582403] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13162.465352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13162.472565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13162.487189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13162.982376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13163.000340] Lustre: Skipped 1 previous similar message [13163.004103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13163.008090] Lustre: Skipped 2 previous similar messages [13165.621839] Lustre: server umount lustre-MDT0000 complete [13166.623249] LustreError: 425167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.109779] LustreError: 423325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13168.136145] LustreError: 423325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13168.696654] LustreError: 423307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.3@tcp arrived at 1783981542 with bad export cookie 3661099710194430546 [13168.718857] LustreError: 423307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13170.661987] LustreError: 423305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783981544 with bad export cookie 3661099710194429650 [13170.680895] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13170.683882] LustreError: 423305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13172.205802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13172.211167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13172.239307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13173.218981] LustreError: 424055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13173.232234] LustreError: 424055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13175.266828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13175.274310] Lustre: Skipped 1 previous similar message [13177.064848] Lustre: server umount lustre-OST0000 complete [13180.390120] LustreError: 424055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13180.401410] LustreError: 424055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13184.158039] Lustre: server umount lustre-MDT0001 complete [13194.558743] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [13197.372517] Key type lgssc unregistered [13197.683804] LNet: 427010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13197.691886] LNetError: 427010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13198.761124] LNet: Removed LNI 192.168.202.103@tcp [13199.721270] Key type .llcrypt unregistered [13199.724500] Key type ._llcrypt unregistered [13224.545572] Key type ._llcrypt registered [13224.551117] Key type .llcrypt registered [13224.651893] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [13225.916927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13225.944083] alg: No test for adler32 (adler32-zlib) [13226.931260] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [13227.111876] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [13228.761128] Key type lgssc registered [13230.023358] Lustre: Echo OBD driver; http://www.lustre.org/ [13237.754601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13237.771715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13239.359471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13244.372452] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13249.535284] LustreError: 428434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13249.559082] LustreError: 428434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13252.472328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13252.958724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13256.934318] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13263.404654] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13268.496467] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13274.879260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13275.154312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13281.003307] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13284.481900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13287.345229] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13294.049751] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13294.055395] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13294.065652] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13295.073789] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13295.084070] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13295.097054] Lustre: Skipped 1 previous similar message [13297.375176] Lustre: server umount lustre-MDT0001 complete [13299.313054] LustreError: 428430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13305.313741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13305.320550] LustreError: 428433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13305.339162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13305.351722] LustreError: 428433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13305.360735] Lustre: Skipped 1 previous similar message [13310.384778] Lustre: server umount lustre-OST0000 complete [13310.434779] LustreError: 429161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13316.225521] Lustre: server umount lustre-MDT0000 complete [13331.098354] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [13333.526543] Key type lgssc unregistered [13333.774175] LNet: 432118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13333.782602] LNetError: 432118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13334.827612] LNet: Removed LNI 192.168.202.103@tcp [13335.656531] Key type .llcrypt unregistered [13335.659783] Key type ._llcrypt unregistered [13359.113390] Key type ._llcrypt registered [13359.115586] Key type .llcrypt registered [13359.191660] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [13373.005555] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [13374.159648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13374.174562] alg: No test for adler32 (adler32-zlib) [13375.146144] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [13375.369272] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [13377.040170] Key type lgssc registered [13378.116356] Lustre: Echo OBD driver; http://www.lustre.org/ [13410.673487] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [13425.501143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13425.564138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13426.994045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13427.035044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13427.158202] Lustre: lustre-MDT0000: new disk, initializing [13427.290067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13427.331567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13432.087618] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13448.194054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13448.599497] Lustre: lustre-OST0000: new disk, initializing [13448.609860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13448.615746] Lustre: Skipped 1 previous similar message [13448.620948] Lustre: 437153:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13448.716260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13455.689349] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13458.496858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13458.504016] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13458.522786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13465.171673] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13478.247084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13478.372390] Lustre: 436137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13478.433943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13478.493176] Lustre: lustre-MDT0001: new disk, initializing [13478.598974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13478.639249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13478.650930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13483.333652] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13492.628487] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13492.862625] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13497.740616] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13497.948467] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13502.579107] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.3@tcp (stopping) [13502.948096] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13502.965644] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13507.682420] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.3@tcp (stopping) [13507.690397] Lustre: Skipped 2 previous similar messages [13508.380762] Lustre: server umount lustre-MDT0001 complete [13518.317056] LustreError: 436148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13518.327818] LustreError: 436148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13523.431834] LustreError: 436142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13523.465435] LustreError: 436142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13528.547226] LustreError: 436148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13528.557935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13528.579473] Lustre: Skipped 1 previous similar message [13528.586574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13528.592076] Lustre: Skipped 2 previous similar messages [13531.934705] Lustre: server umount lustre-OST0000 complete [13533.666177] LustreError: 438767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13541.614844] Lustre: server umount lustre-MDT0000 complete [13553.569247] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [13555.944821] Key type lgssc unregistered [13556.223507] LNet: 440900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13556.236177] LNetError: 440900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13556.258508] LNet: Removed LNI 192.168.202.103@tcp [13557.111653] Key type .llcrypt unregistered [13557.115646] Key type ._llcrypt unregistered [13597.159787] Key type ._llcrypt registered [13597.161216] Key type .llcrypt registered [13597.269416] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [13597.556981] Key type .llcrypt unregistered [13597.560325] Key type ._llcrypt unregistered [13613.473883] Key type ._llcrypt registered [13613.476557] Key type .llcrypt registered [13613.615472] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [13615.028263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13615.044862] alg: No test for adler32 (adler32-zlib) [13616.081445] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [13616.323808] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [13618.000252] Key type lgssc registered [13618.978295] Lustre: Echo OBD driver; http://www.lustre.org/ [13644.070083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13644.106916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13645.393522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13645.453627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13645.507914] Lustre: lustre-MDT0000: new disk, initializing [13645.578795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13645.603600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13649.515739] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13660.546648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13660.680381] Lustre: 444278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13660.734730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13660.739410] Lustre: Skipped 1 previous similar message [13660.823612] Lustre: lustre-MDT0001: new disk, initializing [13660.921943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13660.953676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13660.974750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13665.611412] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13674.857315] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13680.562650] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13686.166193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13686.405330] Lustre: lustre-OST0000: new disk, initializing [13686.408626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13686.413405] Lustre: 446243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13686.499789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13691.444987] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13691.952130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13691.967050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13692.070677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13699.802204] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13707.235028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13707.246972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13707.286975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13707.748918] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13707.782939] Lustre: Skipped 1 previous similar message [13707.791131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13707.794116] Lustre: Skipped 1 previous similar message [13712.870781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13712.881054] Lustre: Skipped 2 previous similar messages [13712.971230] Lustre: server umount lustre-MDT0000 complete [13716.727191] LustreError: 444267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783982090 with bad export cookie 15411865195559198817 [13716.759192] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13717.163327] Lustre: server umount lustre-MDT0001 complete [13731.467752] Lustre: server umount lustre-OST0000 complete [13740.893263] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:35:13 (1783982113) [13747.986068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13748.371540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13752.670297] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13753.828951] LustreError: 448273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13753.848783] LustreError: 448273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13758.953246] LustreError: 448274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13760.718875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13761.000567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13765.235247] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13772.110393] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13776.523704] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13795.569841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13795.751868] Lustre: Found index 0 for lustre-OST0000, updating log [13795.958429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13801.498497] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [13807.869566] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13820.546834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13828.077876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13828.089798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13828.114599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13833.706370] Lustre: server umount lustre-OST0000 complete [13850.599859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13850.634227] Lustre: Skipped 1 previous similar message [13850.645202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13850.653379] Lustre: Skipped 4 previous similar messages [13860.834666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13860.837290] Lustre: Skipped 4 previous similar messages [13863.392157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13863.710071] Lustre: server umount lustre-MDT0000 complete [13865.954453] LustreError: 448270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13865.966965] LustreError: 448270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13867.148727] LustreError: 450116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783982240 with bad export cookie 15411865195559200210 [13867.151275] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13867.159169] LustreError: 450116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13867.232728] LNet: 1 local NIs in recovery (showing 1): 192.168.202.103@tcp [13867.496983] Lustre: server umount lustre-MDT0001 complete [13874.984303] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:37:27 (1783982247) [14485.585257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14485.889753] LustreError: 454708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14485.945471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14489.586883] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [14491.108645] LustreError: 454709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14495.897994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14496.175970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14499.233961] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [14504.236262] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14507.880252] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14525.545476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14525.709677] Lustre: Found index 0 for lustre-OST0000, updating log [14525.850635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14529.727476] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [14534.393068] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14535.765498] Lustre: server umount lustre-OST0000 complete [14542.310685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14542.320702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14542.333570] Lustre: Skipped 1 previous similar message [14542.338673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14542.823076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14544.161083] LNet: 1 local NIs in recovery (showing 1): 192.168.202.103@tcp [14544.230167] Lustre: server umount lustre-MDT0000 complete [14544.866138] LustreError: 454703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14544.877317] LustreError: 454703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14546.688851] LustreError: 454688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783982920 with bad export cookie 15411865195559201288 [14546.694608] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14546.696536] LustreError: 454688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14546.902199] Lustre: server umount lustre-MDT0001 complete [14570.852895] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [14577.029099] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [14601.906610] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:49:34 (1783982974) [14614.397517] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [14616.339939] Key type lgssc unregistered [14616.541111] LNet: 461309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14616.551969] LNetError: 461309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14746.608065] LNet: Removed LNI 192.168.202.103@tcp [14747.375815] Key type .llcrypt unregistered [14747.377677] Key type ._llcrypt unregistered [14758.329084] Key type ._llcrypt registered [14758.331382] Key type .llcrypt registered [14758.414814] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [14759.358654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14759.369328] alg: No test for adler32 (adler32-zlib) [14760.313607] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [14760.472190] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [14762.088742] Key type lgssc registered [14762.870659] Lustre: Echo OBD driver; http://www.lustre.org/ [14765.508415] LNet: Added LNI 192.168.202.103@tcp100 [8/256/0/180] [14766.094592] LNet: Added LNI 192.168.202.103@tcp101 [8/256/0/180] [14766.630719] LNet: Added LNI 192.168.202.103@tcp102 [8/256/0/180] [14767.165704] LNet: Added LNI 192.168.202.103@tcp103 [8/256/0/180] [14767.786798] LNet: Added LNI 192.168.202.103@tcp104 [8/256/0/180] [14768.388243] LNet: Added LNI 192.168.202.103@tcp105 [8/256/0/180] [14769.004221] LNet: Added LNI 192.168.202.103@tcp106 [8/256/0/180] [14769.570523] LNet: Added LNI 192.168.202.103@tcp107 [8/256/0/180] [14770.164332] LNet: Added LNI 192.168.202.103@tcp108 [8/256/0/180] [14770.632807] LNet: Added LNI 192.168.202.103@tcp109 [8/256/0/180] [14771.193732] LNet: Added LNI 192.168.202.103@tcp110 [8/256/0/180] [14771.842033] LNet: Added LNI 192.168.202.103@tcp111 [8/256/0/180] [14772.465748] LNet: Added LNI 192.168.202.103@tcp112 [8/256/0/180] [14773.077659] LNet: Added LNI 192.168.202.103@tcp113 [8/256/0/180] [14773.639867] LNet: Added LNI 192.168.202.103@tcp114 [8/256/0/180] [14774.181436] LNet: Added LNI 192.168.202.103@tcp115 [8/256/0/180] [14774.700572] LNet: Added LNI 192.168.202.103@tcp116 [8/256/0/180] [14775.238130] LNet: Added LNI 192.168.202.103@tcp117 [8/256/0/180] [14775.811671] LNet: Added LNI 192.168.202.103@tcp118 [8/256/0/180] [14776.379528] LNet: Added LNI 192.168.202.103@tcp119 [8/256/0/180] [14776.975551] LNet: Added LNI 192.168.202.103@tcp120 [8/256/0/180] [14777.611911] LNet: Added LNI 192.168.202.103@tcp121 [8/256/0/180] [14778.126163] LNet: Added LNI 192.168.202.103@tcp122 [8/256/0/180] [14778.664447] LNet: Added LNI 192.168.202.103@tcp123 [8/256/0/180] [14779.145913] LNet: Added LNI 192.168.202.103@tcp124 [8/256/0/180] [14779.646621] LNet: Added LNI 192.168.202.103@tcp125 [8/256/0/180] [14780.211946] LNet: Added LNI 192.168.202.103@tcp126 [8/256/0/180] [14780.710970] LNet: Added LNI 192.168.202.103@tcp127 [8/256/0/180] [14781.288312] LNet: Added LNI 192.168.202.103@tcp128 [8/256/0/180] [14781.932584] LNet: Added LNI 192.168.202.103@tcp129 [8/256/0/180] [14782.497912] LNet: Added LNI 192.168.202.103@tcp130 [8/256/0/180] [14783.085368] LNet: Added LNI 192.168.202.103@tcp131 [8/256/0/180] [14783.682306] LNet: Added LNI 192.168.202.103@tcp132 [8/256/0/180] [14784.322469] LNet: Added LNI 192.168.202.103@tcp133 [8/256/0/180] [14784.829555] LNet: Added LNI 192.168.202.103@tcp134 [8/256/0/180] [14785.463369] LNet: Added LNI 192.168.202.103@tcp135 [8/256/0/180] [14790.246684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14790.259969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14791.341311] Lustre: 464319:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [14791.396359] LNetError: 464341:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [14791.414991] LNetError: Protocol error connecting to 192.168.202.103@tcp113 at host 192.168.202.103:988: is it running a compatible version of Lustre? [14791.424412] LNetError: Skipped 6 previous similar messages [14791.475826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14791.493942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14791.544351] Lustre: lustre-MDT0000: new disk, initializing [14791.589791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14791.598929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14792.238099] LNet: 2 local NIs in recovery (showing 2): 192.168.202.103@tcp128, 192.168.202.103@tcp129 [14792.239865] LNetError: 464340:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [14792.263361] LNetError: 464340:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14792.273228] LNetError: Protocol error connecting to 192.168.202.103@tcp109 at host 192.168.202.103:988: is it running a compatible version of Lustre? [14792.283881] LNetError: Skipped 1 previous similar message [14794.608123] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [14802.278386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14802.389968] Lustre: 464338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14802.443569] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14802.447327] Lustre: Skipped 1 previous similar message [14802.535228] Lustre: lustre-MDT0001: new disk, initializing [14802.576263] Lustre: 465142:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [14802.584593] Lustre: 465142:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14802.693491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14802.724835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14802.738338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14806.165502] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [14812.193988] Lustre: 465711:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [14812.210066] Lustre: 465711:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14813.068732] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14817.038404] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14821.397935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14821.569240] Lustre: lustre-OST0000: new disk, initializing [14821.571644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14821.574127] Lustre: 466315:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14821.628820] Lustre: 466320:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [14821.633744] Lustre: 466320:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14821.663126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14821.924275] LNetError: 464354:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [14821.932136] LNetError: Protocol error connecting to 192.168.202.103@tcp127 at host 192.168.202.103:988: is it running a compatible version of Lustre? [14823.970803] LNetError: 461779:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [14823.975057] LNetError: Protocol error connecting to 192.168.202.103@tcp109 at host 192.168.202.103:988: is it running a compatible version of Lustre? [14825.612185] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [14829.089308] LNetError: 461780:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [14829.097919] LNetError: 461780:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14829.104270] LNetError: Protocol error connecting to 192.168.202.103@tcp126 at host 192.168.202.103:988: is it running a compatible version of Lustre? [14829.111749] LNetError: Skipped 2 previous similar messages [14829.587749] Lustre: 466892:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [14829.598338] Lustre: 466892:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14829.653562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14829.660030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14829.756603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14831.833143] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14837.281143] LNetError: 464353:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [14837.290753] LNetError: 464353:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [14837.296442] LNetError: Protocol error connecting to 192.168.202.103@tcp125 at host 192.168.202.103:988: is it running a compatible version of Lustre? [14837.303400] LNetError: Skipped 4 previous similar messages [14840.292584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14840.295228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14840.301736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14840.309976] Lustre: Skipped 1 previous similar message [14842.250753] Lustre: server umount lustre-OST0000 complete [14845.410669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14845.418474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14845.419323] Lustre: Skipped 1 previous similar message [14845.422741] Lustre: Skipped 1 previous similar message [14850.529324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14850.535376] Lustre: Skipped 2 previous similar messages [14850.859737] Lustre: server umount lustre-MDT0000 complete [14853.482789] LustreError: 464327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783983227 with bad export cookie 6196542966072036825 [14853.489859] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14853.495812] LustreError: 464327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14853.669192] LNetError: 461781:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [14853.677779] LNetError: 461781:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14853.682840] LNetError: Protocol error connecting to 192.168.202.103@tcp123 at host 192.168.202.103:988: is it running a compatible version of Lustre? [14853.690274] LNetError: Skipped 7 previous similar messages [14853.757501] Lustre: server umount lustre-MDT0001 complete [14861.120409] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [14863.122191] Key type lgssc unregistered [14863.358824] LNet: 468224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14863.367621] LNetError: 468224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14864.421186] LNet: Removed LNI 192.168.202.103@tcp [14864.424775] LNetError: 468224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14864.441264] LNet: Removed LNI 192.168.202.103@tcp100 [14865.506582] LNet: Removed LNI 192.168.202.103@tcp101 [14865.510871] LNetError: 468224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14865.518901] LNetError: 468224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14865.524524] LNet: Removed LNI 192.168.202.103@tcp102 [14866.528637] LNet: Removed LNI 192.168.202.103@tcp103 [14867.552158] LNet: Removed LNI 192.168.202.103@tcp104 [14867.554861] LNetError: 468224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14867.561318] LNetError: 468224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14867.566604] LNet: Removed LNI 192.168.202.103@tcp105 [14868.576590] LNet: Removed LNI 192.168.202.103@tcp106 [14869.600144] LNet: Removed LNI 192.168.202.103@tcp107 [14870.624177] LNet: Removed LNI 192.168.202.103@tcp108 [14871.650954] LNet: Removed LNI 192.168.202.103@tcp109 [14871.653295] LNetError: 468224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14871.657247] LNetError: 468224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14871.661711] LNet: Removed LNI 192.168.202.103@tcp110 [14872.672151] LNet: Removed LNI 192.168.202.103@tcp111 [14873.696169] LNet: Removed LNI 192.168.202.103@tcp112 [14874.720176] LNet: Removed LNI 192.168.202.103@tcp113 [14875.744151] LNet: Removed LNI 192.168.202.103@tcp114 [14876.768546] LNet: Removed LNI 192.168.202.103@tcp115 [14877.792164] LNet: Removed LNI 192.168.202.103@tcp116 [14878.817805] LNet: Removed LNI 192.168.202.103@tcp117 [14879.840211] LNet: Removed LNI 192.168.202.103@tcp118 [14879.844570] LNetError: 468224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14879.854313] LNetError: 468224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14879.865199] LNet: Removed LNI 192.168.202.103@tcp119 [14880.928149] LNet: Removed LNI 192.168.202.103@tcp120 [14881.954838] LNet: Removed LNI 192.168.202.103@tcp121 [14882.976139] LNet: Removed LNI 192.168.202.103@tcp122 [14884.000144] LNet: Removed LNI 192.168.202.103@tcp123 [14885.025668] LNet: Removed LNI 192.168.202.103@tcp124 [14886.048263] LNet: Removed LNI 192.168.202.103@tcp125 [14887.072136] LNet: Removed LNI 192.168.202.103@tcp126 [14888.096266] LNet: Removed LNI 192.168.202.103@tcp127 [14889.120152] LNet: Removed LNI 192.168.202.103@tcp128 [14890.145523] LNet: Removed LNI 192.168.202.103@tcp129 [14890.146922] LNet: Removed LNI 192.168.202.103@tcp130 [14890.149613] LNet: Removed LNI 192.168.202.103@tcp131 [14890.151785] LNet: Removed LNI 192.168.202.103@tcp132 [14890.153485] LNet: Removed LNI 192.168.202.103@tcp133 [14890.155684] LNet: Removed LNI 192.168.202.103@tcp134 [14890.166730] LNet: Removed LNI 192.168.202.103@tcp135 [14890.704693] Key type .llcrypt unregistered [14890.706059] Key type ._llcrypt unregistered [14905.174941] Key type ._llcrypt registered [14905.179742] Key type .llcrypt registered [14905.245716] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [14914.813254] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [14915.643279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14915.661066] alg: No test for adler32 (adler32-zlib) [14916.580971] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [14916.723826] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [14918.336105] Key type lgssc registered [14919.078596] Lustre: Echo OBD driver; http://www.lustre.org/ [14940.061837] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:55:13 (1783983313) [14950.606711] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [14952.295775] Key type lgssc unregistered [14952.491802] LNet: 472286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14952.501556] LNetError: 472286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14952.511483] LNet: Removed LNI 192.168.202.103@tcp [14953.057336] Key type .llcrypt unregistered [14953.059688] Key type ._llcrypt unregistered [14963.039652] Key type ._llcrypt registered [14963.042233] Key type .llcrypt registered [14963.121039] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [14963.755142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14963.774418] alg: No test for adler32 (adler32-zlib) [14964.686792] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [14964.820696] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [14966.464171] Key type lgssc registered [14967.092363] Lustre: Echo OBD driver; http://www.lustre.org/ [14972.521417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14972.542620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14973.739262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14973.756088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14973.800431] Lustre: lustre-MDT0000: new disk, initializing [14973.847657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14973.864938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14976.691164] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [14983.596021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14983.658993] Lustre: 473477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14983.691593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14983.693753] Lustre: Skipped 1 previous similar message [14983.742851] Lustre: lustre-MDT0001: new disk, initializing [14983.821620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14983.852306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14983.858661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14986.332813] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [14992.495353] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14995.823832] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14999.800932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14999.959451] Lustre: lustre-OST0000: new disk, initializing [14999.962166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14999.966330] Lustre: 475444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15000.031995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15003.794826] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15009.990377] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15010.320667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15010.326305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15010.398089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15012.238570] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing lnet_if_list [15015.164652] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15015.179858] LNet: Added LNI 192.168.202.103@tcp10 [8/256/0/180] [15018.310692] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15018.327719] LNet: Added LNI 192.168.202.103@tcp11 [8/256/0/180] [15021.011531] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15021.031082] LNet: Added LNI 192.168.202.103@tcp12 [8/256/0/180] [15023.569080] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15023.593474] LNet: Added LNI 192.168.202.103@tcp13 [8/256/0/180] [15026.559764] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15026.591210] LNet: Added LNI 192.168.202.103@tcp14 [8/256/0/180] [15029.775235] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15029.795830] LNet: Added LNI 192.168.202.103@tcp15 [8/256/0/180] [15032.625393] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15032.644328] LNet: Added LNI 192.168.202.103@tcp16 [8/256/0/180] [15035.988220] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15036.008292] LNet: Added LNI 192.168.202.103@tcp17 [8/256/0/180] [15039.118084] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15039.135426] LNet: Added LNI 192.168.202.103@tcp18 [8/256/0/180] [15042.125481] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15042.153221] LNet: Added LNI 192.168.202.103@tcp19 [8/256/0/180] [15045.671774] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15045.686453] LNet: Added LNI 192.168.202.103@tcp20 [8/256/0/180] [15049.151556] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15049.167482] LNet: Added LNI 192.168.202.103@tcp21 [8/256/0/180] [15052.420290] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15052.435202] LNet: Added LNI 192.168.202.103@tcp22 [8/256/0/180] [15056.107989] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15056.137215] LNet: Added LNI 192.168.202.103@tcp23 [8/256/0/180] [15059.551642] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15059.576663] LNet: Added LNI 192.168.202.103@tcp24 [8/256/0/180] [15063.467727] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15063.489528] LNet: Added LNI 192.168.202.103@tcp25 [8/256/0/180] [15067.627158] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15067.652957] LNet: Added LNI 192.168.202.103@tcp26 [8/256/0/180] [15071.748428] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15071.769883] LNet: Added LNI 192.168.202.103@tcp27 [8/256/0/180] [15075.506721] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15075.526035] LNet: Added LNI 192.168.202.103@tcp28 [8/256/0/180] [15079.075941] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15079.100755] LNet: Added LNI 192.168.202.103@tcp29 [8/256/0/180] [15082.034425] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15082.052915] LNet: Added LNI 192.168.202.103@tcp30 [8/256/0/180] [15085.244948] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15085.264783] LNet: Added LNI 192.168.202.103@tcp31 [8/256/0/180] [15089.353993] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15089.372020] LNet: Added LNI 192.168.202.103@tcp32 [8/256/0/180] [15093.368657] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15093.387676] LNet: Added LNI 192.168.202.103@tcp33 [8/256/0/180] [15097.477500] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15097.496102] LNet: Added LNI 192.168.202.103@tcp34 [8/256/0/180] [15100.882328] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15100.901567] LNet: Added LNI 192.168.202.103@tcp35 [8/256/0/180] [15104.620630] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15104.635721] LNet: Added LNI 192.168.202.103@tcp36 [8/256/0/180] [15108.781258] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15108.805576] LNet: Added LNI 192.168.202.103@tcp37 [8/256/0/180] [15112.716696] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15112.738732] LNet: Added LNI 192.168.202.103@tcp38 [8/256/0/180] [15116.166561] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15116.184343] LNet: Added LNI 192.168.202.103@tcp39 [8/256/0/180] [15120.042669] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15120.056925] LNet: Added LNI 192.168.202.103@tcp40 [8/256/0/180] [15123.814323] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15123.832767] LNet: Added LNI 192.168.202.103@tcp41 [8/256/0/180] [15127.828801] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15127.852660] LNet: Added LNI 192.168.202.103@tcp42 [8/256/0/180] [15131.730187] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15131.744714] LNet: Added LNI 192.168.202.103@tcp43 [8/256/0/180] [15135.674612] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15135.688955] LNet: Added LNI 192.168.202.103@tcp44 [8/256/0/180] [15139.581744] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15139.597865] LNet: Added LNI 192.168.202.103@tcp45 [8/256/0/180] [15143.221571] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15143.238487] LNet: Added LNI 192.168.202.103@tcp46 [8/256/0/180] [15147.537672] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15147.564488] LNet: Added LNI 192.168.202.103@tcp47 [8/256/0/180] [15151.559674] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15151.577397] LNet: Added LNI 192.168.202.103@tcp48 [8/256/0/180] [15156.163138] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15156.174910] LNet: Added LNI 192.168.202.103@tcp49 [8/256/0/180] [15160.608825] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15160.629443] LNet: Added LNI 192.168.202.103@tcp50 [8/256/0/180] [15173.500110] Lustre: server umount lustre-MDT0000 complete [15173.603098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15173.606890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15174.113553] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15175.723269] LustreError: 476654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783983549 with bad export cookie 13446541084101304958 [15175.724721] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15175.731207] LustreError: 476654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15175.889413] Lustre: server umount lustre-MDT0001 complete [15187.476722] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [15192.344489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15194.592602] Lustre: 472750:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783983552/real 1783983552] req@0000000057ff966e x1870642112737408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783983568 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15194.611737] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15194.617588] Lustre: Skipped 1 previous similar message [15195.617066] Lustre: 472751:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783983553/real 1783983553] req@000000005b807085 x1870642112737536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783983569 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15200.736171] Lustre: 472751:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783983558/real 1783983558] req@00000000bd5da53c x1870642112737664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783983574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15200.800683] LustreError: 472748:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a80cab3f x1870642112738560/t0(0) o250->MGC192.168.202.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15200.965722] LustreError: 475438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15200.975200] LustreError: 475438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [15201.008954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15203.386241] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15205.090969] LustreError: 499641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15205.109520] LustreError: 499641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15207.136105] Lustre: 472750:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783983564/real 1783983564] req@0000000097fa9393 x1870642112738048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783983580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15207.169935] Lustre: 472750:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15207.921021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15208.145754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15210.177609] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15213.548519] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15214.030359] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15214.569219] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15217.208943] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15228.898267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15228.901950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15228.907885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15229.922977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15229.926414] Lustre: Skipped 1 previous similar message [15230.255214] Lustre: server umount lustre-OST0000 complete [15234.018566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15234.020906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15234.025987] Lustre: Skipped 1 previous similar message [15234.028982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15238.452639] Lustre: server umount lustre-MDT0000 complete [15240.169326] LustreError: 499647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15240.188051] LustreError: 499647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15240.569761] LustreError: 473469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783983614 with bad export cookie 13446541084101307254 [15240.573877] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15240.580498] LustreError: 473469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15240.771218] Lustre: server umount lustre-MDT0001 complete [15246.975064] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [15248.828352] Key type lgssc unregistered [15249.108372] LNet: 502417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15249.115672] LNetError: 502417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15250.146611] LNet: Removed LNI 192.168.202.103@tcp [15250.148458] LNetError: 502417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15250.154033] LNet: Removed LNI 192.168.202.103@tcp10 [15250.157491] LNet: Removed LNI 192.168.202.103@tcp11 [15250.160245] LNet: Removed LNI 192.168.202.103@tcp12 [15250.162914] LNet: Removed LNI 192.168.202.103@tcp13 [15250.165752] LNet: Removed LNI 192.168.202.103@tcp14 [15250.168752] LNet: Removed LNI 192.168.202.103@tcp15 [15250.171474] LNet: Removed LNI 192.168.202.103@tcp16 [15250.173604] LNet: Removed LNI 192.168.202.103@tcp17 [15250.176406] LNet: Removed LNI 192.168.202.103@tcp18 [15250.178876] LNet: Removed LNI 192.168.202.103@tcp19 [15250.181478] LNet: Removed LNI 192.168.202.103@tcp20 [15250.183551] LNet: Removed LNI 192.168.202.103@tcp21 [15250.186105] LNet: Removed LNI 192.168.202.103@tcp22 [15250.188468] LNet: Removed LNI 192.168.202.103@tcp23 [15250.190616] LNet: Removed LNI 192.168.202.103@tcp24 [15250.193581] LNet: Removed LNI 192.168.202.103@tcp25 [15250.196187] LNet: Removed LNI 192.168.202.103@tcp26 [15250.198186] LNet: Removed LNI 192.168.202.103@tcp27 [15250.200311] LNet: Removed LNI 192.168.202.103@tcp28 [15250.202771] LNet: Removed LNI 192.168.202.103@tcp29 [15250.205156] LNet: Removed LNI 192.168.202.103@tcp30 [15250.207721] LNet: Removed LNI 192.168.202.103@tcp31 [15250.209834] LNet: Removed LNI 192.168.202.103@tcp32 [15250.211868] LNet: Removed LNI 192.168.202.103@tcp33 [15250.214240] LNet: Removed LNI 192.168.202.103@tcp34 [15250.215920] LNet: Removed LNI 192.168.202.103@tcp35 [15250.217576] LNet: Removed LNI 192.168.202.103@tcp36 [15250.222661] LNet: Removed LNI 192.168.202.103@tcp37 [15250.224886] LNet: Removed LNI 192.168.202.103@tcp38 [15250.226903] LNet: Removed LNI 192.168.202.103@tcp39 [15250.228455] LNet: Removed LNI 192.168.202.103@tcp40 [15250.231267] LNet: Removed LNI 192.168.202.103@tcp41 [15250.233922] LNet: Removed LNI 192.168.202.103@tcp42 [15250.236217] LNet: Removed LNI 192.168.202.103@tcp43 [15250.238674] LNet: Removed LNI 192.168.202.103@tcp44 [15250.241175] LNet: Removed LNI 192.168.202.103@tcp45 [15250.247311] LNet: Removed LNI 192.168.202.103@tcp46 [15250.250117] LNet: Removed LNI 192.168.202.103@tcp47 [15250.252762] LNet: Removed LNI 192.168.202.103@tcp48 [15250.255207] LNet: Removed LNI 192.168.202.103@tcp49 [15250.263564] LNet: Removed LNI 192.168.202.103@tcp50 [15250.788150] Key type .llcrypt unregistered [15250.790426] Key type ._llcrypt unregistered [15262.498244] Key type ._llcrypt registered [15262.500355] Key type .llcrypt registered [15262.576807] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [15270.711064] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [15271.455348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15271.474885] alg: No test for adler32 (adler32-zlib) [15272.441954] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [15272.604391] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [15274.192149] Key type lgssc registered [15274.834569] Lustre: Echo OBD driver; http://www.lustre.org/ [15295.001887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15295.029350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15296.199509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15296.228044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15296.292358] Lustre: lustre-MDT0000: new disk, initializing [15296.349471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15296.358618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15298.905317] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15304.560508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15304.610686] Lustre: 505600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15304.644101] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15304.646877] Lustre: Skipped 1 previous similar message [15304.685626] Lustre: lustre-MDT0001: new disk, initializing [15304.761322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15304.790127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15304.800327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15306.715441] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15311.382331] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15314.116921] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15317.691599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15317.810910] Lustre: *** cfs_fail_loc=90f, val=0*** [15317.857505] Lustre: lustre-OST0000: new disk, initializing [15317.860553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15317.865305] Lustre: 507615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15317.908554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15318.419770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15318.426535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15318.484294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15320.742770] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15325.683309] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15333.861180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15333.861539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15333.864796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15333.864893] LustreError: Skipped 1 previous similar message [15333.877807] Lustre: Skipped 1 previous similar message [15335.557176] Lustre: server umount lustre-OST0000 complete [15338.978791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15338.994753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15338.997474] Lustre: Skipped 2 previous similar messages [15343.849200] Lustre: server umount lustre-MDT0000 complete [15345.975591] LustreError: 507614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783983719 with bad export cookie 5428651807291150579 [15345.981726] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15345.988913] LustreError: 507614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15346.181144] Lustre: server umount lustre-MDT0001 complete [15351.023196] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [15352.321576] Key type lgssc unregistered [15352.470505] LNet: 509517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15352.474790] LNetError: 509517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15352.484600] LNet: Removed LNI 192.168.202.103@tcp [15352.910070] Key type .llcrypt unregistered [15352.911527] Key type ._llcrypt unregistered [15362.996770] Key type ._llcrypt registered [15362.998338] Key type .llcrypt registered [15363.052445] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [15363.536584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15363.646178] alg: No test for adler32 (adler32-zlib) [15364.580254] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [15364.715106] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [15366.304161] Key type lgssc registered [15366.795541] Lustre: Echo OBD driver; http://www.lustre.org/ [15370.467417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15370.480333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15371.728515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15373.782932] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15378.258569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15378.521443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15380.866681] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15384.581353] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15387.425246] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15390.601638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15390.785274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15393.681978] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15397.283730] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15412.148834] loop0: detected capacity change from 0 to 204800000 [15412.219298] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15412.223985] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15412.230493] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15414.931490] loop0: detected capacity change from 0 to 204800000 [15415.012287] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15415.018783] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15415.024548] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15417.458174] loop0: detected capacity change from 0 to 204800000 [15420.544464] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15420.625575] Lustre: fs15246-OST0000: new disk, initializing [15420.627699] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15420.632084] Lustre: 514375:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15420.660493] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15424.134188] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15429.156256] loop1: detected capacity change from 0 to 204800000 [15431.533065] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15431.578251] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15431.610655] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15431.653180] Lustre: fs15246-MDT0000: new disk, initializing [15431.715921] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15431.729556] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15431.783932] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15432.801377] Lustre: 515497:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d1843a27 x1870642532094208/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783983822 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15433.567583] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15441.893445] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15441.898671] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15441.921964] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15443.946272] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15443.959324] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15449.062603] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15449.462117] Lustre: server umount fs15246-MDT0000 complete [15453.983345] Lustre: server umount fs15246-OST0000 complete [15459.304380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15459.311485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15459.313746] Lustre: Skipped 1 previous similar message [15464.424303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15464.428643] Lustre: Skipped 1 previous similar message [15465.149549] Lustre: server umount lustre-OST0000 complete [15469.538345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15469.539452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15469.543765] Lustre: Skipped 2 previous similar messages [15473.271304] Lustre: server umount lustre-MDT0000 complete [15474.658282] LustreError: 514806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15474.665145] LustreError: 514806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15475.271239] LustreError: 510745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783983848 with bad export cookie 18123778444933171449 [15475.274227] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15475.275944] LustreError: 510745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15475.446376] Lustre: server umount lustre-MDT0001 complete [15480.513365] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [15481.858707] Key type lgssc unregistered [15482.013677] LNet: 518590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15482.022351] LNetError: 518590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15482.032370] LNet: Removed LNI 192.168.202.103@tcp [15482.446658] Key type .llcrypt unregistered [15482.447906] Key type ._llcrypt unregistered [15492.745367] Key type ._llcrypt registered [15492.746837] Key type .llcrypt registered [15492.800220] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [15493.310466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15493.389081] alg: No test for adler32 (adler32-zlib) [15494.358112] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [15494.516505] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [15496.152174] Key type lgssc registered [15496.719587] Lustre: Echo OBD driver; http://www.lustre.org/ [15516.257320] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [15522.007184] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [15540.826773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15540.847231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15541.983784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15542.007096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15542.061986] Lustre: lustre-MDT0000: new disk, initializing [15542.107809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15542.117362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15544.220165] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15549.650183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15549.704386] Lustre: 523029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15549.731603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15549.733796] Lustre: Skipped 1 previous similar message [15549.774127] Lustre: lustre-MDT0001: new disk, initializing [15549.818810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15549.838803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15549.845061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15551.746970] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [15556.607888] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0